Hacker News new | past | comments | ask | show | jobs | submit
schiffern | on: One-minute frequency disturbances from synchronization in power systems
The corresponding concept in gears is to avoid meshing two gears unless the number of teeth are co-prime, because this will cause uneven wear patterns. If the tooth counts are coprime, then each tooth will contact every tooth on the other gear the same number of times, effectively "wear leveling" the teeth.

In this instance, one fix is to fire the timer every 59.XXX seconds, so it will naturally drift out of phase with the minute hand of the clock.

If work needs to be done exactly every 60 seconds, this is fixed by having a WAIT(RAND(60e6)) microseconds during the first timer initialization, so if 10,000 units start simultaneously they don't try all at once. This is good practice in most (all?) systems. Hint: test power and infra outage/reconnect, which can unintentionally sync timers.

This is why many software will do automated updates every 17 hours: 17 is coprime with 24.

Classic "thundering herd" mitigation tactics.

Edit: Tempting to blame IoT, but (since this is news) I don't think anyone knew there was something to fix here. We have the tools however.

nik282000 | parent | next
I get a disturbing amount of equipment that does not take into account the inrush current of starting many loads at the same time. While each individual device is protected by a breaker that can handle the inrush of it's load the main breaker for the equipment is not sized to handle every load starting at the same time. The fix is to include logic that starts things up one at a time but convincing engineers, project managers, and machine owners that they didn't think of everything is an uphill battle.
3eb7988a1663 | root | parent | next
We used to have a turbopump that would trip the breakers every time you turned it on from cold. The pump seemingly attempted to spin up to max RPM before settling down to a more sustainable load. The only way to get the thing to stay on was to turn it on once, let it warm itself up on the initial attempt, trip the breakers, and try it another cycle now that it had some momentum and would pull slightly fewer amps. Sometimes took a few attempts to get that rolling start working.
loading story #50328524
loading story #50332087
wowczarek | root | parent | next
I assume you are talking beyond what (in the DIN rail world) type B/C/D breakers are meant to handle?

It's sort of a solved problem - the ones I'm aware of are things like LynTec sequenced panels built exactly for this, and Siemens' SENTRON semiconductor CBs where you can control then curve to your spec.

Convincing stakeholders though, is almost a job of its own.

loading story #50328720
| root | parent | next
{"deleted":true,"id":50327758,"parent":50327431,"time":1793554719,"type":"comment"}
cowlevel | root | parent | next
what? Inrush current is well known in electronics engineering.
loading story #50327944
loading story #50334806
mkj | parent | next
59 seconds will still have second alignment. Maybe it needs to be prime milliseconds or microseconds.
kazinator | root | parent | next
But seconds are 60X higher frequency than minutes. This is almost 6 octaves higher. The grid will low-pass-filter that out much better than the minute period. (You can see that trend in the diagrams given in the article: the amplitudes all get lower with decreasing period to the left, where the 60 second blip is a an outlier.)

Secondly, the amplitude drops. Events that were concentrated on the xx:xx:00 times are now distributed into xx:xx:xx times. If the overall number of events is the same, each :xx time slot now receives 1/60th of the events that were concentrated on the :00 time. If power decibels are relevant here (20 log 10 x), that's a 35 dB reduction in amplitude of the driving signal.

35 dB less amplitude, shifted to a 6 octave higher frequency that the grid doesn't react to very well can be expected to basically make the problem vanish.

This is much like the situation of soldiers avoiding rythmic marching when crossing a bridge to randomize their footstrikes.

vlovich123 | root | parent | next
It depends on a lot of factors, but even second granularity gives you a 60x reduction in the thundering herd. Whether more is needed or even possible often depends on the specific problem domain. For example, if your task takes 20ms, us granularity doesn’t help. Indeed it makes it worse if you’re using randomization because there’s a lot more microseconds than seconds so you can end up with a lot of work that ends up in practice still getting scheduled together due to other delays.

You want your jitter to be significantly >> than the time your task takes as otherwise you get non intuitive degradation that can reintroduce the problem you’re trying to solve. It’s a balancing act picking the jitter that drops off the thundering herd enough and one that is still greater than the time your work takes.

jyounker | root | parent
I think you should re-read the post you're responding to. It said 59.XXX seconds, and given the reasoning immediately after that, it's clear that XXX is intended to be a randomly chosen number of microseconds.
mkj | root | parent
That was edited in later
vlovich123 | parent | next
Small nit: Wait(50+rand(10)) is probably closer to what you want to get about a minute between executions. As stated your proposed execution has an average about 30s execution interval with a variance of about 17s. If you’re talking about a random sleep on a 1 minute interval, this actually results in a possibility of 120s between tasks, also not what you want.

But yeah, as soon as you start using Rand most people make all sorts of mistakes. And if you’re using C Rand, don’t. It has other statistical problems.

jagged-chisel | root | parent
He’s still allowing for exactly 60s between executions. He’s saying stagger the startup by a random amount up to 60s. Then we get what we want every 60s, but they’re all just staggered a random amount.
loading story #50326527
HPsquared | parent | next
Coprime meaning "no common factors".
loading story #50334796