Coordinated disturbances haunt grids worldwide. Here’s how researchers solved the mystery.

For years, mysterious electrical anomalies have been flummoxing grid managers around the world. Every 60 seconds, the frequency on the grid would drop a tiny bit.

Even weirder: The sags happened at the exact same time on power grids across countries and continents.

It was a mystery that Dirk Witthaut, a researcher at Forschungszentrum Jülich in Germany, couldn’t pass up.

Germany’s largest transmission system operator, Tennet Germany, brought the puzzle to Witthaut, and it took years of investigative and modeling work to determine the cause. From Europe to Australia, the U.S. to Malaysia, the same pattern shows up in the data.

Such frequency drops aren’t unheard of, but they are troubling. And the regularity with which they have been occurring can make the phenomenon feel even more ominous.

Balancing the grid

The grid’s orderly operation relies on the frequency being kept within a certain range. If the oscillations get too large, or if grid operators don’t know the cause and respond improperly, they can lead to blackouts. Even if the sags don’t cause blackouts, they can damage equipment in power plants if they go unchecked, especially in hydropower turbines.

Hydropower plants have historically been a main driver of so-called ultra-low frequency oscillations (UFLOs). Every turbine vibrates, and if those vibrations occur with a certain regularity, known as the resonant frequency, they can magnify, sometimes to dangerous levels. Think of a wine glass breaking before an opera singer’s mighty pipes — that’s resonance.

So Witthaut and his team started investigating hydropower, the most obvious potential culprit. They used a computer model of a hydropower plant control system on the Scandinavian grid to test the hypothesis.

While they managed to find conditions that could generate a resonance that peaked once per minute, they couldn’t explain why it persisted for so long. Typically, resonances in mechanical systems will come and go, and so will the accompanying signal on the electrical grid. But this frequency drop kept happening minute after minute after minute. Every 60 seconds, without fail.

It then dawned on the team that anything operating with such clockwork required, well, a clock. How else could they explain the uncanny timing?

Searching for clues

To that hypothesis, the researchers started close to home, installing more than 80 smart power quality meters around the Jülich campus. The research site is outfitted with medium- and low-voltage distribution lines, transformers, solar panels, and battery energy storage systems that serve as uninterruptible power supplies (UPS). It also has a high-performance computer system as well as the usual office computers powered by much smaller UPS devices. The power quality meters sampled a range of attributes, including frequency, voltage, current, and power, once every three seconds.

Before long, the research team had its answer. The measurements recorded on the transformer attached to the battery energy storage system peaked every minute, just like on the bigger power grids.

It turns out that many stationary batteries, from data center UPSes to large grid-scale batteries, check their state-of-charge every minute. Since these devices are internet-connected, their clocks are in near perfect synchronicity. Every time one checks its charge, the disturbance is imperceptible on the wide grid. But when they all do it together, that disruption becomes apparent.

To find evidence that UPSes were the main culprit on the grid, Witthaut and his team turned to data provided by grid operators. They found their answer in Ireland.

There, data centers have grown to consume a large fraction of the country’s electricity. By 2022, data centers represented 18% of the load on Ireland’s grid, and each facility is served by a fleet of UPSes. Sure enough, between 2016 and 2021, the minute-by-minute frequency drop on the Irish grid quadrupled from 3 milli-Hertz (mHz) in 2016 to 12 mHz in 2021. More recently, some drops have exceeded 20 mHz.

Ireland’s 50 Hz grid is built to withstand swings of plus or minus 200 mHz, though the grid operator would rather limit the swings to 100 mHz. In that range, 20 mHz isn’t a lot, but data centers aren’t the only loads on the grid. If other loads also cause the frequency to sag, the extra bump from every UPS checking its state-of-charge could push the grid toward blackout conditions.

The solution

The problem is entirely preventable. Rather than check state-of-charge exactly on the minute, Witthaus and team recommend that UPS and grid-scale battery manufacturers add some randomization to the schedule.

While UPSes are driving the problem today, there’s a risk that the phenomenon could persist. The researchers also tested IoT devices. In devices as diverse as a smart oven and Raspberry Pi clusters, they were able to detect periodic power fluctuations. The fix is also a simple software patch, but many IoT devices go unpatched. A recent study found that less than 3% of IoT devices are running the latest firmware.

Regulators may soon have to step in to demand changes, but they can only address their corner of the grid. Given the global nature of the issue, it’ll ultimately require global coordination. The mystery may be solved, but the problem could remain for years to come.