BOSTON (Reuters) - Data Center Alley, a 30-square-mile stretch outside Washington D.C. and home to more than 200 data centers, consumes roughly the same electricity as Boston. So power company officials were alarmed when a big chunk of those centers - 60 of them - suddenly dropped off the grid one day last summer and switched to on-site generators.

The mass reaction was triggered by a standard safety mechanism across the data center industry, intended to protect computer chips and electronic equipment from damage caused by voltage fluctuations. But it caused a huge surge in excess electricity, according to federal regulators and utility executives.

The magnitude of the imbalance forced grid operator PJM and local utility Dominion Energy ( D ) to scale back output from power plants to protect grid infrastructure and avoid a worst-case scenario of cascading power outages across the region.

The near-miss - reported here in detail for the first time - forced federal regulators to recognize a new vulnerability of America's electrical grid: unannounced disconnections by data centers.

"As these data centers get bigger and consume more energy, the grid is not designed to withstand the loss of 1,500-megawatt data centers," John Moura, Director of Reliability Assessment and System Analysis for NERC, told Reuters in an interview. "At some level it becomes too large to withstand unless more grid resources are added."

Historically, grid operators have planned for large power plants tripping offline. But the rapid expansion of data centers processing the vast amounts of information used for AI and crypto mining is forcing grid operators to plan for new contingencies and complicating the already difficult task of balancing the country's supply and demand of electricity. 

"What it tells us is that the behavior of data centers has the potential to cause cascading power outages for an entire region,"  said Alison Silverstein, a former senior adviser to the chairman of the U.S. Federal Energy Regulatory Commission.

The event last July 10 occurred near the D.C. suburb of Fairfax, Virginia, an area known as Data Center Alley for its concentration of facilities serving Microsoft ( MSFT ), Google and Amazon ( AMZN ). About 70% of the world's internet traffic flows through the area.

A month after the incident, the North American Electric Reliability Corporation (NERC), the federal regulator for grid reliability, founded a taskforce to study en masse disconnections by data centers and crypto miners.

For this story, Reuters examined thousands of pages of regulatory documents and interviewed about a dozen industry executives to determine the origins of the fault - a failed surge protector on Dominion's Ox-Possum 230-kilovolt line near Fairfax, Virginia - and its spread across the area.

NERC reviewed the incident in a report in January but did not disclose the exact location of the fault, the number of data centers involved, or how PJM and Dominion worked to rebalance the grid's supply and demand of electricity. 

NEAR-MISS EVENTS INCREASING

The number of near-miss events like the one in Data Center Alley has grown rapidly over the last five years as more data centers come online.

The amount of power used by data centers has tripled over the past decade and could triple again by 2028, according to a report produced by the Lawrence Berkeley National Laboratory for the Department of Energy in December.

A Reuters review of disclosure filings by the Electric Reliability Council of Texas (ERCOT), the state's main grid operator, identified more than 30 near-miss incidents since 2020, triggered by big energy users like data centers and crypto miners switching offline.

In December 2022, a failed transformer at a substation in west Texas caused nearly 400 crypto miners, data centers and oil and gas production facilities to unplug without warning. 

The mass exodus produced an oversupply of nearly 1,700 megawatts of electricity - equivalent to about about 5% of the grid's total demand - and forced 112 megawatts of power generation to shut down, according to ERCOT.

The risk of power outages will only grow as new data centers come online, the NERC forecast in a December report. Nearly all of the United States will face higher risks of energy shortfalls over the next 5 to 10 years, the report said.

The regulator urged utilities to consider updating federal reliability standards for data centers and crypto miners.

A CONTROVERSIAL FIX

Many data centers are engineered by their operators to switch to local generators at the smallest hint of a problem on the grid to minimize the risk of an interruption to services like Google search or crypto mining, according to NERC.

Some grid operators have proposed requiring data centers to "ride through" routine voltage dips without disconnecting. But data center operators are opposed because of the risk of damaging electronic equipment and cooling systems.

ERCOT last year withdrew a proposal that would have imposed ride-through restrictions on data centers and crypto miners after facing pushback from an industry group, the Data Center Coalition.  

The group, whose members include Amazon ( AMZN ), Google, and Meta, cited costs and the risk of damaging computer chips and cooling systems exposed to fluctuating voltage levels.

"Data center hardware and power supplies, similar to other electronics, are very sensitive to power supply stability," the coalition said in January 2024 comments filed with ERCOT.  

"Deviating from this range will deteriorate the optimal performance, reduce longevity, or damage the components beyond repair."

The coalition declined to comment for this story. Amazon ( AMZN ), Google and Meta did not return messages seeking comment. ERCOT did not return messages seeking comment.

There is "high potential" for the magnitude of these disconnection events to grow as larger operations plug into the Texas grid, ERCOT operations engineer Patrick Gravois said in a December presentation to NERC's Large Load Task Force.

Gravois said the grid operator is still working to determine exactly what prompts big users of electricity to unplug from the grid, so that it can avoid surprises.

Ari Peskoe, director of the Electricity Law Initiative at Harvard Law School, said regulators could require data centers to ride through voltage dips - but that could risk Big Tech decamping for states with more relaxed rules.

Jim Simonelli, chief technology officer for Schneider Electric's secure power division, said utilities and the data center industry have a lot of lessons to be learned from what happened outside Washington DC this past July. 

"One thing that doesn't exist yet for the data center industry is how to be grid-friendly," Simonelli said.