Part of Grid Network »

The Grid Professionals Group covers electric current from its transmission step down to each customer's home. 

WARNING: SIGN-IN

You need to be a member of Energy Central to access some features and content. Please or register to continue.

Post

Are Manchurian Candidates Lurking in the Electric System?

image credit: Credit: Wikimedia Commons

There are worries afoot in the electric utility world. The issue is the integrity of the grid and the possibility that foreign suppliers of bulk power equipment (BPE) may have introduced the technical equivalent of Manchurian candidates into the hardware that manages the system. 

This represents a departure from previous concerns which have emphasized software and paid more attention to attacks aimed at the computer systems of electric utilities than to their hardware. They get millions of these attacks every day and have worked relentlessly to protect against them. 

Now a new front has opened. The battle has moved from the world of internet technology (IT) to the hardware itself, to BPE. Leading the charge to draw attention to systems whose vulnerability may have been overlooked is Joe Weiss, a professional engineer, a veteran of the Electric Power Research Institute in Palo Alto, California, and now an independent consultant. 

Weiss said in a blog, which went viral in the world of utility engineers last week, “Why would attackers hit defenses head-on when they can simply bypass them?” And that is exactly what they’re doing, he believes. 

On May 1, President Donald Trump issued the far-reaching Executive Order 13920 which prohibits the purchase of major BPE from potential adversaries, later named by the Department of Energy (DOE) as China and Russia, among others. China is the primary supplier of BPE to American utilities. Then, on July 8, the department issued a request for information about what the electric utilities purchase and from where. It appears the government is attempting to scope the problem. 

Initially, many in the industry thought the executive order was just another shot in the Trump administration’s trade war with China. But not so. It signaled what may be a big vulnerability not only in installed equipment, but also equipment that is on order.

China has become the primary supplier of heavy equipment for utilities, particularly big transformers. While these have no moving parts, Weiss believes they can have “backdoors” through which an adversary could catastrophically alter their operation.

The key, he says, may be the sensors which can send false readings and bring about major disruption, and send parts of the grid haywire. 

Transformers are critical to the distribution of current. They boost voltage to compensate for line losses and ultimately step down the voltage for local distribution.  

This vulnerability story began after terrorist attacks of 9/11, when a trend to look at the security of the electric grid turned to a greater concentration on IT and, some argue, away from the old regime of operational technology (OT), where engineers took responsibility for the security of their equipment. A cultural division opened, as I was told by the one of the nation’s top computer experts in academia. 

Underlying this shift in responsibility are the workhorses of modern industry, programmable controllers, part of the larger Industrial Control Systems (ICS). These are the automated systems which do the work of managing operations in modern industry, including utilities. 

The worry for the electric utility industry is that these devices that manage the grid could be manipulated without showing up as an attack.

There is precedent for this kind of attack: The Stuxnet virus that disabled centrifuges at Iran’s Natanz nuclear facility in 2010. The United States and Israel didn’t go after the facility’s computer system – an attack which would’ve been detected -- but rather after the controllers governing the centrifuges.  

Last year, something big was discovered, and details are sketchy: A Chinese-made transformer at a large investor-owned utility was found to have counterfeit parts and, perhaps, backdoors through which the integrity of the grid could’ve been compromised. Alarm bells rang at the departments of Homeland Security (DHS) and Energy.

A similar or identical transformer made by JiangSu HuaPeng Transformer Company, Ltd., a family owned company with a small office in San Jose, Calif., was seized by agents of the DHS and DOE and hustled straight to Sandia National Laboratory in Albuquerque, New Mexico. upon its arrival at the Port of Houston.

This transformer had been destined for the Western Area Power Administration’s Aluit Station, near Denver. WAPA is one of the power distribution systems owned by the government through the DOE. 

What, if anything, has been discovered in the transformer hasn’t been disclosed. Everything is cloaked in secrecy, my sources tell me.  

 

Llewellyn King's picture

Thank Llewellyn for the Post!

Energy Central contributors share their experience and insights for the benefit of other Members (like you). Please show them your appreciation by leaving a comment, 'liking' this post, or following this Member.

Discussions

Matt Chester's picture
Matt Chester on Jul 20, 2020 12:47 pm GMT

Last year, something big was discovered, and details are sketchy: A Chinese-made transformer at a large investor-owned utility was found to have counterfeit parts and, perhaps, backdoors through which the integrity of the grid could’ve been compromised. Alarm bells rang at the departments of Homeland Security (DHS) and Energy.

This is indeed a scary situation, especially as tensions between China and the U.S. remain high. I wonder why this story hasn't made more waves outside of the energy industry

Jim Tarpey's picture
Jim Tarpey on Jul 21, 2020 12:41 pm GMT

This indeed has been an issue raised back in the 90's when the US manufacturers went out of business as the foreign manufacturers undercut their pricing.  That opened the door for foreign actors to work their way to low bid and take control over the market.  I looked at the executive order as something that was needed a long time ago, and now, maybe a bit too late.  The bulk power system is extermely critical and the move away from fossil fuel plants to restart the system in an event makes it even more important to assure the bulk system is protected.  I am very concerned that it will take another blackout for us to realize that politicians will have pushed the technical limits of the electic system too far and pushed manyfacturing of critical components off shore - the utilty companies, RTOs and engineers will ultimately be blamed for allowing it to happen.  

Matt Chester's picture
Matt Chester on Jul 21, 2020 4:23 pm GMT

What was the response (besides, ultimately, inaction) when this was brought up in the 90's? Were the reasons for punting and not addressing the very real problem the same then as they have been for the past decade? 

Jim Tarpey's picture
Jim Tarpey on Jul 22, 2020 8:21 pm GMT

The concern was that if we lost a bulk system transformer, for example, the time to get an exact replacment was going to be delayed by about 18 months at that time and the system would already be in an N-1 state for an extended period of time.  Considering that the US manufacturers were being undercut by the foreign factories, there was no "Made in USA" incentive to keep them afloat, all of them shut down their operations, which also pushed smaller sized transformers off shore.  The industry response was to develop contingency plans and spare transformer inventories, which ultimately cost customers more and as one size did not fit all, lots of unique contingencies and bulk system upgrades were done for higher level contingency cases.  And, at the time there was a concern as to the counties involved and where how strong their alliances would be with the US in the future.  If there were cyber issues back then, there might have been a stronger effort to do something to keep the US factories in business.

Randy Long's picture
Randy Long on Jul 21, 2020 8:33 pm GMT

Perhaps Mr. King could put out a little more detail on the transformer in question? I have worked in the Utility Business for 15 years and the last 5 or 6 have seen increased awareness of geopolitical tensions and issues impact the grid. I fear (and hope I'm wrong) that CT's, Relays (not domestic ones), PT's, and some of the control and signaling infrastructure has probably already been compromised. Could I be off base? Sure, but there are a lot of transformers, generators, cap. banks and just by sheer volume, some have to be compromised. The risk to the grid is moderate, but it is still there. The interconnections (WESTERN, EASTERN, ERCOT) are (hopefully) hardened. I'm still hopeful that domestic manufacturing hasn't been impacted, but it's anyone's guess.

Matt Chester's picture
Matt Chester on Jul 21, 2020 8:55 pm GMT

If hypothetically some of this infrastructure you mention is compromised, would that mean the EO is too little too late? What would be the best strategy to tackle the threat from the view that 'it's already here and plugged in'?

Randy Long's picture
Randy Long on Jul 21, 2020 9:35 pm GMT

hmmm...That's a loaded question. NIST and CISA have guidelines regarding supply chain security and integrity. Are these fail safe? No. However, the mantra that "it's already here and plugged in" might be a little much. "Trust, but verify is more like it."

If, say in the last 5 years we could audit (I know the compliance folks will hate me) the infrastructure that has been installed/repaired/replace, that could be somewhat of an assurance the grid is ok. That's probably an exercise that most utilities/coops/muni's can't/won't do. Like I said before, I would assume the individual interconnects are hardened, but once you get granular, say RTO/ISO, Transmission Orgs., BA's, etc, it's going to be a heavy lift to do that audit. And honestly, some manufacturers are VERY strict on embedded systems (ie basic security hashing, dongles, etc.) and some are a little light on the security side. I could definitely see the NERC offering a little guidance on this. 

I have yet to find Mr. Weiss's original blog posting, but I'd like to do a deep dive on this one. 

 

Matt Chester's picture
Matt Chester on Jul 22, 2020 10:34 am GMT

Asking loaded questions is what I like to do! Thanks for the thorough and careful answer in response!

Here's a link to Joe Weiss's blog that Llewelyn referenced in his post: https://energycentral.com/c/iu/ics-cyber-security-second-coming-maginot-line-%E2%80%93-and-chinese-have-breached-it

Get Published - Build a Following

The Energy Central Power Industry Network is based on one core idea - power industry professionals helping each other and advancing the industry by sharing and learning from each other.

If you have an experience or insight to share or have learned something from a conference or seminar, your peers and colleagues on Energy Central want to hear about it. It's also easy to share a link to an article you've liked or an industry resource that you think would be helpful.

                 Learn more about posting on Energy Central »