The mission of this group is to bring together utility professionals in the power industry who are in the thick of the digital utility transformation. 

WARNING: SIGN-IN

You need to be a member of Energy Central to access some features and content. Please or register to continue.

Post

A great supply chain security presentation - part I

I recently put up a post describing some excellent points regarding firmware security, that were made in a paper written by Matt Wyckhouse of Finite State. However, I’m not done with Matt; he seems to be a fount of interesting observations. In this post and the next two parts (which won’t necessarily appear consecutively), I want to discuss Matt’s excellent presentation at the Midwest Reliability Organization’s annual security conference (virtual this year, of course) two weeks ago. This presentation made some really excellent points about supply chain cybersecurity, although as you’ll see I think Matt also missed – or at least underemphasized – an important point.

The title of Matt’s presentation was “Don’t trust – verify everything: Scaling product and supply chain risk assessments through firmware analysis”. The slides – and maybe the recording – should be posted on MRO’s website in the near future, but they’re not there now. Matt addressed three important points[i], which I’ll discuss in the three parts of this post.

The first point has to do with determining “foreign adversary” influence. As you probably know, the May 1 Executive Order is focused on that problem. Its stated purpose is to place DoE in charge of finding threats to the Bulk Power System that might be found in products with some degree of “influence” from six countries, including China and Russia.

First off, there are no products sold by Chinese (or any of the other five foreign adversaries) companies that are used to control or monitor elements of the US Bulk Electric System, as Kevin Perry and I documented in this post five days after the EO came out. Moreover, the only BES products we could identify that might even be assembled in China were Dell and HP servers and workstations used in Control Centers. And since the same servers and workstations are used by literally every industry, it’s hard to see how the Chinese could launch an attack on the US grid through such devices.

So now, we need to look at “influence” in a general way – that is, cases where a hardware or software manufacturer is somehow under the influence of the Chinese, even though it isn’t headquartered in China and even though its products aren’t manufactured or even assembled there.

But in his presentation, Matt pointed out that there’s no good way to state what constitutes influence, without including many companies that are highly unlikely to be under foreign adversary “influence” in any meaningful sense. Here are some of the points he made:

·        Not buying Huawei is an easy choice; the US government has already made that choice for you. But no Huawei products are now used to monitor or control the US BES.

·        But how about Honeywell? They’re a longtime critical infrastructure and defense supplier, and some of their products are definitely used on the grid. Yet some of their products include components from Huawei, Dahua, Hikvision and HiSilicon – all banned Chinese companies. Should you stop buying from Honeywell? (Tom says: I hope not. They’re my former employer and I still think highly of them)

·        How about Siemens? They’re a huge supplier to the power industry and many other industries. Yet they have 21 R&D hubs in China and over 5,000 R&D and engineering staff there. Should you stop buying from Siemens because of that?

·        Let’s say somebody located in China contributed to an open source software product (which probably happens every day, of course). Do we need to ban that product? More importantly, how would we ever verify the location or even the identity of all contributors? And remember, the whole idea of open source is that there are many eyes looking at the code. Even if say a Chinese People’s Liberation Army soldier placed some sort of backdoor in an open source project, there’s no guarantee at all that it wouldn’t be removed before the code was made available for download.

As Matt points out, the bottom line is that it’s a losing proposition to try to ban products based on connections to a particular country. Connected devices have complex global hardware and software supply chains. If we don’t want adversary countries in our supply chains, where do we draw the line? Any line we draw will only change the attackers’ tactics. Of course, it’s perfectly acceptable to say that no Chinese products can ever be installed in a position where they could be remotely controlled to execute an attack on the BES. As I’ve already said, there aren’t any Chinese products in such a position now, but I have no problem in saying they should never be installed in the future as well.

However, it’s important that no devices from China be banned, if there’s absolutely no way they could be controlled remotely – or even pre-programmed to misoperate at a certain time in the future – to impact the BES. Any device that could be misoperated to have a BES impact at a later date would have to have some sort of logical engine built into the device: a microprocessor, FPGA, etc. Yet about 20 of the 25-odd devices listed in the EO are controlled by an external device like a relay (if at all); they don’t have any logic built into them. So how could they possibly be subject to a supply chain cyberattack?

One of these “non-logical” devices is a transformer. A transformer, taken by itself, operates purely according to the laws of physics; it neither requires external commands to operate, nor is there any built-in logic that could change its behavior.[ii] Yet, in the wake of the EO, transformers manufactured in China have been pointed to as some sort of dagger pointed at the heart of the US grid, when in fact they are no more likely to be subject to a supply chain attack than my steam iron is.

I want to point out that I’ve never heard of a vulnerability or backdoor that was deliberately planted in critical infrastructure equipment by a nation-state in order to attack the US. On the other hand, the US has definitely done this to other countries. For example, the mother of all supply chain attacks was conducted by the US and resulted in a huge pipeline explosion in the Soviet Union in 1982, which played a role in the collapse of the USSR eight years later. And there are suspicions that the backdoor that was found in Juniper routers in 2015 was actually planted by the NSA.

Of course, this doesn’t mean that other countries wouldn’t try to do the same thing to us. But the question is why they would do this, given that the discovery that an adversary had deliberately caused a serious critical infrastructure disruption would very likely be taken as an act of war. And there’s no country in the world, other than perhaps Russia, who would be able to “win” a war with the US.

It simply amazes me that people talk about these attacks as if they’re very likely to happen, when they’re almost impossible to imagine. Sure, we need to take steps to prevent such attacks, no matter how improbable they are. But what’s much more probable (although even then, not very probable) is that a couple bright teenagers in India would be able to cause a grid outage by exploiting a garden-variety vulnerability in firmware or software. This is a much greater risk. But of course, it’s not addressed at all in the EO. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.


[i] And other points as well. But I found these three points the most interesting. Not surprisingly, they all tie in with issues that have been discussed a lot in the industry lately, especially in relation to the May 1 Executive Order. 

[ii] Transformers sometimes have load tap changers (or dissolved gas analyzers), which are controlled by built-in microprocessors and might themselves be attacked to impact the BES, although it’s very hard to imagine how an attack on either one could result in anything more than a small local outage. But load tap changers (LTCs) are often manufactured by third parties, so any attack would really be on the LTC, not on the transformer itself. However, load tap changers aren’t even listed in the EO. One big manufacturer of LTCs is GE. Maybe LTCs from China should be banned, but not Chinese transformers themselves.

Discussions

Matt Chester's picture
Matt Chester on Oct 23, 2020

But what’s much more probable (although even then, not very probable) is that a couple bright teenagers in India would be able to cause a grid outage by exploiting a garden-variety vulnerability in firmware or software. This is a much greater risk. But of course, it’s not addressed at all in the EO. 

Agreed on this-- the most likely 'groups' to make attacks are those that are outside any official state function or even aligned company, but rather an undefined group wither perilously uncertain motives. But certain for anyone that could do this, the motive to use their ill-intentioned skills would be something they could benefit from-- finding a financial angle, which is a outcome we've seen via hacking. I don't see the immediately obvious benefit that would come from then attacking the grid other than being able to say that they did as a point of 'pride'

Tom Alrich's picture
Tom Alrich on Oct 23, 2020

How about ransom? I've heard that at a conference in around 2006 someone from the CIA said a Brazilian city was blacked out until a ransom was paid to the attackers.

How about market manipulation? Essentially, that's what Enron did in California in the 2000-2001 crisis. And since there's a forward market in electric power, someone could say go long and then cause outages that would raise prices.

Tom Alrich's picture

Thank Tom for the Post!

Energy Central contributors share their experience and insights for the benefit of other Members (like you). Please show them your appreciation by leaving a comment, 'liking' this post, or following this Member.

Get Published - Build a Following

The Energy Central Power Industry Network is based on one core idea - power industry professionals helping each other and advancing the industry by sharing and learning from each other.

If you have an experience or insight to share or have learned something from a conference or seminar, your peers and colleagues on Energy Central want to hear about it. It's also easy to share a link to an article you've liked or an industry resource that you think would be helpful.

                 Learn more about posting on Energy Central »