The mission of this group is to bring together utility professionals in the power industry who are in the thick of the digital utility transformation. 

Post

Tim Roxey and the "end of days"

Tom Alrich's picture
Supply chain Cybersecurity Risk Management and NERC CIP-013 consulting Tom Alrich LLC

Currently with Tom Alrich LLC, I provide strategy and compliance consulting to electric power industry clients and vendors to the power industry, focusing on the NERC CIP cybersecurity standards....

  • Member since 2018
  • 185 items added with 43,062 views

Note from Tom: I’ve moved my email feed from FeedBurner (who’s getting out of this business in July) to Follow.It. If you aren’t getting my posts by email anymore, just hit the Subscribe button in the top right. And if you’d like to start receiving these posts in your email inbox, also hit the Subscribe button.

The day after I put up my post describing why it’s literally impossible for a single cyberattack (or a single set of coordinated cyberattacks) to shut down the entire US grid, I was pleased to receive an email from Tim Roxey, former NERC VP and CSO, on the subject. As usual, he brought a really interesting perspective to the topic.

Your access to Member Features is limited.

To briefly summarize my post, I said that

a)      You can divide the assets in the Bulk Electric System into three types: generation, distribution substations and control centers, and transmission substations and control centers.

b)     Generation and distribution are fairly easily dismissed as attack vectors, leaving transmission substations and control centers as the likeliest vectors.

c)      However, I showed that penetrating the control systems in transmission substations and control centers would be extremely hard (and has never been accomplished in North America), even in the case of a single asset, due in part to the rigorous controls required by the NERC CIP standards.

d)     But attacking a single asset won’t get you very far if your goal is to bring down the entire grid. I estimated that you’d have to carry out a very well-coordinated attack on at least 40 transmission assets (10 in ERCOT, and 15 in both the Eastern and Western Interconnects. And if you want to include Quebec in your continent-wide blackout, then you have to add at least 10 assets there, since Quebec has its own grid – plus I know it’s stretching the truth a lot to say that Alberta is connected to the Western Interconnect. I believe it’s just in recent years that there’s been any connection at all, and even now I think it’s just one line. You’d probably have to attack at least 10 assets in Alberta as well, for a total of at least 60 all told), and even that is probably a woeful underestimate.

e)     I said this would be simply impossible. My reasoning – which I should have stated – was that OT networks are incredibly diverse in the power industry. The devices on the networks are quite variable, as are the configuration and technologies behind the networks.

Of course, other industries might consider it very inefficient to have so much diversity, since it means that suppliers can’t realize the huge economies of scale that for example Dell, HP and Cisco have realized on the IT side. There’s no doubt this is true, but at the same time it makes it literally impossible for the grid to be the subject of a massive, coordinated attack.

This diversity wasn’t planned, of course. It just happened because decision-making is so decentralized in the power industry. I’ve always said that planning is great, but in the end there’s no substitute for dumb luck! The industry - and North American power users - have benefited greatly from that dumb luck.

Tim wrote in to say he agreed with my general argument, but (and here I’m paraphrasing him) I’d overlooked another type of assets: IT assets. In fact, the only thing that generation, distribution and transmission operations have in common is that they all rely on IT assets, not just OT ones. A coordinated attack on IT assets throughout the industry could conceivably be the vector for a takedown of the entire North American power grid.

Tim’s point was that, since it would be normal to expect IT networks to be fairly homogeneous, that means those networks – and the devices attached to them - might well be the vector that would enable an attack to occur. However, once again the power industry has saved itself because of diversity. This time, it’s not diversity in the technologies involved in the IT networks – they literally all run IP, I’m sure, on Intel-standard devices. There’s no DECNet or Novell IPX anymore, although I can remember when these were present in lots of IT networks. And the machines on the networks almost all run Windows, with some Linux. No MS-DOS, MacOS, VMS, etc.

So where does the diversity come from? It’s in the network architecture. Electric utilities have realized the benefits of network segmentation, firewalling off different areas of the network, different WAN technologies, etc. None of this is great for pure efficiency, but it’s great for preventing a small number of hackers from carrying out a massive, simultaneous attack on lots of different grid assets in every Interconnect. And it would take such an attack to bring down the US (or North American) grid in its entirety.

Here is what Tim wrote. Lots of wisdom in here!

Tom Yes – Scalability is directly related to variability in the environment. Very little variation – broader span. Larger variability, then more effort for each unique piece of variability.  

If an environment is very homogeneous, then a successful exploit at one interface is likely useful at a second or third interface. 

1.      Homogenous is bad.

a)      Network architecture using the same make and model for all switches, hubs, routers, servers, etc.

b)     Desktop environment consistent across the enterprise.

c)      Lack of principle of least privilege. 

d)     Lack of application White Listing. 

If an environment is Heterogeneous, then a successful exploit at one interface does not necessarily mean it will work at a second or third interface.   

2.      Heterogeneity is good. 

a)      Network architecture mixed with different vendors supplying parts of the environment. 

b)     A desktop environment consisting of different Operating Systems. 

c)      Full implementation of the principle of least privilege 

d)     Full implementation of application whitelisting 

In number 1, the Adversary only needs to understand one (or a few) different types of network technology. Perhaps the same firewalls are used everywhere for segmentation. In this case, the same exploit used for layer 1 is useful for layers two and layer 3. 

If the victim changes firewalls at every boundary level, then the Adversary must deal with a different set of exploits for each of the different levels. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Tom Alrich's picture
Thank Tom for the Post!
Energy Central contributors share their experience and insights for the benefit of other Members (like you). Please show them your appreciation by leaving a comment, 'liking' this post, or following this Member.
More posts from this member

Discussions

Spell checking: Press the CTRL or COMMAND key then click on the underlined misspelled word.
Bob Meinetz's picture
Bob Meinetz on Jun 10, 2021

"And the machines on the networks almost all run Windows, with some Linux. No MS-DOS, MacOS, VMS, etc."

And because the machines on the networks almost all run Windows (yet the machines are of different architecture) there are plenty of holes to exploit. There is no number of firewalls that can change that.

"...it’s great for preventing a small number of hackers from carrying out a massive, simultaneous attack on lots of different grid assets in every Interconnect."

A small number of hackers...on which planet?

Get Published - Build a Following

The Energy Central Power Industry Network is based on one core idea - power industry professionals helping each other and advancing the industry by sharing and learning from each other.

If you have an experience or insight to share or have learned something from a conference or seminar, your peers and colleagues on Energy Central want to hear about it. It's also easy to share a link to an article you've liked or an industry resource that you think would be helpful.

                 Learn more about posting on Energy Central »