The mission of this group is to bring together utility professionals in the power industry who are in the thick of the digital utility transformation. 

Post

How could a billing system attack shut down an OT network?

Tom Alrich's picture
Supply chain Cybersecurity Risk Management and NERC CIP-013 consulting Tom Alrich LLC

Currently with Tom Alrich LLC, I provide strategy and compliance consulting to electric power industry clients and vendors to the power industry, focusing on the NERC CIP cybersecurity standards....

  • Member since 2018
  • 200 items added with 48,401 views
  • Jul 22, 2021 10:22 am GMT
  • 151 views

 

Yesterday, I attended an excellent webinar on a topic I’ve been waiting to have someone explain to me, “Consequence Driven Cyber Informed Engineering (CCE) – Resilience Strategies”. It was sponsored by Midwest Reliability Organization (MRO), and featured two longtime friends of mine: Jodi Jensen of WAPA and Sam Chanoski of INL. Since a recording will be available on MRO’s website soon, I won’t try to reiterate what was said in the webinar, other than saying it’s worth your while to listen to the recording.

What inspired me to write this post was Jodi’s statement, regarding the Colonial Pipeline ransomware attack, that Colonial had said that they shut down the actual pipeline (i.e. their OT network) because of the loss of their billing system (which was on the IT network). Of course, the IT network was compromised, so it had to be shut down and the machines rebuilt.

Your access to Member Features is limited.

Colonial insisted that their OT network wasn’t affected by the ransomware, but they had to shut it down anyway due to the loss of their billing system. Jodi wondered why the billing system was essential to operations. In other words, couldn’t they have continued shipping petroleum products through the pipeline and worried about billing later?

I wrote three posts after the Colonial incident: Here, here, and here (in that order). In all three of them, I discussed possible reasons why the OT network (and pipeline) had to be shut down, even though the ransomware didn’t penetrate it. I also linked to a post I wrote last October, describing an incident in 2018 in which a major utility – a BA for a multi-state area – had to shut down their Control Centers (i.e. an important part of their OT network) for up to 24 hours and run the grid from cell phones, when their IT network was hit by a ransomware attack that required rebuilding 12,000 computers from scratch.

Just like in the case of Colonial, the utility swore the ransomware never penetrated their OT network (and I have no reason not to believe them), but they couldn’t take the chance that just one machine in the Control Center had been compromised. If that had happened, that one machine might have then compromised all of the IT network when it was restarted, requiring another huge shutdown and rebuild (and I’m told that this becomes much less fun the second time around, to say nothing of the third or fourth time). Which is why they shut down and rebuilt all the systems in the Control Centers as well.

I brought up that incident because this might have been another reason why Colonial shut down their pipeline. And after I wrote the second post, one of the most prolific commenters on my posts, a person named Unknown, wrote in once again to say

Like you, I also believe that Colonial shut down because they could not accurately bill customers or track their customers' assets (i.e. refined petroleum products).

Pipelines are like banks and oil in the pipeline is like cash in the bank. If a bank loses its ability to track who gave them cash (or who they loaned it to), then there is no point opening the doors, even if they can safely store the money in the vault.

Unknown wrote this because I had pointed out in the post that the Washington Post had said in an editorial (which I paraphrase), “If they had kept their pipelines operating while the IT network was down, they wouldn’t have been able to invoice their customers.” I added, “And it’s safe to say that Colonial doesn’t feel that it should deliver gasoline through their pipeline solely as a charity.”

Unknown was pointing out that it was more than the wish to avoid operating as a charity that motivated Colonial to shut down. They don’t own the gasoline they ship in their pipeline, any more than a trucking company owns the furniture they ship or a bank owns the money in its vaults. If either one loses track of what’s been entrusted to them, the trucking company or bank has to repay the entire amount (and certainly with consequential damages) to whoever shipped the furniture or deposited the money.

In other words, this isn’t like an electric distribution utility, which – at least for a brief period of time – owns the electric power they’re distributing to their customers (I’ll omit discussion of Retail Choice here). That utility has to keep the lights on, no matter what it costs them, and if they can’t bill during an emergency, they can usually bill later (the meters needed for billing are all on the OT network, so presumably an IT network shutdown wouldn’t affect them anyway). Colonial isn’t obligated to keep the cars in Georgia full of gas (nor are they paid to do that, of course). They obviously can’t keep shipping gasoline if it’s likely they’ll end up having to pay the full cost of the gas to the shippers.

I concluded my third post on Colonial by articulating the first law of nature that I’ve ever identified. Tom’s First Law of OT Networks says that “an “operations-focused” company – as opposed to an information-focused company like an insurance company or a consulting firm – will be forced to bring their OT network down if their IT network falls victim to a ransomware attack.”

I’ve been told that this can’t be considered as a new law of nature because there are already enough of those. How about Newton’s Laws of Motion? They’ve been around since the 1600s, and Einstein showed they’re not applicable in extreme conditions. Why not drop one of them, and put my law in its place? Seems sensible to me…

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Tom Alrich's picture
Thank Tom for the Post!
Energy Central contributors share their experience and insights for the benefit of other Members (like you). Please show them your appreciation by leaving a comment, 'liking' this post, or following this Member.
More posts from this member
Discussions
Spell checking: Press the CTRL or COMMAND key then click on the underlined misspelled word.
Matt Chester's picture
Matt Chester on Jul 22, 2021

As the entire utility sector and energy systems get more and more complex, it seems that's just opening the door where there are more places to attack-- billing system this time, next time ??

 

Obviously that's not an argument to not advance those systems technically, but rather highlighting the fact that cybersecurity may have been a moderate priority in years past must now be pretty much a #1 priority of all teams across the whole system, identifying and closing all new and existing vulnerabilities that come about

Tom Alrich's picture
Tom Alrich on Jul 26, 2021

Yes, except maybe we need to think about this in a new way. Tim Roxey (former NERC CISO and a legend in the industry) made some excellent comments to me, which I'll share with the eagerly-awaiting world in a post I hope to put up on Monday.

Matt Chester's picture
Matt Chester on Jul 26, 2021

Looking forward to it, Tom! Thanks for the great insights, as always

Get Published - Build a Following

The Energy Central Power Industry Network is based on one core idea - power industry professionals helping each other and advancing the industry by sharing and learning from each other.

If you have an experience or insight to share or have learned something from a conference or seminar, your peers and colleagues on Energy Central want to hear about it. It's also easy to share a link to an article you've liked or an industry resource that you think would be helpful.

                 Learn more about posting on Energy Central »