The mission of this group is to bring together utility professionals in the power industry who are in the thick of the digital utility transformation. 

WARNING: SIGN-IN

You need to be a member of Energy Central to access some features and content. Please or register to continue.

Post

Presidential Executive Order 13920 was not due to a malware event - recent and upcoming events will discuss the event

There continues to be a lack of understanding about control system cyber security. What’s worse, there is a growing schism between network security/threat analysts and electrical, mechanical, control system, safety and other domain system engineers. This gap was laid bare in dozens of recent blogs discussing Presidential Executive Order 13920. It should be noted this is not just a US electric utility issue but is international in scope and affects industries beyond just electric.

Presidential Executive Order 13920 was not the result of a malware cyber event.  Rather, it was the result of hardware backdoors that could not be detected by network security’s overly narrow focus on Ethernet and IP protocols.  Network security’s narrow focus overlooks cyber attacks that take place beyond the focus on Ethernet and IP protocols, such as sensors, serial data streams, and protocols that are neither Ethernet or IP.

Apparently, the network cyber security community did not see the transformer issue or the Executive Order coming because neither are related to malware on the networks. Ironically it wasn’t a network threat analyst who saw this issue coming but Rebecca Smith from the Wall Street Journal. You can also find the April 28, 2019 blog on hacking transformers of interest - https://www.controlglobal.com/blogs/unfettered/large-electric-transformers-are-subject-to-cyber-attacks-which-can-cause-outages-of-months-to-years/ as it was evident since working on Aurora in 2011 that transformers could also be at risk to cyber attacks. 

SANS prepared a 19 page report on https://www.controlglobal.com/blogs/unfettered/emergency-executive-order-13920-response-to-a-real-nation-state-cyberattack-against-the-us-grid/ in which SANS took strong exception to the blog post. In particular, they pointed to a lack of direct confirmation of concerns about a hardware vulnerability in the transformer supply chain. The SANS report was based entirely on a network assessment (see https://ics.sans.org/ics-library ICS Defense Use Case 7). Unfortunately, this wasn’t a network problem nor, as mentioned, was it detected by network security. The transformer issue was not the first time network security analysts overlooked non-Ethernet-based threats. The indirect evidence is disturbing and is well presented in Rebecca Smith’s Wall Street Journal article. The direct evidence comes from the site.

The Department of Energy (DOE) has yet to comment publicly on the inspection of either transformer. What has been released by DOE to date such as the DOE FAQs do not relate specifically to the transformer case. In the case of the transformers, the attackers were able to avoid the network security protections which is why it was not detected by network threat analysis or inspection. It is also why network security was outside the scope of the Executive Order. The lack of malware involved in this nation-state attack has the network threat analysis and security community in a tizzy.

Surprisingly (or maybe not), news reports in Chinese media. Global Times on June 3rd, claimed there is nothing to the Wall Street Journal’s story (https://www.globaltimes.cn/content/1190466.shtml). They claim it is just US disinformation in the trade war. Jiangsu Huapeng (the Chinese transformer manufacturer) categorically denies that its transformer was seized. The company says it was delivered to the end user in Colorado last summer and that Jiangsu Huapeng received payment in full. However, there are pictures of the first transformer installed at the substation in Colorado as well as the second transformer that was seized at the port of Houston and taken to the Sandia National Laboratory that discredits the Chinese claim. The transformer pictures and technical issues associated with the Executive Order will be discussed during the July 30th presentation to SURFA (see below).

The transformer issue was not the first time network security analysts overlooked non-Ethernet-based threats. Process sensors (e.g., pressure, level, flow, temperature, voltage, current, etc.) have no cyber security or authentication (see https://www.controlglobal.com/articles/2020/cybersecurity-for-field-devices/?utm_campaign=CGU_2020_Enews_Campaign&utm_medium=email&_hsmi=89865000&_hsenc=p2ANqtz-8Fba63D6MVTZGQAQt6NuXQf-WQpeCXU4tNSUD0FcPrpaZl1XFybaEowgU6zJInqfrfof0YpKllJIACzvh-WZFK2en0YA&utm_content=89865000&utm_source=hs_email) . However, the network security community generally refuses to acknowledge these gaps. The monitor and detect network technology generally overlooks breaches and other unintentional issues that occur outside of Ethernet/IP domain. Moreover, as mentioned in https://www.controlglobal.com/blogs/unfettered/the-connection-between-the-isa84-annex-h-on-process-sensor-cyber-security-and-presidential-executive-order-13920, process sensors can be used to compromise the transformers. 

Everyone benefits when network security, physical security, and engineering experts work together. They did so with Stuxnet. However, the same can’t be said for defending control systems. Many attackers are aware of this gap and will develop their scenarios to attack where there is no monitoring such as the transformer case. The lack of understanding of the engineering issues was manifested in the 2017 Dragos whitepaper on CRASHOVERRIDE. The analysis addressed the remote opening of the breakers, not the reclosing of the breakers. Yet, the Aurora vulnerability and resultant long-term damage comes not from opening the breakers but from reclosing the breakers out-of-phase (https://www.controlglobal.com/blogs/unfettered/the-aurora-vulnerability-still-being-shunned-by-the-electric-industry-where-is-the-education/). This is an engineering issue, not common to network system analysts.

When incidents like the transformer issue arise, include the domain experts and you will get a better understanding of what can or cannot be expected to happen with the equipment. Relying on network security experts who focus on the Ethernet/IP band may be necessary, but it is certainly not sufficient. The SANS/Dragos transformer and CRASHOVERRIDE reports are clear examples. If the network security analysts work with domain engineers and technicians, the scope of such inquiries will be wider and more likely to identify and prevent unintentional incidents or malicious breaches.  

There are T&D experts that could have been consulted but neither the Dragos report nor Honeywell’s Sinclair Koelemij’s Aurora blog appeared to do so. One T&D expert in particular was a substation manager for many years and involved in one of the only two Aurora hardware demonstration projects with DOD. He reviewed this blog, the Dragos report, and Sinclair’s blog. Suffice it to say, there are technical errors with the Dragos report and Sinclair’s blog.

Here are some other podcasts and presentations that may be of interest to those following the stories about grid security:

June 16, 2020, Civil Defense Radio posted an interview I did with Preston Schleinkofer May 13, 2020 on Presidential Executive Order 13920. As the interview predated the CSO and Wall Street Journal articles, there was no mention of either the name of the utility or there were two transformers involved. The interview can be found at Civil Defense radio - http://civildefenseradio.com/joe-weiss-on-electronic-control-systems-security/

July 15th 10amPacific, I will be giving a presentation on control system cyber security for the Purdue Cerias summer seminar series – “Cyber security of control systems – what needs to be done” (https://www.cerias.purdue.edu/news_and_events/events/security_seminar/summer)The focus will be on what makes control systems unique and will touch on the Executive Order.

July 30th 11amPacific, will be a panel session for the Society of Utility Regulatory Financial Analysts (SURFA) on the Presidential Executive Order 13920. Panelists will be Dave Batz from EEI, Phil Jones who was President of NARUC, and myself to address the technical issues and implications. This will be a very important session as these are the state regulators and utility financial analysts reporting to the Boards. The presentation will address real risk – system/equipment impact and physical consequences - not network vulnerabilities. Webinar details will be provided later.

 

Original article published here. 

Joe Weiss's picture

Thank Joe for the Post!

Energy Central contributors share their experience and insights for the benefit of other Members (like you). Please show them your appreciation by leaving a comment, 'liking' this post, or following this Member.

Discussions

Richard Brooks's picture
Richard Brooks on Jun 24, 2020 2:24 pm GMT

Joe, I find the SANS DUC 7 authors analysis to be flawed on several levels, but the most egregious flaw is their use of "Absence of evidence as evidence of absence" as proof for their "no link" claim between your initial assertion of the Executive Order on 5/1 and the seizure of a transformer destined for WAPA. I've followed, and trusted SANS for many years and have always held their reporting in high regard and authoritative. But DUC 7 fails to reach the high level of integrity and credibility that SANS usually produces. I sincerely hope that SANS will uphold their high level of integrity and credibility by retracting DUC7.

FYI: I attempted to raise my concern on LinkedIn with one of the DUC7 authors and have now been blocked from all further engagement. It takes real courage to "stand under the arch" with conviction to show real confidence in your work, but it requies no fortitude to censor those who ask you to "stand under the arch" and defend your position. I surely hope that SANS will do the right thing and retract DUC 7.

Matt Chester's picture
Matt Chester on Jun 24, 2020 4:18 pm GMT

FYI: I attempted to raise my concern on LinkedIn with one of the DUC7 authors and have now been blocked from all further engagement. It takes real courage to "stand under the arch" with conviction to show real confidence in your work, but it requies no fortitude to censor those who ask you to "stand under the arch" and defend your position. I surely hope that SANS will do the right thing and retract DUC 7.

Yikes. That's not a great signal for the direction this debate is going to go as more information and direction comes out

Richard Brooks's picture
Richard Brooks on Jun 24, 2020 5:53 pm GMT

I agree, Matt. The cybersec universe needs more honest and open dialog not censorship, if we are to be successful in our quest to protect the BES from the bad guys.

Bob Meinetz's picture
Bob Meinetz on Jun 24, 2020 5:09 pm GMT

Richard, one might apply your "absence of evidence as evidence of absence" to the threat of backdoor vulnerabilities from Chinese equipment itself. I'm unfamiliar with the details of the Executive Order and the delivery of the transformer, but two other general principles apply:

1) In any system, an increase in complexity introduces a corresponding compromise of both security and reliability, and

2) The Precautionary Principle - when the impacts of a planned course of action are unknown but potentially catastrophic, an abundance of caution is warranted.

Computer security expert Bruce Schneier:

"Supply-chain security is an incredibly complex problem. U.S.-only design and manufacturing isn’t an option; the tech world is far too internationally interdependent for that. We can’t trust anyone, yet we have no choice but to trust everyone. Our phones, computers, software and cloud systems are touched by citizens of dozens of different countries, any one of whom could subvert them at the demand of their government. And just as Russia is penetrating the U.S. power grid so they have that capability in the event of hostilities, many countries are almost certainly doing the same thing at the consumer level."

Richard Brooks's picture
Richard Brooks on Jun 25, 2020 4:47 pm GMT

I agree with Bruce's analysis, this is why we need to be more diligent in the steps we can take to secure the supply chain. Edison Electric Institute (EEI) advises the BES responsible entities to require their vendors to supply a Software Bill of Materials (SBOM) for their software objects: Model Suppply Chain Procurement Langauge V2.0, May 2020

Get Published - Build a Following

The Energy Central Power Industry Network is based on one core idea - power industry professionals helping each other and advancing the industry by sharing and learning from each other.

If you have an experience or insight to share or have learned something from a conference or seminar, your peers and colleagues on Energy Central want to hear about it. It's also easy to share a link to an article you've liked or an industry resource that you think would be helpful.

                 Learn more about posting on Energy Central »