Posted to Energy Central in the Digital Utility Group
Audra Drazga's picture
Vice President of The Power Industry Network, Energy Central

I am the VP of the Energy Central Power Industry Network.  In this role, I help to connect professionals in the power industry through the development and management of topic-specific community...

  • Member since 2012
  • 1,027 items added with 645,733 views
  • May 17, 2021
  • 1291 views

The cyberattack on the Colonial Pipeline had a widespread impact and surprised many people, but for those working in energy cybersecurity it wasn't terribly shocking to see vulnerabilities in our energy systems exploited. Now that the pipeline is flowing again but the focus remains on energy security, what should be the specific and actionable priorities of all stakeholders in the energy system to prevent the next attack from being even more impactful?

Your access to Member Features is limited.

The July 28 Presidential Memorandum on Cybersecurity aims to create a baseline set of cybersecurity requirements/goals for all critical infrastructure. The inter-dependencies across critical infrastructure demands that a baseline cybersecurity "safety net" is in place. https://energycentral.com/c/gr/how-do-biden-administrations-planned-cybersecurity-performance-goals-affect

I agree with David. As long as utility controlling computers are connected, the treat exists. The bad actors will always be a few steps ahead of the cyber teams.

Audra, conspicuously missing from this conversation is where the blame for Colonial's ransomware attack (and 11,000 other recent ransomware victims) lies: in vulnerabilities of Microsoft Exchange Server. What actions should stakeholders in energy systems take? The first should be to install another server, then hire a competent IT staff to configure and maintain it.

Disconnect any and all utility controlling computers entirely from the internet.

There are indeed several consequences of this both in terms of cost and side effects.

However. The cost is much lower than continuing this menace.

It is not complicated but it will indeed take time and sweat.

John Benson's picture
John Benson on May 20, 2021

Hi David:

I believe most electric utilities and federal control systems already do this, and have for a number of years.

However, cyber-spies have identified a work-around. It's called promiscuous (unmanaged) USB-drives. That's why the computers used in many of these control systems only have a single USB ports that are inside computer enclosures, strict rules for accessing these ports with only tightly managed USB-drives (and/or other devices).

-John

Bob Meinetz's picture
Bob Meinetz on Aug 5, 2021

John, since before there was an internet nuclear plants have had a strict "air lock" around control systems. The only wires that connect with the outside world are wires that send raw power out and take raw power in.

Why connect to the outside world to power a nuclear reactor? In case of generator or other balance-of-plant malfunctions, it's needed to shut the plant down. If the grid shuts down, on-site batteries instantly kick in to power a shutdown. Diesel generators take over soon thereafter.

Nuclear plant control systems are primitive by today's standards - by design. That's why in a nuclear plant control room, there are no USB ports, disk drives, wi-fi, bluetooth, Netflix, YouTube, Hulu. A very boring place to work, by design.

I would guess that power systems at coal, natural gas, plants, etc. have similar protections in place proportional to the risk they represent. But rest assured, that for every potential catastrophe at a nuclear plant that haunts the sordid imaginations of anti-nuclear activists, there are teams of smart people who have already considered it, and developed a robust defense against it.

I posted the paper linked below as part of the “The Future of Electric Power in the United States” series. Most of this part covers cyber security and related resiliency issues. Although this is for the Electric Utility Industry, I’m guessing that FERC also regulates the petroleum pipeline industry, and will use their authority to extend regulations like NERC’s Critical Infrastructure Protection (CIP) regulations to the pipelines, and perhaps other energy-related functions.

https://energycentral.com/c/gr/future-electric-power-united-states-%E2%80%93-part-3

The pipeline industry and many other industries (including the electric utility industry) are involved in  the National Council of ISACs. ISACs are Information Sharing and Analysis Centers. These are briefly covered  in the above linked “the Future…” document, but mainly as focused on the Electricity ISACs. The document linked below is an older paper that I updated about a year ago. It contains a more general and thorough descriptions of ISACs in section 3.2.7.

https://energycentral.com/c/iu/cyber-security-basics-rev-b

-John

 

A

Audra,

What you are going to see is a knee jerk reaction to this event, I can tell you exactly what the next steps will be because I've seen it hundreds of times. First, the company will throw millions of dollars in tools, cyber firms and rearchitect of the network, they might even remove a few bodies from the management realm, then the government will push for more regulation and oversight and then you'll see endless case studies of how to make it better and never happen again but yet it will.

I did 5-week blog about "Cybersecurity is broken" last year which laid out exactly what was wrong with the cybersecurity industry as a whole.  Part of the problem is what Mr. Meinetz touched on in his post.  Microsoft does have a lot of problems but they had such a great marketing thrust in the 90's that made everyone Microsoft Junkies and it will probably never change, it's too far engrained in our culture that it will never go away, at least not in our lifetime.  Cybersecurity in general is much bigger than just the OSs, it goes much deeper and is evolving all of the time.  

I was dealing with an Incident Response of ransomware the Monday before Colonial reported they had been hit with ransomware.  It too was the Darkside ransomware.  The company that had called us was a smaller (45M) company and not a large pipeline but they were hit with the same malware.  I just finished the root cause analysis (RCA) over the weekend and it came down to a person clicking on a phishing email (pretty certain it happened the same way at Colonial).  However, what was interesting is that the company was running Barracuda Email Security Gateway which didn't detect it as bad, then the company was only using Microsoft Defender as its AV protection.  We also found out that they never did annual security training, they never did patching, they didn't spend the money (approx. 4K a year) to use a better AV solution, weak passwords, and it was a flat network.  The one thing they did right was they did do backups......but it was on the same network.  They were fortunate that the first action I told them when they called us was to disconnect the backup server immediately.

Not wanting to go down a rabbit hole but where did they fail?  No security training so the person wouldn’t have clicked on the link?  The email security tools which employs signature-based protection (which I am not a believer in) with the email protection?  Problem with signature-based protection is you need to know about the problem before you can stop the problem.  No patching?  Flat network? And on and on.

As my Dad told me, “life is hard, but it’s harder when you make dumb decisions”.  Cybersecurity is hard, but it’s much harder if you don’t do anything or take the bare minimum approach.  I like the list that Mr. Brooks provided but I feel it’s missing some items and it’s at the 20K foot level.

It comes down to practicing good cyber hygiene.  Annual security training, patch management program, configuration management program, business continuity planning, incident response plan, proper network security segmentation, updated tools, vulnerability scanning, penetration testing, backup plans, multiple backups, etc…..  and there are probably more.

Bob Meinetz's picture
Bob Meinetz on May 19, 2021

Aaron, it seems two simple steps:

1) Stripping attachments from all emails originating from outside sources, and
2) Forcing employees to use secure passwords

would prevent 90% of ransomware attacks. Would you agree?

Steve Lindsay's picture
Steve Lindsay on May 21, 2021

Bob, I would argue strongly that eliminating passwords altogether is a much more secure solution.  We wrote an article (and by "we" I clearly mean my much more intelligent cohort Danny Vital) on using digital certificates over PKI to authenticate, validate and authorize a user (or device).  The article also addresses shared accounts as mentioned below.  By making it much easier to login in and out, a digital certificate helps to eliminate shared accounts so commonly found in OT environments.

The tech has been in use by the government and military in the form of PIV and CAC cards for 15 years now.

You can read it here: https://energycentral.com/c/iu/water-cyberattack-highlights-need-strong-...

My two cents,

Steve

Aaron Fansler's picture
Aaron Fansler on May 21, 2021

Not just attachments but hyperlinks as well.  Yes, using secure passwords but also not using shared accounts, which I see all the time in OT environments.  Those would greatly decrease the ability for ransomware to infiltrate a network.  The other one is network enclaving / segmentation, if you decrease the ability for it to migrate from asset to asset then you will be limiting the effects.    

Audra, the first step is using a secure operating system that isn't a first target for attackers: https://energycentral.com/c/cp/colonial-pipeline-attack-linked-microsoft... An attack on national defense systems would certainly be more impactful: "WHILE MICROSOFT CONTINUES to trumpet the success of its NT operating system over Unix-based systems, the US Navy is having second thoughts about putting NT at the helm. A system failure on the USS Yorktown last September temporarily paralyzed the cruiser, leaving it stalled in port for the remainder of a weekend. "For about two-and-a-half hours, the ship was what we call 'dead in the water,'" said Commander John Singley of the Atlantic Fleet Surface Force. The warship was testing its new Smart Ship system, which uses off-the-shelf PCs to automate tasks that sailors have traditionally done themselves. "The Navy started the Smart Ship program with three essential goals in mind: improve combat readiness, reduce crew workload and operating costs, and to do it safely," said Singley." Sunk by Windows NT https://www.wired.com/1998/07/sunk-by-windows-nt/

Audra, The World Economic Forum released a guidance document for Board members and Corporate Officers of Oil and Gas pipeline companies on 5/17/2021.

I highly recommend reading the entire document, it contains some really useful guidance, which applies across industries.

Here are the key principles:

PRINCIPLE 1 Responsibility for cyber resilience
The board as a whole takes ultimate responsibility for oversight of cyber risk and resilience. The board may
delegate primary oversight activity to an existing committee (e.g. risk committee) or new committee (e.g.
cyber-resilience committee).
PRINCIPLE 2 Command of the subject
Board members receive cyber-resilience orientation upon joining the board and are regularly updated on
recent threats and trends – with advice and assistance from independent external experts upon request.
PRINCIPLE 3 Accountable officer
The board ensures that one corporate officer is accountable for reporting on the organization’s capability to
manage cyber resilience and on progress in implementing cyber-resilience goals. The board ensures that
this officer has regular board access, sufficient authority, command of the subject matter, experience and
resources to fulfil these duties.
PRINCIPLE 4 Integration of cyber resilience
The board ensures that management integrates cyber-resilience and cyber-risk assessments into the overall
business strategy and into enterprise-wide risk management, as well as budgeting and resource allocation.
PRINCIPLE 5 Risk appetite
The board annually defines and quantifies business risk tolerance relative to cyber resilience and ensures
that this is consistent with the corporate strategy and risk appetite. The board is advised on both current and
future risk exposure as well as regulatory requirements and industry/societal benchmarks for risk appetite.
PRINCIPLE 6 Risk assessment and reporting
The board holds management accountable for reporting a quantified and understandable assessment
of cyber risks, threats and events as a standing agenda item during board meetings. It validates these
assessments with its own strategic risk assessment using the board’s cyber-risk framework.
PRINCIPLE 7 Resilience plans
The board ensures that management supports the officer accountable for cyber resilience through
the creation, implementation, testing and ongoing improvement of cyber-resilience plans, which are
appropriately harmonized across the business. It requires the officer in charge to monitor performance and
to regularly report to the board.
PRINCIPLE 8 Community
The board encourages management to collaborate with other stakeholders, as relevant and appropriate, in
order to ensure systemic cyber resilience.
PRINCIPLE 9 Review
The board ensures that a formal, independent cyber-resilience review of the organization is carried out annually.
PRINCIPLE 10 Effectiveness
The board periodically reviews its own performance on

Audra Drazga's picture
Audra Drazga on May 17, 2021

Dick,

Thanks for sharing this.  Curious do you think an annual review will be enough?  It seems like this is an ever-moving target.

Richard Brooks's picture
Richard Brooks on May 18, 2021

Audra, I think this might depend on the industry. DoD may require more frequent reviews by Gas Pipelines that are  serving critical facilities.