The mission of this group is to bring together utility professionals in the power industry who are in the thick of the digital utility transformation. 

Post

Has CIP-013 worked?

image credit: © Adam121 | Dreamstime.com
Tom Alrich's picture
Supply chain Cyber Risk management - emphasis on SBOMs and VEX documents Tom Alrich LLC

I provide consulting services in supply chain cybersecurity risk management and am now primarily focused on software bills of materials (SBOMs) and VEX (Vulnerability Exploitability eXchange). I...

  • Member since 2018
  • 329 items added with 96,535 views
  • Apr 27, 2022
  • 696 views

This item is part of the Special Issue - 2022-04 - Cybersecurity 2022, click here for more

When FERC mandated in July 2016 that NERC develop a supply chain cybersecurity risk management standard, just about everyone (including me) focused on the fact that it was probably the first supply chain cybersecurity standard outside of the military – in the US and possibly anywhere. This was true, but NERC CIP-013 (produced in response to FERC’s order) was also one of the first cybersecurity risk management standards: i.e., a standard whose explicit goal was establishment of a cyber risk management program. I think that will be CIP-013’s most important legacy. This is why I say that:

Your access to Member Features is limited.

A brief, admittedly biased history of the CIP standards

No NERC standards before CIP version 1 had even mentioned anything about risk. And why should they have? They were all about physical actions: trimming trees under Transmission lines, balancing load and supply in real time, etc. The results of these actions could be objectively measured and needed to be kept within certain operating limits. Risk management had nothing to do with those standards. They were based ultimately on the laws of physics, which in themselves don’t understand risk.

However, the team that drafted CIP version 1 knew that cybersecurity was different. They knew there’s just about zero historical data that will let you make predictions like, “If we decrease our patching interval from 45 to 30 days, our chances of suffering a cyberattack – that could cause an adverse grid event – will go down by 15%.”

Instead, the only thing you can say with certainty is, “Patching more frequently will in general lower the risk that we will be compromised through an unpatched software vulnerability.” Does that mean a utility should patch every hour, even if it means shutting down the rest of its operations and putting its entire staff to work on patch management? No. Like every organization, electric utilities have limited resources and have many other risks (both cyber and otherwise) that they need to manage. They need to balance their various risk management activities against each other, so they can reduce as much overall risk as possible, given their available resources. Oh, and they have to keep the lights on at the same time.

In other words, cybersecurity is at heart about risk management. An important principle of risk management is accepting risks that are too expensive to mitigate, when weighed against the benefits that would be realized by mitigating them. Sure, a meteor strike on your headquarters would have a devastating effect on an organization (to say nothing of their people). Does that mean that every organization in the world should spend every extra dime (shekel, euro, rupee, etc.) they have on protecting their headquarters against a meteor strike?

No, because risk is a combination of likelihood and impact. The impact of a meteor strike would be huge, but the likelihood is so small that the risk itself is close to zero. No organization that I know of spends anything to fortify their headquarters (or anything else) against meteor strikes. By the same token, there’s no amount of spending on cybersecurity that would allow any organization – electric utility, dry cleaners, pizza parlor, the US military, etc. – to say it was perfectly cybersecure. There ain’t no such thing. There will always be some residual risk that the organization needs to accept.

For this reason, the CIP v1 drafting team included in many of the CIP v1 requirements the words “…or acceptance of risk…” In other words, if the utility believes that the cost of fully complying with the words of the requirement would outweigh whatever benefits (in risk reduction) would be achieved by compliance – and can document their reasons - they would have the option of not fully complying with those words. And they wouldn’t be held in violation of the requirement.

But FERC wouldn’t have any of this. When they approved CIP v1 – after 17 months of consideration – in early 2008, they said, somewhere in their 800 (or so)-page Order 706, that they wanted NERC to start work on a new version that would, among many other things, eliminate the wording about acceptance of risk. And in the meantime, NERC needed to audit as if those forbidden words weren’t there (by the way, if you want to see a very impressionistic history of NERC CIP that I wrote in 2018, go here).

But this caused a problem: The team that drafted CIP v1, believing that the language about acceptance of risk would remain in the requirements they were drafting, deliberately made the requirements quite prescriptive. After all, if a NERC entity found they couldn’t follow the prescriptive wording of the requirement, they could always just accept the risk. What could possibly go wrong?

As it turns out, a lot. By eliminating  “acceptance of risk” but not changing the overly prescriptive nature of the CIP v1 requirements, FERC left NERC with the worst of both worlds: hard-edged prescriptive requirements with no means of “softening” them through risk considerations.

Over the first four or five years of CIP enforcement – and as CIP v1 was replaced by v2 and then v3 - there were loud and growing complaints from NERC entities about the amount of time and money they were being required to spend to comply with rigidly prescriptive requirements. Yea, great was the weeping and wailing and gnashing of teeth over NERC’s “zero-tolerance” (i.e. non-risk-based) enforcement of the CIP requirements.

It wasn’t until CIP version 5 came into effect in 2017 that there were any CIP requirements (let alone entire standards) that at least implicitly allowed NERC entities to take risk into account: these pioneering requirements, both part of CIP v5 (which was a complete rewrite of all the CIP standards), were CIP-011-5 R1 (Information Protection) and CIP-007-5 R3 (Anti-Malware)[i].

But the watershed event in moving to a risk management approach was FERC Order 829, which ordered NERC to develop a supply chain security standard in July 2016. Even though NERC had never – since their unfortunate experience with “acceptance of risk” - officially referred to “risk” in any standard, in Order 829, FERC specifically called for a “supply chain (cyber) risk management” (my emphasis) standard. And they specifically warned against “one size fits all” – i.e. prescriptive – requirements.

Why this change of heart? For one thing, FERC certainly had heard the complaints about the then-current standards (when they issued Order 829 in 2016, the industry was still complying with CIP version 3, although they were preparing for CIP v5, which came into effect on July 1, 2017), and knew that the last thing NERC entities needed was another highly prescriptive CIP standard.

But I believe (and believed in 2016) that the main reason why FERC wanted the new standard to be risk-based was because there was simply no other way to do it. Remember, even though the burden of compliance with CIP-013 falls on electric utilities (and other bulk electric system entities like federal power marketing agencies and independent power producers), the standard is really aimed at the suppliers of the hardware, software and services that those utilities rely on to operate the BES.

Let’s say a utility decided to hold all of those suppliers to a very high standard of cybersecurity, e.g. ISO 27001. For some suppliers, such as the supplier of the energy management system (EMS) that literally runs the utility’s own corner of the grid, it makes sense to require this. However, for other suppliers, such as perhaps the vendor of maintenance services in a power plant, this would be overkill.

If the utility tried to force the maintenance services vendor to comply with ISO 27001, they would quickly find themselves looking for a new vendor. A vendor isn’t going to spend many times the profit they may make from a customer in order not to lose that customer; they’ll simply try to find another customer (perhaps in another industry) that isn’t so demanding. And if the vendor can’t find another customer to replace the utility, they would still be much better off financially than if they’d agreed to pay for an ISO 27001 audit just to keep one customer.

And this is the problem with supply chain risk management: The organization has to convince their vendors to incur costs in order to keep them as a customer. They’ll never do that if they require them all to comply with the same high standards, rather than tailor their requirements to the degree of risk posed by each vendor. A supply chain risk management standard like CIP-013 has to be risk-based, if it is to have any chance of succeeding.

The NERC standards drafting team (SDT) took Order 829 to heart when they developed CIP-013. In fact, in my opinion, the SDT went a little too far. CIP-013 consists of a grand total of five sentences (although with a number of clauses), divided into three requirements:

  1. The first part of requirement R1 (R1.1) tells the utility to develop a “supply chain cyber security risk management plan(s)”. The plan needs to “..identify and assess cyber security risk(s) to the Bulk Electric System from vendor products or services resulting from: (i) procuring and installing vendor equipment and software; and (ii) transitions from one vendor(s) to another vendor(s)..”
  2. The second part of R1 (R1.2) identifies six risks that need to be included in the plan, such as “Disclosure by vendors of known vulnerabilities related to the products or services provided to the Responsible Entity”. While all six of these risks are important ones, they weren’t developed as some sort of comprehensive catalog. Rather, the drafting team (and I attended a number of their meetings) simply gathered in one place six statements about particular risks that FERC made at various random points in Order 829. These six risks are by no means the only six that need to be included in the utility’s supply chain cyber risk management plan. But these are the only risks that FERC specifically required to be included in the plan; the utility is free to decide for themselves what other risks should be included.
  3. R2 requires the utility to “..implement its supply chain cyber security risk management plan(s) specified in Requirement R1.” That’s literally all it says.
  4. R3 requires the utility to review its plan every 15 months.

I just summarized the entirety of CIP-013. The utility has lots of freedom to develop their supply chain cyber risk management in R1, but in R2 they have no freedom at all: they have to implement the plan as written. However, this isn’t as bad as it sounds: If the utility decides they made a mistake in their original plan – or if they realize that changed circumstances require a change in the plan - they’re free to change it at any time; they just have to document the changes they made and why they made them.

When NERC entities were starting to think about CIP-013 compliance, I know some of them made a very understandable mistake (especially for NERC entities): They focused on the six items in R1.2 and considered these to be the total of what’s required by CIP-013. After all, those six requirement parts were the most like the existing CIP requirements; why shouldn’t they be the only real “requirements” in CIP-013?

However, taking that attitude was based on completely ignoring the wording at the beginning of CIP-013 R1: “Each Responsible Entity shall develop one or more documented supply chain cyber security risk management plan(s)… The plan(s) shall include:..” This is followed by the text of R1.1 and R1.2, meaning that what’s described in both R1.1 and R1.2 needs to be included in the plan. So, without a doubt, the plan needs to include the six R1.2 items. But it also needs to include R1.1.

What threw these utilities off was probably the fact that R1.1 doesn’t prescribe anything in particular: It just requires the entity to develop a plan to “identify and assess” supply chain cybersecurity risks. In other words, it’s up to the entity themselves to determine what are the risks they face. Of course, that idea was a 180-degree change from all previous NERC standards. If you were complying with any of the CIP version 3 standards, or the FAC, BAL, etc. standards, and you told an auditor that you should be allowed to decide what goes in your compliance plan, not NERC, how far do you think that would get you?...You’re right – not very far at all.

For this reason, it’s understandable that NERC entities wouldn’t know what to do with a requirement that says it’s up to them to decide what goes into their supply chain cyber risk management plan. What’s to prevent an entity from simply including the six R1.2 items in their plan? In other words, how could they be found non-compliant if that’s all they put in their plan?

They can be found non-compliant because they’d have to convince the auditor, as part of their R1.1 compliance evidence, that they searched high and low to find any other supply chain cyber risks that applied to them, beyond the six risks in R1.2 – and they just couldn’t find any others at all. If I were the auditor and that evidence were prevented to me, I’d just ask some questions like:

  1. What about SolarWinds-type attacks? Don’t you think you should be concerned about those? More importantly, what have you done that makes you immune to such attacks?
  2. What about Log4j-based attacks? Have you determined that you don’t have any log4j at all in your environment (even in a component of a component of a component)?
  3. How about the risks identified in the NATF Criteria? They include the six R1.2 items, but they go well beyond those (there are 60 Criteria now), including:
    1. The risk that a software or device supplier won’t follow a secure development lifecycle (criterion #47);
    2. The risk that a supplier might install a backdoor while developing a product and not remove it before they ship it to you, leading to your being compromised (criterion #15);
    3. The risk that a product supplier might not conduct 7-year background checks on its employees (criterion #4); etc.

These are all supply chain cyber risks. The entity will have to convince the auditor that they at least considered all of these risks when they were developing their plan. And if they didn’t include them in the plan, they’ll have to provide some justification for not including at least each of the 60 NATF Criteria. From what I’ve heard about how CIP-013 is being audited, and from the presentations I’ve seen by regional auditors on the subject, I really don’t think an entity that only included the R1.2 risks in their plan wouldn’t find the auditor to be fairly skeptical.

In retrospect, it would have been good if the drafting team had included a list of say ten areas of supply chain security risk that the entity needs to consider in their plan; if they decide one or more of those areas don’t apply to them, they need to document that fact. These areas might include:

  1. Software supply chain risks, including the risk that a malicious party might have implanted a backdoor in the software while it was being developed, as in SolarWinds; or the risk that a serious vulnerability would be identified in a widely used open source component like Log4j, making it difficult even to find all the vulnerable instances on your network.
  2. Inadequate protection of the supplier’s remote access system (i.e. no MFA). DHS said in 2018 that at least 200 vendors to the power industry had been penetrated by the Russians through their remote access systems, in attempts to penetrate US electric utilities through them.
  3. Inadequate anti-phishing training and other anti-phishing measures, making the vendor a possible vector for attacks aimed at the utility. In early 2019, the Wall Street Journal published a great article on how the Russians were penetrating vendors to the power industry through phishing attacks, then utilizing that access to penetrate electric utilities; the article listed four utilities that had been penetrated this way.

Had the drafting team included this list in R1 (and I certainly never even thought to suggest it to them), this would have made it clear to utilities that the purpose of R1 was to allow them to decide the risks that were most important to them, given their configuration of assets and vendors. Being able to allocate your resources toward the risks that are most important is a key element of supply chain cyber risk management.

And there would have been another benefit: Including this list in the requirement itself would have made it auditable. That is, the auditors would have been able to determine whether or not the utility gave serious consideration to each of these areas of risk, based on the documentation they were shown. If the utility hadn’t even considered each of these areas, they would have been eligible for a PNC (potential non-compliance) finding in an audit.

However, as far as I know, a substantial majority of NERC entities (who have medium or high impact BES Cyber Systems, since they are the only ones in scope for CIP-013 so far) did a good job of identifying and assessing supply chain cyber risks in R1, in spite of there not being a list of risks in the requirement itself. This is because organizations – especially the North American Transmission Forum (NATF) – stepped up to provide the guidance that the drafting team didn’t want to include in the requirement itself.

To get back to the question in the title, has CIP-013 worked? Yes, it has. It’s worked both as a supply chain cybersecurity standard, and as a cybersecurity risk management standard. It could have been better, but it could also have been a lot worse.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. 


[i] They were joined by CIP-010-2 (now -3) and CIP-003-7 (now -8), which are also risk-based requirements – although neither of them mentions risk, either.

Tom Alrich's picture
Thank Tom for the Post!
Energy Central contributors share their experience and insights for the benefit of other Members (like you). Please show them your appreciation by leaving a comment, 'liking' this post, or following this Member.
More posts from this member
Discussions
Spell checking: Press the CTRL or COMMAND key then click on the underlined misspelled word.
Barry Jones's picture
Barry Jones on May 3, 2022

The answer to this question depends on what you interpret "worked" to be. If "worked" translates to putting language in place to require something with the name Supply Chain, i would say yes. If "worked" translated into preventing a Solarwinds or other supply chain issue, then no.

There are a couple of inherent problems in CIP-013 (and CIP standards across the board) and that is poor obscure language and requirements. Some language can mean everything and nothing at the same time. Catch me outside some day Tom and we can discuss (It's part of my Grassy Knoll theory of CIP compliance).

CIP-013 for example requires in R1.2.5. - Verification of software integrity and authenticity of all software and patches provided by the vendor for use in the BES Cyber System. So what's the problem here? The goal of supply chain is to identify cyber security risks before implementing technologies. From a software management perspective verifying software integrity and authenticity occurs during operational patching phases when a patch is acquired for use. A SysAdmin does this in CIP-007 R2 or CIP-010 R1 where they verify patches, their applicability and implement the changes. I argue 1.2.5 belongs there to keep the standard requirements aligned with internal security programs. This is a minor point because one can say "just move the process governance activity of verifying software in CIP-013 into your CIP-007 or CIP-010 processes." But then i'd ask why put in there to begin? The more aligned entities can get their governance and internal processes, the more efficient they are and are more understandable to auditors. While this is a minor issue there are greater issues which are not addressed. The same occurs in R1.2.6 around remote access. Those controls occur when the system is operational and in CIP-005 (and soon CIP-003 R6) and part of the access design in CIP-005 (and CIP-003).

CIP-13 also does not verify security controls of vendors. It does elicit responses to questions from entities. For example, an entity may ask the supply chain vendor "do you notify us of vendor-identified incidents...(Requirement 1.2.1)." The answer may depend on who responds to that question. Is it a Sales Engineer completing that question? Someone in Legal? Is it a security person? Or compliance admin? And when that question is answered, what is the expected evidence an entity shows compliance to an entity? A contract? An internal procedure used by the vendor to notify? A screen capture of a vendor's machine that notifies an entity? Vendors may not want to share their internal processes due to potential litigation? Cloud providers surely won't. Lest they end up in court explaining why they didn't follow their procedures ending up in a PV for an entity. And at the end of the day the agreement or contract doesn't stop anyone from compromising a vendor system. It goes much deeper and broader than that example. Question? Will the regulators share with entities when/if their systems are compromised? Probably not. Should they?

Lastly, the part that is missing from CIP-013 is SDLC. Software development source and quality is not required. Entities have to trust the source of the software to be safe to use. So an entity can use an authorized patch developed by anyone anywhere and provided to the vendor. CIP-013 does nothing to stop authorized bad patches. Think Solarwinds. There's no verification, so there's no control. We will trust the vendor to make sure their processes are sound. And given the pressures on cost and maintenance, such verifications may not occur.

CIP-013 is a start, but does not address key risks in supply chain or cloud.

 

Tom Alrich's picture
Tom Alrich on May 4, 2022

Barry, regarding your point about R1.2.5 and R1.2.6: There's a difference between supply chain security and procurement security. Securing your supply chain is a continuous process; it isn't something that you just do before and during procurement. Every time you interact with a vendor's people or systems, you need to consider security. 

You may have forgotten that the drafting team originally just had system-to-system remote access in R1.2.6, but then also added it to CIP 5 R2.4 and 2.5. They also just had the software integrity and authenticity verification in R1.2.5 and then they added it to CIP 10 R1.6 (actually, a different SDT drafted the changes to the other standards). So they're actually addressed in both places, but the difference is that, in CIP 3-11, the entity has to document every instance of compliance, whereas in CIP 13, they just have to document that they have a good plan and they're following the plan. I wouldn't mind seeing all the CIP standards being made like CIP 13, rather than the other way around.

Regarding vendor security controls, it's absolutely incumbent on the entity to verify, to their satisfaction, that the vendor is doing things they promised to do. In most cases, I think a questionnaire is sufficient, but if an entity really doesn't trust a vendor, by all means they should audit them. Of course, if an entity ignores a lot of evidence that a vendor has seriously deficient security and pretends there's no problem, they might well be found in violation of R2. But FERC, in order 829, pointed out that they didn't want a prescriptive, "one-size-fits-all" standard; this approach is consonant with the Order.

I would also have liked to see SDLC specifically called out, probably in R1.2. In fact, I would have liked to have the SDT develop a set of supply chain security disciplines (like SDLC, vendor remote access, secure manufacturing, etc.) and required that the R1 plan address each one of those (this is in retrospect. I attended a number of the SDT meetings and never once brought this up!). But R1.1 does require the entity to identify and assess supply chain risks. I think any entity - especially after SolarWinds - that doesn't address SDLC risks at all has really missed the boat, although I wouldn't say they're in violation. If the set of disciplines was in R1 and the entity didn't address one or two of them at all (even to say "This one doesn't apply to us, because..."), then I think they could be held in violation. 

But supply chain cyber risks are, along with perhaps ransomware, the most serious cyber risks worldwide nowadays (with software supply chain cyber risks in particular at the top in importance). All electric utilities should be following CIP 13, regardless of whether they have to or not. And they should try to make their R1 plan as comprehensive as possible, since this is so important. Even more so than when FERC ordered the standard in 2016.

Tom Alrich's picture
Tom Alrich on May 4, 2022

By the way, I'm currently leading two NERC SCWG teams revising the guidelines we put out in 2019, on SCRM lifecycle and vendor risk management lifecycle (I led the two teams that developed the original versions). Each team meets twice a month and we have some good discussions on questions like these. It would be great to have you join in when you can. Just send me an email at tom@tomalrich.com.

Of course, this goes for anyone else in the NERC community who'd like to participate.

Richard Brooks's picture
Richard Brooks on May 3, 2022

I believe NERC's Supply Chain Working Group survey results answered this question, here are some telling responses:

“We all cringe when we know we have a to make a purchase.”

Supply Chain is a global issue for all critical infrastructure and not just electric utilities

Any solution the federal government devises needs to be useful for all sectors, not just ours

Implementing different requirements will only serve to frustrate vendors

The supply chain requirements as written are reasonable. It's the audit oversight piece that has everyone worried

Our organization has spent significant time attempting to comply with the SCRM standards without assurance that compliance has been or can be achieved;

John Benson's picture
John Benson on May 9, 2022

Very good post, Tom.

The bad news is that supply chain cyber-security risks are really tough, because a utility rarely has specific insight into most supply chain firms' cyber-security practices. The exception being (or should be) any firm that is involved in that utility's cyber infrastructure.

The good news regarding risk is, each industry has certain mandated due diligence requirements, especially if they are a publicly traded company. Even though these are complex and vary from sector to sector, it might be good for a medium-to-large utility to (1) identify parts of their supply chain that might cause a major disruption, (2) investigate the due diligence requirements in those sectors, (3) reference those requirements in any future contracts and (4) negotiate any issues that a given critical-supply-company has with imposing those requirements in a contract.  Over time, as with all things, the utility will develop skill at accessing and resolving these risks.

-John

Barry Jones's picture
Barry Jones on Jun 12, 2022

Exactly! Those risks and mitigation due diligence actions are paid for by ratepayers and will also lead to market pressures driving security. I will stay in my lane and trust but verify my supplier to the extent possible - but do not have unlimited resources to evaluate the mine that made the metals that were used in the component. This is why off-prem computing (cloud) is so interesting. It can put risks to critical infrastructure in someone else's wheelhouse - same as supply chain.

Tom Alrich's picture
Tom Alrich on Jun 13, 2022

Barry, I disagree that the cloud puts risks somewhere else. You might ask Capital One if they were able to offload those risks. The risks belong to the owner of the data, regardless of whether the data are in the cloud or on premises.

 

Tom Alrich's picture
Tom Alrich on Jun 13, 2022

Sorry, John. I just saw your comment now. NERC CIP-013 requires all 4 of the actions you point to.

Get Published - Build a Following

The Energy Central Power Industry Network is based on one core idea - power industry professionals helping each other and advancing the industry by sharing and learning from each other.

If you have an experience or insight to share or have learned something from a conference or seminar, your peers and colleagues on Energy Central want to hear about it. It's also easy to share a link to an article you've liked or an industry resource that you think would be helpful.

                 Learn more about posting on Energy Central »