The mission of this group is to bring together utility professionals in the power industry who are in the thick of the digital utility transformation. 

WARNING: SIGN-IN

You need to be a member of Energy Central to access some features and content. Please or register to continue.

Post

A powerful example of why we need SBOMs

I’ve written several posts recently on the idea of software bills of materials. I titled those posts “Is software bill of materials the answer to all of our problems?” I must confess that this was a bit of an exaggeration. For one, I don’t think SBOMs will solve the problems of climate change or inequality, let alone the problem of teenagers who won’t listen to their parents. And they won’t even solve all of our cybersecurity problems. But what I find most exciting about them is that they offer the key to identifying a whole new class of software vulnerabilities which have always been present but have seldom been visible. Here is a great real-world example.

The example was discussed in an excellent paper (one of a number of excellent papers) produced by the Software Component Transparency initiative of the National Telecommunications and Information Administration (NTIA) of the Department of Commerce – a group, led by Dr. Allan Friedman, that is attempting to hasten the day when SBOMs will be widely available and widely used. The paper is called “Roles and Benefits for SBOM across the supply chain”.

On page 25 of that paper, the authors (members of the initiative from various industries and government agencies) point to the “Urgent/11” vulnerabilities in safety critical systems, which were brought to light in the summer of 2019. To quote the paper, “The initial vulnerability finders identified that the flaws were present in certain versions of Wind River’s VxWorks RTOS (Real Time Operating System). In response, Wind River provided thorough impact analysis of their affected versions of VxWorks.” However, it turns out that wasn’t enough, since “It was later revealed that the actual vulnerabilities were from a supplier to Wind River - named IPnet (whose creator, Swedish company Interpeak, was later acquired by Wind River).”

In other words, while VxWorks was definitely vulnerable, the source of the vulnerability was a component that Wind River had purchased and included in VxWorks. Moreover, that component, IPnet, was also included in a number of software packages from companies including ENEA, Green Hills, Microsoft, Mentor, TRON Forum, and IP Infusion. Those other products were also potentially infected, but of course their users didn’t know that.

How do SBOMs fit into this story? Neither VxWorks, nor any of the components from the other six companies (which are presumably competing RTOS’s), is ever purchased directly by an end user; the user acquires them when they acquire certain “safety critical” device (one example being an infusion pump used in hospitals). When Urgent/11 was believed to be only a VxWorks vulnerability, Wind River notified their customers (the device makers) and provided a patch to them.

But did those device makers notify their users of the vulnerability and provide the patch to them? The paper doesn’t say, but I point you to a quote from a 2017 Veracode study, that I included in the post linked at the top of this post: “A mere 52 percent of companies reported they provide security fixes to components when new security vulnerabilities are discovered.” So if you were a user of one of the devices that included VxWorks, and if those device makers followed the normal pattern, your chances were only 50/50 that you would have been told your device was vulnerable.

But let’s say you had insisted on being provided an SBOM when you purchased the product (and the Food and Drug Administration, which must approve medical devices before they can be sold in the US, announced in 2018 that they will require that an SBOM be provided before they will approve any new device). When you heard about the VxWorks vulnerability, and knowing that it was found in many types of devices, you would have scanned the SBOM and realized VxWorks was indeed included in your device, so you were potentially vulnerable.

At that point, you would have contacted the device’s manufacturer and asked when you would have the patch for the Wind River vulnerability. You would have been told one of two things:

1.      “Fortunately, even though we use VxWorks in our product, we don’t use the module that contains the vulnerability. So you don’t have to worry.” In many cases, a vulnerability in an “upstream” component doesn’t automatically apply to the “downstream” devices or software packages that include that component. For example, when the upstream component is a library of software modules (which was probably the case with VxWorks), not all of those are always incorporated into each downstream device, meaning the device isn’t in fact vulnerable. Or you would have been told:

2.      “Oh yes, we recently received that patch. As soon as we complete testing, we will get it out to you.” At that point, you shouldn’t have hung up. You should have asked “And when will you complete the testing?” You might have heard a reply like “Um, let me see…Yes that’s scheduled to be finished next week, so you should have the patch early the following week.” And if you didn’t hang up then, you might have heard the person call out “Joe, you know that VxWorks patch we got last month? People are starting to ask about it. You have to stop working on our super duper next version, then test the patch and ship it out. I just promised it for two weeks from today…Yes, I know you had a vacation scheduled next week. Let me talk to your wife, I’ll explain this to her...”

So this is one important reason you should start asking for SBOMs: If your Supplier is like half of its peers, you won’t be told about a vulnerability that might affect your systems, even though the supplier should know about it (and there’s no excuse for a software or device supplier not knowing about a vulnerability in a component of their software, unless they don’t even know what components are in their software – meaning there’s no way they can give you an SBOM. Hint: If a supplier can’t even list for you the third-party or open source components of their own software, you should find another supplier).

However, there’s another twist to this example, which points to another need. Once IPnet was determined to be the source of the vulnerability, the developer of that component presumably contacted all of their customers – including the six companies listed above – and sent them the patch. Presumably, those customers sent the patch on to their customers, the device makers. And we’ve already seen that, on average, there’s only a 50/50 chance the device makers would have passed the patch on to their end users (and of course provided support so they could install it correctly).

But what if you had asked for and received an SBOM from the maker of your device, and you read about the vulnerability in IPnet? Would you have known this vulnerability applied to your device and called the device maker about the patch (assuming they hadn’t yet sent it to you)? Not necessarily. Only if you had a “2-level” SBOM would you know there was IPnet in your device. That is, only if you knew about the components of the components in the device’s software would you have called your supplier and shamed them into sending you the patch.

Does this mean you should always ask for a two-level SBOM, not just a one-level one – in other words, a list of the components of the components of the software or device that you purchased and installed? You could ask if the supplier has one, but it’s not too likely they do today, simply because the use of SBOMs is still in its infancy. But the end goal is to have every SBOM include an SBOM for each of its components as well. In fact, it wouldn’t be bad if it went another level or two beyond that, so you would know all of the components of the components of the components of the components of the software you have purchased and deployed.

Is it likely that 4- or 5-level SBOMs will be available in the near future? Definitely not. It would be a huge step if every supplier just provided one-level SBOMs, as well as updated them whenever something changed in their software, let alone more than that. But think about it: If all suppliers (including open source communities) provided complete one-level SBOMs (meaning every component was listed), then almost all SBOMs could easily go down many layers. The reason I say this is that suppliers of components will ex hypothesi all provide SBOMs, meaning the only component SBOMs that wouldn’t be available to your supplier would be those for suppliers that have gone out of business, leaving their products unsupported.[i]

But let’s forget about universality at the moment. When will even a majority of software suppliers provide SBOMs? I don’t know when, but I will point out that Gartner, in a report titled Technology Insight for Software Composition Analysis,  said “By 2024, the provision of a detailed, regularly updated software bill of materials by software vendors will be a non-negotiable requirement for at least half of enterprise software buyers, up from less than 5% in 2019.” In other words, when customers start requiring SBOMs, the suppliers will provide them.

So now you know what you need to do.

 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Are you wondering if you’ve forgotten something for the 10/1 deadline for CIP-013 compliance? This post describes three important tasks you need to make sure you address. I’ll be glad to discuss this with you as well – just email me and we’ll set up a time to talk.

Tom Alrich's picture

Thank Tom for the Post!

Energy Central contributors share their experience and insights for the benefit of other Members (like you). Please show them your appreciation by leaving a comment, 'liking' this post, or following this Member.

Discussions

Matt Chester's picture
Matt Chester on Sep 24, 2020 11:22 am GMT

“A mere 52 percent of companies reported they provide security fixes to components when new security vulnerabilities are discovered.”

I expect that discussion of this figure would result in a lot of shocked faces in board rooms across the country-- I don't think the powers that be recognize the risks in front of them and too quickly adopt an 'out of sight, out of mind' mentality. It sounds like SBOM could be one way to rip off that band-aid of blissful ignorance

Tom Alrich's picture
Tom Alrich on Sep 24, 2020 8:15 pm GMT

You're exactly right, which is why the initiative driven by the Dept. of Commerce is moving ahead. But it's a terribly difficult problem, since the vendors don't want to provide SBOMs unless they're forced to by their customers, and the customers don't now have any way to incorporate SBOMs into their automated vulnerability management process. I'm now working with a company that might lead an SBOM proof of concept for utilities and power industry vendors under the DoC initiative, like the current healthcare proof of concept (and an autos proof of concept starts next month). And the configuration and vulnerability management vendors will have to be involved as well.

Get Published - Build a Following

The Energy Central Power Industry Network is based on one core idea - power industry professionals helping each other and advancing the industry by sharing and learning from each other.

If you have an experience or insight to share or have learned something from a conference or seminar, your peers and colleagues on Energy Central want to hear about it. It's also easy to share a link to an article you've liked or an industry resource that you think would be helpful.

                 Learn more about posting on Energy Central »