The mission of this group is to bring together utility professionals in the power industry who are in the thick of the digital utility transformation. 

Post

Zero-days aren’t the worst of our problems, unfortunately

Tom Alrich's picture
Supply chain Cyber Risk management - emphasis on SBOMs and VEX documents Tom Alrich LLC

I provide consulting services in supply chain cybersecurity risk management and am now primarily focused on software bills of materials (SBOMs) and VEX (Vulnerability Exploitability eXchange). I...

  • Member since 2018
  • 328 items added with 95,884 views
  • Aug 1, 2022
  • 172 views

I want to thank Kevin Perry for forwarding a post by Patrick Miller from last week on the Ampere Security blog. You should read it, but I’ll summarize the post by saying it describes the dangers posed by two “zero-day” vulnerabilities that two researchers found in “…a Moxa Ethernet-to-serial converter with the newest firmware.” Of course, the problem with zero-days is they won’t be picked up by vulnerability scanners, so – absent intrepid researchers who are willing to reach out to the supplier before they exploit the zero-days they found (and aren’t planning on exploiting the vulnerabilities themselves, of course) – neither the supplier nor their customers will have any way of learning about those vulnerabilities.

I don’t want to downplay the importance of this development, since zero-days pose a big risk to any organization, and especially critical infrastructure organizations like electric utilities. An Ethernet-to-serial converter is very unlikely to be found in any but a critical infrastructure organization, since only these organizations are likely to be “blessed” (or cursed, one of the two) with serially-connected devices. If Ethernet-to-serial converters installed in multiple electric power transmission substations were to be compromised, this could potentially cause a serious outage, and potentially the worst kind of outage: a cascading one.

However, while this is a real problem, in May I learned, from discussions sparked by Tom Pace of NetRise in an informal group of SBOM practitioners that I’m part of, that there are at least two even worse problems than individual zero-day vulnerabilities. One is currently only theoretical, but could in the longer term (a few years – that’s not that long) turn into perhaps a huge issue worldwide, were some creative person or nation-state decide to devote a little time to exploit it. It’s hard to imagine any quick practical solution to this problem (perhaps banning production of open source software? That shows how serious the problem could turn out to be). That problem is described in this post

The other problem is an immediate one and is currently huge, although it will be hard even to learn how big it is until the evidence becomes all too clear through successful attacks. But this problem can be solved with relatively easy measures - easy technically, although I’m not sure about politically. This is one of those problems where the cause can be easily traced to the wetware between the seat and the keyboard. I’ll focus on this problem in this post.

In the first post of the series, I described Tom’s initial presentation to our group, in which he described his experience with one in a family of infrastructure devices from a certain manufacturer. These devices are used in critical infrastructure and in – shudder! – the military. Were you to do your due diligence on this device before procuring it, you might well think it had perfect security. After all, when you search on the name of either the device or its manufacturer in the National Vulnerability Database (NVD), you will receive this response: “There are 0 matching records.”

Sounds good, right? Before you purchased the product, you wanted to see if the NVD listed any serious vulnerabilities for it. Not only did no serious vulnerabilities appear, but no vulnerabilities at all appeared. However, in his initial presentation to our group, Tom said he knew that, in a single piece of firmware in the device in question, there are 1,237 unpatched vulnerabilities due to open source firmware components. In fact, Tom later calculated, this time through examining all of the firmware and software installed in the device, that there were at least 40,000 unpatched vulnerabilities[i] in the one device – which of course appears to have no vulnerabilities at all, if you just go by the NVD search results.

These 40,000 are all known vulnerabilities, not zero-days. But the problem is that end users will never learn about these vulnerabilities if they’re not listed in the NVD - and this manufacturer has never done so. As far as the users are concerned, these are zero-day vulnerabilities. In fact, this manufacturer appears to make around 50 devices and not a single one of them has any vulnerabilities listed in the NVD. Moreover, this supplier doesn’t even mention security or vulnerabilities on their web site. They obviously think they have the security problem licked in their devices, since they’ve never identified any vulnerabilities. Perhaps they missed the memo that said if you don’t look for vulnerabilities, you won’t find them…or perhaps their lawyers sent that memo.

What’s even worse is that the supplier itself isn’t hiding anything. They haven’t lied to the NVD or anyone else, since they have probably never identified any vulnerabilities in any of their products. Unfortunately, a search in the NVD that doesn’t find the product at all (as in this case) returns the same error message as if the product was found, but there were no vulnerabilities listed for it. Someone who hasn’t been schooled in the pitfalls of relying too heavily on what the NVD says will likely come away believing the product has zero unpatched vulnerabilities, and they should go ahead and buy it. However, their number is just a little off the true number of unpatched vulnerabilities – by about 40,000.

Here’s the reason why I say this is a bigger problem than any discovery of zero-day vulnerabilities. Even though these 40,000 vulnerabilities are known (meaning they have been assigned a CVE number or another recognized vulnerability identifier), the fact that they’re in this device (and lots of others, to be sure) will never be known to any end user, unless the supplier reports them to the NVD.

However, the supplier won’t report them until they’ve found them; but they’re not looking for them, so it seems they won’t find them. Darn the luck! To the users of this device, these might as well be 40,000 zero-days. But don’t worry; they’ll learn about them when they get hacked.[ii]

What can be done about this problem? I’m glad you asked. The group to which Tom Pace made this presentation (the group is completely informal, but we call ourselves the “SBOM Forum”) is now – largely because of that presentation – hot in pursuit of a solution to the problem behind this and other aspects of the “naming problem”. The naming problem is perhaps the biggest problem plaguing software supply chain security, and SBOMs in particular. It’s definitely a huge part of the reason why SBOMs aren’t distributed to customers in any more than a few trickles today (the exception to this is the European auto industry, which uses SBOMs for open source license management, not software vulnerability management. They are also exchanging their SBOMs through a unique arrangement that would probably – in my non-lawyerly opinion - not even be legal under US antitrust laws).

When we started to pursue a solution (although we didn’t dare use that word at the time) to the naming problem in May, I thought we should be realistic and break the problem into two parts: the part that can be addressed in the “near term” of 2-3 years and the part that will take 5-10 years to address. I advocated that we put off the latter for the moment.

But guess what: It turns out that the near-term solution is the same as the long-term one. And neither is technically hard at all: If everyone were agreed on what needs to be done and were on board with how to implement their part of the solution, it would be up and running in about a week – and I’m only saying that partially tongue-in-cheek.[iii] Of course, “everyone” will never be in agreement on everything, so I’m sure that there will never be a 100% solution to the naming problem. But, as we see it now, there are no big technical obstacles in the way of what we’re proposing, and the funding required (all of which will need to come from the government, of course) is much less than I imagined originally – although I’ll admit we’re not considering cost in our solution. The problem is already exacting a huge cost on the software industry, as well as the general public, due to not being able to easily (or even at all, in some cases) learn about most software supply chain vulnerabilities.

I’m hoping that in a couple weeks, I can share some details on what we’re proposing, although the formal push won’t come until September. We’re already starting to socialize it with people in the government and elsewhere who can help with the implementation (and we’re getting a good initial response). At the moment, we need to finalize the 12-page document we’re working on, before we can seriously discuss this with anybody. I hope we’ll have that done in a week or two.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.


[i] Of course, these are 40,000 vulnerabilities that are identified in software and firmware components found in the product. Since at least 95% of these vulnerabilities are probably not exploitable in the product itself, this means there are “only” 2,000 exploitable vulnerabilities in the device. And since maybe 2/3 (to be generous) of those vulnerabilities are perhaps not serious enough to patch, that leaves over 600 exploitable vulnerabilities that are serious enough to patch, and of course haven’t been patched yet – since this manufacturer evidently never even looks for vulnerabilities, let alone patches them.

[ii] I know that Tom has recently been in contact with the manufacturer about this problem, although there hadn’t been any resolution when I heard from him more than a month ago. But the problem is that this manufacturer is far from alone. There are tons of products that have zero vulnerabilities listed in the NVD, mainly because the supplier has never reported a single one. This is why I said, in one of the posts linked above, that it’s actually safer only to purchase software products or intelligent devices that have at least some vulnerabilities listed in the NVD, since their suppliers obviously take vulnerability management seriously. This isn’t a joke.

[iii] I’ll admit I’m glossing over the fact that we’re solving two naming problems: software and hardware. What I say above is related to software. The hardware side of the problem isn’t technically hard, but it will require a lot of grunt work merging databases. Nothing like that is required on the software side. However, since the NVD includes both hardware and software products and a lot of the components of the naming problem apply to both hardware and software, they both need to be solved, although not necessarily at the same time.

Tom Alrich's picture
Thank Tom for the Post!
Energy Central contributors share their experience and insights for the benefit of other Members (like you). Please show them your appreciation by leaving a comment, 'liking' this post, or following this Member.
More posts from this member
Discussions
Spell checking: Press the CTRL or COMMAND key then click on the underlined misspelled word.
Jim Stack's picture
Jim Stack on Aug 3, 2022

Sounds like the NVD needs to be fixed. I would also suggest 3rd party testing for security issues with each product. Really good companies like Tesla Energy and vehicles invite hackers to find any weakness and pay them when they report it to them. Some have even then been hired to work inside Tesla and make secure products. 

Tom Alrich's picture
Tom Alrich on Aug 8, 2022

Yes, the NVD does need to be fixed - badly. However, help is on the way, as described in this post: https://tomalrichblog.blogspot.com/2022/08/zero-days-arent-worst-of-our-problems.html 

It's good to have pen testing of your software product, but new vulnerabilities are identified all the time. It's up to the product supplier to continually check their product for new vulnerabilities and patch the serious ones. It's up to the end user to do the same thing - although the user can't develop the patches themselves. A lot of users and suppliers do this regularly, but a lot don't. Regularly distributing SBOMs to customers will help, since they make it easier for the customer to learn about new components and the vulnerabilities they might contain.

Get Published - Build a Following

The Energy Central Power Industry Network is based on one core idea - power industry professionals helping each other and advancing the industry by sharing and learning from each other.

If you have an experience or insight to share or have learned something from a conference or seminar, your peers and colleagues on Energy Central want to hear about it. It's also easy to share a link to an article you've liked or an industry resource that you think would be helpful.

                 Learn more about posting on Energy Central »