The mission of this group is to bring together utility professionals in the power industry who are in the thick of the digital utility transformation. 

WARNING: SIGN-IN

You need to be a member of Energy Central to access some features and content. Please or register to continue.

Post

What could have prevented the SolarWinds attacks?

In one of Energy Central’s emails today, I saw a post by Joe Weiss that looked interesting; it was entitled “SolarWinds Orion: The Weaponization of a Network Management System”. For those who are not EC members, here’s the link to the same post on Joe’s blog (BTW, for about 4 or 5 months I’ve been putting almost all of my posts on EC, as well as in this blog. They’re the same in both venues. I’m quite happy with the level of attention my posts have received on EC).

I confess that I’ve only written a few posts about something Joe wrote, and none of them have been positive. So I’m happy to say now that I completely agree with everything Joe says in the post, which points to a mistake sometimes made with network management systems (NMS), and more often with the devices that are controlled by NMS (including UPS, battery management systems, building control systems and power distribution units): they are placed directly on the internet, not even behind a firewall. Of course, this makes them ripe for attack and compromise (especially given the weaknesses of the SNMP protocol used for network monitoring).

However, Joe admits that the SolarWinds NMS that were compromised by the attacks announced in December were almost all (or probably all) behind a firewall. So how could these attacks have been prevented?

If you read any of the 15 or so posts I’ve written about the SolarWinds’ attacks since they were announced in mid-December – or you read some of the huge number of articles and posts that have been written about this subject by others – you’ll probably know the answer to this question: There’s literally nothing a SolarWinds customer could have done to prevent the attack from happening to them in the first place, although they could have lessened the degree of compromise through various measures.

This is because these were pure supply chain attacks. SolarWinds’ development environment(s) was compromised by Russian attackers, who placed an exquisitely designed piece of malware[i] into their software build process. That malware then placed the Sunburst malware into the code of the updates themselves. Sunburst contained a zero-day vulnerability (which is called a backdoor. A backdoor is just a deliberately-planted vulnerability, as opposed to one of the many vulnerabilities that find their way into software every day, simply through poor security practices – or just plain bad luck – on the part of software developers).

This means that, when customers loaded one of the tainted updates (it looks like there were about seven such updates), they loaded Sunburst at the same time. The Russians then took advantage of the backdoor to penetrate the customer’s network and do nasty deeds. But don’t worry: Those customers were mostly unimportant ones – the NSA, DHS, DoE, the National Nuclear Safety Agency, FERC, etc. Fortunately, the Russians didn’t get into the White House football pool server.

Because the Russians had placed the Sunburst malware into SolarWinds updates while they were being built, the updates were signed by SolarWinds. This means that signature verification – or comparison of hash values - didn’t raise any red flags about the updates. And since Sunburst used a zero-day vulnerability, it wasn’t picked up by any malware scanners in antivirus software. There is literally nothing an organization could have done to detect these tainted updates, and thus prevent them from being installed.

So what could have at least mitigated the SolarWinds attacks? Joe mentions one measure – not placing the NMS directly on the internet – that I suspect just about every SolarWinds customer already practices. What else could have been done?

Of course, there’s a lot written about that issue (and Fortress Information Security is conducting a webinar on the topic on Thursday, which will most likely be quite interesting). I would just say that in general, network and server monitoring must have really fallen down – or just not been in place in the first place – in many of the SolarWinds customers who were actually attacked (i.e. they not only applied the tainted update, but the Russians exploited the malware to exfiltrate files or in general do bad things on the network. It seems there were “only” 2-300 of those and perhaps fewer than that – vs. the 18,000 who downloaded one of the tainted updates).

I say this because the Russians stopped planting Sunburst in Orion updates in June, meaning it’s likely they were inside every compromised network for a number of months. Of course, they were certainly very careful, but they finally slipped up and were detected because someone who worked for FireEye noticed an unknown login to their account. Did they make zero mistakes between (at least) June and December with every other customer besides FireEye? That’s not very likely. It seems all of those other customers weren’t looking very hard for evidence of attacks or compromise.

What could have actually prevented the SolarWinds attacks in the first place? Clearly, it has to do with SolarWinds’ controls (or more likely, the lack thereof) over their development network(s). There are two components to this. The first is the technical controls that should have been applied to the development network(s) themselves. Those controls are familiar to most power industry networking people, since they’re very similar to the ones required by the NERC CIP standards to protect the electronic security perimeter and the devices within it (including BES Cyber Systems, of course).

Perhaps the most important of those controls are found in CIP-005-6. These include a) complete separation between the IT network and the ESP/development network, including separate authentication; b) tight control over open ports and services on the ESP firewall, as well as on the network devices themselves; and c) requiring all outside access to devices inside the ESP to be via encrypted VPN, which is terminated at an Intermediate System located in the DMZ between the IT and OT (ESP) networks. Maybe in some cases all of these controls aren’t needed of every software developer (e.g. when the developer produces games). However, in hindsight it’s clear that SolarWinds should have done much more to protect its development networks than it did.

There should also have been controls like those in CIP-004-6, CIP-007-6, and CIP-010-3, including background checks on employees with access to the development network, training for them on appropriate security procedures, strict configuration management, logging on all devices on the network, and perhaps multifactor authentication to important devices.

The second component of controls is SolarWinds’ controls on access to their network in general. Even assuming the Russians penetrated the SolarWinds IT network first, how did they do that? We can’t say today what would have prevented the Russians from penetrating that network, since we don’t know how the network was penetrated. However, at least three possibilities have been raised:

1.      It might have been a supply chain attack through a Microsoft Office 365 reseller, as discussed in this post. In this case, this would be the first documented (that I know of) multi-level supply chain attack, where a supply chain attack was used to penetrate a supplier, and from there another supply chain attack was executed against the customers of the supplier.

2.      It also might have had something to do with the fact that SolarWinds had outsourced a lot of their software development work to organizations in Poland, the Czech Republic and Belarus (what could possibly go wrong with that?). A rogue developer could have placed the Sunburst malware in the update code being developed (although this idea goes against the fact that the Russians developed and deployed a very sophisticated piece of malware called SUNSPOT that did everything that was needed remotely; moreover, SUNSPOT painstakingly covered up what it did. In fact, this was almost certainly better than using a human being to plant the Sunburst malware, since they would have inevitably made a mistake and been detected. The SUNSPOT malware was never detected by Solar Winds until it was too late).

3.      Finally, the Russians could have penetrated a software development tool (presumably by planting malware in the tool developer’s network, which would have played the same role that SUNSPOT did with SolarWinds). Then, if SolarWinds used that tool, the Russians wouldn’t have to penetrate SolarWinds’ development network - they would have already been there! This might be the ultimate supply chain attack, for reasons described in this post. 

But how could users force SolarWinds and similar software suppliers to implement these controls? Let’s be clear: The only way to force them to do anything is with some kind of regulation. That may well be in order, since I think it’s clear (in retrospect, of course) that SolarWinds is as much of a critical infrastructure provider as any electric utility. The same consideration applies to other organizations like cloud providers. I believe that ultimately there will need to be mandatory controls on these organizations, perhaps structured something like what’s required by the recently approved IoT Cybersecurity Improvement Act (which requires NIST to develop a framework for IoT suppliers, rather than specifying specific controls. Moreover, the Act directly applies only to federal purchasing, although there’s a high likelihood that it will in fact serve as a standard for all IoT devices).

So barring regulation, what can we do to get software developers in general to improve their level of development security? The same thing we do regarding anything else we want a supplier to do: nudge them along the path of righteousness. This can include questionnaires; use of contract language where possible; other means of asking them to commit to doing something, like – gasp, shudder! – calling them on the phone and asking them point blank; RFPs and other means. Are you guaranteed to get results using any of these means? Of course not. But what is guaranteed is that you won’t get any results at all if you don’t try. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.


[i] I hope to write a post about that malware soon. There are lots of lessons to be learned from it!

Tom Alrich's picture

Thank Tom for the Post!

Energy Central contributors share their experience and insights for the benefit of other Members (like you). Please show them your appreciation by leaving a comment, 'liking' this post, or following this Member.

Discussions

Spell checking: Press the CTRL or COMMAND key then click on the underlined misspelled word.

No discussions yet. Start a discussion below.

Get Published - Build a Following

The Energy Central Power Industry Network is based on one core idea - power industry professionals helping each other and advancing the industry by sharing and learning from each other.

If you have an experience or insight to share or have learned something from a conference or seminar, your peers and colleagues on Energy Central want to hear about it. It's also easy to share a link to an article you've liked or an industry resource that you think would be helpful.

                 Learn more about posting on Energy Central »