A serious new cloud vulnerability – fortunately, it’s fixed. We might not be so lucky the next time.
If you’re reading this blog for the first time in Substack, welcome! To continue to provide the quality of posts my readers on Blogspot have come to expect, I am now putting up most of my new posts for paid subscribers only on Substack. All 1200+ posts that I have written since 2013 on Blogspot are also available to paid subscribers on Substack.
A paid subscription to my blog on Substack costs only $30 per year or $5 per month; it includes access to all new and legacy posts. Anyone who can’t pay that much should email me, so I can set you up with a free one-year subscription. Enjoy!
A few days ago, Kevin Perry, former Chief CIP Auditor of SPP Regional Entity, sent me the link to this blog post by Haakon Holm Gulbrandsrud of Binary Security. It described in exquisite detail (a lot of which I skimmed over, truth be told) how a flaw in Azure’s API Connection software could allow an attacker with access to one Azure tenant to access the data in a server in an Azure tenant on the other end of an API connection (to understand at least a little about the attack, it helps to skim through Part 1 of the post).
He points out some servers that should be vulnerable to an exploit based on this vulnerability. What servers are they? Nothing very important, of course…just Azure’s Key Vault, SQL databases, Salesforce, Google Mail (formerly Gmail), and presumably many others 😊. In addition, he also warns that the same vulnerability might be present in other cloud environments besides Azure.
Needless to say, this could be a very serious problem; fortunately, Microsoft has already patched it (they also paid Mr. G. a $40,000 bug bounty and sent him to Black Hat, where he described the vulnerability). Given that, should we just smile and say, “All’s well that ends well”? Not really. This was a very serious vulnerability and could have been devastating if the bad guys had discovered it before Mr. G did. In fact, if you read between the lines of his post, he clearly is appalled that such a vulnerability would even be present in a major cloud environment in the first place.
I’m not picking on Microsoft, since other cloud service providers have also made serious mistakes. However, I regard this as a fresh reminder that there are lots of vulnerabilities in the cloud, most of which we have no idea about today. I described three of these vulnerabilities – all of which have led to significant attacks - near the end of this post.
I also described something else in that post: the fact that the most serious risks in the cloud aren’t risks having to do with patch management, configuration management, open ports and services, etc. Those risks are all addressed in FedRAMP, ISO 27001, and Soc 2 Type 2 audits. Since the big CSPs have all passed those audits, it is safe to assume they have mitigated all the risks addressed in those three compliance regimes.
Of course, since the vulnerability described by Haakon only applies to the cloud, it isn’t now, and probably never will be, addressed by any of those compliance regimes. This also applies to the three cloud-only vulnerabilities (all three of which have led to successful attacks. One of those almost brought down Capitol One. Another may have been a key element of the Russian attack on SolarWinds) that I described near the end of this post.
This morning, I remotely attended (and participated in) the last three hours of what I call the “Cloud CIP” Standards Drafting Team’s first face-to-face meeting (it started Tuesday morning). This discussion was particularly important, because the team was trying to make decisions about its direction from now on.
Unfortunately, the team seems to have decided that their main purpose is to develop requirements that address the same risks as do the current CIP requirements, except with modifications to address cloud-based systems. This approach overlooks four important points:
1. Almost all the existing CIP requirements address risks that only apply to on-premises systems. This is because it’s safe to say that all the risks addressed by the current CIP requirements are included in the FedRAMP, ISO 27001 and Soc 2 Type 2 audits that all the major CSPs have already passed (although I think the “Cloud CIP” standards should require NERC to review each CSP ’s audit reports and summarize any negative findings for NERC entities who are customers of that CSP).
2. There will need to be new definitions for cloud-based systems, since the existing definitions – especially the definitions of BCS (BES Cyber System), EACMS (Electronic Access Control or Monitoring System), PACS (Physical Access Control System) and ESP (Electronic Security Perimeter)– apply to devices and networks installed on-premises. For example, a BCS is defined as a grouping of BES Cyber Assets, which are physical devices. The main reason why new standards are needed for cloud-based systems is that a cloud-based system is always deployed on multiple virtual machines housed in many physical servers that may themselves be housed in data centers spread across North America – and those virtual machines are constantly jumping from physical server to physical server as well as data center to data center. There’s no way the CSP can be expected to track all the physical or virtual systems on which every part of even a single BCS was housed throughout the three-year audit period, let alone make sure that all 120-odd CIP Requirements and Requirement Parts that apply to each of those physical and virtual systems have been followed during that period.
3. The two most important definitions that are needed are “Cloud BCS” – i.e., a system deployed in the cloud that has a 15-minute BES impact if lost, compromised, etc. – and “BES SaaS System”, defined as a cloud-based system that supports BES operations and potentially utilizes BES Cyber System Information, but which doesn’t itself have a 15-minute BES impact. Note that BES SaaS systems should only have to comply with the existing requirements that apply to BCSI: CIP-004-7 R6, CIP-011-3 R1, and CIP-011-3 R2 (the first two of these requirements were developed or revised with the cloud in mind, unlike all the other current CIP requirements).
4. Cloud-based risks (like the ones discussed earlier in this post and the post linked above) are potentially much more dangerous than risks to on-premises systems – since a cloud-based risk could potentially affect a huge number of BES systems deployed in the cloud. There will need to be separate CIP requirements (presumably in a new standard) that address “purely cloud” risks. However, first those risks need to be identified.
My main concern is the fourth point. The SDT doesn’t seem to be even considering purely cloud risks now – only risks that are addressed by the current CIP requirements. Moreover, given that they have so much on their plate already, it isn’t likely they’ll take six months or so off to identify cloud risks (I estimate they would need to devote at least that amount of time to thoroughly researching cloud risks that could affect the Bulk Electric System, since there’s obviously no such information available today).
This is why I’m calling for volunteers concerned about protecting power grid assets deployed in the cloud to join an unofficial working group; the group will identify purely cloud risks for the SDT. We won’t draft requirements; we will simply identify and clarify cloud risks that could affect the Bulk Electric System, since these will probably not otherwise be addressed in the “Cloud CIP” standards. We will publish those risks and allow the SDT to decide whether to develop requirements based on them (of course, we will recommend that they do this).
This invitation is open to anyone who works for a NERC entity (e.g. an electric utility or IPP, including renewable power producers), the NERC ERO, FERC, a platform CSP, an existing SaaS provider or a software developer that is moving toward a SaaS model (it seems that just about every software producer in the world falls into one or the other of the last two categories). However, note that no participants will be presumed to be speaking for the organization they work for – just for themselves. We’re also open to consultants and anyone else with an interest in this topic. I’d especially like to have cloud security experts participate, whether or not you have specific electric power industry experience or knowledge.
If you fall into one of the above categories - or if you’re just a user of the power grid who wants to make sure your lights stay on - please contact me at the email address below. The group will start meeting soon.
If you would like to comment on what you have read here, I would love to hear from you. Please email me at [email protected] or comment on this blog’s Substack community chat.