The mission of this group is to bring together utility professionals in the power industry who are in the thick of the digital utility transformation. 

Post

Water Cyberattack Highlights the Need for Strong Authentication

Danny Vital's picture
Senior Cybersecurity Engineer XTec

Hi, I've been working in IT for the past two decades. Twelve of these years have been working specifically in the area of cybersecurity. During my work in tech I've played various different roles...

  • Member since 2021
  • 14 items added with 3,920 views
  • Mar 3, 2021
  • 949 views

On Tuesday February 2 a massive trove of usernames and passwords was placed online. The attack is now known as the COMB breach, Compilation of Many Breaches. These types of attacks have been happening for decades. An attacker gets access to a password repository and then publishes the hack. This by itself is not new but the scale of this attack is. This collection included over 3.2 billion accounts and it also included the username and passwords for personnel that operate the Oldsmar water plant in Florida. An attacker took notice and utilized this now publicly available username and password combination and started tweaking the additives to the counties water in a bad way.

Just days later as the cyberattack on the Oldsmar, FL water treatment facility on Super Bowl weekend made abundantly clear, the need for stronger authentication into the cyber-physical systems that manage our critical infrastructure is imperative.  Our nation’s critical infrastructure, especially energy and water, are controlled more and more from SCADA and OT, or operational technology, systems that enable change by remote users. Sometimes we don’t have the time or know how to properly lock these external facing systems down. There’s often a shortcut that’s taken when it comes down to identifying persons at shared systems.  The opportunity for bad actors, whether from a hacker, adversarial foreign state or disgruntled internal employee, is often protected by nothing more than a username and password.

Despite the known vulnerabilities of passwords – and we won’t rehash all the known statistics of how bad they are here - their use is still prevalent throughout electric utilities. NERC CIP regulations do require the use of multifactor authentication (MFA) in certain circumstances, but also go to great length to govern password complexity.  Every utility should look at how they can eliminate the use of passwords not only in critical OT systems, but also in all traditional IT systems as well.

At the end of the day there are many ways that this type of attack may have been avoided. You really need multiple layers protecting your utility and we’re not beyond stating that many security components need to work together as part of a larger whole. One area that would have been a major contributor to deterring this attack and/or making it that much more difficult is the outright replacement of passwords with something much stronger.

A word on shared accounts

Although shared accounts provide a faster way for staff to switch between shifts, access environments and keep track of passwords this is a poor practice in any environment. You’re giving this same ease of use to your attacker. There is less effort required to target individual accounts and most times compromising one password means gaining entry into various systems. This mind frame has to change. They used shared accounts at Oldsmar.

A word on shared passwords

If multiple people use the same account they’ll also have the same password. Possibly to multiple systems, this means more places that this can be captured during weak authentication. Key loggers, phishing attempts and rogue notes left around the office are all ways that passwords can be stolen. If more people are using it the likelihood of it being stolen goes up as well. Everyone needs their own PIN or password that they’ve setup for themselves. There’s little that can be done in today’s world without your own password. The operators have an online banking account, e-mail, Amazon, Facebook, Twitter and other passwords … you get the idea. The concept of each person having one to get into a system should not be a hurdle, get your own. Even better get something beyond a password which is vulnerable in many ways. Many knew the password to the shared accounts at Oldsmar.

Password Managers

Password managers sell themselves by providing the benefit of protecting many different passwords from many different accounts with a master password and some levels of data encryption. You’re able to make the password as long as you want and introduce as many special characters as the authenticating application supports. All you’re really getting here is the introduction of complexity requirements within the tool itself. All the major problems with passwords are at the root of this solution resulting in an identity management tree that just won’t hold up.

So what are some of the things utilities and others should be looking at to replace the use of passwords. Let’s take a look at some of the alternatives that are available today as username and password replacements.

One Time Passcodes

One-time passcodes (OTP) are a frequently used and often inexpensive way to comply with rules regarding MFA (Multi-Factor Authentication). They add a layer of authentication that makes it more difficult for a remote access hack by using two factors – something you know (username/password) and something you have (typically a mobile phone). OTP can be susceptible to a social engineering or Man in the Middle attack. The passcode themselves are easily intercepted, rerouted and if you have enough information to determine the sequence, an attacker may even determine the next number. There are many documented cases of hackers who have contacted mobile phone providers in a social engineering attack and had a specific number ported to another phone in order to steal OTP as well as other important data.

NIST deprecated the use of OTP in 2017 for all government and military functions.  If the government is not willing to absorb the risk of an OTP, utilities and other critical infrastructure entities shouldn’t either.

Hardware Tokens

Hardware tokens provide the Something You Have factor. These are most commonly seen as an access card for building access but include other forms such as a USB key or a FOB. When using some of these devices you’ll have to know the PIN or password that you’ve setup for yourself while also being in possession of the physical token itself. This is a good way to implement multiple factors of authentication within your organization. It’s less risky to require physical possession since sometimes it’s resource intensive to try to duplicate the identifying information on the token and having the token in hand is sometimes sufficient enough of a hurdle for an attacker. This method does have limitations.

For a determined attacker or those that have unlimited resources available, duplication of some hardware tokens is possible. Some card-based systems such as those for building access offer limited protection to the information that is stored on them. These devices may be lost or stolen. If you only require physical possession for access or entry this is a big problem. There are ways to improve upon the authentication model that hardware tokens provide us. It’s best to implement multi-factor authentication and knowing that what you’re storing in the token is quality is best.

Hardware tokens are another form of MFA. By issuing a hardware token, such as a USB key or a fob, an organization can more securely lock down an application, network or other point of digital entry. A hardware token certainly meets the requirements for MFA.  You still have to know a PIN or password and the token itself must be in your physical possession. The possibility of a social engineering, Man in the Middle or other type of attack is much less than with an OTP, however security risks are still ever present with a hardware token.  A small token can be lost, or worse, stolen rather easily.  A token that only requires having possession of the token can easily result in a breach. 

Certificate Based Authentication over PKI

A good piece of information to encode on hardware tokens is a digital certificate. A digital certificate is one of the better ways to identify yourself to services and systems today. You’re taking identifying attributes from an individual and associating them to the digital certificate that will in turn be used to authenticate, validate and authorize access. These digital certificates may also be used to encrypt e-mails and digitally sign documents. These certificates are obtained from a certificate authority and an environment of third party trust. One thing to keep in mind is that not all certificate authorities are created equally so you need to keep the organizations practices in mind and if you’re issuing these certificates internally you should audit your security to identify any risks to mitigate them. These digital certificates may be placed within the secure storage of a device or they may be placed within special hardware like we’ve mentioned.

Used in combination with other authentication factors, possession of a digital certificate is one of the more secure approaches to authenticating personnel at critical systems.

A good model that incorporates a physical smart card with the use of a digital certificate issued from an environment of trust is the PIV model. The use of a PIV card in critical infrastructure can secure IT and OT systems as well as physical spaces for any utility with one unified credential.  This would allow an organization to move to a passwordless environment across the board including substation control houses, security operations centers, distributed energy platforms or even customer service and financial systems.  The PIV SmartID card can also add a layer of security wherever a personnel badge or swipe card is used. In the most secure areas of a utility, the use of all three factors of authentication adds a layer of security that cannot be matched with other access systems. 

For mobile applications, these same credentials can be extended for the user in what is termed a Derived Credential.  This allows an employee to use a mobile phone, tablet or laptop in the field without having to use or necessarily be in possession of the SmartID card.  This allows an organization to maintain high levels of security while adopting  a more mobile and modern workforce.

Our utilities, whether it is electric, water or gas are critical to the safe operations of our country. Utilities, power producers and other critical infrastructure organizations should be moving away from shared accounts. They should implement one of the solutions listed above. If you’re in an area that is deemed critical you should only really be considering the highest level of security. We really need to avoid another Oldsmar type attack since they are completely preventable.

Danny Vital is the Senior Cybersecurity Engineer for Critical Infrastructure Initiatives with XTec, Inc.  For questions, comments or to find out more information, please contact him at dvital@xtec.com.

 

Danny Vital's picture
Thank Danny for the Post!
Energy Central contributors share their experience and insights for the benefit of other Members (like you). Please show them your appreciation by leaving a comment, 'liking' this post, or following this Member.
More posts from this member
Discussions
Spell checking: Press the CTRL or COMMAND key then click on the underlined misspelled word.
Matt Chester's picture
Matt Chester on Mar 3, 2021

The Oldsmar situation provided some moderate buzz in this area, but it seems like because crisis was successfully averted that it didn't raise more national/industry alarm bells like I would have expected. I'd hate for the worst to have to happen for the proper attention to be paid to these risks-- do you think the leaders across the industry are doing what they should and this observation was more one of the press not overly pressing on the issue, or is there a sense that this was swept by (especially with ERCOT's failures in the ensuing weeks taking much of the attention anyway)? 

Danny Vital's picture
Danny Vital on Mar 3, 2021

A major outage like ERCOT definitely does take all the attention away and rightly so. The scale and timing of the outage were catastrophic. Unfortunately credential theft which includes a password breach like in Oldsmar still make up a large percentage of cyberattacks today. Regardless of industry leadership attention to this, the problem is not going away on its own. There are good pushes by several security focused national groups and industry associations but sometimes this does not translate into a priority for some organizations and their leadership. This is especially worrisome when the preventative measures are available in house with current technology or via cost effective solutions. 

Richard Brooks's picture
Richard Brooks on Mar 3, 2021

Danny, thanks for providing such a comprehensive analysis of the various approaches available for access control. One item in particular I want to emphasize from your analysis is this point: "One thing to keep in mind is that not all certificate authorities are created equally so you need to keep the organizations practices in mind and if you’re issuing these certificates internally you should audit your security to identify any risks to mitigate them."

Totally agree, in fact, six organizations submitted a FERC comment filing that talks to this very point. I'm thinking XTec would have been a good addition to this FERC filing.  I'm very interested in your thoughts on the filing, which you can access through this link: https://energycentral.com/c/ec/ferc-filing-recommending-financial-incentives-voluntary-adoption-sbom-energy

Danny Vital's picture
Danny Vital on Mar 3, 2021

Hey thanks for reading the post and contributing to the comments. Glad we have some common ground, if you're getting your certs from somewhere else you need to utilize a trusted third party. One thing that I enjoyed reading in the filing with FERC is that you refer to the NIST Cybersecurity framework. I've been a long time advocate for a standards based approach to solving problems and what NIST puts out there including the Cybersecurity Framework is top notch. Access control is a huge problem for SBOM. One part of the problem is authentication then you have to make sure they only have the rights they should have. You don't want configuration changes being made where they're not needed nor give attackers a way circumvent these tools to their favor. Your also should have a way to detect tampering and the suggested use of digital signatures is a good way to do this. Incentivizing utilities to move to a better internal security posture is a great idea and one XTec would rally behind. Reach out to us next time. 

Richard Brooks's picture
Richard Brooks on Mar 3, 2021

We are in agreement Danny, SBOM confidentiality and integrity must be supported in a production implementation.

Audra Drazga's picture
Audra Drazga on Mar 4, 2021

Great post Danny. I am looking forward to our PowerSession that we will be hosting in April that will discuss how using PIV cards to help with this issue. I will add a link to the session once we put it live. For our community members reading this - Save the date on your calendars for Thursday, April 15th at 2:00pm ET.

Matt Karber's picture
Matt Karber on Mar 8, 2021

As "old-school" as this sounds, it may be necessary for critical systems such as utilities, to have a manual, physical disconnect from the internet and use internal systems that can operate in a crisis without that connection until appropriate steps are taken.

Danny Vital's picture
Danny Vital on Mar 9, 2021

Hi Matt, yes you're right if it's stopping me from taking corrective action I don't want it in the way. The area I'm mentioning is with regards to unauthorized users gaining entry from a distance. We have more of the grid coming online every day. Do you have any recommendations on how this should be done/if this should be done safely? Thanks for adding to the conversation. 

David Svarrer's picture
David Svarrer on Mar 10, 2021

I am not writing this in order to offend anyone. However. What I am going to write, MAY offend someone - yet - it is meant to be formal and not emotional. 

======

It is obvious, that the utility sector both within water, energy, distribution and many with them - have not employed sufficiently simple thinking people to handle their IT and security. 

The following is not a joke. It is the reality: 

Why on EARTH do you good utility firms have ANY network, connected to ANY function which controls switching gear, water treatment or ANYTHING ELSE what so ever???

I would like to see that malware which can jump across the air to the server structures. 

Who have said to you that you need to have your systems connected to the internet? 

Have you never heard about how you can update your systems without being on the internet all the times? 

Have you never thought about that you change your systems to LINUX, such that you are not sitting in the "Microsoft Update Trap" - which keeps the entire world of utility companies, health care facilities and others - tapping their fingers on the tables, nervous for what the next connectivity to the internet may be made of? 

Have you never studied a bit deeper what it is, which is REALLY your need, in terms of running your facilities? 

Have you not thought about why you may need to "update your operating system all the time" - for SECURITY - says Microsoft, Kaspersky, McAfee and the entire bunch of blood sucking dragons. 

ANd your incompetent, educated-beyond-oblivion CIO's are joining the voices of these big companies, because - they really really enjoy their high salaries from having done CISSP, CRISP, and what do you have. 

ALL these gymnastics are not relevant - including OS-updates - when you are not on the internet. 

Have you never thought about why on EARTH a computer should begin to fail, after even 10, 15 or 20 years without any update, when nothing is being added to it? 

Have you thought about what it would cost you to swap your energy systems directly over to Linux (millions of USD, indeed) - compared to the nearly BILLIONS you are using now by opening your systems widely up for the underworld, after which your CISO, CIO and other unscrupulous security bosses in your bread tries with cellutape, Gaffa-tape, superglue and what do you have to prop something into the wide gapped swiss cheese holes this policy is putting into your security? 

Wake up, utility bosses! Take control of your systems, and demand that they are being switched off from the internet-grid. 

You can even - at miniscule cost - provide the few thousands of kilometers of fibre-optic cable, aligned with your high voltage cables - and indeed - INDEED - never ever rented out for "normal internet use" - ONLY and SOLELY used to interconnect all your power stations. 

If you create such a very large scale INtra-net between all your sites, then you do not even need to be on any "INTERNET". 

If you are damn hell-bent on remaining running on the largely defunct Windows systems - then have a conversation at sufficiently high level and demand from Windows, that you receive DVD's or Flash-disks which are written/burned at Windows's operating system department - and ensure that these flash-disks are now installed centrally where you maintain either your plant-IT, or your Intra-net. From there you can distribute safely. 

Remove any and all USB-port based or other external connectivity to your plant system - such that nobody can poke holes into your security....

Create playstations in the utility - segregated from the grid-based machinery - such that your good staff indeed can watch a Youtube movie or do other online leisure activity - outside of your secured network. 

Plug all RJ45 plugs (Internet cable plugs) - for instance with a good clot of superglue or other glue or resin. 

You have so many many many thousands of opportunities and ways in which you can protect yourselves, so all these thousands (literally) of conferences, meetings, and worried security facial expressions - are rather useless. 

Get on with a simple life.

Read my lips: You DO really not need any internet connecting any of your grids controlling computers. If you really think that you do, then feel welcome to tell me (reply here!) and I will indeed read everything you say and give you my best response.

I have worked for the better part of my professional career with critical mission and one of the things we learn is to reduce risk before we do anything else. I would be a fool, if I was called to any utility and recommended any of the current setups: Connect everything to the internet and then try to disconnect the sensitive parts via software? O tempora O mores! 

Sincerely

David Svarrer

Rational Intuitive Limited

CEO

Danny Vital's picture
Danny Vital on Mar 10, 2021

Hey thanks for taking the time to put your thoughts together on this post. I agree with you on several parts. This seems like a good starting point for much broader discussions for the community with regards to many security issues. We’re all here to learn from one another so let’s make this a useful thread. Here are some of my thoughts and questions to you.

You’re right much of what’s out there on the grid was never designed with security in mind. The industry is working backwards to try to make them secure.

There’s no better security than a completely isolated network. One that lives in complete isolation and one that will never be connected to or be plugged into. This is not what’s happening in the field.

You still have to update Linux. There are entire lines of CVE’s dedicated to patches for vulnerabilities to Linux operating systems.

A vulnerability is still a vulnerability, Regardless of your connection to the internet.

Unauthorized access from within your intranet is still unauthorized access. Authentication, validation and authorization are areas that still need improvement.

You still need to have personnel connect to these systems. Personnel that more and more are crossing the boundaries between the internet connected IT systems and your OT. Despite what may have been physically done to the hardware or implemented through security policy.

You’re writing from the perspective of someone with plenty of experience. There are many teams that lack this knowhow and need the help to design such systems. Unfortunately other priorities usually take over.   

There are plenty of topics to pick from here for the community to take a deeper dive on. Again thanks for making this a more interesting post.

David Svarrer's picture
David Svarrer on Mar 17, 2021

Dear Danny Vital, 

You are mentioning the same arguments which are genuine and true arguments, however, with still a tough but true way to deal with them - and which a lot of others are mentioning, that: 

1) The staff need access

Yes - however, it is much less likely that staff will do "insider jobs", if the risk of getting caught is enormous. It is much easier to be an "insider", if you can provide any sort of access from an "outsider". Then the risk is not that big. Therefore, physically segregated systems for all internet, and complete, physical cut lines to the internet

2) Personnel need to connect and how now is that done?

Yes, indeed - no problem - as long as they connect via the intranet, which can be spread all over an entire continent, as an INTRANET - where there is no single point of connectivity to any INTERNET. This can be maintained by sufficient programming of a crawler - which can do all the necessary testing of the network, and the network's boundaries - on top of naturally physically ensuring that the cables for the internet and intranet are not even pulled around within the same physical room. That means: TWO IT-rooms. Not ONE. There are many small things like this which are then necessary - but - the cost of this double-standard, so to say - is by factors of 10 or more less than the current menageri, where chaos reigns, and where power stations and other societally crucial equipment is hanging to dry out there on the internet, just waiting for the next hacker to arrive. 

3) Crossing boundaries between Internet and operations

No problem - just ensure that there are two systems. All internet related stuff - including all letter writing, white collar jobs etc. - let it happen on the internet. If there is a dire need to cross those boundaries, then put a protocol filter between, where absolutely nothing can pass unless stuffed into the protocol. Make that protocol converter be running from a micro computer (Arduino board ie.), and lock it (physical switch on the board) against being reprogrammed. We are discussing here, that hackers who may be able to get in, will discover that they cannot rewrite any code, therefore they cannot break further in.

4) Unuthorized access from within the system (!)

While this could be an issue, it is rarely not. The most occurring form of unaurhorized insider access is when people share passwords and login credentials. There are means to stop this, completely - namely by simple fingerprint gadgets, where the cumbersomeness stops. Also a lot can be done by letting people KEEP their dam dam passwords, and not force them to change, which makes them write them down. Two-level-authorization - where the person's personal mobile phone is being used to receive a OTP can easily be implemented too. 

5) Linux systems has vulnerabilities too and need upgrades

Yes, you are absolutely right - however - the vulnerabilities are rarely - very rarely - of a nature such that again insiders would benefit from them. They are 99% of the time, or maybe more (experts can tell) - vulnerabilities of the nature I argued against in my response to the article itself - namely where the vulnerable system (critical mission operations) are put on systems which are connected in one way or the other to the internet. THerefore, fixing these or not, is not an issue. 

Yes, I have a bit of experience, having taught CISSP - and then in the more than a decade after that, having worked from time to time guiding insurance companies, financial institutions, health care facilities on the same. 

WHy, by the way, are all of those so dam dam dam secretive about the problems? What is this privacy hysteria we are suffering from? We all end up suffering when for instance my contracts states that I must keep it totally secret that I even work for them???? DO you have this also where you work, Mr. Vital? 

If we cannot share our experiences, including names of the corporate etc. etc. - then we cannot learn. Why are we so shy to discuss our various down falls? 
 

Well. Have a wonderful week ahead.  Questions welcome!

Sincerely

Rational Intuitive Limited

David Svarrer

 - member of Infinity Point BA.

Audra Drazga's picture
Audra Drazga on Mar 17, 2021

Great conversation on this post - Thanks Danny for sharing. For all those following we will be hosting a PowerSession on "How to Improve Security and Reduce NERC CIP Compliance Costs Using Smart Identification" on Thursday, April 15th at 12:00pm MDT.  I would love it if you could join us and better yet share this out with your colleagues and invite them to join too.  To register follow this link. Panelists include: 

• Danny Vital, Senior Cybersecurity Engineer - XTec

• Dwight Williams, Principal - GreySky, LLC

• Tom Alrich, Supply chain Cybersecurity Risk Management and NERC CIP-013 consulting - Tom Alrich LLC

This should be a very good discussion.  I hope to see you there. 

Paul Korzeniowski's picture
Paul Korzeniowski on Mar 23, 2021

Interesting points, highlighting the challenges that utilities as well as other organizations face in protecting sensitive information/systems. The reality is that the hackers only need to find one flaw in complex systems often spanning millions of lines of code. The utility has to harden all of it; the tiniest flaw can be catastrophic.

Passwords represent the trade-offs in securing systems. As systems have become more sophisticated, the bad guys have had more success in finding their way in. IMO, we are reaching a point that modern password requirements (long character fields with complex sequences) are creating usage barriers. Who can remember their password when you use it once a year or your mother's address when she was in middle school?

Two factor authentication (a password plus say a code sent to your cell phone) is becoming more common. It can be inconvenient at times but is becoming faster and more effective as technology matures.

FWIW, I had hoped that biometrics would become more common as smartphone usage gained traction several years ago. It still seems to be less consistent than users would like, so  usage has not caught on.

 

Steve Lindsay's picture
Steve Lindsay on Mar 24, 2021

Paul - colleague of Danny's at XTec here.  We focus of providing customers with the most secure access technologies - and these can also be the most frictionless as well.  We offer a biometric option where a user inserts their SmartID card and uses a fingerprint template to login.  This is especially useful in areas like substation control houses or operations centers where multiple engineers are often logging into the same system.  Too often they use shared passwords.  This eliminates the need for shared passwords and makes for an easy authentication process.

Get Published - Build a Following

The Energy Central Power Industry Network is based on one core idea - power industry professionals helping each other and advancing the industry by sharing and learning from each other.

If you have an experience or insight to share or have learned something from a conference or seminar, your peers and colleagues on Energy Central want to hear about it. It's also easy to share a link to an article you've liked or an industry resource that you think would be helpful.

                 Learn more about posting on Energy Central »