The mission of this group is to bring together utility professionals in the power industry who are in the thick of the digital utility transformation. 

WARNING: SIGN-IN

You need to be a member of Energy Central to access some features and content. Please or register to continue.

Post

Forgotten Cyber Holes That Still Plague Utilities

image credit: © Leowolfert | Dreamstime.com

This item is part of the Cybersecurity - Special Issue - 04/2020, click here for more

Abstract:

Electric utility communications within a substation can be one of the most complex systems, depending on the utility, the number of substations owned by the utility, and age of each substation. Unlike data centers or other IT communication systems, Operational Technology (OT) communications within a substation utilize multiple types of communication transports, cable links and protocols that are added or layered into an existing system, based on the need. Over time, as new substation equipment is installed either within the yard or the control shelter, additional communication circuits can be provisioned and installed without having to overhaul the entire substation communications system. Depending on when the circuit was provisioned, the speed and bandwidth requirements, and the destination of the circuit, the type of communications technology that was utilized can vary drastically, including the cabling used to link equipment. Unfortunately, as utilities work to address cyber vulnerabilities, this tangled layer of communications equipment often creates the potential to overlook areas of exposure, or rather a cyber hole, especially where older legacy devices are prevalent. Communications with and between these devices must not be overlooked, as they need to be passively monitored in a similar manner as network devices.

Article:

Electric utility communications within a substation can be one of the most complex systems, depending on the utility, the number of substations owned by the utility, and age of each substation. Unlike data centers or other IT communication systems, Operational Technology (OT) communications within a substation utilize multiple types of communication transports, cable links and protocols that are added or layered into an existing system, based on the need. Over time, as new substation equipment is installed either within the yard or the control shelter, additional communication circuits can be provisioned and installed without having to overhaul the entire substation communications system. Depending on when the circuit was provisioned, the speed and bandwidth requirements, and the destination of the circuit, the type of communications technology that was utilized can vary drastically, including the cabling used to link equipment. Unfortunately, as utilities work to address cyber vulnerabilities, this tangled layer of communications equipment often creates the potential to overlook areas of exposure, or rather a cyber hole, especially where older legacy devices are prevalent. Communications with and between these devices must not be overlooked, as they need to be passively monitored in a similar manner as network devices.

As we advanced from analog to digital, network communications provided us with the potential to move large quantities of data to monitor substation activity back to a centralized operation center. With these advancements in place, most within the electric utility industry including utility executives, promote the notion that their operations are adequately secured from an IT perspective. But this can be far from the truth in most instances as significant vulnerabilities remain within the remote substations in the world of OT communications where the controls equipment resides. Even with all the technological advancements over the last 20 years, there still remains a plethora of substation serial and analog OT controls equipment that is neither identified as part of network asset management nor monitored for command and control activity.

Serial and analog equipment tend to fall out of the purview of most IT groups within a utility as a result of the IT culture and industry. It is often assumed that equipment within a substation is upgraded on a regular basis similar to IT equipment where a 3-5 year upgrade cycle or less is normal depending on both the equipment’s use and the industry where it is utilized. What tends to go unnoticed by most utility IT professionals is that the culture of the utilities on the operations side for over a hundred years was to select and install equipment that can function efficiently for decades. Much of the legacy electric utility equipment requires testing or calibration on an annual basis at most.  Mechanical, electro-mechanical, analog or digital equipment conversions and upgrades take place very infrequently, much to the surprise of the IT staff. This mentality also conflicts directly with today’s throwaway society where phones, TVs, computers and other equipment receive frequent upgrades simply to keep pace with the latest enhancements to the latest applications. Even for those who purchase and expect to use that device until the hardware fails, find it hard to resist the temptation to upgrade. A recent example of a forced change was when Netflix pushed an upgrade to their system that caused the streaming capabilities for some Blu-ray players to become completely incompatible overnight.

Unlike substation drawings, which typically display the physical installation of the equipment, most utility communications drawings display either a physical or logical diagram of the installed equipment. In addition, edge, end or other IoT equipment are also typically identified within these drawings. What tends to be inadvertently left off are the analog, TDM, or serial equipment in use. This can include controls and SCADA equipment including teleprotection relays, RTUs, meters, and switchgear, just to name a few. Generations of communications equipment, depending on the age of the substation, may also end up excluded. Most legacy utility and industrial equipment built since 1969 was manufactured with a serial interface that served multiple functions. It was typically used for sending control signals between devices, data output, or for a technician to connect to and make changes or adjustments to equipment settings. The number of serial interfaces on a piece of equipment often varied depending on the manufacturer and function of the device.

This older OT equipment is easily overlooked by an IT group conducting an overhaul or upgrade of the OT communications system, since most of the equipment in its’ native configuration does not contain a network interface. Even still, this equipment in most cases can be sending back data to an Operations Center and be directly tied to the operation of the electric system. Because of costs and the low bandwidth constraints of the communications equipment used by electric utilities in the 80’s and 90’s, the number of devices and the data retrieved was limited to the controls and essential metering required for operations. As technology developed, dial-up modems were installed and used to retrieve data outside of the typically closed OT utility communications system. These modems were very simple devices with a phone line jack and a single serial port to connect to the equipment. Most dial-up modems used a single user ID and password to gain access to the connected equipment and provided the ability for the engineer to remotely connect to a field device in the substation. This was cool stuff at the time for the engineer, but even then, there were never enough modem lines available to connect to all of the devices, so available data points and records had limitations.

As compared to other industries, the development of network communications into the substations has been an extremely slow process. The most significant strides have taken place in the last 5 to 10 years. But with it has come the use of the serial to Ethernet converter that provides the opportunity to tunnel serial communications from a substation device through a utility network to another serial device on the receiving end, or to a terminal emulator program on a desktop computer. This ability opened the possibilities to connect and bring back even more data to an operations center or an engineer’s desktop. The serial to Ethernet converter quickly became the go to device for many integrators (including myself), because it had the ability to bring back larger amounts of data, retrieve records, or make changes to the equipment settings remotely. The serial to Ethernet converter allowed the integrator to put a device onto a network with little to no effort.

Up until a few years following 9/11, the culture within many industries including the electric utility, advocated for open communications as the norm. Any requested access to a device or system was routinely granted to just about any user’s desktop where it was needed. Typically, all that was required was a common user ID, password, and the IP address of the serial to Ethernet converter connected to equipment on the far end. Firewalls, multi-factor authentication, individual user IDs and complex passwords were rarely in use, if at all. This was not out of negligence, but out of culture.

The threat of a cyber-attack in the early 2000s was extremely low. Perhaps it was naive to think that way by technology professionals, managers, and even the communication engineers that recommended automation to simplify monitoring the operation. If the question was raised regarding how secure a serial to Ethernet converter was, the answer was, “why would anyone outside of the utility want to connect to a utility device, let alone know what to do once they were connected?” Unlike a dial-up modem that could be accessed from anywhere in the world, and that were subject to war-dialers looking to find open unsecure modems, a serial to Ethernet converter appeared to be safely inside the utility OT communications system. The length that a nefarious individual would have to go to gain access to the utility OT communications system, and reach a specific serial device within a substation, seemed impossible. Or so we thought.

In 2015, a modular type of malware called Industroyer, was used in the first successful cyber-attack that took place to create a blackout of a Ukraine power system. The malware was designed with four payload components which were able to gain direct control of switches and circuit breakers in the electrical substation to create a blackout. While there have been many papers written on the incident, the part that has consistently disturbed me was that the malware payload enabled an attack on both serial and network equipment within the utility system. This should have served as a call to action for everyone concerned about utility OT communications security. But it did not. Specifically, here in the U.S., we need to work to secure all serial communications within our most critical ICS industries.

RS-232, RS-485, and other serial communications have been the backbone of communications between equipment and operators for most utilities. Even as network communications began to grow and become popular within the electric industry, many electric utility manufactures added a RJ-45 port to their equipment to accommodate a possible Ethernet connection. But the connection that it makes does not constitute the equipment as IP enabled.

While serial to Ethernet converters have been extremely convenient with providing a link between legacy equipment and the utility IT network, these devices and the equipment that they connect to are typically omitted from most network drawings since they are not a pure network-based device. The ratio of serial ports and other communication connections to a single-network connection can easily range from a 30 to 50 to 1 ratio, if not higher, depending on the size and age of the substation or system.  This ratio alone quickly explains as to why these non-network devices were intentionally left off most communication drawings. To make matters worse, many devices within a substation are interconnected via a serial or some other direct-wire connection, to the point where should an attacker be able to gain access to the controls equipment, the potential to traverse via serial, direct-connect and other non-network connections in order to strike other equipment becomes highly possible. The control or damage that can be done is unthinkable.

While network anomaly detection for Operational Technology (OT) has advanced in the past five years for IP enabled devices to prevent attackers from accessing critical infrastructure without detection, serial communications remain unprotected. This is the case, even though the technology to monitor serial infrastructure has been developed and tested. The problem is that the demand to roll out this technology is severely lacking across all industries. The number of serial devices and connections still in use can vary considerably depending on the utility. In addition, the number of substations within their system, or the age of the substation differs as well. But here’s the clincher, most utilities with substations older than 10 or 15 years have serial equipment that accounts for 60% to 80% of the devices within their substation. This range is even higher for substations that have not had any type of equipment upgrade in the past 15 years.

As networks continue to be integrated into substations the problem will only compound. For mid to large size companies, the cost of pulling out and replacing 60% to 80% of their non-network equipment within a utility system cannot be justified either as a business case within the company, or within a rate case to a utility commission. Typically, the change-outs have only happened in substation new-builds or during a control shelter replacement, where all the old utility equipment is migrated over to the new equipment shelter. These two options usually require a heavy cost on the part of the utility, in addition to requiring a large amount of time and workforce. Upgrading 3 to 6 substations per year would be considered a success for most utilities. However, for smaller utilities having 15 to 25 substations, these upgrades to their entire system can take place in the relatively timely manner of a few years. Unfortunately, for larger utilities owning tens if not hundreds of substations, this pace will not suffice to adequately cover the entire system.

So, how do we address this resilience problem?

Solutions that should be applied to help remedy cyber holes in any utility: 

  • First and foremost, install Passive Network Monitoring equipment starting with all critical substations
  • Second, install Serial Monitoring equipment starting with all critical serial equipment
  • Review monitoring logs to ensure that each system is adjusted properly
  • Test each system at least once a year to ensure proper function

Solutions that should be applied to identify and prevent future cyber holes:

  • Apply Asset Management techniques for properly illustrating and documenting communications links between all equipment
  • Work from a Standards base approach
  • Apply a System Design approach to Equipment Standards
  • Apply proper Supply Management techniques when purchasing equipment
James Moralez, P.E.'s picture

Thank James for the Post!

Energy Central contributors share their experience and insights for the benefit of other Members (like you). Please show them your appreciation by leaving a comment, 'liking' this post, or following this Member.

Discussions

Matt Chester's picture
Matt Chester on Apr 29, 2020 4:11 pm GMT

Thanks for sharing, James-- you can bet that 'forgotten' cyber holes are anything but forgotten by any malicious actors looking to access their systems. 

For a typical utility that might be plagued with these vulnerabilities, what sort of capital and time investment do you think would be necessary to catch up?

James Moralez, P.E.'s picture
James Moralez, P.E. on May 5, 2020 6:25 am GMT

Time or Money. The monitoring like any technology, has a direct cost associated with implementation, but can provide immediate protection. Whereas Asset Management, Standards Base Approach, System Design and Supply Chain Management techniques when implemented into a utility's existing proceedures and processes will require less cost, but will require much more time to yield protection results and future security upgrade savings. Both need to be applied to meet short and long-term cybersecurity goals.

Get Published - Build a Following

The Energy Central Power Industry Network is based on one core idea - power industry professionals helping each other and advancing the industry by sharing and learning from each other.

If you have an experience or insight to share or have learned something from a conference or seminar, your peers and colleagues on Energy Central want to hear about it. It's also easy to share a link to an article you've liked or an industry resource that you think would be helpful.

                 Learn more about posting on Energy Central »