RECOMMENDATIONS FOR NEXT GENERATION UTILITY TECHNOLOGIES (Control and monitoring gas and electric with AI 'bots" and more...)
- Nov 15, 2021 5:08 pm GMT
We all know technology is rapidly changing industries and services. The electric and gas industry is no different. Not only office and accounting systems but a greater focus on migrating critical Operational Technology (OT) and/or Industrial Control Systems (ICS) systems - used to monitor and control gas and/or electric generation, transmission and distribution substations, generating plants and Control Centers – off-prem, to cloud service and supply chain providers.
The results are centralized and virtualized gas and electric generation and transmission Control Centers and cloud hosted gas and electric applications - where remote operators either individually or at control centers, monitor and control vast global swaths of gas and electric services. This also considers future automated virtual control centers where programmed AI applications operating in cloud data centers which ingest millions and billions of data points from field devices and combine with societal, economic and environmental conditions to determine, monitor and control gas and electric supplies and distribution - whether it be IoT distributed local energy sources or traditional generation to distribution chains. This transformation driven by technology, the cost to businesses and ratepayers, and a need for improved reliability and security of gas and electric systems at large.
What exactly is cloud though? Peeling it back you will find massive data centers with millions of computing platforms, hardware, software and applications collecting and moving data. It’s the Walmart of data. The proposition of a reduction in expensive in-house IT services combined with robust operational reliability from redundant data centers (physically located around the world) and high-speed communication paths is inevitable. Technology at electric and gas utilities is converging with legacy operational technologies (OT) and ICS. This includes replacing legacy equipment with digital based solutions such as AI operators and substation robotics which ingest data, correlate control and monitoring data from IoT equipment, and alert for meaningful operational, maintenance and security values, as well as trending to forecast future load, voltage and gas needs based on societal, economic and environmental considerations.
Cloud data center services and the virtualization of hardware platforms and their software applications are revolutionizing the supply and distribution of gas and electric power. Cloud technologies and their centralized hardware and software will allow the operation of gas and electric Control Centers from anywhere in the world and will impact and shape the people, processes and technology of these entity’s and customers alike. Given the current COVID-19 pandemic, we are already seeing the value of these considerations.
Looking ahead further automated energy management systems (EMS), automated transmission switching, line drones and robotics for line and equipment maintenance, robotic systems for substation relay testing, falling conductor protection systems to monitor and prevent wild fires, self-healing grid outage applications, and virtualized operations are potentials. The convergence of technology into these traditional operational technology (OT) areas will transform the gas and electric business
Cost – Virtualization and Cloud
A primary driver of virtualization and Cloud technologies is cost. Virtualization allows Entity’s to reduce the number of hardware platforms and their applications and services by moving software applications and their services to condensed “virtualized” hardware platforms. It is in essence the centralization of software applications and their services to hardware platforms. This reduces an Entity’s cost of support labor, hardware and software license fees. In addition to applications, it allows a cost-effective solution for the storage of company data – core records and system backup files (data).
Combined with the proposition of virtual control center “bots” whom monitor and control the gas and electric supplies, make switching decisions, shift voltage and load for a region or area, determine next day and next month usage, and take socio, environmental and economic considerations into account when operating the grid, and you have an extremely appealing path forward.
From an Entity’s perspective Cloud providers are able to better acquire and retain qualified personnel to support large Cloud data center environments and ensure the systems are configured and patched, monitored and maintained accordingly. However, pressed against this thought are the realities of the Colonial pipeline, Twitter, ADP, USOPM and other cyber breaches.
While virtualization allows the consolidation of applications and services onto hardware platforms, Cloud condenses hardware (virtualized and non-virtualized assets) and their applications and services to a centralized data center physical location. Cloud service providers are the “big box” stores for data services, containing in some cases millions of software applications on thousands of hardware platforms. Gas and electric Entity’s see the opportunity to eliminate locally owned corporate data centers in favor of Cloud providers data centers. Cloud providers provide a more cost-effective support model resulting in an exponential growth in Cloud services per Gartner’s 2019 forecast (table 1).
Cloud Business Process Services (BPaaS)
Cloud Application Infrastructure Services (PaaS)
Cloud Application Services (SaaS)
Cloud Management and Security Services
Cloud System Infrastructure Services (IaaS)
BPaaS = business process as a service; IaaS = infrastructure as a service; PaaS = platform as a service; SaaS = software as a service
Note: Totals may not add up due to rounding.
*Source: Gartner (April 2019)
Availability – Virtualization and Cloud
A second driver of virtualization and Cloud technologies is availability. Cloud data service providers build multiple Cloud data centers and are replicating data between data centers and across regions. This ensures reliable and available access to applications and services. This allows gas and electric entity’s to ensure their key systems and applications are accessible and reliable.
Security – Virtualization and Cloud
A third driver of virtualization and Cloud technologies is physical and cyber security. From the Entity’s perspective, Cloud data service providers have designed and implemented strong and robust physical cyber security controls. Physical access to Cloud data centers is very strong and Cloud providers prevent breaches of physical security or face reputational damage or loss of customers.
Cloud providers however are not perfect. Entity critical infrastructure applications (OT/ICS) are high risk targets. We already have seen cloud data breaches and compromises such as Twitter, ADP, Colonial, USOPM, Ukraine, etc… Cloud providers are supply chain hosts and must do better at ensuring their target rich environments are well protected. This may be difficult with critical infrastructure because of potential litigation between an entity and cloud provider. If a regional black out occurs because of a change or compromise to a cloud system, the result may be years of litigation. So more has to be done within this realm.
Infrastructure - Virtualization of Cloud based SCADA and ICS
Many gas and electric entity’s have migrated their corporate systems (email, web, databases, storage) applications to Cloud service provider data centers and some are contracting to migrate security systems and OT asset management and/or change management applications to Cloud providers.
Migrating critical core infrastructure such as electric and gas SCADA monitoring and control applications or industrial control systems (ICS) instrumentation, telemetry, or plant information systems offers the same cost benefit; however, these systems are critical to the Bulk Electric System (BES) and require greater risk considerations. Areas of impact for deploying and operating Cloud-based SCADA and ICS are:
- Control Centers
- Data Center
- Communication Networks
- Data and Information Protection
- Local Employment
The greatest cost advantage of virtualization and Cloud services is in virtualizing an Entity’s gas or electric Control Centers. Cloud services allow an entity to consolidate their existing Control Centers systems and personnel. It also paves the way to IA virtual operators. Currently entity certified operators can control and monitor the BES components and field conditions from one region using SCADA and ICS hardware and software applications which resides at a cloud service data center in another region.
An entity can also see valuable benefits in evaluating regional labor costs for operating staff. Add to this the move to AI virtualized operators who can take in billions of data points in making decisions about gas and electric services. Software developers will continue to develop less complex multi-functional applications for virtualized environments and provide “one stop” applications to monitor and control gas and electric systems. This also may reduce the numbers of operating personnel required to operate critical infrastructure systems.
Cloud Data Centers
Entity’s who migrate existing SCADA (EMS/DCS) and ICS applications to Cloud data center service providers are transferring the ownership and maintenance of critical infrastructure platforms and applications to Cloud platforms. They can eliminate or reduce the ownership and management of their existing data centers.
Entity’s may also choose to store BES Cyber System Information (BCSI) on Cloud hosted platforms.
The bulk of generated electricity occurs at stationary “as built” plant locations. The primary cost benefit from the virtualization of hardware and Cloud services for generation occurs from the consolidation of generation Control Centers to a single generation Control Center while generation SCADA (DCS) is operating as a service at a Cloud provider location. Similar to transmission, generating operating personnel can operate plants using Cloud-based SCADA (DCS) and plant support applications such as PI, vibration, environmental and chiller plant monitoring and control applications.
Local generation and balance of plant hardware such as programmable logic controllers, meters, heat, water testing sensors and other devices can be networked and eventually virtualized to communicate state data up to entity applications at Cloud provider locations, however the cost benefit currently does not exist.
As with generation, the cost benefit of virtualization and Cloud services to substations may be realized in virtualized and Cloud-based applications. RAS scheme applications, relay testing applications, and virtualized internet protocol-based relays, meters, condition-based maintenance applications, telemetry and communications processors may be virtualized to consolidate hardware. Manufacturers and software developers are designing micro devices which communicate their state to Control Center applications over IP (internet protocol) in order to allow Entity’s to proactively monitor and respond to system events.
Critical gas and electric applications require guaranteed network paths in order to transmit and carry data and voice services from end-point generation, substation and communications field devices to Control Center and Cloud data center based applications. This is critical to monitoring the BES for anomalies, and for the coordination of outages and issues between an Entity’s assets, their customers, and regional coordinators or regulators.
Data and Information Protection
Of critical importance is the protection of data streams between electric and gas critical infrastructure and cloud data centers and operational control centers. As critical infrastructure hardware and software become more generic and as more devices move to centralized large scale operations, and the more we move to an Internet of Things (IoT), the protection of data from end point devices is critical. Especially where local distribution sources converge with traditional SCADA/ICS data and as systems become more centralized. Data owners must protect critical infrastructure data from being compromised and ensure the integrity of the data stream. Actors whom area able to compromise the data stream for major centralized systems can alter load, voltage or other telemetry, or real-time data, and create a potential cascading event across multiple regions. Entity’s and cloud providers must ensure the protection of real-time critical infrastructure operating data. A compromise to a social media platform is low risk issue compared to the risk from a compromise to critical infrastructure systems.
Entity’s also must protect consumer data and information. As more electric vehicles, homes and small businesses use technology, so is the ability of system owners to collect information from consumer end points. Voice and video data, usage history, socio-economic status, health, and family or business information can be used to discriminate or spill creating a reputational risk to consumers, entity’s or vendors. Data from electric vehicles, refrigerators, HVAC systems, water systems – from wired and wireless monitoring – is at risk. It is therefore necessary that entity’s and their supply chain cloud vendors must ensure the protection of such data.
Another potential effect of the virtualization and move to Cloud services is the loss of technology positions in Entity cities and areas. Systems migrated to Cloud providers no longer require Entity’s to implement or maintain data centers and/or local support personnel; or acquire hardware or software from local suppliers.
Security and Cyber Security Risks
Virtualized and Cloud SCADA and ICS applications are critical infrastructure. It is imperative that industry and regulators understand and acknowledge that while cost effective, transferring critical infrastructure applications to Cloud service providers can have major impacts and implications to ensuring reliable electric and gas operations and services. Entity’s must implement controls to establish;
- Physical and Cyber Security
- Operating Service Agreements
- Reliable Network Communications
- Administrative Controls
- Personnel and Organizations
By managing risks to operations, reliability and safety, utilities and Cloud providers can provide interlocking security controls and ensure data and system confidentiality, integrity and availability.
Physical and Cyber Security
Utility security physical and cyber security and information assurance programs must be enhanced to consider, establish and/or test the security controls at Cloud data centers and between Control Center and end-point generation, substation, communications locations.
Gas and electric Entity’s must understand that Cloud data and assets can be easily replicated from one data center to another. This includes regional and international duplication. Entity’s controls should consider security controls around data at rest or in transit. This includes BCSI.
Operating Service Agreements
Entity and Cloud support agreements must include controls and processes for ensuring an entity can evaluate, test and document security controls, system redundancy, and the coordination and communication for projects and changes. These support agreements must establish metrics to ensure reliable, secure and available hardware platform applications services and controls to identify issues and restore reliable SCADA and ICS services at Cloud locations. This includes programs for supply chain, asset, change management and security event monitoring.
Entity’s may include contractual remedies however these do not act as a preventive or detective controls as in most cases an operating or security event will have occurred before any contractual obligations occur.
Reliable Network Communications
As previously stated, Entity’s must ensure redundant, secure and robust voice and data communications paths between Cloud service provider and Control Center and end-point generation, distribution and communications locations.
Entity’s must develop and maintain administrative controls to operate and troubleshoot SCADA and ICS systems outside of their immediate control or physical access. Cloud service providers currently do not have experience or understanding of the impacts from operating or maintaining gas and electric SCADA and ICS applications and Entity’s must establish process controls to ensure Cloud providers management of change processes do not adversely impact an Entity’s SCADA and ICS systems and in-turn its BES footprint.
Management of change process controls must include steps to manage, communicate, schedule, perform, test and verify the integrity of changes. They must also ensure standardized processes for problem resolution of operational, system and security events between plant (substation, generation), Cloud data center and Control Center locations and their hardware, applications and systems, as well as the restoral of critical services.
Entity’s must also ensure Cloud service providers have establish controls for software development. Software development presents risks to the security of SCADA and ICS applications and their platforms.
Entity’s must ensure corrective actions programs to monitor and identify systemic process issues and provide corrective and mitigating actions for Cloud providers.
Personnel and Organizations
Because of the disparate distances and functions between Cloud systems and a utility’s virtualized transmission Control Centers and SCADA/ICS/OT substation locations, the potential for silo’s is a risk.
Personnel, maintenance support departments, and management at Cloud providers and utilities must align and agree to ensure standardized operational practices for Cloud service provider personnel such as software developers, contractors and support personnel. Personnel must have the experience necessary to support and maintain gas and electric SCADA and ICS applications which reside in Cloud data centers and must be aware of the operational impact of these systems upon an Entity’s BES assets and operating footprint.
Entity’s must establish controls for accountability to ensure Cloud data service providers establish, maintain and follow processes. Entity’s must be able to communicate and coordinate outages and status with their customers, Reliability Coordinators (RC) and regulators across times zones and regions.
Within the conversation are automated or virtual operators making control and monitoring decisions. This is where technology may shine, but move forward cautiously.
Generator, gas and electric substation operating environments are dangerous places. Electric transmission Control Center operators make grid switching decisions that can be life and death propositions. An operator at a virtualized electric transmission (or gas or generation) Control Center in Missouri, performing or scheduling transmission switching for substations in Arizona, but using a Cloud-hosted EMS SCADA/ICS system located in Seattle or Australia, must ensure its systems are configured correctly and that administrative processes have strong coordination and communication controls so that switching decisions and operations do not injure workers.
Likewise, a technician or software developer for a Cloud data center must ensure that changes to Cloud-hosted SCADA/ICS hardware and software do not adversely impact operations and safety in either Missouri or Arizona. The last thing Cloud providers or utilities want is a major outage or injury from a change. Because these locations and owners may be physically apart, a focus needs to be on sound processes.
Compliance - Gaps and “Seams” in the Cloud Governance
The role of NERC governance and oversight with cloud technologies and operations is complex. There are currently knowledge, skills and experience gaps between Entity Information Technology (IT), Operational Technology (OT), and cloud service providers personnel. The convergence and standardization of technologies in OT and IT arenas is helping to close this skills gap, however this process is slow due to rate-cases and return on equity.
Highlighting the risk is that Cloud service providers have little or no core knowledge or experience with gas and electric operational environments or maintaining critical gas and electric SCADA and ICS applications and systems. As mentioned above, cloud data center hosted SCADA and ICS applications and platforms are critical infrastructure. Without prescribed and verifiable controls, the risks of adverse cascading operational, safety or security events or incidents are amplified.
Lastly, the old adage to never put all one’s eggs in a basket is a more than justified analogy. Government, regulators and entity’s must ensure a percent of SCADA and ICS systems exist in locations which are not subject to massive cyber security or operational cascading events. The Covid-19 pandemic is a prime example of a potential impact to the stability of the grid.
NERC and the regions must be prepared to measure CMEP processes across non-entity companies, processes, personnel and environments. In order to do this NERC must work with industry to define the controls within the standards and requirements. This will allow regulators and entity’s alike to review and verify controls implemented at third-party cloud provider locations. NERC and the regions will need to align better to develop the CMEP processes to communicate and guide the verification of controls.
NERC should revamp the registration process. In the current state, registrations ultimately drive the risk profiles and functions within the standards and requirements. Registering Transmission Operators or Owners (TO/TOP) for example sets the basis for entity’s assets, functions and risk. Rather than basing these functions on registration, NERC should consider establishing a process to identify risks directly related to the Electric Grid based on operating footprints, functions and impacts to critical infrastructure, people and cities.
NERC must also address issues within the Standards Authorization Requests (SARs) and Standards Drafting processes to include identified controls as well as ensure are new standards or revisions are functionally mapped within “like” subject areas. They must also ensure that Standards Drafting teams work collaboratively in developing content.
The standards drafting processes must better align standards to entity functions. This is challenging because of the various footprints, technologies and organziations of entity’s. Transmission Operations standard TOP-1-4, requirements R20 to R24 for example are misplaced set of requirements which contain prescriptions for entity’s to establish “redundant and diversely routed data exchange infrastructure.” The subject requirements here are technology architecture and design based and do not map to Transmission Operational standards. This results in inefficiencies internal to entity’s organizational support and compliance program processes. These requirements are better mapped to a Technology Standard.
CIP standards are also mapped to areas which do not match entity or security framework functions (such as NIST or ISO). This creates inefficient silo’d and mismatched controls and processes resulting in ambiguous interpretations, misalignments with entity information assurance programs, ambiguous audit translations and functional process issues. Transient Cyber Assets (TCAs) are identified and mixed with CIP-010 change management processes, when they should occur in CIP-002. Low impact assets are defined in CIP-002 but managed in CIP-003 which is reserved for policies, and EACMS and PACS are obscurely identified in CIP-002, CIP-005 and CIP-006, but not explicit. The common thread among these is asset management (generally associated with CIP-002).
Similar mapping issues occur in Supply Chain concepts where supply chain program concepts are mixed with change management (in CIP-010), security patch management (in CIP-007 R2) and vendors systems and access (CIP-005). These concepts are better aligned with other CIP standards.
Realignment of Standards and Requirements to Information Assurance Program Areas
A proposed solution to these mismatches for CIP could be mapping SARs and drafting initiatives to security framework (NIST, ISO, etc..) defense-in-depth sub-program areas. This includes but is not limited to:
- Asset management (identification and risk of software and hardware assets)
- Change management (schedule, test, perform, verify change, systems administration)
- Access Management (electronic access and accounts)
- Security Architecture (preventive technical controls (IDS/IPS, firewalls)
- Software patch management
- Security Event and Incident Response
- Information Security
- System Restoration (disaster recovery/BRCP)
- Physical Security
- Personnel Risk Management (physical and electronic/employees, contractors, vendors)
- Governance, Risk and Compliance
- Supply Chain (procurement and sourcing)
Aligning CIP standards to NIST and ISO sub-program areas results in more efficient processes, reduction in resource loads, enhanced communications and functions and the reduction of risk areas and violations.
New Technology Standard and Requirements – IOT-001-1
New technologies areas such as virtualization, cloud and robotics will not map to current CIP or Operations and Technology standards and requirements. A new Standard body, such as the NERC Information and Operational Technology (IOT-001-1) standards and requirements should be developed to allow the existing SAR, drafting team and Standards Efficiency Review (SER) processes to migrate existing IT and OT centric standards to technology functional areas. This in-turn aligns to entity’s organizations, processes and technologies; reducing resource, process and audit constraints. As will, this allows the SAR and Standards Drafting processes to migrate the existing non-functional CIP or O&P standards and requirements to an IT/OT standard body.
Entity’s have established Internal NERC Compliance (ICP), operations, engineering, maintenance and cyber security programs. Aligning and mapping those programs will result in industry efficiencies and cost savings.
Auditing Critical Infrastructure in the Cloud
Cloud providers may be resistant to allowing entity’s (or regulators) the ability to verify operational or security controls, let alone be reviewed for audits or establish corrective actions for issues identified at Cloud provider locations. Litigation may occur. This may be a sticking point to the proposition of regulated SCADA and ICS applications in Cloud data center environments, however there seems to be no alternative given the risks from Cloud based SCADA and ICS applications and services to the Bulk Electric System.
Because NERC regulation does not yet address Cloud service providers, Entity’s and regulators must train, contract and retain personnel and expand their programs to verify and audit Entity’s Cloud serviced SCADA and ICS. Regulators must ensure they understand Cloud technologies, operating environments and the advances in technologies. How will regulations regulate AI virtual grid operators? How will cloud providers and entity’s protect virtual operators?
Technologies change rapidly and entity personnel as well as regulators will be required to keep current on operations, risk, cyber security, process controls and technology. Entity personnel, audit and enforcement personnel alike must understand Cloud and ICS environments, the hardware and software used, the business process controls and the risks to the grid. This requires continual education and training effort. Because data centers are replicated, the audit scope may physically exceed current NERC regions.
There are strong drivers toward the automation, virtualization and implementation of Cloud based technologies and services for gas and electric entity’s. Juxtaposed to these drivers is a nation’s grid, it’s stakeholders, technologies, personnel and interconnects to Canada and Mexico and the world. A complex mix of large, medium, small private, public and community footprints.
Entity’s must ensure they are not front-page news as the next compromised entity. This may lead to costly litigation where service was lost, security or safety compromised or where cascading events occurred.
Implementing secure and reliable Cloud services is critically important due to the potential for silo’s between Entity and Cloud organizations, and an ever-growing base of malicious actors who breach systems, steal data, cause public relations reputational and financial risks to gas and electric Entity’s.
For organizations moving to Cloud based gas and electric OT SCADA applications, the risks to the grid at large are even greater. These risks highlight the need for entity’s to be more vigilant than ever in leveraging their information security programs and professionals in order to ensure the protection of valuable information assets and data. It is also critical to maintain a regulatory backstop for critical services.
About the author
Barry Jones has over 24 years’ experience at gas and electric companies working in network and cyber security engineering and operations, NERC CIP compliance programs and as a Principal Consultant developing policy and administrative controls for private Entity’s and the federal government. Experience includes direct experience with IT, OT and ICS SCADA systems located in transmission gas and electric Control Centers, electric substations, generation plants, communications buildings, data centers and manufacturing facilities and with enterprise, local and legacy operating systems and applications on virtualized, standalone systems.
No discussions yet. Start a discussion below.
Get Published - Build a Following
The Energy Central Power Industry Network is based on one core idea - power industry professionals helping each other and advancing the industry by sharing and learning from each other.
If you have an experience or insight to share or have learned something from a conference or seminar, your peers and colleagues on Energy Central want to hear about it. It's also easy to share a link to an article you've liked or an industry resource that you think would be helpful.