The mission of this group is to bring together utility professionals in the power industry who are in the thick of the digital utility transformation. 

WARNING: SIGN-IN

You need to be a member of Energy Central to access some features and content. Please or register to continue.

Question

Has your utility completed a successful collaboration between IT and OT, or do you know of any publicly available examples of such collaborations?

I’m actively involved with the IEEE Power and Energy Society’s Technical Committee on Power System Communications and Cybersecurity (PSCCC), and right now I’m chairing a task force on Utility IT-OT Cybersecurity Challenges in Roles and Terminology. As we gather information for this task force, I’m actively seeking out input from practitioners in the utility space, both in IT and OT, on what sort of collaborations they’ve undergone, what challenges arose from these arrangements, and more. So if you have completed such a collaboration, I’d also specifically ask:

  • Was the collaboration cyber security related?
  • What made the collaboration successful?
  •  How did you resolve the differences in cybersecurity philosophy that exist between IT and OT?

Answers

Theo, NERC CIP-010-3 R1, Part 1.6, Software Verification requirements does indeed cross cut IT/OT on cybersecurity. The focus being on software objects, without distinction of where that software will be deployed, i.e. an HMI Windows desktop or an RTU or any other device within a BES control infrastructure, is a great use case showing an IT/OT intersection. The CIP-010-3 SAG-PM™ cybersecurity risk assessment software that analyzes software objects before deployment can be used by IT and/or OT personnel to detect harmful software objects and report a trust score to help decision makers decide to install a software object, or not.

Theo Laughner's picture
Theo Laughner on May 26, 2020 2:44 pm GMT

Thanks for the information!

 

I don’t have any recent experience to provide. At a major utility at which I previously worked, though, we addressed the IT/OT divide quite a few years ago. One of the first steps was to bring the OT group that supported their applications (e.g EMS), kicking and screaming, into the IT organization. That resulted in them following project management processes and also a broadening of their enterprise architecture perspective. I had previously worked with the OT group on a few minor projects that involved getting data out of their systems. That resulted in some clunky but useable interfaces that got around the security challenges.

From a security standpoint the utility's IT Security group addresses both IT and OT security. Aside from the challenge of securing the grid in general the issue of data flowing between the two networks was a big one. Firewalls were the basic solution. So from the collaboration standpoint that was mostly addressed through the organizational alignment. Obviously there were some collaboration challenges at the lower levels more from an “it’s mine and you can’t touch it” standpoint as well as some technical issues.

One of the issues comes with applications that span the divide. The primary one being OMS since it uses data from both the Customer application and potentially from EMS/SCADA or other OT applications. At this particular utility, OMS runs on the IT side. I have talked with other utilities where it runs on the OT side especially if it runs under the wing of ADMS. At my utility, the OT connection for OMS did not come until a few years after its initial implementation so not really a concern at that time.

Theo Laughner's picture
Theo Laughner on May 26, 2020 2:49 pm GMT

I'm curious if organizational realignments helped develop cultural understandings at the utility you mention?  One of the things we are attempting to address is capturing differences in terms like reliablity which to IT may mean 95%+ uptime, where as for OT this is more like %99.999+.  

on May 26, 2020 10:05 pm GMT

The short answer is yes. Amazingly, or perhaps not, I think there were a lot of people in IT who didn’t even realize the OT side existed or if they knew they existed didn’t understand what they did. So a lot of effort has to go into gaining that understanding including what differences may exist in areas like reliability and security. On the OT side because they were so siloed things like integration and related architectures weren’t a real concern to them at the time. So on that side they need to understand that they were part of a larger enterprise and that the time of living in isolation was no longer. Getting  over the “we are different” or “we are special” hurdle can take a while. Just saying “no you are not” is not a good approach. Understanding what the differences are and if they are truly differences takes effort. Particularly in the reliability and security areas risk analysis is important as well as understanding that one size doesn’t fit all.

Side story relative to the lack of knowing about OT … at one point early on in the consolidation one of my colleagues on the OT side gave a presentation on what the grid was all about to people on the IT side. When he came to talking about substations and asking if anyone knew what their function was the answer came back “that’s where the electricity is stored”. Obviously a good laugh was had on that one. That was well before the time of battery storage which now does exist in the substations so I guess the person with that response was just a very forward thinker.

Below is some work we did around how data should be viewed from a convergence viewpoint. We have done work around security as well.

Below is a refreshed OT/IT Digital Orchestration (we have stopped using convergence as it implies interoperability which has been problematic for solution providers to agree upon). However what is essential is the orchestration of data, the security context and critical information streams.

Theo Laughner's picture
Theo Laughner on May 26, 2020 2:50 pm GMT

This is really excellent insight!  Is any of this public domain or available to use within the guide?

Ben Ettlinger's picture
Ben Ettlinger on Jun 1, 2020 7:27 pm GMT

Here at NYPA it's an Enterprise Architecture and Engineering Group initiated effort. At NYPA we have begun to embrace the difference between OT and IT while at the same time learning to better collaborate and work together. One of the keys for the IT department was understanding that the OT environment operates with different requirements and SLAs and in many cases requires a fully redundant 24x7x365 HA architecture.  At the same time the OT group is subject to much stricter oversight and regulation from external agencies.  The key however was to embrace the similarities and the interdependencies.  At NYPA the Infrastructure and Networking groups from IT provide a lot of direct service to OT and the plants. An important part of that collaboration was for the IT staff to physically visit the plants and get to know the local OT staff.  The plant staff has said several times they have found great value in having central IT staff visiting the site and getting a firsthand look at the facilities.  This was critically important in a most recent upgrade of our enterprise storage platform which involve equipment room build out and installations at each plant.

     Additionally we have been holding two separate OT/IT coordination meetings one at the Director level and one for the line staff.  The director level meeting addresses longer term and broader collaboration for projects, planning, strategy, and procurements while the other meeting focuses on the day-to-day/week-to-week tactical issues. Most utilities have the executive level meeting, as did NYPA; however, when we instituted the bi-weekly line staff meeting (with no senior leaders invited) we were able to begin laying the foundation for true collaboration.  We keep a simple agenda with only three topics as follows:

 

Near term (next 30 days) IT projects and initiative that could impact OT network or staff – (presented by IT)

Near term (next 30 days) OT Project needs and coordination issues for IT awareness and assistance – (presented round robin by each OT attendee)

Any other topic raised by the group – oftentimes OT/OT site coordination issues that are not addressed in any other forum.

 

The most important message was IT reaching out and saying ‘what can we (IT) do to help you?’  This included providing resources, following up on technology/procurement approvals, and oftentimes just connecting the right people to have a conversation. We have begun to see the most convergence around the security and infrastructure disciplines with IT providing the majority f the support for the sites.  Another recent change implemented 2-3 years ago was that our Solution Engineering group provides Solution Engineers to work directly on OT project to assist with document and detailed design efforts.  These staff focused on bridging the communication gap between OT/IT and collect/develop the majority of the documentation IT requires for technology projects.  This was definitely seen as positive by the OT group as it helped them get there work done and provided them with well-organized, detailed as-built/as-configured system information.

 

We believe our success is based on establishing a forum for communication and collaboration. Over the past two years it ‘feels’ like the IT/OT tension has lessened considerably and when anything does flare up we have a forum for addressing it.  Our guiding principle for collaboration/convergence has been acknowledging that working together is imperative to the organizations success and outcomes cannot be about winning/losing.  Most importantly  our IT group has learned we need to approach every opportunity with a ‘how can we get this done’ mindset instead of always being seen as a barrier or the group that always tries to say ‘No’.

 

 

Tap Into The Experience of the Network

One of the great things about our industry is our willingness to share knowledge and experience.

The Energy Central Q&A platform allows you to easily tap into the experience of thousands of your colleagues in utilities.

When you need advice, have a tough problem or just need other viewpoints, post a question. Your question will go out to our network of industry professionals and experts. If it is sensitive, you can post anonymously.