Post

EnergyIoT Article 3 – IoT Architecture Big Picture

Posted to GridIntellect, LLC – A Veteran-owned Company in the Digital Utility Group
image credit: Image by Stuart McCafferty - No copyright
Stuart McCafferty's picture
Chief Architect, eaaS Siemens Smart Infrastructure

2021 Cleanie Award winner. Siemens Smart Infrastructure CTO Office, technologist, distributed energy expert, researcher, author, and climate change warrior. Genuinely focused on doing good for...

  • Member since 2018
  • 108 items added with 100,921 views
  • Apr 29, 2019
  • 4188 views

EnergyIoT Article 3 – IoT Architecture Big Picture

By Stuart McCafferty, Contributions from Eamonn McCormick and David Forfia

Disclaimer:  The viewpoints in this article and others in the series are the personal views of the authors and in no way are meant to imply or represent those of the companies they work for.

This is the third in a series of articles introducing an EnergyIoT Conceptual Model that leverages modern architectural elements, such as virtualization, containerization, orchestration, rich semantic information models, message buses, DevOps  microservices, and the cloud.  This architecture:

  • Is elastic and scalable
  • Is service-oriented
  • Is event-driven
  • Enables edge computing as well as more traditional centralized computing
  • Is data-centric
  • Is secure
  • Uses Internet Protocol (IP) addressing scheme
  • Uses rich semantics for communication (not points or register-based) – interoperability
  • Is extensible and designed to be adapted in the future
  • Is redundant and fail safe
  • Can be deployed on premise, in the cloud, distributed on the grid, or any combination of those options
  • Employs distributed hierarchical control and coordination
  • Is designed with expected communication losses

The future grid must transform from a top down to a bottom up hierarchical paradigm and become much more nimble and adaptable.  The systems need to be able to scale to thousands, hundreds of thousands, or millions of assets.  The technologies implemented need to become more aligned with Industrial Internet of Things (IIoT), to include virtualization and containerization, standard information models and message buses, DevOps techniques, microservices, and the flexibility and scalability provided by cloud.  The systems should be designed with the intent of fully leveraging the latest mobile and wired communication technologies (e.g. 5G) delivering improved communication capabilities but continue to safely run autonomously when communication is interrupted. 


New assets need to provision themselves and become active, dispatchable system elements quickly and simply.  The system needs to be secure and resilient to local outages, whether the failure is device, weather or an intentional physical or cyber-attack, and prevent cascading events.  Transmission and distribution systems must remain in balance by leveraging both distributed generation and demand response DER resources.  Customers of all sizes and new business entities should be able to choose to participate in markets allowing open access.  All of this must be done carefully and practically - enabling "brown field" legacy assets and systems to continue to operate while the transformation occurs.  The EnergyIoT Conceptual Architecture we will be describing in this article series is designed with these fundamental ideas as its guiding principles.

What is happening here?

In its most simplistic terms, the energy industry is experiencing the merging of IT and OT.  This transformation is the poster child for the Industrial Internet of Things (IIoT).  When considering the sheer scale, huge volume of data, opportunities for analytics and Artificial Intelligence, the amount and distance of distributed assets, the electric power industry is literally the “Poster Child” for IIoT.

Figure 1- Energy, the Poster Child for IoT

The conceptual drawing in figure 2 is a simple representation of some very complex system of system relationships.  The figure is abstract enough to easily represent the big picture, but also be able to consider each of the components individually and have a good understanding of what role they perform, the data they generate, and how each element interacts with other ecosystem components.  The architecture is event-driven and data-centric.  At its core, the architecture is loosely-coupled and includes state-of-the-art security and access control techniques that minimize the probability of cascading events due to natural disruptions or a physical or cyber-attack. 

Figure 2- EnergyIoT Conceptual Model
(developed for this series with the GridWise Architecture Council)

Each grouped set of items within the two clouds and box are the “domains” of the conceptual architecture.  The two IT domains (Energy Systems Cloud and Energy Services Cloud) include virtual components of the EnergyIoT ecosystem – software, middleware, core services, and data.  The IT domains can live anywhere – on-premise at utility data centers, hosted in the cloud, or deployed on physical grid assets.  The OT domain includes the physical components – the grid, sensors, and machines.  The physical and virtual need to work hand-in-hand.

The Stack View

Another way to consider the power of this architecture is through a technology stack view. 

EnergyIoT Model Stacked View

Figure 3- Energy IoT Reference Architecture
(developed for this series with the Gridwise Architecture Council)

Figure 3 shows the Operational Technology (OT) layer of physical grid Assets at the bottom part of the stack.  The Services/DevOps layer resides in the middle and abstracts the Assets from the Roles layer on the top.  The Roles layer includes the businesses, consumers and other stakeholders who ultimately use and operate the assets.  The key point of this diagram is that most of what is represented already exists.  In fact, the green-colored services in the middleware can be found today on any of the major cloud vendor’s sites.  The yellow-color shaded rectangles are new services that must be developed and tuned specifically to support the operational and compliance requirements of the electric power industry.  There are probably some new services yet to be discovered, but a large number of building blocks are available today.  It will not be a simple or instantaneous transition and inevitably there will be problems to be solved, but assembling an energy-specific Services layer and developing the missing yellow data-centric and associated services in Figure 3 can help propel industry-wide adoption of the EnergyIoT architecture ecosystem we will be discussing throughout this series of articles.

 Operational Technology DomainOperational Technology (OT) Domain

The OT Domain includes all the physical assets that make up the electric power grid.  This includes generators, wires, substations, switches, poles, transformers, capacitor banks, remote terminal units, taps, solar panels and inverters, energy storage and any other devices yet to be invented.  As DER penetration grows, the grid will need to become more nimble and adaptable – a “neural grid” – that has assets that communicate and cooperate with one another to support stability and resilience.  It includes the following sub-domains:

  1. Sensors and Measurement:  Required for situational awareness for systems, devices and people.
  2. Bulk Generation: Needed to support bulk and baseline power and ancillary services for reliability.
  3. Telecom Infrastructure:  Essential to support grid asset interaction.
  4. Aggregators and Community Choice:   Provides a means for all in the community to participate regardless of financial means.
  5. Smart Homes, Buildings & Cities:  Required systems and assets to communicate and/or dispatch assets in homes, buildings, and cities.
  6. Distributed Energy Resources:   Enables all forms of distributed generation, energy storage, or demand response to operate and interconnect safely.
  7. Security:  Prerequisite physical and cyber security delivered through applications, devices and processes.
  8. Electric Transportation:  Necessary systems and devices to support any form of electrified transportation.
  9. Neural Grid:  Vital technologies that provide for safe, reliable, nimble and adaptive grid in timeframes faster than humanly possible.

 This domain will be discussed in more detail in “EnergyIoT Article 4 – Operational Technology (OT) Domain”.

Energy Systems Cloud Domain Energy Systems Cloud Domain

The Energy Systems Cloud Domain is a utility perspective of the application systems, devices, knowledge and processes required to plan and operate the grid.  It is broken out by its sub-domains:

  1.  Planning:  Systems to enable forecasting, and short-term, and long-term planning.
  2. Construction and Maintenance:  Required systems to support construction, geo-locating, and maintaining infrastructure.
  3. Comms and Security:  Required systems for voice and data communications, and physical and cyber security services.
  4. Customer:  Required systems to enable customer engagement, program enrollment, interconnect, settlement, and billing.
  5. Markets:  Required systems to enable transmission and future distribution markets.
  6. Operations:  Required systems to enable safe and efficient operations of the transmission and distribution grid networks.

 This domain will be discussed in more detail in “EnergyIoT Article 5 – Energy Systems Domain”.

 

Energy Services Cloud (DevOps) Domain (Green Cloud)

The Energy Services Cloud or DevOps Domain is the heart of this architecture. The “Green Cloud” services provide a “common abstraction layer” between the physical and virtual, dramatically simplifying when changes occur to the OT physical model.  Interoperability and streamlined communications between services, systems, and grid assets is delivered as a service.  It supports an “energy-specific” rapid development environment, data repositories, and analytics.  Based upon the services provided by the major cloud vendors, these existing structures, tools, and services support the DevOps techniques, scale, and virtualization necessary to support a DER-rich, bi-directional energy flow, dynamic and adaptive grid.  As a “data-centric” architecture, the Energy Services Cloud is designed with data and supporting data services as the centerpiece of the entire architecture.  This domain includes the following services: 

  1. Smart Contracts, Digital Ledger Technology:  Provide trustless, immutable transaction tracking systems utilizing technologies like Blockchain.
  2. Structured Data:  Allow storage and retrieval of relational data.
  3. Unstructured Data:  Deliver storage and retrieval of raw data files like NoSQL.
  4. Digital Twin Agent:  Enable the virtual emulation of assets and groups of assets.
  5. Security and Identity Management:  Automates encryption, identity management, relationship management, and access management.
  6. Source Code Management:   Allows developers to maintain source code development, modification, and collaborative development while ensuring configuration management.
  7. Orchestration:  Provision, execute, and choreograph processes.
  8. Microservices:  Standardizes services to containerize (and orchestrate virtualization), communicate and translate between standards, middleware operations, and other common tools for energy-specific DevOps environment.
  9. Artificial Intelligence and Analytics:  Amplifies operation and design engineer’s abilities for analysis and optimization.

 This domain will be discussed in more detail in “EnergyIoT Article 6 – Energy Services (DevOps) Domain”.

 Conclusion

Recognize that this is a completely different architectural construct from today’s hub and spoke architecture.  The grid of the future is coming quicker than you think. New methods and technologies to scale and support rapid grid configuration changes are needed.  This EnergyIoT Conceptual Architecture is designed to leverage today’s most advanced technologies, uses existing standards, and leverages the services from cloud providers to provide extensibility and scalability that is future-proof.  This is a moonshot opportunity for our generation.  It cannot be done by one company alone.  It will take an army of people with similar vision and passion to solve today’s electric grid problems – and create the inclusive, de-carbonized, plug-and-play “grid of tomorrow”.

Let’s start a conversation.

The Article Series will continue with deeper dives into each of the three different domains.  Please see our next article on Thursday, May 2 on Energy Central, “EnergyIoT Article 4 – Operational Technology Domain” for more details on the physical grid assets.

The rest of the article series can be found here:  

 

About the Authors

Eamonn McCormick, Chief Technology Officer, Utilicast

Eamonn McCormick is the CTO at Utilicast, a leading energy industry consultancy. Eamonn is a passionate believer in the bright future of the energy industry and the importance of collaboration as the foundation for solving for our current industry challenges. He is a results driven technology leader with a track record of success. He has implemented strategic technology change at several large energy companies over the last twenty years in the areas of wholesale markets, transmission and energy distribution primarily. In addition Eamonn is currently chief architect of the Energy Block Chain consortium.

Stuart McCafferty, IoT Architect, Black & Veatch

Stuart McCafferty is an accomplished Smart Grid technical executive with an innovative history, strong relationships in the utility and vendor communities, business and partner development, platform and solution design, go to market planning and execution, and practical application of existing and emerging/disruptive technologies. Prior to B&V, he was VP of EnergyIoT for Hitachi America, where he led the architectural design of a distribution system platform supporting microgrid and Distributed Energy Resource (DER) related businesses.  At B&V, Stuart supports the utility, technology, and vendor communities in strategy and pragmatic application of DER that combines IoT best practices and technologies with energy standards and protocols.

Thought leader in the Internet of Things (IoT), Big Data, Cloud Computing, Artificial Intelligence (AI), Machine Learning, and connected home with practical application within the Smart Grid ecosystem. Expert in utility IT/OT and the application of DER and microgrids for resilience, economics, and reliability.

Stuart is a US military veteran, Air Force Academy graduate, an Energy Fellow for community resilience at the National Institute of Standards and Technology (NIST), an Energy “Expert” for Energy Central, and Vice Chair of the Open Field Message Bus (OpenFMB) user group.

 David Forfia, Gridwise Architecture Chair

David is the Chair of the GridWise Architecture Council since 2015 and has been a council member since 2013.

The GridWise Architecture Council (GWAC) is a team of industry leaders who are shaping the guiding principles of a highly intelligent and interactive electric system. The Council is neither a design team, nor a standards making body. Its role is to help identify areas for standardization that allow significant levels of interoperation between system components. More about the Council can be found at www.gridwiseac.org

David is the current chair of the Technical Advisory Committee and a former member of the Board of Directors of the Smart Electric Power Alliance.  He was also Chair of the SGIP Board of Directors from 2015 until 2017, and as a board member beginning in 2011.
 
In his current role, he is the Director of Technology Architecture and IT Transformation at the Electric Reliability Council of Texas (ERCOT).  He began his career at Austin Energy Director of Information Technology Services for Austin Energy and was Deputy Director and Chief Information Officer for an $18B pension fund. He holds a BBA from the University of Texas at Austin and an MBA from St. Edward’s University.

 

Discussions
Matt Chester's picture
Matt Chester on Apr 30, 2019

I've really enjoyed this series of articles, Stuart, I think you're onto something extremely important and are pushing for it in a really timely moment. 

"The future grid must transform from a top down to a bottom up hierarchical paradigm and become much more nimble and adaptable"

This quote sticks out to me and could not be more pertinent to the utility industry. The power sector has long been somewhat of an inertia-filled legacy industry, and we are no doubt in the midst of the need (and delivery) of the greatest amount of evolution the sector has seen since its early days. I'm hoping leaders across the industry have this same mindset (or perhaps adapt it after reading your pieces) as we look to tackle the types of challenges  you lay out in the coming years. 

Stuart McCafferty's picture
Stuart McCafferty on Apr 30, 2019

Thank you again, Matt.  This is not going to be a simple transition, but even my most traditional-thinking architect friends agree that our current situation is untenable and we simply cannot scale to the dimensions anticipated with DER and electric vehicles entering the system.

David, Eamonn, and I will also be the first to say that our proposed solution is not perfect.  We don't know what we don't know.  But, the whole point of this series is to get people thinking and talking.  If that is all that happens, then we probably failed, but I think that once people do start talking, we will see the change begin to happen.  It's common sense and it isn't rocket science.  But, it is disruptive change nonetheless.

I'm trying to respond to any comments we get so we keep the conversation and engagement going.  I read ALL of them.  I sincerely appreciate your feedback and if you see issues with anything we are saying in these articles, please do not worry about hurting our feelings.  All of us are trying to learn from this - and discovering the ommissions and mistakes is all part of that.

Thanks, Stu

Jake Brooks's picture
Jake Brooks on May 3, 2019

Congratulations Stuart, Eamonn and David, on another brilliant article. You have managed to integrate a slew of existing concepts into a something much bigger, a living ecosystem. One of the questions that this article prompts is how to think about the interactions between systems clouds and services clouds in other sectors besides energy. From an enterprise perspective, the energy ecosystem will often be closely tied together with telcomm, IT, fintech, transportation and other IoT ecosystems - even in the energy specific services layer you and others have described. I think your model is likely adapatable enough to handle this, but certain business drivers may push through the edges of the envelope relatively quickly.

Another question is what forms of regulation will be appropriate in such a complex system, and how to prevent abuse. Inevitably there will be social equity issues, largely on how to share cost responsibility. Better to address them early on rather than have them crop up in the form of objections to otherwise good ideas.

It's probably a blessing and a curse that this construct is built largely with existing standards. A third question will arise on how to determine whether an existing standard is holding back development of important opportunities.

Entrepreneurs in the sector will be searching for indications of where business opportunities exist for new forms of integration and orchestration. It could be that the most successful energy companies will be those who are most effective, not in assembling physical assets, but in disseminating meaningful, effective control over appropriate network functions to the widest number of active users of the grid. Business success may be defined by how well entrepreneurs enable rapidly-evolving networks of people. A fourth question arises from this: What is the best approach for enabling consumers to manage increasing levels of complexity? This issue was developed in an article on orchestration I shared last last year. (See https://bit.ly/2PBC8lL)

There is lots of interesting work ahead and you've proposed a road map. It's terrific the way you are openly inviting input from all commentators. Many others are working in a similar vein - and this kind of interaction is a sure way to build the inherent strength of the evolving new system.

Stuart McCafferty's picture
Stuart McCafferty on Apr 30, 2019

Hi Jake, thanks for the feedback and questions.  There are way more questions than any of us have answers.  But, putting them out there for people to ponder and work with other is absolutely the correct approach.

Some of the regulation questions are probably pretty straight-forward.  I know David Forfia has some pretty good perspectives there with his experience.  There are some really sharp commission folks out there that are probably having a lot of these conversations already.  Hopefully, we can get the right questions in front of them to come up with the best answers.

The standards I was referring to were semantic and not protocols.  So, I actually am pretty comfortable with the international work done in this area and recognize that they were developed by large bodies of people and companies - and were governed in a consensus process.

The entrepreneurs you spoke of are the true wildcard in all of this.  It's exciting to think about opening up an energy ecosystem that entices creative minds to participate and bring unimaginable new benefits to the stakeholders.  Probably the thing that excites me the most about this idea, frankly.  And, when innovation is encouraged, things get simpler, not more complex.  The Uber example we used in the second article is a pretty good idea of what can happen when a disruptive force brings not only new and better capabilities, but also simplifies the interface with the customer.  But, you are right, there is going to be a lot of change and we will need that innovation, artificial intelligence, ROBOTS! to make a lot of this work smoothly (my opinion).

I have been writing like mad to try to get the last of these completed, so I apologize for not responding quicker.  As I said previously in another comment I responded to, I read all of these and respect that you spent the time to make a comment.  So, I want to thank you once again and please continue to bring up these provocative questions and make us and all the other readers think.

Thanks Jake, Stu

 

 

Jim Horstman's picture
Jim Horstman on Apr 30, 2019

A good start on the arcthictecture perspective. A few comments.

Since you state that this could be implemented on prem or in cloud identifying two of the domains as 'clouds' seems strange.

I am not sure why you would include the term Dev Ops in the services domain along with source code management. Those would seem to be incongruous with the domain.

Geospatial seems oddly placed in the C&M sub domain. Geospatial capabilities are utilized broadly not just for C&M. Perhaps they should be considered a service?

Weather, or more broadly following the work done in the IEC CIM, Environment could also be considered a service. The use of environmental information is increasing not only for forecasting but also in areas such as operations for predicting outages.

In spite of the IT and OT are merging statement they are still shown as distinct domains. Perhaps it is time to move on from using those terms.

I'm not a big fan of the 'digital twin' concept as it strikes me as just another 'buzz term' but somewhere, perhaps in the center of the IT domain should be the network model and it's use and management. EPRI currently is working on extending their previous effort in this space to the distribution level in large part due to DER and its impact.

I'm looking forward to the upcoming articles!

Rahul Krishna Pandey's picture
Rahul Krishna Pandey on May 1, 2019

What a wonderful insights by Stuart McCafferty, Eamonn McCormick, and David Forfia on Energy IoT Conceptual Model that leverages modern architectural elements, such as virtualization, containerization, orchestration, rich semantic information models, message buses, DevOps  micro services, and the cloud.  The systems need to be able to scale to thousands, hundreds of thousands, or millions of assets ------ that is really needed ….. to visualize a big picture IoT architecture.

Stuart McCafferty's picture
Thank Stuart for the Post!
Energy Central contributors share their experience and insights for the benefit of other Members (like you). Please show them your appreciation by leaving a comment, 'liking' this post, or following this Member.
More posts from this member

Get Published - Build a Following

The Energy Central Power Industry Network is based on one core idea - power industry professionals helping each other and advancing the industry by sharing and learning from each other.

If you have an experience or insight to share or have learned something from a conference or seminar, your peers and colleagues on Energy Central want to hear about it. It's also easy to share a link to an article you've liked or an industry resource that you think would be helpful.

                 Learn more about posting on Energy Central »