The mission of this group is to bring together utility professionals in the power industry who are in the thick of the digital utility transformation. 


Energy Data in 2020

image credit: © Photopal604 |
Douglas Fant's picture
Counsel SunZia Transmission LLC


  • Member since 2020
  • 2 items added with 1,461 views
  • Jan 31, 2020 12:00 pm GMT

This item is part of the Special Issue - 2020-01 - Predictions & Trends, click here for more

1. Introduction- The Smart Grid and Big Data.

The key word in electric power systems in 2020 is “disruption”. The energy sector generally has remained insulated from the impact of technology while technology has been fundamentally altering the landscape in many other industries. But for the energy sector, that is true no more. If you want to stay competitive and relevant in energy, then realize that information and communication technologies are now impacting power systems.

In the data arena, business goals dictate any entity’s data needs and selection of technology which will help achieve those goals. Entities need to be responsive to customer demands for more flexibility, reliability, better cost control, and innovative service models, collectively referred to as the “Smart Grid.” “Energy-as-a-service” business models may also facilitate customers’ energy efficiency, as well as reduce costs, and stoke additional best practices in energy management for them.

Your access to Member Features is limited.

Remember too, that carbon emission reduction and sustainability are driving forces and purposes behind the Smart Grid. The Smart Grid’s decentralized structure will employ distributed generation units in the distribution system as a means for the utilization of widespread renewable energy.

Wind and solar energies will likely be dominant energy sources for the next generation power grid. However, their intermittent characteristics impede their utilization in a stable way. To deal with such intermittency and to improve dispatch planning, maintenance scheduling and regulation, accurate and reliable resource forecasting has become a key issue too. Organized systemic data will be required to meld these activities into distribution system operations.

The Smart Grid will provide numerous opportunities for application of data analytics. Data mining, for example, could help create new feedback loops, enhance planning, and offer accurate system status reports. Smart Grid innovation should drive useful integration of distribution system data sets across legacy applications, build useful new analytical frameworks, and encourage automated processes to achieve such outcomes.

Interpreting and cataloguing data mined from the Smart Grid will drive these changes. That goal however ll remains easier stated than accomplished.  

2. Data Volumes Will Continue to Rise.

The volume of data being generated in the power sector has grown tremendously over the past few years due to deployment of Smart Grid technologies. These technologies can meter individual customers and individual grid equipment throughout the distribution system including transformers, switches, capacitor banks, voltage regulators and other equipment. This information may then be relayed back to the entity via communications systems. 

The volume of data generated by these technologies is often referred to as Big Data. “Big Data” includes data sets too large to store and analyze using traditional database technology. What will likely be needed is distributed systems to store data in different locations, connect them by networks, and then bring them together by software. To give one example of the amounts of data generated in distribution systems, the Advanced Metering Infrastructure in the State of New York with seconds- data resolution produces roughly 127.1 terabytes of electric consumption data per day.

Similarly on a macro scale, global internet traffic has tripled since 2015 and is expected to further double by 2022 to 4.2 zettabytes per year (4.2 trillion gigabytes) (Cisco 2019). The number of mobile internet users is expected to increase from 3.6 billion in 2018 to 5 billion by 2025, while the number of Internet of Things (IoT) connections is expected to triple from 7.5 billion in 2018 to over 25 billion by 2025 (GSM Association 2019).

Solutions for storing, handling, and assembling such system data is an open issue needing some sort of resolution for the expanded application of Big Data analytics to the Smart Grid.

3. Not there yet- Lacking Tools as Well as Data Standardization.

Data analytics seeks to extract valuable information from historical system data for the purpose of guiding current real-time operation and maintenance activities. The true significance of data analytics will be achieved once we understand, adapt, and incorporate to a new data management environment. However, that has yet to occur as the data management and computation structures and tools in the electric power sector are yet to go through an evolution in order to adapt to the new concepts and applications in data analytics.

And then there’s the data itself. Energy data now come from variety of power system sources and span a wide range of locations, types and applications. Additionally, many forms of grid data are generated at rapidly and in high volumes.  Some sort of IEC 61724 Working Group- or Green Button Alliance-like effort is needed to make such data available.

Unfortunately many power system sensors, including even many emerging and state-of-the-art sensors, the majority of data is either not logged, or are overwritten very quickly. For example, in most protection relays and related sensors, the data collected is discarded shortly after internal use. Additionally, almost all state-of-the-art power quality sensors tend to store the voltage or current wave forms for only a few cycles before and after an event is detected.

4. We Have Met the Enemy and He is Us.

The electric power sector needs to defeat its own current method of thinking about handling data. Several essential features have to come together to make power system data analytics a viable option. First, the analytics needs to continue to emerge in many sectors. Second, major advances in both hardware and software tools and platforms focused upon power systems must increase. Finally we need to increase affordability of non- enterprise data center data acquisition, communication, and storage.

That is, encourage continued development of data analytics in the form of predictive analytics (domain and off-domain data forecasting), data mining and machine learning (classification, regression, clustering), artificial intelligence (cognitive simulation, expert systems, perception, pattern recognition), statistical analysis, natural language processing, and advanced data visualization. The majority of these tools and techniques are exploratory in nature and do not require us to pre-determine what one expects to look for or see in the data.

Data needs to be viewed as an important asset, which will require questioning and redefining  traditional viewpoints and practices towards that data.

4. Access to Data Still a Challenge too.

Access to data remains a challenge. The majority of measured data in power systems are intended to be used close to the point which they are generated, or are trapped in silos, and are not intended to be carried to and stored in an enterprise data center, as opposed to the data analytics mentality in the information and technology sector.

Data science practitioners assert that 80% of the time is involved in acquiring and preparing data. The data silos and isolated stockpiles of power system data which are restricted from any non-intended use or user, loom as barriers to data analytics development. Data availability across entities aside, different divisions within entities themselves such as distribution system operators, or transmission operators often cannot access to data from each others’ division. Some of the drivers are market-driven, some could be related to structural shortcomings. with limited resources, applications and databases have been optimized for their main function and specific teams, and incentives are not sufficient to encourage data sharing. In addition the sensitivity over sharing power system operational data may also be traced to cyber security concerns.

5. Conclusion.

The link between data analytics and the Smart Grid is increasing. Yet deployed software applications for exploiting distribution system data sets still remain few in number. Many distribution system infrastructure-types of open issues also need to be addressed before software engineers can create the availability of data in reality. Finally data analytics and the Smart Grid are a complicated combination, and require a dose of mathematics, information and telecommunication technologies, computer science, and electrical engineering. Coordinating these elements and designs will accelerate utilization of the ever-increasing volumes of data generated by the Smart Grid.

Douglas Fant's picture
Thank Douglas for the Post!
Energy Central contributors share their experience and insights for the benefit of other Members (like you). Please show them your appreciation by leaving a comment, 'liking' this post, or following this Member.
More posts from this member
Spell checking: Press the CTRL or COMMAND key then click on the underlined misspelled word.
Matt Chester's picture
Matt Chester on Jan 31, 2020

I see a lot of utilities collecting massive treasure troves of data, but then not making full use of them-- it seems like you agree with that. Is this a problem with leaders at top thinking they've already achieved digitization and not caring about the meat on the bone they're missing? Or are there technological holdups? 

Douglas Fant's picture
Douglas Fant on Jan 31, 2020


Good questions. I believe the issue to be primarily one of characterization- how does one view system data, its purpose and use? System data should actually be seen as a new form of 21st century infrastructure- just like wires, switches, and capacitors, etc. Ignore your system data and you are actually underbuilding your power system.  That is, data is infrastructure. In addition, as you note, more data standardization (format) and data availability need to occur, in order to garner the software engineers' interest and time. Sincerely, Doug F.

Chris Law's picture
Chris Law on Apr 4, 2020

Thanks, great share.

I frame what you have outlined into the "Data Barrier". The most important point is you are not really using your data until is being used within an operational process - this is actually where you will find the biggest "hold-ups will ultimately lie" assuming of course you have converted the data into something useful.

Get Published - Build a Following

The Energy Central Power Industry Network is based on one core idea - power industry professionals helping each other and advancing the industry by sharing and learning from each other.

If you have an experience or insight to share or have learned something from a conference or seminar, your peers and colleagues on Energy Central want to hear about it. It's also easy to share a link to an article you've liked or an industry resource that you think would be helpful.

                 Learn more about posting on Energy Central »