The mission of this group is to bring together utility professionals in the power industry who are in the thick of the digital utility transformation. 


ARPA-E funded platform NI4AI is democratizing grid data

image credit: Title: NI4AI Project Pillars, Author: PingThings Inc., Source:
Laurel Dunn's picture
Power Systems and Data Enthusiast, National Infrastructure for AI on the Grid

My work helps decision makers to get the most out of their data. I build data-driven models for evaluating risk and reliability. I specialize in climate resilience, wildfire mitigation, and...

  • Member since 2020
  • 2 items added with 1,439 views
  • Oct 12, 2020

The National Infrastructure for Artificial Intelligence (AI) on the Grid -- or NI4AI -- is a three-year ARPA-E initiative designed to enable breakthroughs in data analytics for the grid. 

In the past, limited data access has created barriers to advancing the analytical capabilities of the industry. NI4AI is geared at catalyzing the rapid development and deployment of AI tools to improve every aspect of the grid. The project is taking a three pronged approach:

  1. The platform offers a high-performance environment that makes it easier to work with large volumes of time series data, and to collaborate.

  2. The data captures different aspects of grid behavior relevant to developing analytical tools to address problems on the grid. 

  3. The community includes data analysts and practitioners working to find new opportunities for data to solve practical problems for utilities.

The project is led by a tech startup, PingThings, and leverages a platform they developed called PredictiveGridTM. The platform is used in production by utilities and by researchers, and is being available through NI4AI to make it easier for a broader community of data analysts to work with large volumes of data. The University of California, Berkeley is also a collaborator, and is developing educational content and tutorials to demonstrate different techniques analysts can use to get started.

Data analytics can improve every aspect of the grid

Aging infrastructure, climate risks, cyber threats, renewables, and changing loads make situational awareness essential to maintaining a safe and reliable grid. These changes on the grid will lead to situations that operators have not experienced in the past. For example, they may introduce new dynamics and transients that conventional measurement systems like SCADA and AMI simply aren’t fast enough to capture.

Real-time information and high-frequency sensor data can allow grid operators to detect and respond to emerging risks before they become problematic. Innovative solutions using AI have demonstrated that real-time data from AMI, phasor measurement units (PMUs), and new continuous point-on-wave sensors can alert decision-makers to potential issues, allowing them to take more timely and more effective actions.

One barrier to extracting value from sensor data is that the volume is simply too great. Analytical and visualization tools are necessary to synthesize raw data into information about the system that can help decision-makers figure out how to respond. But analyzing terabytes of data poses a computational challenge for most researchers and practitioners. The NI4AI platform was built by computer scientists to make these large amounts of data easily accessible to practitioners so that they can more readily analyze data to answer questions that come up on the job.

Three pillars of NI4AI: Data, Platform, and Community

The Data

The platform provides open access data from dedicated NI4AI sensors, previously collected grid data, and anonymized utility data “scrubbed” to remove sensitive information. These data support the development of new analytical tools to address issues on the grid by connecting analysts with the data they need to detect problems.

Algorithms are only as good as the data that train them, and some use cases may require more training data than a single utility possesses. Data shared via the platform can lead to higher quality solutions that benefit the industry as a whole.

Open access data can also be used to compare different solutions to the same problem. Transparency in evaluating which algorithms work and which don’t can minimize duplicative work across utilities, and allows analysts to build on one another’s ideas rather than starting from scratch. Open access data can allow analysts to more readily discuss data, share code, and compare results. NI4AI will be hosting hackathons and competitions to encourage knowledge exchange and collaboration among data analysts that are using the platform.

The Platform

The data volume from modern sensor networks requires high-performance infrastructure and tooling. Data historians commonly in use today simply weren’t developed with AI and machine learning in mind. Also, most tools don’t address measurement data quality, such as missing or erroneous measurements. NI4AI is developing standardized and automated mechanisms to clean data and provide feedback about data quality in real time.

Central to NI4AI is a novel,  high-performance platform to ingest, store, cleanse, compress, visualize, and process sensor data for AI applications. The project leverages a commercially available platform called PredictiveGridTM, which is used in production by utilities to allow decision-makers to work more effectively and efficiently with big data. The platform capitalizes on best practices established in other industries. Specifically, the platform is purpose-built for high volumes and high-resolution time series data, and is optimized to minimize computing resources to rapidly read and write data. This eliminates many of the computational barriers that have traditionally made it difficult for analysts in the industry to get the most out of time series data on a daily basis.

The Community

Anyone who works in energy appreciates the grid’s complexity. Operating the grid involves making choices that come with inevitable trade-offs. Prioritizing environmental sustainability may come at a cost to reliability. Mitigating wildfire risk may mean causing blackouts. For the industry to use data to inform better, more timely decisions about when to take one action versus another, the industry needs practitioners who really understand the system to become skilled data analysts as well.

NI4AI is facilitating knowledge exchange between grid experts about results that can be achieved using data analytics. The project is hosting data competitions and hackathons focused on addressing specific challenges on the grid. The project is also offering workshops and tutorials to build awareness about new opportunities for using data, and about the solutions that have already been demonstrated. 

By streamlining data transfer, code-sharing, and collaboration across institutions, the platform can also streamline the process of establishing collaborations among a diverse community of experts in power systems, data science, and software development.

Becoming an agent for change in the industry

NI4AI will enable industry professionals to use data more effectively in their jobs. Practitioners can support this transition by using the platform, participating in training sessions, contributing data, hosting a sensor pilot, or sharing insights about analytical tools that would be valuable to them on the job. Readers can learn more by signing up at, or can reach out directly to

Experts agree that digitalization is just beginning to hit the industry. Developing the skills to work with big data will help practitioners become change agents in their organizations. NI4AI is offering professional development workshops on data visualization and analytics geared at training practitioners to use big data on the job. Look for sessions hosted by CIGRE, NASPI, and IEEE SmartGridConn this fall. Go to to learn more.

Matt Chester's picture
Matt Chester on Oct 12, 2020

The community includes data analysts and practitioners working to find new opportunities for data to solve practical problems for utilities.

Can you give some examples of common problems that this would help solve? What opportunities might have utility executives salivating if they knew more about it?

Laurel Dunn's picture
Laurel Dunn on Oct 13, 2020

Great question! There are plenty of applications utilities have found that are worth salivating over. Some examples: locating power quality issues, mitigating wildfire risk, incipient failure prediction, and generator model validation.

But the biggest value doesn't come from any single application -- it comes from empowering decision makers to use data to answer questions that might otherwise leave them stumped. 

If you need to make a decision and aren't sure which option is best, odds are that between AMI, SCADA, PMUs, relays, and other datasets -- your utility has data that could give you confidence that a particular decision is the right one to make. The can lead to fewer judgement calls, stronger rate cases, and can prevent opportunities (or issues) from being overlooked.

Laurel Dunn's picture
Thank Laurel for the Post!
Energy Central contributors share their experience and insights for the benefit of other Members (like you). Please show them your appreciation by leaving a comment, 'liking' this post, or following this Member.
More posts from this member

Get Published - Build a Following

The Energy Central Power Industry Network is based on one core idea - power industry professionals helping each other and advancing the industry by sharing and learning from each other.

If you have an experience or insight to share or have learned something from a conference or seminar, your peers and colleagues on Energy Central want to hear about it. It's also easy to share a link to an article you've liked or an industry resource that you think would be helpful.

                 Learn more about posting on Energy Central »