Part of Grid Network »

The Grid Professionals Group covers electric current from its transmission step down to each customer's home. 


Analytics and the Future of Demand Management: Realizing Grid Potential

image credit: © Lovelyday12 |
Rachel Slover's picture
Cleantech Corner Contributor , UNC IE

Rachel Slover is a senior at UNC Chapel Hill majoring in Mathematics with minors in Poetry Writing and Environmental Science & Studies. She sees cleantech as an exciting intersection of...

  • Member since 2020
  • 1 items added with 3,975 views
  • Dec 2, 2020

This item is part of the Data Analytics & Intelligence - Winter 2020/21 SPECIAL ISSUE, click here for more

Analytic barriers to the smart grid

The energy and utilities sector is on the brink of myriad technological advancements essential to our country’s social, economic, and environmental future. Over the past decade, we’ve hung in the threshold of an exponential increase in grid capability, and now utilities are bringing a clean-energy smart-grid closer to reality with the widespread addition of renewables and storage technology.

However, the industry risks derailing this progress due to an inability to balance energy supply with burgeoning demand. If we fail to curb the current demand trajectory, it will rapidly overtake supply. This demand will only increase as the age of electrification continues to proliferate in new areas of our economy. Even worse, our current grid is already straining to keep up, with power fluctuations due to blackouts alone causing our economy to hemorrhage “upwards of $150 billion annually” [1]. In order to handle this surge in demand, utilities need to foster a robust grid capable of superior demand management prediction through advanced analytics.

Your access to Member Features is limited.

Historically, utilities have functioned in a “data-rich, information-poor environment”, and this data-richness is increasingly permeating due to the ‘data tsunami’ flooding in from smart meters and other novel grid technology [2]. With a single smart meter able to produce the same amount of data in 25 days as the combined data from the past 100 years, the problem is clear – the landscape is changing, and modeling and analysis practices must change along with it [3]. Without revitalization, forecasting models will remain unable to utilize the sheer amount of information, leaving this potential for heightened precision unrealized. 

Looking to demand response

A major opportunity for grid improvement arises upon examining demand trends. Currently, the most significant portion of electricity costs results from a very small portion of hours each year [4]. While these demand peaks have traditionally led utilities to turn on emissions-intensive “peaker plants” in an effort to increase supply, a focus on lowering demand – a tactic called demand response (DR) – provides greater potential for smart-grid proliferation and grid stability.

DR programs not only help stabilize the grid but also carry environmental and economic benefits. By reducing the use of peaker plants, the significant emissions attributed to their use decrease. Further, DR not only offsets the operation of these power plants but also prevents new peaker plants from being built, avoiding new emissions that would have resulted from both use and construction [5]. Aside from their negative environmental impact, peaker plants are also prohibitively expensive to run – upwards of 10% of total energy costs result from these peak demands that occur less than 1% of the time [5]. This increases costs for all customers, as retail rates are not reflective of real-time price, and as such, decreasing the need for costly peaker plants will reduce prices across the system [5].

Though utilities have employed rudimentary DR since the 1970s, it lacked the analytic capability endemic to today's DR and was therefore dampened by slow response times and sparse grid penetration limited to infrequent use by large industrial customers [5]. Yet as technology has improved, widespread DR has begun to play an increasingly important role. Unlike early attempts, modern DR offers both commercial/industrial and residential consumers an opportunity to play an active role in grid operations by adjusting their load to meet grid-wide energy needs. Even though these large commercial/industrial consumers can have a greater effect, widespread residential implementation will still prove essential to proper grid enhancement.

However, current programs have not yet attained the caliber required to move the grid forward. Many DR programs require manual consumer participation, wherein consumers are asked via phone call, email, app alert, etc. to lower their usage during peak events in exchange for critical peak rebates, avoidance of price surges, and other financial incentives [6]. As a result, even today with our advanced capabilities, Smart Electric Power Alliance (SEPA) reports that most demand response is rarely activated, with an average of only 7-9 DR residential-space activations per year [7].

While this approach has helped to incite the DR transition, customer engagement has hit a wall. According to Southern California Edison spokesperson Robert Laffoon Villegas, as more utilities implement frequent DR events, fewer and fewer customers participate [8]. DR cannot realize its potential if customers are asked to bear the brunt of the active decision making – the norm of DR must shift to automation.

Data for demand management

To shift to data-centric DR, utilities need to make numerous changes. As previously mentioned, the ‘data tsunami’ means that the problem isn’t insufficient access to data – it’s improper utilization and analysis. According to Tim Fairchild, Director of Global Energy and Manufacturing Industry Practices at SAS, “all throughout the grid there is so much data coming out that is available to analyze – more than they’ve ever had before…[to quote] our CTO: ‘data without analytics is value not yet realized’” [9].

Historical data usage has missed this value, emphasizing statistical reporting based on a limited subset of available data that often proves inadequate to understand the intricacies of customer behavior. At the time this was sufficient, but this fair performance was mainly due to the lack of volatility in the data or the variables affecting customer demand, such as metrics based on the past 20-25 years of mostly stable economic times [3]. Now, old forecasting models often underperform, and as grid response is only as strong as the analytics backing it, those models need modernizing. 

Fairchild points to a shift approximately 10 years ago when it became clear that current techniques would not suffice: “You started seeing a lot more solar on rooftops and all kinds of things changing in the home needed to do a lot more forecasts a lot more often.” This shift spurred data scientists at SAS to develop a targeted program to forecast load, unique in its ability to consider all time horizons. “Whether you’re trying to do next hour, next day, next year, 5 years, 30 years…[it’s] all in the same package.” What further sets the program apart is its array of forecasting methods; SAS’s forecaster tool combines time series, regression, and neural networks to let the program itself decide which offers the best estimate. This provides an easy-to-understand output that allows forecasters to fully comprehend their results, which Fairchild emphasizes as essential. Any forecasting program’s output must be transparent, defensible, and repeatable, for forecasting models and human oversight will continue to intersect [9].

Across the industry, companies are realizing the importance of shifting to real-time data in grid operations. Betty Watson, Senior Director of Policy and Market Design at Modern Energy, a holding company focused on clean energy, agrees with the shift away from models, arguing that the energy sector must, “stop using models as much and use real-time data more.” She doesn’t discount the merit of models thus far but reminds people that the very concept of models is to approximate, and “all models are wrong, but some are useful” [10]. As even our most accurate model is still an approximation, we need to ensure we are basing that approximation on the most robust data available – data that considers all categories that influence hour-by-hour and day-by-day usage. This often includes load, weather, economic scenarios, growth, time of week (weekend versus weekday), and time of year (season, holiday, big events) [9].

We have the capacity to make this adjustment, we just must understand, as suggested by David Pope, Senior Manager of Pre-Sales of U.S. Energy at SAS, that data can no longer be limited to hindsight – “Instead, analytics is foresight” [11]. For the first time since the inception of utility data collection, we are capable of this foresight. In this, we must realize that humans are often incapable of rapid processing required by utilities, and so we must increasingly look to software and hardware to help solve our demand problems. Take as one example, a micro-shift in demand (where emissions drop to zero for fleeting moments to then increase abruptly). Though traditional techniques are unable to adjust for these conditions while preserving grid stability, continuous and widespread DR analytics could shift such loads around as needed without consumers losing their baseline comfort level [7]. The smart-grid both facilitates and requires this kind of widespread and self-sustaining communication.

Artificial intelligence

Artificial intelligence (AI) is essentially programmed, simulated human thought that employs insight from fields such as computer science, neuroscience, economics, information theory, statistics, psychology, control theory, and optimization [5]. In recent years, AI has begun to prove increasingly essential to comprehensive and timely decision-making, leaving some to claim that if data is the new oil, “AI is the new refinery,” vital for any data-rich industry to function at full potential [1].

As DR encompasses highly complex tasks and frequently needs real-time decision making that surpasses human capability, AI is poised to be the key to widespread DR [12]. In fact, existing AI forecasting already shows promising returns in the energy sector. Compared to classic models, automated AI forecasting prevents more errors while providing 20-30% greater precision – precision which may be attributed to an ability to constantly self-correct its forecasting model [13]. Applications across the grid include load forecasting, price forecasting, scheduling and control (both aggregator and consumer), determining the most effective pricing/incentive mechanisms, and customer segmentation [5].

Among the most commonly used AI subgroups are machine learning and its subset, neural networks. Machine learning defines AI technology wherein the program learns from and corrects through provided datasets, and is useful for identifying patterns that it then uses to make predictions in uncertain environments [14].  Neural networks or artificial neural networks (ANNs) are a subset of machine learning made up of interconnected “neurons” that relay information between each other to process information, going through multiple data passes in order to gain deep understanding [14].

Shen et al. (2012) performed a literature review of AI use in DR, listing the common methods employed as supervised machine learning, unsupervised machine learning, reinforcement learning, evolutionary algorithms, swarm AI, ANNs, deep learning (ANNs with deeper processing layers), coalitional game theory, mechanism design, and automated negotiation. As evidenced, there is no best-practice branch of AI to employ across the board – the best-equipped method depends on the prediction to be made. However, the literature most frequently listed neural networks, which were also the only method employed across all facets of forecasting, indicating it may be the most useful when one method does not clearly prove superior [5]. An in-depth explanation of these techniques is beyond the scope of this paper, but Shen et al. (2012) provide a thorough description.

One company creating AI solutions for the future of the grid is Veritone, an AI tech company based in Costa Mesa, California. According to Veritone Chief Energy Scientist Dr. Wolf Kohn, Veritone’s technology predicts the future using "current data from multiple sources – weather sensors, load demand systems, and distributed grid devices – to predict optimal energy dispatch, allowing utilities to accurately and dynamically meet future grid demand, whether minutes, hours or days ahead” [15]. Through their aiWARE™ operating system (OS) and their Forecaster, Optimizer, and Controller programs, Veritone is bringing the clean-grid future towards the present.

According to Sean McEvoy, SVP of Business Strategy at Veritone, their AI is able to “...develop a forecast model for each residence in the community, which includes the individual home load by the minute, hour, and day and incorporating weather forecasting to predict the solar generation capacity”. In this way, Veritone can then “synchronize all homes in the community to aggregate the available power and transfer it from one home to another based on peak demand needs. In essence, the community becomes a virtual power plant” – an unprecedented development in grid stability potential [16].

Creative demand management solutions to an unnoticed problem

Some problems plaguing the grid are extensive yet not readily apparent, as issues that have persisted for years are often considered the norm. Such was the case for the problem tackled by Atom Power, creators of the Atom Switch – the world’s first solid-state digital circuit breaker. Manual circuit breakers essentially hadn’t been updated since their creation, providing an opportunity for Atom Power to significantly improve grid reliability, safety, and demand management.

Traditional circuit breakers physically break the connection in cases of dangerous power surges, but opening speed is often too slow, causing arcs that can lead to dangerous fires. In contrast, the Atom Switch switches power in about 80 microseconds – orders of magnitude faster than manual breakers [17]. Not only does this have obvious implications for safety but it is also essential for managing demand. Ryan Kennedy, CEO of Atom Power, explains that “to do demand management, if you can create a solid-state circuit breaker, then there are no arcs, everything is digitally controlled…it's not even like making an improvement – it’s night and day – and all of a sudden now, you have demand management across the whole portfolio.” When asked how the Atom Switch helps with demand management, he pointed out that solving demand management is “inherent in the nature of the product – as a semiconductor-based circuit breaker, it's built in by default,” since the visibility inherent in digital control enables much swifter and smarter load shedding.

Though up until now, the Atom Switch had been limited to the commercial/industrial sector, Kennedy says that by 2021, they will expand to the residential sector, and due to metering built into the product, it can easily replace smart-meters if needed (though widespread replacement of smart meters was never Kennedy’s goal) [17]. Such grid technology also decreases cost and materials replacement, as the Atom Switch has programmable amperage from 15 to 100 that can be easily adjusted [17]. It also allows for better implementation of renewables, as digital breakers make it nearly effortless to transfer between utility and home solar (or other renewable power) without loss of service; as Kennedy puts it: “using a solid-state absolutely key to distributed energy”. 

Implications for the analytics shift

By 2030, the Edison Electric Institute predicts upwards of 18.7 million electric vehicles (EVs) will be on the roads, a transition for which we currently lack infrastructure [18]. A change of this magnitude requires a massive grid overhaul – one that analytics can provide. Without significant improvements in efficiency and demand management, there is no way our grid can handle the surge in demand resulting from such an influx of charging needs. Yet this plan can be realistic with the grid-strengthening abilities of AI-controlled demand management and charging technology such as the 2021 Atom Switch update to function as an EV charger (making charging much cheaper and more reliable) [17].

Further bolstered by analytical support, the distributed grid layout necessitated by the shift to renewable energy becomes possible. Unlike the past centralized model relying on a few large central producers, the distributed grid is characterized by a wide variety and number of energy sources. With renewable energy less reliable than old power plants, the grid needs plenty of sources to draw from and a sophisticated ability to shift load. This will require unprecedented speed that can only be accomplished through a smart-grid equipped with advanced analytic abilities [14].

Kennedy describes how the grid currently makes it so that “when you lose utility, you simultaneously lose your solar [or other residential renewable].” This is due to islanding, a phenomenon in which the renewable can feed into the utility when it’s down – “If somebody is working on the line downstream they could be’s a pretty serious flaw in the system”. The Atom Switch is able to remedy this flaw by allowing for harmless and easy shifting between sources, creating a greater incentive for homeowners to invest in their own residential renewable energy. Further, the traditional centralized model feeds electricity in a unidirectional flow of power, whereas intricate DR models require bidirectional flow for optimized communication [5].

Bidirectional flow is also critical to harnessing all clean energy sources available, such as residential renewables, yet according to Betty Watson, we are leaving a lot of possible energy left unutilized. “We’re already seeing a lot of distributed solar and a lot of distributed storage go onto people’s houses, that is...a huge asset sitting on the system,” she states – but this is a huge asset that we just aren’t using. This is why Modern Energy is a firm proponent of a distributed grid; balancing renewables in such a way “activates the demand side of the supply and demand equation…that provides us with a lot more resources...[and] there are a lot of places where that’ll be a lot more cost-effective than continuing to only focus on large, centralized energy…Moving to a distributed grid gets us the balance that we need but in a much more cost-effective and equitable way”.

Security and the digital grid

Whenever something is made digital, one must consider security risks. Many are wary of the security threats made possible by AI, though it must be noted that AI itself can often bring security strength. While digital infrastructure without AI is at heightened risk, AI can protect the grid against cyberattacks, steering electricity around compromised parts of the grid until vulnerabilities can be remedied [1].

Companies involved in the digital shift are already meticulously planning for these security challenges. Ryan Kennedy states that security is a top priority at Atom Power, mentioning how anyone working on something digital needs to ensure it can’t be made to do something it was not intended to do. “The least risk is someone just ‘monkeying’ around with it and turning things on and off, the biggest risk is someone adjusting the firmware changing breakers to 2000 amp breakers that never trip and cause fires.” They addressed the risk head-on with “both hardware and software measures [so the switch] legitimately couldn’t do that.”

“The other concern is someone hacking in and doing things – it’s been an evolution and will continue to be, and it is for everyone, not just for us. We’ve done things like putting in hardware that make it as difficult to hack into our system as it is to hack the most robust systems out there today.” Atom Power is even trying to solve the “biggest security risk” – the user, by requiring password changes and other protective measures to avoid simple hacking [17]. Yet Kennedy emphasizes that security is never truly solved, as new threats constantly arise. While AI and a digital grid can be confidently secure, it will only remain as such with dedicated technicians ensuring that security measures continuously evolve.

What does this mean for the future of the digital grid?

“I think it’s a directionally correct type of question – yes we’re moving towards [the digital grid], want to be a little careful of how you make that transition,” says Mike Smith, Industry Principal Marketing Manager at SAS, and he is right to be cautious of overzealous attempts to change the grid overnight. For the foreseeable future, he believes that there will still be “a large electromechanical or analog component to how the grid is managed”. In terms of the grid becoming fully-digital, he says “we’re moving towards that,” but we can’t pretend that we’ve nearly arrived.

Watson echoes this sentiment of measured improvement and suggests a “crawl, walk, run” approach to grid enhancement – we achieve nothing by naively assuming we can instantaneously create a new system. Evident of this is the relative newness of AI technology, as according to Zpryme Director of Research Chris Moyer, "It is not yet clear AI will be mature enough in the next five years to ten years" to completely direct the grid [19]. Yet as we work to make AI integration possible, the grid will incrementally become more reliable and resilient. Despite the need for a gradual approach, AI heavyweight Veritone still firmly believes the grid will reach full digitization, suggesting that “traditional grids will be broken down into multiple microgrids that can function autonomously or in unison (AI synchronized)” [16]. The gradual path can still lead to the digital grid, we simply must always ponder the possibility of a higher caliber of analytics so as to never become stuck seeing problems to be solved as intrinsic and unchangeable.

Though the days of one-size-fits-all are gone for utility analysts, our grid problems share a common path from which specific solutions will branch off – intentionality in analysis. Further, with emphasis on the limits of human capability that advanced analytics can solve, we must remember the immeasurable human-impact on these breakthroughs. To end, look to Tim Fairchild’s refrain about the essential backing for demand management: “The game-changer is us, it’s people”. Demand response, for all its reliance on capable technologies, relies at its core on our implementation of solutions. 


[1] Steelberg, Chad. “Why AI Is Our Best Bet To Save Green Energy In The US.” International Business Times, October 10, 2020.

[2] Farrell, Alyssa. “Innovate and Optimize: The Power of Analytics in Today’s Utility”. SAS, 2011.

[3] “How Does Forecasting Enhance Smart Grid Benefits?” SAS, 2016.

[4] Fields, Spencer. “Demand Response Programs Explained.” Solar News. EnergySage, April 19, 2019.

[5] Shen, Bo, Ghatikar, Girish, Chun Ni, Chun, and Dudley, Junqiao. “Addressing Energy Demand through Demand Response: International Experiences and Practices”. Environmental Energy Technologies Division, Ernest Orlando Lawrence Berkeley National Laboratory. 2012.

[6] U.S. Department of Energy Office of Electricity. “Demand Response.” Accessed October 10, 2020.

[7] McCormick, Gavin, and Peter Bronski. “Demand Response Is Evolving. Again.” SEPA, June 20, 2019.

[8] Trabish, Herman. “The New Demand Response and the Future of the Power Sector.” Utility Dive, December 11, 2017.

[9] Fairchild, Tim (Director of Global Energy and Manufacturing Industry Practices, SAS) and Smith, Mike (Industry Principal Marketing Manager, SAS). Video interview by author. October 13, 2020. 

[10] Waston, Betty,  Senior Director of Policy and Market Design, Modern Energy. Telephone interview by author. October 21, 2020. 

[11] Geschickter, Chet and Pope, David. “Managing Data to Maximize Smart Grid Benefits: Insights from a webinar hosted by Electric Light & Power.” SAS, 2012.

[12] Antonopoulos, Ioannis, Valentin Robu, Benoit Couraud, Desen Kirli, Sonam Norbu, Aristides Kiprakis, David Flynn, Sergio Elizondo-Gonzalez, and Steve Wattam. “Artificial Intelligence and Machine Learning Approaches to Energy Demand-Side Response: A Systematic Review.” Renewable and Sustainable Energy Reviews 130 (September 2020).

[13] Cognet, Olivier. “Why Utilities Are Using AI Digital Solutions for Price Forecasting.” Schneider Electric Blog, June 25, 2020.

[14] “The Autonomous Grid in the Age of the Artificial Intelligence of Things.” SAS, 2019.

[15] “Predictive AI Tools Aim to Help Utilities Make Transition to Green Energy.” Smart Cities World, October 7, 2020.

[16] McEvoy, Sean, SVP Business Strategy, Veritone. Email correspondence with author. October 30, 2020.

[17] Kennedy, Ryan, CEO, Atom Power. Video interview by author. October 27, 2020.

[18] “Electric Transportation Benefits Customers and Communities.” Edison Electric Institute, 2020.

[19] Trabish, Herman K. “The Biggest Numbers Game in the Power Sector: Data Analytics and the Utility Community of the Future.” Utility Dive, March 25, 2019.

Matt Chester's picture
Matt Chester on Dec 2, 2020

While this approach has helped to incite the DR transition, customer engagement has hit a wall. According to Southern California Edison spokesperson Robert Laffoon Villegas, as more utilities implement frequent DR events, fewer and fewer customers participate [8]. DR cannot realize its potential if customers are asked to bear the brunt of the active decision making – the norm of DR must shift to automation.

Interesting point, Rachel-- do you think the failure to move to the 'next step' is from lack of awareness on the part of the utilities or lack of buy in from the customer? Are the utilities using the customer-driven DR as a proving ground first before running into the next step? 

Rachel Slover's picture
Thank Rachel for the Post!
Energy Central contributors share their experience and insights for the benefit of other Members (like you). Please show them your appreciation by leaving a comment, 'liking' this post, or following this Member.
More posts from this member

Get Published - Build a Following

The Energy Central Power Industry Network is based on one core idea - power industry professionals helping each other and advancing the industry by sharing and learning from each other.

If you have an experience or insight to share or have learned something from a conference or seminar, your peers and colleagues on Energy Central want to hear about it. It's also easy to share a link to an article you've liked or an industry resource that you think would be helpful.

                 Learn more about posting on Energy Central »