Senior decision-makers come together to connect around strategies and business trends affecting utilities.


California Energy Crisis 2.0: it’s time to use our smart meters

image credit: This work is a derivative of "Berkeley and Berkeley campus-004" by Christian Haugen, used and licensed under CC BY 2.0 by Samuel Golding. Source image:
Samuel GOLDING's picture
President, Community Choice Partners, Inc.

Political economist and executive management consultant focused on market reform, community partnerships and competitive power agency realignments: governance structures / oversight / stakeholder...

  • Member since 2019
  • 6 items added with 3,591 views
  • Nov 9, 2020

Note: this op-ed was originally published in California Current on November 2, 2020. 

The rolling blackouts, according to thePreliminary Report on Causes of August Rotating Outages, were fundamentally caused by a California Independent System Operator software glitch and community choice agencies, utilities and direct access providers collectively under-scheduling customer demand in the Day-Ahead Market by ~7% to 8% on August 14th and 15th.

Ordinarily the under-scheduling wouldn’t have been a problem, because how much demand Load Serving Entities choose to schedule in the Day Ahead Market should never impact the stability of the market — by design. That’s because the CAISO Day Ahead Market is a two-step process: the Integrated Forward Market is a financial trading market into which Load Serving Entities submit customer demand schedules, after which the Residual Unit Commitment process is run using CAISO’s own internal load forecast to schedule generation resources sufficient to maintain physical stability.

On August 14th and 15th, CAISO’s own internal load forecast was fairly spot-on and should have been used to schedule resources sufficient to maintain system reliability. Because of its software glitch, however, the grid operator allowed exports to be scheduled in the Day Ahead Market on the basis of the lower demand forecasts submitted by Load Serving Entities. Once they realized the mistake, it was too late to curtail exports at short notice in the Real-Time Market, not the least because doing so could have jeopardized the reliability of neighboring Balancing Areas. They count on receiving the exports to maintain their own supply-demand balance.

In short, we sold off power that we knew would be needed, and consequently had to shed load in the Real-Time Market to maintain the supply-demand balance.

CAISO has identified and fixed the glitch. But everybody seems to agree that it’s worth understanding why Load Serving Entities did not submit more accurate load forecasts in the Day Ahead Market. While the impacts of extreme weather events on how customers use electricity are inherently difficult to forecast, that alone doesn’t explain the magnitude of the load forecast shortfall.

A key root cause may be that utility smart meters have yet to be fully utilized to provide what should be their most basic function: the timely provision of customer interval load data to Load Serving Entities for use in market operations.

A Straight-Forward Market Flaw!

As far as market design flaws go, this one is refreshingly straight-forward. Ratepayers paid tens of billions of dollars over the last decade to install smart meters and the Advanced Metering Infrastructure that collects and analyzes data. Installing meters and then not actually using the data to forecast short-term load is like buying a computer and then using it as a doorstop:


Image credit:

Almost every retail customer in California has a smart meter that records how much electricity is being used on an hourly or 15-minute basis. This granular data is extremely valuable in terms of enhancing the accuracy of short-term load forecasts. These meters also provide more complex, advanced functionality.

To participate in the CAISO Day-Ahead Market, every Load Serving Entity has to submit a day-ahead forecast of their retail customers’ hourly demand by 10 A.M. However, not all Load Serving Entities have equal access to their customers’ most recent smart meter data:

  • The Investor Owned Utilities collect and use smart meter data to update their forecasts several times a day;
  • Community choice agencies serve the mass market of customers who left utility service and are, in contrast, entirely dependent upon utilities to provide them with data from their customers’ meters.
  • Electric Service Providers mostly serve large commercial and industrial customers on Direct Access and have the statutory right to put in their own retail meters and collect their own meter data if the utilities aren’t doing a good job of it.

The Problem: Delayed Access to Smart Meter Data

Therein lies the problem: Pacific Gas & Electric, Southern California Edison and San Diego Gas & Electric do not currently provide community choice agencies with sufficient access to smart meter data to use in CAISO day-ahead forecasting.

Apart from undermining the efficiency of market operations and jeopardizing system stability, this is also one of several inter-related barriers to innovative rate design and decarbonization for community choice agencies. For further details, refer to page 6 onwards of this regulatory filing I wrote earlier in the year along with these comments by Silicon Valley Clean Energy.

This problem has become very large indeed, because community choice agencies are responsible for scheduling more load than Investor Owned Utilities in the CAISO market from here on out:


Image credit: Samuel Golding

Common infrastructure should be used to support the public good. The data generated by the network should not be withheld from different types of Load Serving Entities on a discriminatory basis. It should be fully utilized to support the efficient, clean and reliable operation of the electrical grid.

That’s why, by law, community choice agencies are supposed to receive smart meter data at the same time that the Investor Owned Utilities do:

  • SB 790 and Public Utilities Code 366.2(c)(9) and (10) explicitly requires the IOUs to provide community choice agencies “with appropriate billing and electrical load data, including, but not limited to, electrical consumption data as defined in Section 8380”;
  • Section 8380 defines “electrical consumption data” as including the “incremental and monthly meter-specific electricity data” that is “made available as part of an advanced metering infrastructure”, and further requires that access to this data be “to the extent produced by that infrastructure”.

To translate the requirements of Section 8380 into operational rules, the key question is this: what data is being “produced” by the utilities’ “advanced metering infrastructure” on an “incremental” basis?

The Solution: Equal Access to Smart Meter Data

Ask a simple question, get a simple answer: the system has a communications mesh network that is constantly downloading and validating the 15-minute or hourly usage data from different batches of smart meters, such that all customers' smart meters end up getting read each day. That’s the data that Pacific Gas & Electric, Southern California Edison and San Diego Gas & Electric access and put to good use in CAISO market operations as soon as it is collected by the mesh network and validated. Community choice agencies should be allowed to immediately access that same data as well — so that their forecasting and CAISO market demand bids make use of the most recent smart meter data. 

Providing community choice agencies with the data that utilities are already collecting and using for their own load forecasting on a routine basis is not challenging from a technical perspective. The straight-forward solution would be to establish a common Application Programming Interface (API) that allows Load Serving Entities to access their retail customers’ most recent smart meter data, or a Statewide Data Platform.

That’s what Texas did years ago in establishing the Smart Meter Texas platform. Regulators in Europe and Australia have gone even further, by deploying more advanced “Local Flexibility Market” platforms that not only exchange customer data, but additionally allow aggregators to optimize demand flexibility and the intelligent dispatch of Distributed Energy Resources — integrated into wholesale market operations — across multiple utility territories and even multiple countries.

California’s Advanced Metering Infrastructure has a lot of under-utilized functionality that would be unlocked under a statewide “Local Flexibility Market” platform approach. In that regard, Section 8380 also gives the Commission blanket authority to order the Investor Owned Utilities to disclose smart meter data and separately allows the utilities to do so for “system, grid, or operational needs”. Public Utilities Code 366.2(c)(9) also allows the Commission to order the utilities to additionally disclose “other data detailing electricity needs and patterns of usage… in accordance with procedures established by the commission” to community choice agencies. 

In other words, the Legislature has given the Commission all the authority it needs to ensure that we are collectively putting our smart meter and system data to good use in market operations.

CAISO fixed their glitch; it’s up to the Commission to fix this one.

Matt Chester's picture
Matt Chester on Nov 9, 2020

Almost every retail customer in California has a smart meter that records how much electricity is being used on an hourly or 15-minute basis. This granular data is extremely valuable in terms of enhancing the accuracy of short-term load forecasts. These meters also provide more complex, advanced functionality.

If you had to guess, what portion of customers are benefitting at all today from this data being collected? Are they being given access to it if they want it to provide to a third party? Or is this data really just gathering dust (figuratively speaking)? 

Samuel GOLDING's picture
Samuel GOLDING on Nov 9, 2020

PG&E, SCE and SDG&E each implemented Green Button (Download My Data and Connect My Data) in response to a 2014 CPUC order (D. 14-05-016) to ostensibly allow customers to download their data and to authorize third-parties for data sharing purposes. According to the annual Smart Grid Report to the Legislature, they collectively processed ~ 70k individual authorizations in 2019.

However, the utility-specific implementation of the GBC standard, the customer authorization process, the extent of data made available, and the third-party onboarding / vetting process is all far less standard than one might expect. Taken as a whole, it's a severe fragmentation of the retail market that drives up transaction costs and creates barriers to innovation.

That's why third parties have been strenuously advocating for standardizing / expanding access in Application 18-11-015 (the "click through" docket). That's also why UtilityAPI and Silicon Valley Clean Energy (a Community Choice agency) recently launched a data sharing platform called the "Data Hive", which wouldn't be necessary if the utilities were providing sufficient functionality to support the retail market. Here are a couple links:

All of that aside, the point of my article is that the most recent customer usage data isn't being made available to Community Choice agencies quickly enough to use in CAISO day-ahead market demand forecasting and scheduling. 

Samuel GOLDING's picture
Thank Samuel for the Post!
Energy Central contributors share their experience and insights for the benefit of other Members (like you). Please show them your appreciation by leaving a comment, 'liking' this post, or following this Member.
More posts from this member

Get Published - Build a Following

The Energy Central Power Industry Network is based on one core idea - power industry professionals helping each other and advancing the industry by sharing and learning from each other.

If you have an experience or insight to share or have learned something from a conference or seminar, your peers and colleagues on Energy Central want to hear about it. It's also easy to share a link to an article you've liked or an industry resource that you think would be helpful.

                 Learn more about posting on Energy Central »