EM&V: How COVID-19, Business Models, Policy Frameworks, and Technological Change Have Impacted DSM Evaluation Approaches in OntarioPosted to AESP in the Energy Efficiency Group
- Jan 15, 2021 11:43 pm GMTJan 15, 2021 11:42 pm GMT
- 497 views
The last few years have brought about significant disruption for energy efficiency program evaluation in Ontario. At the beginning of this year, there was already much change, due to recently updated DSM frameworks, evolving technologies, company mergers, and shifting political priorities. In the early spring, this situation was further compounded by the COVID-19 pandemic. Despite these stacking challenges, the evaluation community has generally persevered and brought about innovation to the industry. This article - developed from interviews with four industry leaders in Ontario - hopes to dive into some of the specifics around these challenges with the hope of bringing universal insights and recommendations to evaluators throughout North America.
The Pandemic’s Effects on DSM Evaluation
In the middle of March 2020, when the Canadian provinces started their lockdowns, the evaluators had to adapt quickly to the pandemic and the challenges it introduced. At this moment, the Ontarian evaluators were in the middle of the 2019 evaluation cycle. The lockdowns led to an immediate stop to site visits due to business closures and physical distancing measures. This meant a lack of data collection for metric development, such as realization rates. Other aspects of the evaluation approach were less impacted, such as net-to-gross ratio development and process evaluation, as these were largely accomplished through phone interviews and surveys. For participants where phone interviews and surveys were not possible, some evaluators filled in data gaps using historical adjustment factors.
The restrictions on on-site visits have been relaxed over the past month as Ontario moved into a partial economic re-opening. However, there is still uncertainty on how survey participants perceive participation due to COVID-19 stresses - which can cause further delays and bias samples. There is also the added temporal challenge. Many program evaluations have been delayed due to COVID-19. Some of these same programs cannot be extended due to regulatory deadlines. These delays have required the evaluators to move faster than normal.
The challenges vary by sector and size. On the electric side, in the medium and large commercial and industrial sectors, evaluators have had less trouble going back to normal. The residential side was already evaluated mostly remotely through desk reviews. The area of the most substantial concern is the small C&I sector, which has been most disproportionally affected by the crisis as many small businesses closed down temporarily or permanently.
The evaluator’s data gathering approach has changed to adapt to the challenges in site-visits. For some projects, especially in the residential space, evaluators have moved to desk reviews. Also, some are using simulation modeling tools, which can be useful if calibrated correctly. Hybrid solutions are also possible -- one evaluator used a combination of phone interviews and consumption data to estimate realization rates.
Other methodological approaches have evolved in the evaluation space. Another evaluator mentioned a novel approach that his team is using to determine the realization rate while reducing the number of site visits to a minimum. By looking at the last ten years of data, they calculated the highest acceptable ratio of desk reviews to site visits to ensure the actual historical realization rate. This approach may lead to fewer site visits and increased cost-effectiveness post-pandemic. It is uncertain that this approach would have been developed outside of the pandemic, which suggests innovation brought about by these challenging times. The evaluator hopes to present this finding in an upcoming AESP conference.
Shifting DSM frameworks and Company Mergers
After the provincial election of 2018, the Progressive Conservative Party of Ontario formed the government following the previous Ontario Liberal Party government. This changeover led to some sweeping changes in the electricity sector. Before the transition, the Local Distribution Companies (LDCs) were largely responsible for designing and delivering energy efficiency programs. After the Conservatives came to power, they shifted the majority of the responsibility for energy efficiency programs to the Independent Electricity System Operator (IESO). This shift was brought about through the 2019/2020 CDM Interim Framework (IF), which replaced the Conservation First Framework (CFF).
In addition, there is the possibility that the principal savings target metric changes. This has precedent. From 2011 to 2014, the metric was kW demand savings; currently, the principal metric is energy savings. In general, the shifting of framework metrics reflects changes in economic and societal realities but also forces the evaluators’ processes to adapt.
There are some broad lessons that the changing frameworks give us. Principally, the frameworks and their underlying policies should be crafted in such a way that is supported by official mandates and have formalized direction. If that direction is going to change - like prioritizing peak demand or GHG reductions - the new objectives should be clear. In other words, consistent policy in the longer-term is desired, although it is well recognized that this can be difficult due to medium-term governmental changes.
An updated natural gas DSM framework is expected to be released at the end of the year. Similar to the electric DSM framework, evaluators are ready to change their approach depending on the adjustments made in the framework.
Technological changes are also impacting DSM evaluation approaches in Ontario, and many expect them to evolve the industry further in the next few years. Advanced Metering Infrastructure (AMI) is a technology that has much potential for evaluators. However, it is expected to be a few more years before the datasets stemming from the technology are widely adopted. Currently, it’s mostly used in pilots and for behavioral and demand response programs, not for larger energy efficiency programs. However, if energy efficiency is used more and more to address system needs and capacity constraints, AMI data will become more critical as it is, by nature, temporal and can address system peaks. However, there are currently no AMI data provisions under the Interim Framework. This will need to be ameliorated before AMI data can be used widely to measure the capacity savings from EE effectively.
There are other energy technologies down the road - such as renewables and storage - that are coming under the evaluators’ radar. But according to the interviewees, these are not currently impacting evaluation or policy frameworks. It is unclear how these will evolve DSM program evaluation in the future. One of the most significant disruptions could be the role that heating electrification will have on natural gas consumption. In the future, if there is going to be a regulatory nudge away from natural gas, or prices continue to fall, there needs to be a level of flexibility in EM&V protocols, portfolios, and frameworks. In addition, regulators and evaluators will need to provide impacts from a dual-fuel perspective.
The last big challenge that could play out soon is that of shifting electric consumption and peak demand due to COVID-19. Energy providers may soon see sectoral consumption shifts – as white-collar workers move their office from commercial buildings to their homes. (Add to this heating electrification and DERs -- for even more complications in future demand forecasts.)
These consumption shifts might lead to significant changes in the cost-effectiveness and performance of programs by sector. For example, if a large population now shifts to working from home, the residential sector may gain importance in EE portfolios. Aggregate energy consumption might also change, as people begin to heat and cool homes for one or two workers, replacing much higher worker-dense office buildings. Evaluators need to be aware of these broader macro-level consumption shifts and incorporate them in their analyses.
The challenges listed in this article can have significant impacts on the evaluators’ workflow and quality of the evaluation itself. They can also lead to innovation and process improvements.
As is seen in this current crisis, adaptation and ingenuity have allowed evaluators to make the best out of the situation. This has been realized principally through keeping abreast of the latest approaches, staying practical, developing methodologies when needed, and, most importantly, being flexible on short timeframes. And the benefits of these new approaches will pay dividends well into the future, no matter the political, economic, or public health challenges the industry faces.
David Baumann brings 12 years of experience in program impact evaluations, energy modelling, and scientific programming. Since joining Dunsky Energy Consulting in 2016, he has contributed to multiple impact evaluation projects, harnessing his skill set in Big data, scientific programming and visualization analytics to extract valuable insights from data.
This article is republished from the August 2020 issue of Strategies, AESP’s exclusive magazine for members. To receive Strategies, please consider joining AESP.