EM&V: How COVID-19, Business Models, Policy Frameworks, and Technological Change Have Impacted DSM Evaluation Approaches in Ontario

Posted to AESP in the Energy Efficiency Group
AESP HQ's picture
Leadership AESP

Founded in 1990, the Association of Energy Services Professionals is the leading organization in the U.S. and Canada serving the networking and professional development needs of energy...

  • Member since 2011
  • 236 items added with 223,885 views
  • Jan 15, 2021



DavidBaumann_1639651.jpgBy David Baumann

The last few years have brought about significant disruption for energy efficiency program evaluation in Ontario. At the beginning of this year, there was already much change, due to recently updated DSM frameworks, evolving technologies, company mergers, and shifting political priorities. In the early spring, this situation was further compounded by the COVID-19 pandemic. Despite these stacking challenges, the evaluation community has generally persevered and brought about innovation to the industry. This article - developed from interviews with four industry leaders in Ontario - hopes to dive into some of the specifics around these challenges with the hope of bringing universal insights and recommendations to evaluators throughout North America.


The Pandemic’s Effects on DSM Evaluation

In the middle of March 2020, when the Canadian provinces started their lockdowns, the evaluators had to adapt quickly to the pandemic and the challenges it introduced. At this moment, the Ontarian evaluators were in the middle of the 2019 evaluation cycle. The lockdowns led to an immediate stop to site visits due to business closures and physical distancing measures. This meant a lack of data collection for metric development, such as realization rates. Other aspects of the evaluation approach were less impacted, such as net-to-gross ratio development and process evaluation, as these were largely accomplished through phone interviews and surveys. For participants where phone interviews and surveys were not possible, some evaluators filled in data gaps using historical adjustment factors.

The restrictions on on-site visits have been relaxed over the past month as Ontario moved into a partial economic re-opening. However, there is still uncertainty on how survey participants perceive participation due to COVID-19 stresses - which can cause further delays and bias samples. There is also the added temporal challenge. Many program evaluations have been delayed due to COVID-19. Some of these same programs cannot be extended due to regulatory deadlines. These delays have required the evaluators to move faster than normal.

The challenges vary by sector and size. On the electric side, in the medium and large commercial and industrial sectors, evaluators have had less trouble going back to normal. The residential side was already evaluated mostly remotely through desk reviews.  The area of the most substantial concern is the small C&I sector, which has been most disproportionally affected by the crisis as many small businesses closed down temporarily or permanently.
Another new challenge is that of determining baseline savings in billing analysis-based impact evaluations. With COVID-19 changing consumption patterns, it proves challenging to calculate the savings from measures when the underlying baseline has shifted. Solutions to a shifting baseline are currently being developed but are luckily not urgently needed. A shifting baseline may become more of a concern in 2021 when 2020 evaluations are performed on the new baseline. The baseline may shift again in 2021 and beyond, back to pre-COVID-19 norms, or into a different shape, further challenging future analyses.

Finding alternative approaches to site-visits

The evaluator’s data gathering approach has changed to adapt to the challenges in site-visits. For some projects, especially in the residential space, evaluators have moved to desk reviews. Also, some are using simulation modeling tools, which can be useful if calibrated correctly. Hybrid solutions are also possible -- one evaluator used a combination of phone interviews and consumption data to estimate realization rates.
For those participants who require site visits, evaluators are starting to adopt virtual site visit tools. These are possible due to the increasing availability of web apps and the penetration of smartphones. While some evaluators were already exploring this option before the pandemic, the need skyrocketed during the crisis, leading to a much higher interest in the solution. It will be interesting to see if virtual site visits maintain their market position in a post-COVID world.
The method by which participants are contacted may also be shifting. Before the pandemic, work phone numbers were the principal way of reaching potential participants. Since the pandemic, one evaluator found that web surveys and email correspondence are more effective, as more participants are working from home, less likely to have access to their work phone numbers, and are at their computers more often.

Other methodological approaches have evolved in the evaluation space. Another evaluator mentioned a novel approach that his team is using to determine the realization rate while reducing the number of site visits to a minimum. By looking at the last ten years of data, they calculated the highest acceptable ratio of desk reviews to site visits to ensure the actual historical realization rate. This approach may lead to fewer site visits and increased cost-effectiveness post-pandemic. It is uncertain that this approach would have been developed outside of the pandemic, which suggests innovation brought about by these challenging times. The evaluator hopes to present this finding in an upcoming AESP conference.
The following section steps away from the pandemic and speaks to some of the regulatory challenges that have faced the industry in Ontario.

Shifting DSM frameworks and Company Mergers

After the provincial election of 2018, the Progressive Conservative Party of Ontario formed the government following the previous Ontario Liberal Party government. This changeover led to some sweeping changes in the electricity sector. Before the transition, the Local Distribution Companies (LDCs) were largely responsible for designing and delivering energy efficiency programs. After the Conservatives came to power, they shifted the majority of the responsibility for energy efficiency programs to the Independent Electricity System Operator (IESO). This shift was brought about through the 2019/2020 CDM Interim Framework (IF), which replaced the Conservation First Framework (CFF).
The change of frameworks has led to some challenges. When the CFF was terminated, budgets were reduced, and evaluators had to adapt quickly. One solution was to simplify the scope from the measure to the program level. In addition, to work within the restricted budget, the evaluators used historical adjustment factors such as realization rates and net-to-gross ratios instead of developing up to date factors.
The IESO and the government are continuing discussions on a post-2020 framework and a proposal will soon be brought forward for consideration. The new framework* may consider fuel-switching in the savings calculations, requiring piggy-backing the electric and gas utility program DSM evaluations. (Which would require an agreement with the gas companies and the regulator.) Any move in this direction would create extra complexity for evaluations, mostly around participant data sharing.

In addition, there is the possibility that the principal savings target metric changes. This has precedent. From 2011 to 2014, the metric was kW demand savings; currently, the principal metric is energy savings. In general, the shifting of framework metrics reflects changes in economic and societal realities but also forces the evaluators’ processes to adapt.

There are some broad lessons that the changing frameworks give us. Principally, the frameworks and their underlying policies should be crafted in such a way that is supported by official mandates and have formalized direction. If that direction is going to change - like prioritizing peak demand or GHG reductions - the new objectives should be clear. In other words, consistent policy in the longer-term is desired, although it is well recognized that this can be difficult due to medium-term governmental changes.
On the natural gas side, there have been significant corporate structural changes. In 2019 there was a merger between the two biggest gas providers, Union Gas and Enbridge.  The merger consolidated institutional knowledge and made evaluation more streamlined, plus there is now added consistency in the way that data is collected and reported. Lastly, the evaluator now has a single touchpoint to seek additional data or for general communications with the utility.

An updated natural gas DSM framework is expected to be released at the end of the year. Similar to the electric DSM framework, evaluators are ready to change their approach depending on the adjustments made in the framework.

Shifting technologies

Technological changes are also impacting DSM evaluation approaches in Ontario, and many expect them to evolve the industry further in the next few years. Advanced Metering Infrastructure (AMI) is a technology that has much potential for evaluators. However, it is expected to be a few more years before the datasets stemming from the technology are widely adopted. Currently, it’s mostly used in pilots and for behavioral and demand response programs, not for larger energy efficiency programs. However, if energy efficiency is used more and more to address system needs and capacity constraints, AMI data will become more critical as it is, by nature, temporal and can address system peaks. However, there are currently no AMI data provisions under the Interim Framework. This will need to be ameliorated before AMI data can be used widely to measure the capacity savings from EE effectively.

There are other energy technologies down the road - such as renewables and storage - that are coming under the evaluators’ radar. But according to the interviewees, these are not currently impacting evaluation or policy frameworks. It is unclear how these will evolve DSM program evaluation in the future. One of the most significant disruptions could be the role that heating electrification will have on natural gas consumption. In the future, if there is going to be a regulatory nudge away from natural gas, or prices continue to fall, there needs to be a level of flexibility in EM&V protocols, portfolios, and frameworks. In addition, regulators and evaluators will need to provide impacts from a dual-fuel perspective.

Shifting Peaks and Consumption

The last big challenge that could play out soon is that of shifting electric consumption and peak demand due to COVID-19. Energy providers may soon see sectoral consumption shifts – as white-collar workers move their office from commercial buildings to their homes. (Add to this heating electrification and DERs -- for even more complications in future demand forecasts.)

These consumption shifts might lead to significant changes in the cost-effectiveness and performance of programs by sector. For example, if a large population now shifts to working from home, the residential sector may gain importance in EE portfolios. Aggregate energy consumption might also change, as people begin to heat and cool homes for one or two workers, replacing much higher worker-dense office buildings. Evaluators need to be aware of these broader macro-level consumption shifts and incorporate them in their analyses.

Adapting to Future Challenges

The challenges listed in this article can have significant impacts on the evaluators’ workflow and quality of the evaluation itself. They can also lead to innovation and process improvements.

As is seen in this current crisis, adaptation and ingenuity have allowed evaluators to make the best out of the situation. This has been realized principally through keeping abreast of the latest approaches, staying practical, developing methodologies when needed, and, most importantly, being flexible on short timeframes. And the benefits of these new approaches will pay dividends well into the future, no matter the political, economic, or public health challenges the industry faces.

*Currently, there is no certainty around the programming beyond 2020.  Therefore, any examples given are speculation.
The author wishes to give a special thanks to Henri van Rensburg (Nexant), Alice Herrera (IESO), Tamara Kuiken (DNV GL) and Raegan Bond (Dunsky Energy) for their insights and contributions to this article.


David Baumann brings 12 years of experience in program impact evaluations, energy modelling, and scientific programming. Since joining Dunsky Energy Consulting in 2016, he has contributed to multiple impact evaluation projects, harnessing his skill set in Big data, scientific programming and visualization analytics to extract valuable insights from data.  
This article is contributed by the AESP EM&V Topic Committee.

This article is republished from the August 2020 issue of Strategies, AESP’s exclusive magazine for members. To receive Strategies, please consider joining AESP.


Back to top


AESP leads a vibrant community of professionals dedicated to improving energy efficiency through learning, networking and knowledge sharing.
AESP HQ's picture
Thank AESP for the Post!
Energy Central contributors share their experience and insights for the benefit of other Members (like you). Please show them your appreciation by leaving a comment, 'liking' this post, or following this Member.
More posts from this member


Spell checking: Press the CTRL or COMMAND key then click on the underlined misspelled word.

No discussions yet. Start a discussion below.

Get Published - Build a Following

The Energy Central Power Industry Network is based on one core idea - power industry professionals helping each other and advancing the industry by sharing and learning from each other.

If you have an experience or insight to share or have learned something from a conference or seminar, your peers and colleagues on Energy Central want to hear about it. It's also easy to share a link to an article you've liked or an industry resource that you think would be helpful.

                 Learn more about posting on Energy Central »