Lessons in program design from two New York Smart rate pilotsPosted to AESP in the Energy Efficiency Group
- Apr 13, 2021 12:00 am GMTApr 12, 2021 11:56 pm GMT
- 319 views
This article is republished from the November 2020 issue of Strategies, AESP’s exclusive magazine for members.
The Smart Home Rate Pilot (SHRP). Born out of New York’s Reforming the Energy Vision (REV) initiative, Con Edison and affiliate, Orange and Rockland, are executing the SHR pilot. Its design reflects the Public Service Commission’s goal to use REV demonstration projects to test new technologies, prove new business models, and stimulate innovation, particularly around distributed energy resources (DER).
Launched in 2019, SHR is testing how alternative rate structures can provide price signals to residential customers with AMI and smart home technology to deliver greater control over energy use and costs. This pilot will evaluate load changes in response to the dynamic time-varying rates, as well as measure bill impacts and gauge customer engagement with smart home rates and technologies. In this pilot, participants receive smart thermostats connected to home energy management technologies, which help them maximize savings on the SHR rates.
The pilot was designed as a Randomized Control Trial (RCT) with customers assigned to one of two rates (Rate A or Rate B) or to the Control Group. The pilot’s enrollment goal is 450 participants with active thermostats per rate.
The word ‘pilot’ implies continuous learning and fine-tuning. A successful pilot design requires agility – it must have flexibility incorporated from the start, allowing the utility to adjust the approach as needed to meet its goals. Additionally, a tight, well integrated cross-functional team supports that need to be agile for pilots of this scale and complexity. In their first and second years, both SEP and SHR made changes in pilot design.
Recruitment-Driven Design Changes
Smart Energy Plan
The SEP recruitment and enrollment strategy was originally designed as a two-phased process occurring across two waves, two billing plans, seven rates, and multiple messaging treatments in each of the targeted geographies — beginning with SI/W, followed by Brooklyn.
This deliberately staggered approach included the implementation of initial small-scale tests - called Test & Learns – with a subset of target customers used to determine the relative impact of various offer features on customer acceptance of the base rate.
These small-scale tests indicated whether Con Edison was likely to achieve its target enrollment rates and helped determine which messaging and features to include in the marketing materials used to recruit customers during the second phase. The winning approaches were sent to the remaining larger target customer pilot population in each area.
This staggered approach allowed the team to apply learnings from early waves to inform subsequent waves for ongoing operational optimization and smoother rollout. It also allowed them to leverage customer feedback and insights in order to fine-tune recruitment and enrollment and continuously improve the customer experience.
SEP’s first year encountered several changes in its original design. One change included the replacement of the small-scale SI/W Test & Learn with a MaxDiff analysis (an analytic approach using best-worst scaling to gauge survey respondents’ preference scores for different features). Another change was the shift from a two-waved to a three-waved recruitment rollout, including the addition of a small incentive testing phase to assess the impact of a $25 incentive on SI/W enrollment rates.
Key changes resulting from these design modifications included the following:
Additionally, Year Two of the pilot saw an unforeseen deployment disruption: COVID-19. As a result, Brooklyn’s full-scale pilot recruitment was paused, and the rate launch delayed. ICF worked with Con Edison to adjust operations accordingly. Impacted areas included project schedules, project duration, call center handling, IT system plans, campaign deployment, and creative development. Critical to this last activity, COVID-19 shifted customer sentiment and energy use behavior in new and unknown ways. To investigate how these changes impacted Con Edison’s target population for this pilot, ICF implemented an agile message testing effort. The results of ICF’s testing informed the creative strategy utilized for final recruitment activities, as well as for Con Edison’s ongoing engagement efforts in Year Three and beyond.
Smart Home Rate
At the start of SHR recruitment, a targeted list of Con Edison and Orange & Rockland customers received a direct mail letter and email. While customers began signing up, enrollment was lower than anticipated and attrition rates at key steps in the process were higher than expected in the original program design. Tracking metrics by recruitment channel indicated that, while recruitment material was piquing customer interest and the online portal allowed for the collection of important enrollment data, there were challenges to getting customers all the way through the process. Contributing factors to customer attrition included:
After the initial phase of recruitment completed in early 2020, SHR pilot recruitment paused due to COVID-19. As the team planned for recruitment relaunch, these insights informed several updates to program design aimed at increasing enrollments. This included:
Recruitment Mitigation Planning
When planning recruitment, utilities are strongly encouraged to consider including a mitigation strategy upfront in program planning. The need for mitigation planning became clear as recruitment campaigns were executed, and the team encountered the possibility of one or more of the opt-in rates not hitting the target enrollment rate. For SEP, based on lessons learned from previous opt-in recruitment campaigns, the team developed a mitigation plan prior to the launch of Brooklyn recruitment. When SHR developed a recruitment re-launch plan for fall 2020, it incorporated a mitigation plan into the modified recruitment approach.
For both pilots, a key mitigation strategy has been to monitor enrollments against targets via weekly reporting and enrollment projection analyses. Specifically, the SEP team developed a metric report to track enrollments by rate and wave, comparing each rate’s progress to the recruitment target against the previous campaigns over a comparable time period. SHR employed a similar approach.
Based on these analyses, each pilot team would then determine whether additional recruitment touchpoints would be necessary. Part of this process entailed outlining a creative development schedule that would enable the teams to finalize ahead of time the mitigation recruitment materials for each potential additional touchpoint. Advance creative development was needed so that these recruitment materials could be quickly deployed. This schedule also had to allow customers enough time to receive the touchpoint and enroll before the rate went live.
Technology-Driven Design Changes
Designing pilots with emerging technologies often poses inherent challenges. For SHR, COVID-19 added another dimension of complexity to integrating new technology. The original pilot design allowed customers in Phase One to choose between self-installing their smart thermostats or receiving free, professional installation. Approximately 75 percent of customers selected professional installation. Due to COVID-19, Con Edison transitioned to the self-install method only. The team knew this change could present a significant barrier to installation, so they devised ways to alleviate the friction of customer-directed installation. This resulted in several program design changes, including:
Two years into the deployment of the Smart Energy Plan pilot and one year into the deployment of the Smart Home Rate pilot, the body of knowledge associated with informing, recruiting, driving action from, and building satisfaction with customers on new demand-based rate structures has grown substantially. This article has shared only a few key lessons from a larger set of critical findings of how the target customers and participants think, act, and react in response to billing structures and technology that can provide opportunities to save on energy costs and maintain grid reliability. Con Edison is proud to be among the first utilities to blaze this exciting trail.
Rebecca Roberts is Section Manager of Utility of the Future at Con Edison where she leads a team piloting new rate structures that aim to provide customers with greater choice and control. Rebecca has held various roles at Con Edison, including Manager in its Energy Efficiency & Demand Management department, where she led a team that developed and piloted new energy saving products, services, and technologies.
Candice Tsay is a Senior Planning Analyst at Con Edison, where she is currently responsible for implementing an innovative rate and smart home technology demonstration project. Her career at Con Edison has been focused throughout on developing and implementing new ideas to enable clean energy technologies and demand-side flexibility on the electric grid.
This article is contributed by the AESP Pricing and Demand Response Topic Committee.