Lessons in program design from two New York Smart rate pilots

This article is republished from the November 2020 issue of Strategies, AESP’s exclusive magazine for members. 

The Smart Home Rate Pilot (SHRP). Born out of New York’s Reforming the Energy Vision (REV) initiative, Con Edison and affiliate, Orange and Rockland, are executing the SHR pilot. Its design reflects the Public Service Commission’s goal to use REV demonstration projects to test new technologies, prove new business models, and stimulate innovation, particularly around distributed energy resources (DER).

Launched in 2019, SHR is testing how alternative rate structures can provide price signals to residential customers with AMI and smart home technology to deliver greater control over energy use and costs. This pilot will evaluate load changes in response to the dynamic time-varying rates, as well as measure bill impacts and gauge customer engagement with smart home rates and technologies. In this pilot, participants receive smart thermostats connected to home energy management technologies, which help them maximize savings on the SHR rates.

The pilot was designed as a Randomized Control Trial (RCT) with customers assigned to one of two rates (Rate A or Rate B) or to the Control Group. The pilot’s enrollment goal is 450 participants with active thermostats per rate.

The word ‘pilot’ implies continuous learning and fine-tuning. A successful pilot design requires agility – it must have flexibility incorporated from the start, allowing the utility to adjust the approach as needed to meet its goals. Additionally, a tight, well integrated cross-functional team supports that need to be agile for pilots of this scale and complexity. In their first and second years, both SEP and SHR made changes in pilot design.

Recruitment-Driven Design Changes

Smart Energy Plan

The SEP recruitment and enrollment strategy was originally designed as a two-phased process occurring across two waves, two billing plans, seven rates, and multiple messaging treatments in each of the targeted geographies — beginning with SI/W, followed by Brooklyn.

This deliberately staggered approach included the implementation of initial small-scale tests - called Test & Learns – with a subset of target customers used to determine the relative impact of various offer features on customer acceptance of the base rate.

These small-scale tests indicated whether Con Edison was likely to achieve its target enrollment rates and helped determine which messaging and features to include in the marketing materials used to recruit customers during the second phase. The winning approaches were sent to the remaining larger target customer pilot population in each area.

This staggered approach allowed the team to apply learnings from early waves to inform subsequent waves for ongoing operational optimization and smoother rollout. It also allowed them to leverage customer feedback and insights in order to fine-tune recruitment and enrollment and continuously improve the customer experience.

SEP’s first year encountered several changes in its original design. One change included the replacement of the small-scale SI/W Test & Learn with a MaxDiff analysis (an analytic approach using best-worst scaling to gauge survey respondents’ preference scores for different features). Another change was the shift from a two-waved to a three-waved recruitment rollout, including the addition of a small incentive testing phase to assess the impact of a $25 incentive on SI/W enrollment rates.

Key changes resulting from these design modifications included the following:
1.    The team eliminated enrollment incentives from the recruitment offer to the larger target pilot population in SI/W and Brooklyn (after additional testing within that population resulted in the same findings as SI/W).
2.    The longer time horizon of staggered recruitment extended the operations ramp-up period from three to six months, which allowed the team to better configure technology to meet pilot requirements; hone and refine messaging; implement additional customer care and call center training; identify additional key metrics to track; develop more sophisticated reporting dashboards; and engage other stakeholders as SEP moved from launch through full-scale rollout to the target pilot population.

Additionally, Year Two of the pilot saw an unforeseen deployment disruption: COVID-19. As a result, Brooklyn’s full-scale pilot recruitment was paused, and the rate launch delayed. ICF worked with Con Edison to adjust operations accordingly. Impacted areas included project schedules, project duration, call center handling, IT system plans, campaign deployment, and creative development. Critical to this last activity, COVID-19 shifted customer sentiment and energy use behavior in new and unknown ways. To investigate how these changes impacted Con Edison’s target population for this pilot, ICF implemented an agile message testing effort. The results of ICF’s testing informed the creative strategy utilized for final recruitment activities, as well as for Con Edison’s ongoing engagement efforts in Year Three and beyond.

Smart Home Rate

At the start of SHR recruitment, a targeted list of Con Edison and Orange & Rockland customers received a direct mail letter and email. While customers began signing up, enrollment was lower than anticipated and attrition rates at key steps in the process were higher than expected in the original program design. Tracking metrics by recruitment channel indicated that, while recruitment material was piquing customer interest and the online portal allowed for the collection of important enrollment data, there were challenges to getting customers all the way through the process. Contributing factors to customer attrition included:

• Stringent eligibility requirements – Interested targeted customers had to navigate a two-page, 18 question, self-directed eligibility screen on the enrollment portal. Of those who viewed the portal’s second page of screening questions, 34 percent were deemed ineligible and 12 percent did not complete the page.
• Randomized rate assignment during enrollment – The randomization design meant that customers were not exposed to information about the new rate until the last step of enrollment. Learning about the rate late into the enrollment process may have caused the customer attrition at this step: approximately 46% of customers dropped off at this stage.
• Subscription level selection during enrollment – The design required potential participants randomly assigned to Rate B to select from one of three possible demand subscription and overage charge levels, customized based on each customer’s historical energy usage. Customers found it difficult to make a subscription decision at this early stage of their journey, causing prospective Rate B respondents to drop at almost double the rate of prospective Rate A respondents, who were not required to select a subscription level.

 Another challenge was losing interested customers to the Control Group: The RCT design meant one third of eligible customers attempting to enroll online were assigned the Control Group. This not only removed eligible customers but also caused confusion, with 22 percent of Control Group customers calling in to learn why they couldn’t participate.

After the initial phase of recruitment completed in early 2020, SHR pilot recruitment paused due to COVID-19. As the team planned for recruitment relaunch, these insights informed several updates to program design aimed at increasing enrollments. This included:

• Streamlining eligibility screening on the enrollment portal to remove non-essential questions, bringing the maximum number of questions down to 12.
• Removing demand subscription level selection from the Rate B enrollment flow. Instead of three demand subscription level options, Rate B respondents are now presented with a single default subscription level and overage charge.
• Moving the design from an RCT to a design with a matched control. This new design allows all interested, eligible customers to be placed into either Rate A or Rate B, without the possibility of being randomly placed into the Control Group.

While the SHR pilot’s fall 2020 recruitment campaign is still underway, the team anticipates that these changes will retain a greater number of customers and improve the likelihood of meeting enrollment targets.

Recruitment Mitigation Planning

When planning recruitment, utilities are strongly encouraged to consider including a mitigation strategy upfront in program planning. The need for mitigation planning became clear as recruitment campaigns were executed, and the team encountered the possibility of one or more of the opt-in rates not hitting the target enrollment rate. For SEP, based on lessons learned from previous opt-in recruitment campaigns, the team developed a mitigation plan prior to the launch of Brooklyn recruitment. When SHR developed a recruitment re-launch plan for fall 2020, it incorporated a mitigation plan into the modified recruitment approach.

For both pilots, a key mitigation strategy has been to monitor enrollments against targets via weekly reporting and enrollment projection analyses. Specifically, the SEP team developed a metric report to track enrollments by rate and wave, comparing each rate’s progress to the recruitment target against the previous campaigns over a comparable time period. SHR employed a similar approach.

Based on these analyses, each pilot team would then determine whether additional recruitment touchpoints would be necessary. Part of this process entailed outlining a creative development schedule that would enable the teams to finalize ahead of time the mitigation recruitment materials for each potential additional touchpoint. Advance creative development was needed so that these recruitment materials could be quickly deployed. This schedule also had to allow customers enough time to receive the touchpoint and enroll before the rate went live.

 Technology-Driven Design Changes

Designing pilots with emerging technologies often poses inherent challenges. For SHR, COVID-19 added another dimension of complexity to integrating new technology. The original pilot design allowed customers in Phase One to choose between self-installing their smart thermostats or receiving free, professional installation. Approximately 75 percent of customers selected professional installation. Due to COVID-19, Con Edison transitioned to the self-install method only. The team knew this change could present a significant barrier to installation, so they devised ways to alleviate the friction of customer-directed installation. This resulted in several program design changes, including:

• Increased Incentives – An increased incentive ($50) upon confirmation of a successful installation and registration of the thermostat (vs. original $10 incentive design that was provided upon enrollment).
• Text Message Nudges – In addition to existing email-based nudges to install their thermostat, customers are also now sent reminders via text
• Virtual Installation Support – Phone- and video-based support is provided to customers that request it or have not installed the thermostat after a certain number of days.
• Installation Contingency Measures – The original design did not include significant thermostat installation and registration mitigation “levers” built into it. The updated design builds in the following contingency measures, to be used should installation rates lag:
• Installation Mitigation Bonus Incentive Email Campaign – Targeted email campaign to customers who have not yet installed their thermostat that offers an incentive if they install within a certain timeframe
• Installation Mitigation Outbound Call Campaign – ICF calls customers who have not installed and registered their thermostat and offers virtual install support.
• Installation Mitigation Formal Letter – Customers who have not installed and registered their thermostat receive a formal letter.

Overall, these adjustments directed customers forward in the enrollment and installation journey using “carrots” rather than “sticks” (e.g. if thermostat isn’t installed customer is charged for it). Recruitment and installation are still in progress, but initial results show promise in lessening risk around stranded technology assets and customer attrition.

Conclusion

Two years into the deployment of the Smart Energy Plan pilot and one year into the deployment of the Smart Home Rate pilot, the body of knowledge associated with informing, recruiting, driving action from, and building satisfaction with customers on new demand-based rate structures has grown substantially. This article has shared only a few key lessons from a larger set of critical findings of how the target customers and participants think, act, and react in response to billing structures and technology that can provide opportunities to save on energy costs and maintain grid reliability. Con Edison is proud to be among the first utilities to blaze this exciting trail.   

Rebecca Roberts is Section Manager of Utility of the Future at Con Edison where she leads a team piloting new rate structures that aim to provide customers with greater choice and control. Rebecca has held various roles at Con Edison, including Manager in its Energy Efficiency & Demand Management department, where she led a team that developed and piloted new energy saving products, services, and technologies.

Candice Tsay is a Senior Planning Analyst at Con Edison, where she is currently responsible for implementing an innovative rate and smart home technology demonstration project. Her career at Con Edison has been focused throughout on developing and implementing new ideas to enable clean energy technologies and demand-side flexibility on the electric grid.

Debbie Sassoon is Project Specialist, Utility of the Future at Orange and Rockland Utilities where she is currently responsible for implementation of the smart home technology demonstration project as well as regulatory analysis associated with the implementation of REV and its impact on the company. Debbie has held previous roles at Orange and Rockland, including Manager, Customer Accounting.

Laura Orfanedes serves as the Director of Northeast Marketing Team in ICF’s Commercial Energy Division where she leads her team in designing, implementing, and optimizing energy marketing campaigns and initiatives for clients that include Con Edison, Orange & Rockland Utilities, Central Hudson, Avangrid, National Grid, and the Sponsors of Mass Save. She also serves as a four-time elected member of AESP’s Board of Directors.

Francesca Jones is a Marketing Manager at ICF where she leads marketing and campaign strategy for the energy and utility sectors, with core expertise in new rates and emerging technologies. She has led high-impact marketing and brand campaigns for a variety of clients including Con Edison, National Grid, DTE Energy, Orange & Rockland, Mass Save, NYSERDA, NYPA, and Efficiency Maine.

Leigh Winterbottom has been a Senior Marketing & Communications Manager at ICF for three years and since July 2018, the Project Manager for the Con Edison Innovative Pricing Pilot. Leigh began her work in energy efficiency marketing with a nine-year tenure at the award-winning Efficiency Vermont program – the country’s first state-wide energy efficiency utility at the leading edge of clean energy.

This article is contributed by the AESP Pricing and Demand Response Topic Committee.

Back to top