The mission of this group is to bring together utility professionals in the power industry who are in the thick of the digital utility transformation. 


The “Early & Often” Approach to Analytics

image credit: © Ksenia Kolesnikova |
Michael Smith's picture
Principal, KLN Group

If you have been active in the North American utility IT/automation markets over the last 30+ years, chances are that you have subscribed to a publication, read a research report, or attended an...

  • Member since 2021
  • 21 items added with 9,658 views
  • Dec 8, 2021

This item is part of the Data Analytics & Intelligence - December 2021 SPECIAL ISSUE, click here for more

After over a century of stability and incremental change, the utility industry is now undergoing massive changes. Distributed energy resources (DERs) are changing how the grid operates and how customers engage with their local utility. The pace of growth of electric vehicle (EV) usage is accelerating. Generation sources are no longer exclusively centralized and are becoming decentralized and localized.

One common thread across these and other changes in the utility industry is the presence and role of data and analytics, which present their own challenges and benefits for utility leaders navigating this evolving landscape. As the utility industry posture towards data and analytics has evolved from simply managing masses of data to leveraging that data for predictive and prescriptive insights, utility leaders can see that not only is analytics a “nice to have” it is quickly becoming a requirement to meet the needs of the changes in their operational and customer landscape.

Your access to Member Features is limited.

A leader articulating that he or she wants their organization to embrace data and analytics as a core competency to run the utility is one thing. Recognizing how the organization needs to change to make that embrace last is often quite another. Let’s look at a few examples.

            Starting with the basics, one observation is that a foundational ingredient that is often missing is a data governance plan. A study of the utility data and analytics space conducted by TMG Research in 2021 found that just over half of the respondents (53%) have a formal data governance plan. If an organization wants to scale its analytics use cases and does so without a data governance plan, it becomes a “fake it until you make it,” and when that no longer works, things can get ugly. Data access, security formats, standards, sharing…these all need to be established early and revisited often.

            Two technology areas to also consider early and often are the use of open source tools and the cloud. Looking first at open source tools, these are presenting some interesting opportunities and challenges. On the plus side, there is a lot of new talent coming into the utility industry that has built its collective resume using open source tools. The industry clearly needs to on-board this new talent to meet the needs of that changing landscape noted above. One other facet of open source tools that attracts many utility leaders is that…it’s free!

            A few thoughts on the other side of the open source ledger merit some thought. First is that yes it’s free, but in reality it’s free like a puppy is free. The professional services, frameworks, and additional software and hardware should be considered when looking at open source TCO (Total Cost of Ownership) over a multi-year period. A second consideration is the limitation of open source tools in scaling to the large enterprise use cases that utilities need to execute to be viable in the very near future.

            Similarly, use of the cloud is growing in the utility industry, but this has its share of plusses and minuses, as well. That same TMG study referenced above also points to limited but growing use of the cloud for data storage, and good thing that. The growth of data sets, especially image- and video-based data sets that are now becoming part of many asset inspection, vegetation management, and fire mitigation strategies, would quickly put the traditional corporate server farm out of business very quickly. In addition to using the cloud to store these massive data sets, applications are starting to reside here with increasing frequency. This can enable greater cyber-security and afford more flexibility for the utility, especially with the “burst” capability that cloud solutions provide.

            One area to watch for in migrating to the cloud are costs, which can be less than the traditional on-prem approach on the front end, but over time utilization can really drive costs. One case in point is what might be considered a “dirty little secret” in the cloud marketplace; namely, that cloud sales teams are often compensated on utilization. So, encouraging use of the cloud can be an economical way to meet business goals, but keep an eye on the meter.

            So, what should utility leaders be looking for in this era of data-rich operations that will hopefully yield business value? Here are a couple of ideas that might help. First is alignment. We still find utilities engaging in data and analytics initiatives that are not aligned with the corporate strategy. The best way for an analytics initiative to fail is to start or become misaligned with core corporate goals. More often than not, those high-level goals or strategic imperatives are there – they just need the proper attention from those that are connecting that strategy with the actual analytics work being done.

            Second is to apply the old Steven Covey principal and “start with the end in mind.” Many analytics initiatives start with a single use case as opposed to a large enterprise undertaking, but even in relatively “simple” use case-driven initiatives the solution must be built and delivered to scale to the enterprise. Is there adequate governance? What about data standards and formats? Are there data or analytic sharing opportunities across the traditional organizational or system boundaries? This is another opportunity to hit these and other issues “early and often.”

            Analytics initiatives are not small undertakings. Mistakes will happen, lessons will be learned, and ultimately value will be realized. Just be certain to prepare and focus on the key tasks early and often.

Varun Perumalla's picture
Varun Perumalla on Dec 13, 2021

I have seen two school of thoughts. One, the utilities are ready to undergo the transformation on new technologies like cloud just because the leaders of those organization are sold with the value of new technologies (or sold by marketing presentations given by big companies). Another set of utilities taking more pragmatic approach where they want their own control on their data control center. 


IMHO, I think performing the ROI in moving to new technologies need to be performed more thoroughly and have metrics in place to measure if those objectives are being met. Further, now that all sensitive data is on cloud what all security measures can be taken is a major challenge for utilities today.


With respective to data analytics, standardization is the key.

Michael Smith's picture
Michael Smith on Dec 21, 2021

Agreed, Varun. The ROI is an important part of this, but there needs to be a pretty broad and deep assessment across the financial, security, and reliability requirements. I think that most utilities are somewhere in or near this process, but there is also a leap to be made - it's almost an emotional one - that accepts that there might be new ways of still keeping the lights on.

Ben Ettlinger's picture
Ben Ettlinger on Dec 14, 2021

Yes lack of data governance is a big issue. Jut got out of a meeting with a vendor and followed up with an email to my co-workers railing about the lack of data governance and lack of standards for that particular segment....and on the day after the log4 announcement what do did say about incorporating open source? Open source is open season for malfeasance.

Michael Smith's picture
Michael Smith on Dec 21, 2021

Yes, open source is an interesting conundrum for utility leaders. They love the price tag ("it's free!") until they arrive at the realization that it's actually "free like a puppy." The professional services, frameworks and platforms that are required to be successful at scale will come to bear sooner or later, and again yes, security is an issue.

Kaylyn Mickelsen's picture
Kaylyn Mickelsen on Dec 20, 2021

Michael. I think this was a great post. There are many changes underway in the utility industry and it is important to see the whole picture when dealing with data and technology.

While data governance is important from a technical perspective, the process side tends to have just as many concerns. One thing we often do to support our clients is looking not only at the security issues surrounding data storage and solutions but the data integrity. There are often many individuals working in the same systems or the same data existing in many systems so there is no single source of truth. Not having clearly defined roles and responsibilities within the organization can often lead to poor data quality as well. The data flow needs documented processes and oversight to achieve maximum value for the organization. 

Michael Smith's picture
Michael Smith on Dec 21, 2021

Agreed on your comment about the process side. Utilities tend to be organizations where systems and data come first; the people and process side come later (if at all). Change management needs to be a part of any major analytics initiative, and we are starting to see this at utilities in large systems implementations. Some of the changes that are staring us down right now, like moving from on-prem computing to the cloud, will require significant changes in staffing and business processes. It is an interesting to be working in our industry!

Jim Horstman's picture
Jim Horstman on Dec 30, 2021

Related to the choice of tools is the people who will be using those tools. Most utilities have people with some form of an 'analyst' title who prepare reports. With the advent of the data scientist role many of these people were re-labeled as 'data scientist' even though they received no additional training nor were provided the right tools to fulfill that role. Utilities need to look at the roles involved with analytics both from the creation and the use sides and ensure there is proper training/mentoring for those roles. Otherwise the same old reports which were ignored will be produced using new tools and still be ignored.

Michael Smith's picture
Thank Michael for the Post!
Energy Central contributors share their experience and insights for the benefit of other Members (like you). Please show them your appreciation by leaving a comment, 'liking' this post, or following this Member.
More posts from this member

Get Published - Build a Following

The Energy Central Power Industry Network is based on one core idea - power industry professionals helping each other and advancing the industry by sharing and learning from each other.

If you have an experience or insight to share or have learned something from a conference or seminar, your peers and colleagues on Energy Central want to hear about it. It's also easy to share a link to an article you've liked or an industry resource that you think would be helpful.

                 Learn more about posting on Energy Central »