The mission of this group is to bring together utility professionals in the power industry who are in the thick of the digital utility transformation. 

Post

Dirty Little Secret -- Smart Devices are Consumer Electronics

Gay Gordon-Byrne's picture
President TekTrakker®

Gay Gordon-Byrne is currently the Chair of the Service Industry Association's International Customer Competitiveness Council and a known expert on the contractual and legal issues facing the...

  • Member since 2009
  • 17 items added with 6,413 views
Look under the plastic covers of any smart device, and you will see a printed circuit board and attachments and wires that should be familiar to anyone in the computer industry. This simple fact should give everyone using this equipment pause. Why would this equipment, whose origins and designs have been forged in a throw-away culture of consumer electrics, be durable enough for the demands of the electric utility? The answer is clear. They aren't.

How then do we know what is, and is not, sufficiently durable? At the moment, we don't know and we need to know. The DOE ARRA Funding requests included reporting requirements that might get at this information, but the intention was to monitor the effectiveness of the funding, not the reliability of the emerging technology. NIST and GWAC have worked long and hard on standards for interoperability -- but do not address standards for measuring reliability.

We at TekTrakker know from our work in with large IT organizations that a great deal of data suitable for driving measurements of reliability is already captured in order to do business. This data can be used immediately to measure reliability internally. This same data can then be used for comparison with peers. The essential task is to start measuring.

Utilities know how many meters are installed because they drive billing. Workorder systems connect problems to products. The techniques to use this data to measure reliability are simple and within the means of any organization. It just takes a little imagination, and a little discipline to adopt a culture that allows reliability to be measured and discussed.

What is Reliability?

Reliability is the absence of failure. Engineers and manufacturers routinely measure reliability using Mean Time Between Failure (MTBF). This is the most basic, simplest, and most powerful of all measurements of reliability. Nothing more elaborate is needed. All utilities need to do is keep track of how many (by model) of each new smart device they have installed, and then compare the quantity to the repairs or replacements made (by model). This is both MTBF of the population and, if the element of time is added, it is also failure rate.

In the recent past, utilities didn't measure reliability of specific products because it didn't matter. Lightning arrestors are a great example. Hunks of metal built to specifications are bought, stored, and installed without needing to know the vendor, the data of purchase, or the remaining warranty on the device. If they need to be replaced -- the work crew grabs another one out of the pile and throws the broken one away.

Device Reliability Matters

Enter the electronic era. Smart meters are already posting failure rates, anecdotally, in the 5% per annum range. This is ten times the failure rate of the traditional meter, and the lifecycle of the product has barely begun. Each additional device connecting the meter to the mothership also has a failure rate. The stability of the grid can only be as good as the weakest device -- yet we don't know which devices are weak. Selecting products for reliability is now essential, but the tools for making associations between products and reliability are entirely missing.

Repairs -- Burgeoning Costs

The sheer volume of repairs needed for electronic (smart) devices is going to require a different repair strategy. Even when architected to be networked with alternate data paths, if any device in the grid fails, then some necessary function ceases and a repair or replacement is needed. The explosion of new devices, each with their own dismal failure rate, is going to be a major issue in controlling costs and executing the business case as presented to regulators. There is only one logical approach to controlling repair costs -- don't make as many repairs. Organizations therefore must select products that are the most reliable from available options.

Use MTBF to Demand Reliability

Once reliability is measured, organizations can press for continuous improvements in reliability by using the power of the purchase order. Products with high failure rates can be excised from the system and replaced by those with lower failure rates. Products can be compared on the basis of reliability, and the judgment made if the premium price of a highly durable product is a better selection than a lower cost competitor. Value and total cost of ownership can be brought into the procurement process.

Setting off a reliability war between vendors will be a net benefit to the entire smart grid industry. Every utility should be measuring itself for MTBF at the model level and asking vendors about the failure rate of their equipment and how they know it and when did they know it. Our experience has been that vendors know far less than they want about their own equipment. Until a lot of equipment is installed in the field, they know only what they can learn from bench tests. Pilot deployments are the test bed. Any utility with equipment under pilot is just as knowledgeable as any vendor, provided they keep track. For this reason we suggest, rather vehemently, that utilities should not outsource this information even if they outsource the labor.

Sharing of pilot data between peers is the most effective, efficient, and timely method for the industry to measure the MTBF of products. Utilities will know failure rate in real time, and will have no hidden agenda to avoid trying to mask the failure rate. Utilities of all sizes and forms of ownership can learn from each other. The broader the range of reporting, the better for the industry,

Share and Compare

When it comes to sharing failure data, utilities aren't doing it systematically. No one should fear measuring failure rate. Product failure, when measured correctly, is not under control of the end user. The failure rate of the product as it was designed and manufactured is the foundation upon which all other variables rest.

Peer organizations are an ideal platform for sharing. Surveys have been the traditional method of sharing at the peer level, yet much more could be done if the work was systematic and not unduly burdensome. We have seen several groups attempt to build databases, but the reality of organizing data and getting widespread cooperation has thwarted most attempts.

Get Ready to Share

Just because it is has been hard to do -- doesn't mean it cannot be done. Every utility should start by making sure it can associate problems with models, and be diligent about recording all hardware failures so that the resulting databases can culled. If workorder systems capture failures, make sure the data is not wholly trapped within free form text. Add a few summary fields with drop-down options so that actions can be categorized. Keep the information historically so that management can review changes in the environment over time. Conduct a baseline analysis of the asset database. Make sure that as new products are added to the inventory, that model numbers are kept so that problems can later be associated with the model.

Every one of the suggestions above will pay immediate dividends in allowing self-analysis and at the same time set the stage for sharing data with peers.

Gay Gordon-Byrne's picture
Thank Gay for the Post!
Energy Central contributors share their experience and insights for the benefit of other Members (like you). Please show them your appreciation by leaving a comment, 'liking' this post, or following this Member.
More posts from this member

Discussions

Spell checking: Press the CTRL or COMMAND key then click on the underlined misspelled word.
Bob Amorosi's picture
Bob Amorosi on Dec 17, 2010
"Smart meters are already posting failure rates, anecdotally, in the 5% per annum range. This is ten times the failure rate of the traditional meter, and the lifecycle of the product has barely begun. "

Welcome to the world of electronics all you readers. Great article Gay.

As an electronics design engineer and a semi-expert on smart meter designs, I can tell you that this reliability issue with electronics and its implementation in the smart grid is not news to the smart meter manufacturers, nor to the utility companies.

Designing high reliability into electronics circuits is an old established art. The military and transportation industries have benefited from this for decades. However there are substantial added costs to designing electronics to have high reliability and long MTBFs.

When utility companies go shopping for new technology they put pressure on suppliers and manufacturers to keep prices as low as possible. That usually means NOT designing high reliability into the electronics, or designing just a minimal amount at best. This article is right on because effectively the smart grid devices including smart meters are not substantially more reliable than your average consumer product.

The electronics industry by and large loves this because for consumer products we are accustomed to short lifetimes for products. In all mass produced consumer products it is normal to expect consumers to replace products on a routine basis with next generation products, much more often than the utility industry is willing to bear. Just look around and you can see this, in the computer and cell phone markets it is common to see next generation products introduced multiple times per year.

Why is the utility industry not benefiting from highly reliable smart grid products? Because the utility industry cannot raise the extra money to pay for the extra reliability in these products to begin with unless they were to impose painful rate increases to all their customers to pay for it. The only alternative to raise the extra money is to get handouts from governments.

Until governments wake up and the regulation of the utility industry is reformed to allow them to raise money without saddling all ratepayers with uniform billing rate increases, this situation will never change.

Jack Ellis's picture
Jack Ellis on Dec 21, 2010
Setting aside meters and focusing solely on other "smart" devices, it's not clear higher reliability is worth the added cost, particularly for consumer-level devices. As it is, the business case for smart devices is marginal so long as generation is overbuilt and consumers are "protected" from price volatility. Moreover, the reliability consequences of device failure rates in the 5% range are small since failures are likely to be spread out over time. I could turn this argument around and make a pretty compelling case that any economies of scale associated with giant, 1000 MW power plants are more than offset by the cost of carrying contingency and planning reserves.

Now if you can make the case that even low failure rates have demonstrable health and safety consequences, I might be persuaded to change my mind, but typically any devices that do have health and safety consequences are built to a much higher standard and are prices commensurately.

Steven Rosenstock's picture
Steven Rosenstock on Dec 21, 2010
I would guess if the smart meter pioneer utilities are experiencing that high of a failure rate, they have language in the contracts that makes the vendors responsible for replacements.

Other utilities will then put specifications about failure rates into their smart meter contracts (e.g., a mandate for < 1% failure within the first XX years of operation), and the vendors will make products that meet the tougher specifications.

Murray Duffin's picture
Murray Duffin on Dec 21, 2010
There is something wrong with this discussion. I retired from the microelectronic chip business 10 years ago, and at that time we routinely shipped very complex chips that met quality standards of 3 ppm defective after 3 years of operation, even when harsh environments were expected, for instance ABS controls. Believe me auto companies do not accept unusually high prices from their suppliers. Smart equipment should have a failure rate immeasurably low, unless it is subject to unexpected conditions like surges. Failure to design in surge protection is another issue entirely, and I have no idea how well these equipments are designed. The smart part of these devices is pretty simple stuff. Failure rates of 5% aren't from componenet reliability issues, not even at 0.05%.
Bill Melendez's picture
Bill Melendez on Dec 21, 2010
Interesting discussion on consumer level product reliability. Most consumer electronic products are warrantied for 90 days or less (you must buy an extended warranty). That's because consumers do the "darnest" things to electronics. Consumer products tend to follow the path of least amount of BOM costs to maximize revenue generation. That basically means that the propensity for a product to fail is higher for cheaper products then those that cost more. Reliability can also be stated as the capacity for a product to be sustained. Sustainability is something smart grid companies avoid discussing since it does require clarifying costs not clarifying benefits. The difference between a consumer device and a utility device is the support and sustainability requirements as dictated by regulatory requirements. It these requirements that determine the level of reliability. MTBF rate is a great tool for "burning" in chips within a circuit. All chips and electronic components degrade as they are being used. Hence using MTBF is a general test for "when" that device would most likely fail (not, however, when it will actually fail). While electronic components are normally tested as such for tolerances and failure rates; when combined within a unique circuit, such as a smart meter or smart device, the "combined" circuit may have a higher failure rate then the individual components. Which brings up the fact that devices may pass all tests but fail prematurely once sold and operational.
Don Hirschberg's picture
Don Hirschberg on Dec 21, 2010
I think Murray raises a good point. The only time I ever experienced a failure in any electronic device was in my Timex –Sinclair 2K computer circa 1976, $50 at a Walgreen Drug store. I was delighted with it until the circuit board cracked. Replaced it with a 64kTandy which cost 50 times as much.
Frank Telenta's picture
Frank Telenta on Dec 22, 2010
The recent California Meter Study did not show an difference in Meter Failure Rates from a Smart vs. Traditional Device. Where are you getting this failrue rate of ten times a traditional devices? It sounds great for a Headline but it is not reality. When a meter reads, it reads accurately and when it communicates that data is a nother issue based upon the communication platform. If there are issues with the meter it is on the Communication Platform and not the meter reading accurately.

Be better, be accurate and not an alarmist!!!

Alok Misra's picture
Alok Misra on Dec 22, 2010
As we know it will be some time before the kind of reliability we used to have with earlier meters will return or probably never return!Electronic meter failures in certain cases have been tolerable but such meters have been expensive to say the least.It is matter of manufacturing technology to be employed that determine the life of the meter.Energy meter on the Wireless chip which I proposed some five years ago to a very large Chip Buliderfrom Continent and thety promised they will deliver has not come along yet.Such devices could have been more reliable becuse of the basic construction,What is the Failure rate of Microprocessors in Washing machines?If voltages are firm and no spikes come they can last upto ten years or more. However itall depends on the distribution system to which they are connected.So reliability of Electronic meters can not be seen in isolation- it has to be seen in relation to the system to which they are connected. Even the best of Electronic devices could fail in disturbed cnditions on the grid .
Bob Amorosi's picture
Bob Amorosi on Dec 22, 2010
Good points raised in these comments. One thing not defined is what exactly qualifies as a failure in a smart meter. A failure can be catastrophic where the meter completely shuts down due to surge damage or a component failure. Another failure may have nothing to do with the meter hardware but in any case is blamed on the meter when billing inaccuracies are the fault of software processing of meter data.

Electronics components reliability tends to be very high as posted above, however they can change over time. It is not uncommon today for components with multiple sources to be targets for cheaper less reliable counterfeits being made, usually from the far east. Their overall reliability in a given product circuit design is usually much lower than the individual components if the product is exposed to abuse in harsh environments. Smart Grid devices mounted outdoors or connected to power lines qualify as being in a harsh environment i.e. lightning surges, constant temperature cycling, etc. A certain amount of surge protection goes into a smart meter circuit design, but it is practically impossible for example to protect them from direct or very close lightning strikes.

Large groups of smart meters will turn off in power outages since many designs are not backed up with batteries. Although consumption data is generally not lost in this scenario, it does mean the loss of communication with them in a 2-way meter network.

Consumer devices that communicate with the Smart Grid must be treated like all other consumer products if consumers end up owning and using them. Product service life will depend heavily on the customer’s care in using them.

Kevin Lawless's picture
Kevin Lawless on Dec 22, 2010
Lots of good comments here, but I think there is a tendency to understate the breadth of the issue. Reliability encompasses the entire meter reading and billing system. Failures at any point typically reflect in the bill that customers receive and the the "smart" meter often gets the credit for creating the problem.

To put this in a broader context, it has taken utiltiies and their suppliers over a hundred years to create the level of reliability that exists in the traditional meter reading and billing system. Over many years, hardware has been hardened, software defects have been minimized, business processes have been developed and honed, organizations have been developed to handle day-to-day processes as well as manage "situations" that could reflect on reliability, and staff training has focused on the use of the specific tools in use at a particular utility.

Now in several states, rapid "smart" meter rollouts have been mandated (CA and TX for example). So in a very condensed timeframe, utilities are supposed to essentially replace all the infrastructure related to meter reading and billing including (but not an exhaustive list) meters and related hardware, software, rate structures, business processes, organizations, staff training, customer training, and staffing levels. In addition, as these changes are taking place, the traditional systems have to maintained and kept alive as well.

Now instead of 12 meter reads a year for the typical residential customer, the utility and its vendors may be managing approximately 35,000 reads a year per customer, assuming 15 minute interval reads. Instead of processing 3-5% meter change-outs per year, now 20% is required by mandate and it is not a replace one meter with a similar one meter type of situation. This scale change by itself needs to be managed carefully, but given all the other changes occuring simultaneously, the complexity level is quite high. When the billing system needs a major overhaul or replacement synchronously due to "smart" or demand response rate being mandated at the same time, the complexity in this process change has escalated severely.

Bottom line, measuring the reliability of the meter is just one small aspect of what utilities consider reliability. There are many points of failure in the data communications process, the meter data management systems, the billing systems, the business processes, staff and organizational capabilities and quality of customer communications about the changes. Even if MTBF for the meter equipment itself is very low, there are many other points of failure throughout the process that also need to be measured and hardened before the entire system can be considered reliable. Many measures will necessary to manage the entire process during roll-out as well as after the entire system is up and running.

In many cases, given the complexity of change, the breadth of change, and the pace of change that has been mandated in some cases, I suspect some utility executives would prefer heart surgery.

Gay Gordon-Byrne's picture
Gay Gordon-Byrne on Dec 23, 2010
Dear Readers: Much to consider from your comments. Allow me to comment and clarify:

For Bob, Alok, Bill and Kevin: I don’t think the difficulties of buying reliable equipment over price point are insurmountable. Reliability of smart grid devices should not be difficult to cost-justify if the consequences of failure are connected to the overall costs to operate the grid. If, as we postulate, the failure rate of 5% per year is real – for a utility operating 1 million end points is 50,000 meter failure per year. If each truck roll required to make a repair costs $300, lack of reliability has a very real cost to the utility of $ 15 million dollars per year above and beyond any other causes of meter failure such as vandalism or external damage. Over the ten year useful life of the project, this is $ 150 million dollars – real money in almost any setting. Selecting equipment with lower percentages of problems can clearly be justified – if the comparisons can be calculated.

For Jack: Good point. In Home Devices and “Enabling Technology” at any failure rate can be a disaster for utilities if they are called upon to keep that equipment in service. What provisions are made for service and support of the “energy orb”? If grannie cannot operate her washing machine – will she call Whirlpool or her energy provider? This type of service demand for auxiliary and non-critical devices would quickly challenge any utility. Unless utilities throw support of IHD’s on the consumer electronics vendors, the costs to support IHDs have to be of concern to the utility and by extension, the ratepayer. For Steve: I’ve found that the role of warranty and vendor responsibility is widely misunderstood. With IT products, the buyer has a very short term limited warranty for the product itself, and buys an extended warranty to cover additional services, including onsite repair, optionally. Since utilities typically pull and swap their own equipment problems, the vendor warranty can be longer in term, but remains limited to a parts replacement warranty. The real costs are still borne by the utility – since the vendor does pay for the true cost of rolling a truck for the event. Nor does the vendor experience any of the downstream problems of billing problems or outages. It is far less costly to avoid a warranty repair entirely. For Murray and Frank: Accuracy of the AMI meter is not in question. Completely agree that any weakness in the systems that connect the meter to the utility has to be avoided – and this includes communications equipment in addition to meters. If we have learned anything from 40+ years of electronics manufacturing it is that materials degrade, components fail, mistakes are made, and repairs are always needed. The convergence of IT into ET is a sea change in great need of systematic measurement so as to avoid problems – wherever they may originate.

Gay

Bob Amorosi's picture
Bob Amorosi on Jan 1, 2011
I predict utility companies will avoid supporting most consumer devices that interact with smart grid systems because they will never bear the cost without massive government handouts. Regulation of their businesses will forever preclude them from raising substantial extra money to pay for it. Consumer devices will continue to be very independent of utility company involvement unless specifically funded or mandated by governments.

The utility companies like it this way. They appear to prefer to provide their support for demand management and efficiency upgrade technologies for industrial and commercial customers, since they clearly get a much bigger bang for their support dollar directed towards much larger users of energy compared with individual residential customers. Pity us poor consumers who get left behind. The majority of residential customers will likely only see Time-Of-Use billing with the new smart meters, and little else for a long time to come.

Get Published - Build a Following

The Energy Central Power Industry Network is based on one core idea - power industry professionals helping each other and advancing the industry by sharing and learning from each other.

If you have an experience or insight to share or have learned something from a conference or seminar, your peers and colleagues on Energy Central want to hear about it. It's also easy to share a link to an article you've liked or an industry resource that you think would be helpful.

                 Learn more about posting on Energy Central »