Your Products Are Overrated

Jul 15, 2019
Your Products Are Overrated

Dubious performance claims do a disservice to the industry

By Alex Baker

You might remember the old cartoons depicting various iterations of a tree swing—as marketing requested it, as sales ordered it, as engineering designed it—each panel showing the many ways an order for a simple swing could be botched, with the final panel showing what the customer wanted: a tire, hanging by one rope, from a tree branch. Inevitably these cartoons have engineering designing something nonsensical, and marketing overselling the features, but the cartoons usually treat each corporate department with contempt.

The swing engineers had noble intentions. They tested and spec’d a durable seat and hardware and provided clear documentation informing proper installation while leaving out the extraneous. Proud of their design, they advised marketing that this product will swing higher, faster and for longer than any other. Nevertheless, all is lost as their reputation is one of a pile of rope and wood laying on the ground beneath the tree.

Never did the engineers envision that their top product FAQ would be Where should I attach the swing? Whether it be budget, lack of time, laziness or a twisted sense of humor, they’re compelled to do nothing, content to see their design defeated at nearly every installation. And it was all avoidable, had they included that one exceedingly minor mounting detail they thought would surely be obvious to the customer.

The swing marketers, having a keen sense of the competition, brought to market not one, not two, but three stacked seats per swing, as the company and its competitors convinced customers of the supposed benefits of this “feature.” The tri-seat product invites both teenage and professional acrobatic multi-rider tomfoolery, while legal advises that overstated product claims and the risk of injuries equate to profound legal risk for the company. Yet product demand remains strong, as customers still haven’t realized the product’s inherent limitations. The marketers swiftly ship these silly swings while planning the next generation four-seat model.

This is how I see the solid-state lighting market today.

The SSL general illumination market has run amok with decently engineered LED luminaires hampered by invalid luminous flux maintenance (aka “lumen maintenance”) claims that are statistically dubious at best, fraudulent at worst, and like the swing cartoon, everyone is at fault for the looming customer disappointment. Those who misuse IES TM-21—willfully or unwittingly—are setting false expectations that will result in disappointment and a black eye for this technology. And despite our don’t repeat the mistakes of CFLs mantra, from our big-city factories to the regional metal benders, from publicly traded multi-national corporations to the family-owned legacies, nearly all of you are misusing TM-21 to some degree. The global lighting industry is shooting itself in the foot by throwing away this valuable tool.

IES TM-21-11 was created to help inform product life claims and warranties. It was understood early on that product longevity would be a key selling point of solid-state lighting. The LEDs would not typically fail abruptly (“catastrophically” to some) but would instead simply lose light output and/or shift chromaticity over time. Industry envisioned LED products lasting 50,000 hours. Market transformation programs, eager for cost effective, persistent energy savings solutions without the headaches of fluorescent technology were eager for a new vehicle with which to move the market. Energy Star and the DesignLights Consortium needed metrics that would enable LED product incentives that public utility commissions could approve. Such approval could only be granted if the SSL products could in some way prove their life expectancy. Given the extraordinary life expectations for this new technology, it was clear to all that traditional lamp life testing methods (i.e. burning n lamps until 50% fail) would not be acceptable.

By way of an MOU with the DOE, the IES Testing Procedures Committee (TPC), developed a two-step solution: mathematical extrapolation of empirical luminous flux maintenance data (erroneously “lumen maintenance” at the time). In 2008 the IES published LM-80-08: IES Approved Method: Measuring Lumen Maintenance of LED Light Sources, which described “the measurement of lumen maintenance of inorganic LED-based packages, arrays and modules” within LED lighting products. Three years later the IES published TM-21-11: Projecting Long Term Lumen Maintenance of LED Light Sources, to project the data obtained through LM-80 testing. With these in place it was possible to project how many hours until a given LED source would reach 30% depreciation, or 70% luminous flux maintenance, coined L70. This wouldn’t address LED driver lifetime or other potential luminaire failure points, but it was a start.

Here’s the problem. There is one rule in IES TM-21-11 that is sacrosanct. It is absolutely crucial to the proper use of this projection method. To violate this rule is to venture into the land of make believe where numbers are meaningless, however large and commercially advantageous they may be. To ignore this rule is tantamount to writing down whatever large number you prefer and calling it hours to L70.

As per TM-21 section 5.2.5, it is the famed 6X rule: “luminous flux values must not be projected beyond 6 times the total test duration (in hours) of measured data.” This rule is then referenced nine times in the document. Nine times. In most instances the TM-21 calculations provides three results, in which p is the percentage of initial light output maintained, and D is the LM-80 test duration total in hours divided by 1,000 and rounded to the nearest integer:

Calculated Lp (Dk) e.g. Calculated L70 (10k) = 189,965 hours (Low statistical confidence and not intended for application use)

Projected Lp (Dk) e.g. Projected L70 (10k) = 130,131 hours (Interpolated by temperature, low statistical confidence and not intended for application use)

Reported Lp (Dk) e.g. Reported L70 (10k) > 60,000 hours (Interpolated by temperature and capped at 5.5 or 6 times the LM-80 test duration, high degree of statistical confidence, intended for application use)

TM-21 first produces the Calculated Lp (Dk) for the individual LM-80 case temperature(s). The Projected Lp (Dk) is then calculated, interpolating for the case temperature of the hottest LED package measured in situ within a lamp or luminaire. The backstop for the low statistical confidence in this value is addressed by implementing the 6X cap, resulting in the Reported Lp (Dk), the only TM-21 result with a high degree of statistical confidence and intended for application use.

You’re a specifier examining L70 claims? The longest luminous flux maintenance life a vendor can legitimately claim for 10,000 hours of LM-80 data is L70 (10k) > 60,000 hours, period. The Calculated and Projected Lp values have no practical value in engineering and should not be viewed as legitimate product claims. These values are only useful for ensuring that two TM-21 calculators will produce the same results.

You’re an engineer whose vendor is claiming 300,000 hours and trying to substantiate it with a 10,000-hour LM-80 report? It’s not legitimate—call them on it. Request a photo of how the in situ temperature was measured (often required for certifications). Open a copy of TM-21-11 with them—ask questions—press them to explain. Better yet, tell them there’s been a mistake—you need the 50,000-hour LM-80 report, not the 10,000 hour one.

If you didn’t know this, your ignorance ends here. If you already knew this, have you been misleading your clients with empty, baseless claims? I’m sure you didn’t mean to.

The reasons for this 6X limit (sometimes 5.5X) are explained in detail in TM-21’s Annex D. Would 7X be meaningful? Maybe under some circumstances. 8X? More? Possibly, but these are not compliant with the current version of TM-21, so one cannot legitimately associate the IES or TM-21-11 with such projections. The IES Board of Directors recently noted this industry problem and issued IES Position Statement PS-10-18 in strong support of the statistical analyses underpinning the standard.

Where did things go wrong? The IES did not adequately educate the market on these details. It was not anticipated that L70, L80 and L90 would become terms that engineering, marketing, finance and legal would all need to use and understand. TM-21-11 doesn’t explicitly state that Calculated and Projected Lp values are as meaningful as the serial number on your DVD rewinder. Attempting to correct this and other problems, the TPC issued Addendum A to TM-21 in 2014, replaced with the corrected Addendum B in 2015, both stating:

Calculated and Projected Lp (Dk) “values are set for the purpose of calibrating the calculation tools . . . Due to the statistical uncertainty stated in Annex D, the calculated result in hours beyond reported projection limit set in Section 5.2.5 does not have any practical or meaningful value.” Addendum A stated that these values “are removed from” the TM-21 report. Addendum B was more explicit, stating: “The Calculated and Projected Lp (Dk) are not to be reported.

Sadly, though, addenda are difficult to enforce, and some manufacturers have created a phony competition that others feel compelled to participate in.

Manufacturing friends, I’ve studied your Lp claims for years. A valid TM-21 projection claim of 150,000 hours to LWHATEVER requires 25,000 hours of LM-80 data, nearly three years. With the maturation of SSL technology and commoditization of the market, I’ll accept that today there could be LED packages worth testing for three years. But from my experience in this market I very seriously doubt it, and 150,000 hours to Lp is likely the upper limit of truly legitimate TM-21 projections.

Unfortunately, this is just the beginning of your fantastical claims. There’s no shortage of 200,000-hour claims to L70, L80 and L90, for high bay, roadway, and tunnel luminaires. Roadway luminaire ratings for 250,000 hours to L70 are common.

300,000 hours to L70 for your gasketed low-bay, your roadway luminaire, your area light? That’s more than 34 years of continuous operation. What else is spec’d to last that long? The turbine at the power plant. Show me the 50,000-hour LM-80 report. You actually have a 50,000-hour LM-80 report? That’s 5.7 years of operating time. Does your vendor still sell the LED they supposedly began testing nearly six years ago? You should question that. What does their LM-80 report really mean?

What about a wall pack marketing “Calculated L70 > 560,000 hours” that also cites “10,000 hours of LM-80 testing per IES LM-80-08”? Sixty-four (64) years of useful light output predicted with only 14 months of data? Congrats on entering the horticulture market—those claims can be dried out to fertilize the lawn.

Finally, remember the LED package manufacturer’s press release touting 1 million hours to L70? Were we really to believe a claim of 114 years to L70? Really?

This phony competition to best the other guy’s statistically invalid exaggeration of a life claim has been going on for years, and it’s time to bring it to an end. So, specifiers, put your foot down and demand more information. I’m not suggesting you should require LM-80 reports for your entire fixture schedule, but claims that don’t pass the sniff test deserve your scrutiny. Your client expects your due diligence.

You’re an engineer who doesn’t like TM-21? You’ve two options. The first revision of TM-21, will soon be published as an American National Standard. It promises many improvements including an official IES TM-21 calculator that will be published with the ANSI standard (the Energy Star TM-21 Calculator is but one iteration, though it was never “the” TM-21 calculator). Or, gather your thoughts on ways to improve the standard, collect your data, and approach the IES Testing Procedures Committee with your proposal. Our Society thrives on the expertise and viewpoints of our membership.

Or don’t do anything, just give up, roll over, and let this soon-to-be-ANSI standard become a joke. Your competitor claims 1 million hours to L70? Well then go to market with 2 million. Then what will we trust? Who will we trust? And how?

All right, enough. Time to go swing.