DEV Community

Zoran Stankovic
Zoran Stankovic

Posted on

A firmware fix instead of a recall: root-cause engineering for a deployed wearable

The wearable was already in the market, on athletes, when the reports started. Units power-cycling at random. Unreliable charging while docked. Data going missing during real competition use, which is the worst possible moment. The product was already in customers’ hands. So the two obvious answers, redesign the hardware or recall the units, were both slow and both expensive.

This is the situation nobody wants. It is also where the difference between guessing and root-cause engineering gets very real. We took this device, in exactly this state, and fixed it in firmware. No hardware redesign. No recall.

When a deployed product misbehaves, the tempting move is to chase the symptoms. Add a retry where charging fails. Add a delay where it power-cycles. Patch each visible failure as it gets reported. That road leads to firmware that is a pile of workarounds, and to a problem that keeps coming back in a new shape. We have seen products die slowly that way.

So we went after the cause instead. We stress-tested the device on purpose, to force the failures under controlled conditions, rather than wait for them in the wild. The instability traced back to electrostatic discharge. The device’s input pins did not have enough hardware filtering. And the firmware did not have the signal conditioning to reject the transients that resulted. That single cause explained everything at once: the random power cycling, the charging trouble, and even some inconsistent button behaviour that had been logged as a separate complaint. One root cause, several symptoms. That is usually the shape of these problems. It is also why chasing symptoms one by one never converges.

Once we understood the cause, the fix could be precise. The units were already deployed, so hardware was off the table. The solution had to live entirely inside the existing hardware and its firmware. We built a firmware module for signal stabilisation. It used interrupt-driven validation with defined timing windows, in the range of 80 to 100 milliseconds. So a transient from an ESD event was recognised as noise and rejected, instead of being acted on as a real signal. It dropped into the existing firmware without disturbing the core functions. It removed the random power cycling and the charging instability. The button problem, same root cause, went away with it. The fix was deployed as a firmware update to devices already in use.

Now the honest part, because it would be easy to turn this into a story about firmware saving everything. It does not always. Some field failures have no firmware solution. If the hardware is fundamentally wrong, firmware can sometimes buy time, but it cannot provide a real fix. Pretending otherwise just delays the reckoning and burns the customer’s trust along the way. Part of the value of proper root-cause work is that it tells you honestly which case you are in. You find the cause first. Then you decide whether firmware can address it. You do not just hope a patch holds and find out in the next batch of returns.

Two things made the firmware fix possible here. First, the cause was a signal-integrity problem that firmware could compensate for, which is not always the case. Second, the firmware was built well enough to accept a clean new module without destabilizing the rest of it. That second part is never luck. It is a result of how the firmware was built in the first place.

That is where this connects back to how we work. The whole point of the validation gate in PrecisionPath 7™, our seven-gate development process, is to force failures like this one before the product ships. We use stress testing and EMC and ESD pre-compliance, so they never reach a customer. When something does slip through, the work after launch is the same disciplined root-cause method, now pointed at deployed units. And the firmware-architecture decisions made back at the detailed-engineering gate are what decide whether a clean late fix is even possible. A well-built firmware base is what lets you drop in a stabilization module instead of opening a recall.

A misbehaving product in the field is not automatically a recall. Sometimes it is a hard week of honest root-cause work and a firmware update. The trick is knowing the difference. And you earn the right to the easy ending by building the product properly the first time.

Top comments (0)