Raghu Bharadwaj

Posted on Sep 25 Originally published at techveda.live

Why a Watchdog Is Not a Reliability Strategy

#linux #embedded #kernel #reliability

A hardware watchdog converts an unknown failure into a reboot. That is containment, not a reliability strategy, because the reset names no fault and erases the evidence that would have named it. The minimum a shipping product needs is ordered timeouts — kernel detection thresholds set below the watchdog timeout, so the kernel gets to say what broke before the board resets — plus a persistent console region in RAM, which is the only thing that survives a reset the kernel did not perform itself. The watchdog stays; it just moves to the last rung instead of being the whole ladder.

"We have a watchdog" is one of the most common answers to "what happens if the device locks up in the field", and it is not a wrong answer. It is an incomplete one. A watchdog guarantees exactly one thing: a device that stops petting it will restart. It makes no statement about why the device stopped, whether restarting helps, or whether the same thing will happen again in six hours. Treating that single guarantee as a reliability strategy is the mistake, and it is a mistake that survives to production because the symptom it produces — a device that is usually up — looks like success.

The context: what a reset actually buys

The watchdog's value is real. It handles the failure class where software has stopped making progress and no operator is present, and it does so without requiring anyone to understand the fault first. For a remote device that is worth having.

The limits are equally concrete. A reset acts on one symptom — the petting stopped — so every failure that keeps the petting loop running is invisible to it: a service producing wrong output, a sensor path that has silently degraded, a disk filling up, one of five daemons dead while the other four keep the system looking alive. A reset also destroys the machine state that would explain the fault. RAM is gone, and unless something wrote to persistent storage first, the next boot starts with no account of what happened.

A third limit is easy to miss: a reset has no memory. The board that resets every six hours and the board that has reset once in a year look the same in a naive fleet view — both are up. Without reset-cause accounting, a reliability regression is indistinguishable from a healthy fleet.

The options for a reliability strategy

Option 1 — Reset and forget

Arm the hardware watchdog, pet it, ship. Nothing records why a reset happened.

For: almost no engineering cost, no reserved RAM, no boot-time work, and it genuinely recovers the common lockup case unattended.
Against: every field failure arrives as "it rebooted" with no evidence attached. You cannot tell a watchdog reset from a power glitch, or two different bugs apart. Debugging then depends on reproducing the fault on a bench unit, which for timing- and load-dependent faults may never happen.

Option 2 — Detect, record, then reset

Keep the watchdog, but let the kernel's own detectors run first and make sure their output survives. Linux already ships the detection half. The softlockup detector reports a CPU looping in kernel mode without yielding; its threshold is 2 * watchdog_thresh, and watchdog_thresh defaults to 10 seconds, so that is 20 seconds. The hardlockup detector reports a CPU looping with interrupts disabled, using NMI perf events where the architecture supports them or an SMP "buddy" arrangement where it does not — worth checking on single-core parts, where neither is available. The hung task detector reports a task stuck in uninterruptible D state, controlled by hung_task_timeout_secs, whose default of 120 seconds comes from CONFIG_DEFAULT_HUNG_TASK_TIMEOUT.

By default these detectors log and let the system continue. softlockup_panic and hung_task_panic turn a detection into a panic instead, and the kernel.panic sysctl decides what follows: zero loops forever, a negative value reboots immediately, a positive value reboots after that many seconds, with 60 recommended when a software watchdog is in use. panic_on_oops covers the kernel hitting an oops or a BUG and otherwise continuing in an undefined state; whether it is on depends on whether the kernel was built with CONFIG_PANIC_ON_OOPS.

The recording half is pstore with a ramoops backend: a reserved region of RAM declared in the device tree with compatible = "ramoops". Two of its sub-regions matter here, and they behave differently in a way that decides what a watchdog reset leaves behind. record-size backs the oops and panic records, which the kernel writes through its kmsg dump path at the moment it dies; after the next boot they appear under /sys/fs/pstore as dmesg-ramoops-N. console-size backs a separate region that the ramoops console backend writes continuously as kernel messages are printed, and it appears as console-ramoops-0.

A hardware watchdog reset is not a panic. The SoC simply restarts, so the kernel never reaches its dump path and no dmesg-ramoops record is produced at all. What survives is the console region, because it had already been written message by message. That is the reason console-size is not optional on a product with a hardware watchdog: without it, a watchdog reset leaves pstore empty and you are back to Option 1. The kernel documentation raises this scenario when explaining ramoops' optional software ECC — after a hardware reset triggered by a watchdog, RAM may be somewhat corrupt but is usually still restorable, which is why ECC is offered for exactly this case.

For: resets arrive with evidence attached — a full dmesg record where the kernel panicked, the console tail where the watchdog got there first — and the causes become distinguishable. Faults that only appear in the field become debuggable from the field.
Against: it costs a reserved RAM region, device tree work, and the discipline of reading pstore on every boot and shipping the contents somewhere. The panic sysctls also make the device reboot where it previously limped along — correct behaviour, and still a change your test team will notice.

Option 3 — Contain and degrade

Add defined behaviour per component above Option 2: each service has a declared failure response, the product has a stated degraded mode, and a whole-device reset is the last rung rather than the first. A failed non-essential daemon is restarted and reported; a failed essential one moves the product into a safe, reduced state an operator can observe.

For: most single-component failures never become device-wide outages, and the ones that do are announced rather than inferred.
Against: this is real design work. Somebody has to decide, per component, what "still working" means and what the product does without it — and that cannot be delegated to a timer.

The decision

Option 2 is the floor of a reliability strategy for any product that ships unattended. The cost is a reserved RAM region and some device tree work, and it converts field failures from unreproducible into diagnosable. Option 3 is warranted wherever a reboot has a real cost — an interrupted process, lost buffered data, a visible outage — and that judgement belongs to the product, not the platform team alone.

Option 1 is defensible in one situation: a genuinely stateless device where a reboot is cheap, fast and invisible to the user, and where you have consciously accepted never diagnosing a recurring fault. That situation exists. It is rarer than the number of products relying on it.

Whichever option you choose, one implementation detail decides whether it works. Order your timeouts. If the hardware watchdog fires before the kernel's detectors reach their thresholds, you get the reset without the diagnosis, and Option 2 collapses back into Option 1 while looking configured. With stock values the numbers leave little room: a softlockup is reported at 20 seconds and a hung task at 120, while embedded watchdog timeouts are commonly 30 seconds or less. Read the three values on a real unit:

raghu@techveda.org:~$ cat /proc/sys/kernel/watchdog_thresh
10
raghu@techveda.org:~$ cat /proc/sys/kernel/hung_task_timeout_secs
120
raghu@techveda.org:~$ cat /sys/class/watchdog/watchdog0/timeout
30

Be precise about when that mismatch actually costs you something, because it is easy to overstate. A single task stuck in D state does not stop the rest of the system. If systemd or your petting daemon is still being scheduled, the watchdog keeps being fed, the board never resets, and the hung task detector reports normally at 120 seconds. The ordering only bites when the fault also reaches the petting path — the petting process blocked on the same stuck I/O, a system-wide scheduler or filesystem deadlock, or a health check that deliberately stops petting because it decided the product is unwell. Those are precisely the failures a watchdog exists to catch, which is why the ordering is worth getting right even though it does not apply to every hang. Either lower hung_task_timeout_secs below the watchdog timeout or raise the watchdog timeout above it, and choose the two numbers together.

Then confirm that something actually survives a reset. Mount pstore and look at what is there after a deliberately induced panic, and again after a deliberately starved watchdog:

raghu@techveda.org:~$ mount -t pstore pstore /sys/fs/pstore
raghu@techveda.org:~$ ls /sys/fs/pstore
console-ramoops-0
dmesg-ramoops-0

After a panic you should see both. After a watchdog reset you should see only console-ramoops-0 — and if you see nothing at all, the reserved region is misconfigured. You would rather learn that on the bench than after a field return.

Consequences

You will find out your fleet is less reliable than you thought. Reset-cause accounting usually reveals resets nobody was counting. That is the reliability strategy working, and it still arrives as bad news in a release meeting, so set the expectation before the data does.

Panic-on-detection trades availability for diagnosability. A device configured to panic on a soft lockup restarts where it previously recovered on its own. Usually the right trade for a product you have to support — and still a trade.

The reserved RAM region is permanent. It comes out of the memory budget on every unit for the life of the product, and on a tightly specified board it competes with something real.

Evidence is only useful if it leaves the device. A pstore record nobody reads is the same as no record. Someone must own the path from /sys/fs/pstore to a place an engineer looks, and that is product work, not kernel work.

None of this replaces the watchdog. It is tempting to say the kernel's detectors cover everything a hardware watchdog covers, and that is wrong in both directions. A kernel that has stopped scheduling is exactly what the softlockup detector is built to catch — it runs from an hrtimer, in interrupt context, and does not need the scheduler. What defeats the kernel's detectors is losing the mechanism underneath them: interrupts disabled on every CPU, no NMI and no second CPU for the buddy detector, a clock or power fault, or a detector that was never built in. The hardware timer sits outside all of that, which is why it belongs at the bottom of the ladder rather than being mistaken for the whole of it.

Key takeaways

A watchdog answers one question — did progress stop — and nothing else. That is containment, not a reliability strategy.
Order your timeouts: detection thresholds must sit below the hardware watchdog timeout, or the reset beats the diagnosis — in the failures where the hang also stops the petting, which are the ones the watchdog is there for.
Stock values leave little room. A softlockup is reported at 20 s and a hung task at 120 s; embedded watchdog timeouts are frequently shorter than both.
A watchdog reset is not a panic, so it produces no dmesg-ramoops record. Set console-size as well, and read /sys/fs/pstore on every boot.
Count reset causes. A device that is usually up and a device that is reliable are not the same measurement.

Frequently asked questions

Should I disable the hardware watchdog if I configure the kernel detectors?
No. The kernel's detectors do not need the scheduler — the softlockup detector runs from an hrtimer — but they do need the mechanism underneath them. Interrupts disabled on every CPU, no NMI and no second CPU for the buddy detector, or a clock fault all defeat them. The hardware timer sits outside all of that.

Why would the hung task detector never fire on my board?
If the hang also stops the watchdog being petted, the board resets at its timeout — often 30 seconds — long before the detector's default 120-second threshold. If petting continues, the board does not reset and the detector reports normally. Lower hung_task_timeout_secs below the watchdog timeout, or raise the watchdog timeout above it.

Does a watchdog reset leave anything readable in pstore?
Only the console region. A watchdog reset is not a panic, so the kernel never runs its kmsg dump path and no dmesg-ramoops record is written. The console region is written continuously as messages are printed, so it survives. Set console-size, not just record-size.

What does the kernel.panic sysctl do after a detector panics?
It decides what follows. Zero loops forever, a negative value reboots immediately, and a positive value reboots after that many seconds. The kernel documentation recommends 60 when a software watchdog is in use.

DEV Community