In embedded systems and edge AI, data corruption in an exported model usually points to the usual suspects: an unmanaged pointer, a buffer overflow, a corrupted file write. This time the culprit wasn't memory — it was the IEEE 754 floating-point specification reacting, quite correctly, to a lot of training.
This is a post-mortem on how an exported MNIST model in C++ ended up filled with impossible-looking values like 8.353526e-301, how we isolated the cause, and why the fix matters on microcontrollers without an FPU.
The Problem
While exporting a trained model from Hasaki's C++ pipeline, I noticed anomalous values in the generated header for a standard MNIST network (784 inputs, 128 hidden neurons, 10 outputs, Adam optimizer, 0.3 dropout, 0.0001 L2).
The weights weren't zero, and they weren't normal floats in the expected range either. Some had exponents around e-300, e-301. In a trained network, a weight at that magnitude is not signal — it's noise that shouldn't exist.
Failed Attempts
Before finding the actual cause, we ruled out the usual suspects one by one.
Torn read? First guess was that the exporter had read MODEL.txt while a background training process was still writing to it. Ruled out: training was fully stopped before exporting, and the corruption persisted identically.
Uninitialized variables? Checked whether l2Lambda or neuron.output lacked proper initialization in the C++ classes. Ruled out: l2Lambda had a default member initializer (= 0.0), and neuron.output was already covered by a documented fix from earlier work.
Memory corruption at runtime? Ran the full training pipeline under AddressSanitizer and UndefinedBehaviorSanitizer. Result: zero errors. The binary was completely clean under both sanitizers.
That last result looked like a dead end. It wasn't — it was the clue. If neither sanitizer flags anything, you're probably not looking at a memory-safety bug at all. You're looking at valid, if unwanted, numerical behavior.
Isolating the Variable
To isolate the real variable, we ran two controlled experiments comparing a small dataset against the full one:
- Small dataset (1,600 samples), Debug build: clean.
- Small dataset (1,600 samples), full Release build (
-O3 -march=native -mavx2 -mfma -ffast-math): also clean.
That second result ruled out compiler optimization flags as the cause — a real possibility given how much -ffast-math and FMA fusion can change rounding behavior.
The one variable that actually differed between clean and corrupted runs was the number of weight-update steps: roughly 1,025 on the small dataset versus roughly 30,750 on the full 47,999-sample dataset. Corruption tracked step count, not dataset content, build flags, or random seed.
What We Think Is Happening — and What We Can't Fully Explain Yet
The leading hypothesis: some ReLU neurons die during training — their output stays at zero for every input, so they stop receiving real gradient from the loss function. Their incoming weights, though, are still touched by the optimizer on every step, driven only by the L2 term, which consistently pushes each weight's magnitude toward zero. Left unopposed for tens of thousands of steps, that's a plausible route for a weight to end up far outside the normal float32 range and into IEEE 754 subnormal territory.
That part of the story is consistent with everything we observed: corruption scaled with update count, was deterministic across different random seeds, and was unaffected by build flags.
Where I have to be honest about a gap: I originally wrote this up as simple geometric decay, w -= lr * l2 * w, and did a back-of-envelope check using our actual hyperparameters (lr = 0.001, l2 = 0.0001). That check doesn't hold up. A plain SGD-style decay factor of (1 - lr·l2) per step, compounded over ~30,750 steps, works out to roughly a 0.3% reduction in magnitude — nowhere close to what's needed to push a normal-range weight down into e-300 territory.
Two things complicate the naive picture further:
-
We're using Adam, not plain SGD, and Adam doesn't apply the raw gradient — it normalizes the update by an estimate of the gradient's own variance. For a dead neuron whose only nonzero gradient signal is the small, consistently-signed L2 term, Adam's normalization can produce a step size closer to a fixed fraction of the learning rate itself, rather than a clean multiplicative shrink of
w. That's a fundamentally different dynamic from geometric decay, and we haven't derived a closed-form model for what it actually does towover tens of thousands of steps. - We haven't instrumented this in-training. We know corruption appears, we know it scales with step count, and we know dead ReLU units plus an unopposed regularization term is the most plausible driver in general — but we have not watched a specific weight's trajectory step by step to confirm the exact path it took from a normal value down to a subnormal one.
So the fair summary is: the qualitative mechanism (dead neurons, nothing to counteract the regularizer, monotonic drift toward zero, effect compounding with more update steps) fits the evidence well. The quantitative mechanism — the exact math of how Adam's normalized updates on a near-zero, consistently-signed gradient produce a magnitude of e-300 specifically — is something we have not nailed down, and a naive SGD-style estimate does not reproduce it. That's an open question, not a solved one.
The fix itself doesn't depend on nailing the exact mechanism: we added an explicit flush-to-zero threshold inside cFloatLiteral(), the function in the C++ exporter (ModelExporter.cpp) responsible for formatting every weight and bias before it's written to the generated header. Any value with magnitude below 1e-30 gets snapped to exact 0.0f.
Why an Earlier, Longer-Trained Demo Never Showed This
One question this raises: there's already a working MNIST demo built with an earlier version of the exporter, trained with the same architecture, same Adam optimizer, same L2 value, for up to 50,000 epochs — well past the update count where we saw corruption. Why didn't it show the problem?
Because that demo was exported as int8, not float. The int8 path quantizes every weight against a per-neuron scale and rounds to the nearest integer in [-128, 127] (std::round(floatVal / scale)). A weight at any near-zero magnitude — whether it's a completely normal small value or a subnormal one at e-300 — rounds to integer 0 either way. The quantization step can't represent a subnormal bit pattern at all; it isn't fine-grained enough to distinguish "very small" from "catastrophically small." So whatever was happening to those weights during training was very likely happening in that model too — it just never became visible, because only the float export path preserves enough precision to expose it as garbage instead of silently rounding it away. This part of the reasoning doesn't depend on the exact decay mechanism above — it holds regardless of which specific process drove the weights toward zero.
Results
Verified against the same corrupted checkpoint, before and after the fix:
- Before: exponents as low as e-301, e-302.
- After: minimum exponent of e-23. Zero subnormal values anywhere in the exported header.
- 13,670 weights went from subnormal garbage to exact zero.
These numbers are direct measurements on the exported header, independent of which causal story turns out to be correct.
Lessons Learned
When a sanitizer comes back clean, that's information — change your hypothesis. ASan and UBSan are built to catch memory-safety bugs. A clean run under both doesn't mean "no bug," it means "probably not that category of bug." That's what pushed us from "where's the corrupted memory" to "what's the actual math doing."
Isolating one variable at a time beats reading code by eye. We had three plausible-sounding code-level hypotheses and all three were wrong. What actually worked was running the same training under different controlled conditions (dataset size, build type) until only one variable correlated with the outcome.
A plausible-sounding mechanism still owes you real arithmetic. It's tempting to stop once you have a story that fits the shape of the evidence — dead neurons, unopposed decay, more steps means more corruption. But when I actually ran the numbers on the specific formula I'd written down, they didn't support the magnitude we measured. Catching that gap matters more than the story sounding right; the fix here didn't need the mechanism fully resolved, but the write-up almost shipped with a wrong explanation dressed up as a settled one.
Worth noting, without overclaiming it: subnormal float handling is a well-documented property of many software floating-point implementations — it can require extra branching in the emulation routines compared to normal-range values. We didn't benchmark this specifically on the ESP32-C3's soft-float routines for this project, so I won't put a number on it here. But it's a real enough concern that flushing dead weights to exact zero is worth doing on general principle for anything targeting an FPU-less microcontroller, independent of whatever the exact cycle cost turns out to be.
Next Steps
The flush-to-zero fix addresses the symptom at export time, which is enough for now. The next logical step, if I pick this back up, is instrumenting dead-neuron detection during training itself — logging which neurons produce zero output across an entire epoch, and tracking a sample of their incoming weights step by step — to actually confirm the mechanism instead of inferring it from isolation experiments and an unverified decay formula. That would also let me work out the real closed-form (or at least numerically simulated) dynamics of Adam's normalized update on a small, consistently-signed gradient, which is the piece I currently can't derive cleanly. It would also open the door to real pruning (removing dead connections before export) instead of just zeroing them out after the fact.
Back when I was learning BASIC on a TI-99/4A, a professor of mine used to say: computers do what you tell them to do, not necessarily what you want them to do. Garbage in, garbage out.
Four decades later the hardware moved from 16-bit home computers to 32-bit RISC-V microcontrollers, but the principle hasn't moved a single bit. A network decaying into subnormals over 30,000 steps isn't broken — it's operating exactly within the rules of IEEE 754. The job isn't hoping the hardware forgives sloppy assumptions, and it isn't hoping a good-sounding explanation forgives sloppy arithmetic either. It's knowing the metal — and your own math — well enough that nothing about it surprises you.
Top comments (0)