One Gradient Spike, Six Batches to NaN: Debugging a Deterministic Neural-Network Training Failure

The first visible NaN appeared in the final linear layer at batch 46.

By then, the model was already effectively destroyed.

Six batches earlier, every tensor was still finite. The loss was finite. The gradients were finite. The momentum buffers were finite. Nothing had crossed the clean boundary between a valid floating-point value and NaN.

But the training state was already in a runaway regime.

This is how an aggregate-analysis failure led me backward—from a corrupted checkpoint, to one reproducible batch, to the finite gradient spike that began the collapse.

The experiment

I was running a sequential-training experiment on MNIST using a small CNN.

The experiment compared two training histories:

• SABC: digits 0–4, then digits 5–9, followed by a shared balanced 0–9 relaxation phase.
• SBAC: digits 5–9, then digits 0–4, followed by the same relaxation phase.

As a mechanism-control experiment, I replaced the existing ReLU activation with LeakyReLU while keeping the rest of the protocol fixed.

The setup was:

• SimpleCNN
• no normalization
• LeakyReLU slope 0.01
• SGD
• learning rate 0.05
• momentum 0.9
• no weight decay
• deterministic seeds: 101, 202, 303, 404, 505
• two histories per seed
• ten total runs

Four of the ten runs became numerically invalid:

• SABC seed303
• SBAC seed101
• SBAC seed202
• SBAC seed303

In all four cases, the first known bad checkpoint was epoch 11.

Epoch 10 was the end of the first training domain. Epoch 11 was the first epoch after switching to the second domain.

That timing was suspicious, but not enough to establish a cause. The instability could depend on the interaction between the activation function, learning rate, momentum, domain switch, model state, task order, and random seed.

The validator found it first

I did not discover the problem because the training loop printed NaN.

A later aggregate-analysis validator rejected two checkpoints that should have represented the same model state: the final sequential-training checkpoint and the step-zero checkpoint of the relaxation phase.

At first, this looked like a serialization or bookkeeping issue.

A direct checkpoint audit showed that the affected files contained NaN and Inf values.

For three failed runs, all 421,642 parameters were NaN in the first bad checkpoint. In SABC seed303, the checkpoint contained 421,610 NaNs and 32 positive infinities.

The validator had not caused the problem. It had prevented corrupted runs from entering the scientific analysis.

What I initially got wrong

The original runs had been launched in parallel on one GPU.

My first suspicion was therefore environmental:

• GPU contention
• multiprocessing interference
• nondeterministic CUDA behavior
• a transient scheduling or hardware issue

To test that, I reran one known failing configuration—seed101-SBAC—alone with a detailed numerical tracer.

It failed again at exactly:

• epoch 11
• phase 2
• phase data A
• batch 46
• after the forward pass
• first failing tensor: fc2_logits

The isolated repeat also used the same input hash, label hash, label histogram, and non-finite counts.

The failing logits contained:

• 128 NaNs
• 128 positive infinities
• 384 negative infinities

Reproducing the same failure in isolation, on the same batch and at the same network stage, makes a one-off GPU collision very unlikely.

What actually happened

The first non-finite tensor appeared at batch 46.

But the failure began earlier.

At batch 39, the system still looked ordinary:

• loss: about 3.03
• global gradient norm: about 9.11
• parameter magnitudes still small

At batch 40, the trajectory changed sharply:

• loss jumped to about 44.59
• global gradient norm jumped to about 483.74
• maximum gradient reached about 266.7

Everything was still finite.

After the optimizer step, the largest momentum-buffer value rose from about 9.13 to about 269.93.

The following batches did not recover.

Batch	Loss	Global gradient norm	Max FC1 activation	Interpretation
39	3.03	9.11	2.18	Ordinary-looking state
40	44.59	483.74	17.10	First large finite gradient event
41	1.48 × 10^3	988	913	Runaway growth becomes visible
43	1.04 × 10^5	7.82 × 10^4	5.92 × 10^4	Strong amplification
44	4.71 × 10^8	6.25 × 10^7	1.02 × 10^6	Extreme but finite
45	3.23 × 10^18	1.19 × 10^15	6.87 × 10^15	Model effectively destroyed
46	—	—	1.97 × 10^36	Final logits overflow float32

Immediately before the failing forward pass:

• fc1.weight magnitude was around 10^12
• fc2.weight magnitude was around 10^13
• momentum buffers were around 10^14–10^15
• the maximum FC1 activation was about 1.97 × 10^36

The final linear layer then overflowed float32 and produced NaN and signed Inf values.

The trace supports the following sequence:

large but finite gradient event
→ rapid growth in momentum and parameter scale
→ rapidly growing activations
→ float32 overflow in the final linear layer

Momentum appears to have amplified the instability, but the trace does not establish what caused the initial gradient spike.

Likewise, fc2_logits was the first non-finite tensor, but it was the terminal symptom, not necessarily the origin of the failure.

Finite is not healthy

A standard safeguard is:

if not torch.isfinite(tensor).all():
    stop_training()

That check is necessary, but late.

At batch 45, the model had:

• loss around 10^18
• gradient norm around 10^15
• momentum buffers around 10^14
• activations around 10^15

Every value was still finite.

A binary finite/non-finite check would classify the state as valid. In practice, the model was already beyond recovery.

This suggests a more useful distinction:

1. numerically finite
2. numerically extreme
3. numerically non-finite

The transition from the first state to the second is where the most useful debugging information exists.

By the time NaN appears, the original trigger may be several optimizer steps in the past.

Why the debugging workflow mattered

The investigation followed a simple sequence:

1. Aggregate analysis failed.
2. A pair validator caught a checkpoint inconsistency.
3. A checkpoint audit found NaN and Inf values.
4. A fail-fast numerical tracer was added.
5. A known failure was reproduced in isolation.
6. The first non-finite tensor was localized to one batch and one network stage.
7. Earlier batches revealed the actual runaway dynamics.
8. A finite control run showed that the tracer did not automatically cause failure.

That workflow is reusable.

For numerical debugging, I would now monitor:

• global gradient norm
• maximum absolute gradient
• parameter norms
• momentum-buffer norms
• activation magnitudes
• model state before and after the optimizer step
• deterministic hashes of the input and labels

The hashes were especially useful.

“It failed at batch 46 again” is weaker evidence than:

“It failed at batch 46 again on the exact same input and label tensors.”

What the evidence supports

The careful conclusion is:

Under this sequential-training protocol, LeakyReLU with SGD at learning rate 0.05 and momentum 0.9 showed deterministic, seed-dependent numerical instability in 4 of 10 runs.

The evidence does not show that LeakyReLU universally causes gradient explosions.

It also does not show that momentum, the domain switch, the task order, or the final linear layer independently caused the failure.

The instability may depend on an interaction between:

• activation function
• learning rate
• momentum
• domain switch
• model state
• random seed
• task order

A control run, seed404-SBAC, completed the same requested epoch range without non-finite values.

So the protocol was not universally unstable.

But a 4/10 failure rate made this LeakyReLU branch unreliable for paired scientific analysis under the tested configuration.

The existing ReLU results were not affected by this failure.

Conclusion

The most useful artifact from this experiment was not the NaN checkpoint.

It was the finite trace leading into it.

At batch 39, the system looked ordinary.

At batch 40, the gradient norm jumped from roughly 9 to roughly 484.

At batch 45, the model was still technically finite but already had gradients around 10^15.

At batch 46, a hidden activation reached approximately 10^36, and the final linear layer overflowed float32.

The main lesson is simple:

A finite tensor is not necessarily evidence of a healthy training state.

NaN checks are essential, but they are late alarms.

The final NaN showed where the computation stopped being representable.

The six finite batches before it showed how the model got there.