DEV Community: b1ackjack12

I Slowed My IMU's Clock 4x and Accuracy Went Up

b1ackjack12 — Tue, 03 Nov 2026 01:25:53 +0000

The window-length experiment ended with a clean dose-response: every halving of my HAR model's 2.56-second window cost accuracy, down to −0.87 points and a −1.98 collapse for the worst subject at 32 samples. But cutting the window short is only one of two ways to shrink an input. The other is to keep the full 2.56 seconds and sample them with a slower clock — the same stretch of walking, described by fewer, sparser measurements. For a wearable, that second knob is the interesting one: the sample rate drives the sensor's power draw, the radio bytes, and the model's FLOPs, and 50 Hz is simply what the dataset shipped with. Nobody chose it for the task.

So: the usual harness, 50 paired subject splits, and the 50 Hz windows downsampled to 25, 12.5 and 6.25 Hz over the same time span. Each rate comes in two flavors, because downsampling has a right way and a lazy way — naive striding (x[:, :, ::k], just drop samples) versus proper decimation (low-pass filter first, then drop; scipy.signal.decimate). The gap between those two flavors is itself a measurement, of how much aliasing actually hurts on real signals.

Half the data is free. A quarter of it is a raise

rate	samples	flavor	accuracy	Δ vs 50 Hz	t	worst subject Δ
50 Hz	128	—	92.66 ±2.75	—	—	—
25 Hz	64	naive	92.75	+0.09	+1.6	+0.32
25 Hz	64	anti-aliased	92.73	+0.08	+1.2	+0.21
12.5 Hz	32	naive	92.84	+0.18	+2.1	+0.47
12.5 Hz	32	anti-aliased	93.12	+0.47	+4.3	+1.93
6.25 Hz	16	naive	92.37	−0.29	−2.8	−0.69
6.25 Hz	16	anti-aliased	92.57	−0.09	−0.8	−0.81

Three findings, in ascending order of surprise. First, 25 Hz is simply free: both flavors sit within noise of the baseline, so half the sample rate — half the bytes, half the conv-layer input — costs nothing. Second, at the bottom of the table, 6.25 Hz is where real information finally runs out, and how you get there matters: naive striding loses a statistically solid −0.29 while the anti-aliased version turns that into a statistical nothing (−0.09, t = −0.8). That gap is textbook aliasing made visible — above-Nyquist energy folding back into the band as noise when you skip the low-pass filter.

Third, and the reason this post exists: 12.5 Hz with proper decimation beats the full-rate baseline. Not "matches" — beats, +0.47 with a t of 4.3, 35 wins in 50 splits. And the gain concentrates exactly where this dataset hurts: the worst subject in each split improves by +1.93 on average, the same hardest-subjects margin that every augmentation experiment has been fighting for. A quarter of the data, same time span, better model — with the improvement delivered by the step everyone treats as a chore (the anti-aliasing filter), which at this rate doubles the naive flavor's gain (+0.47 vs +0.18).

The physics was on the label

None of this is mysterious after the fact; it's Nyquist arithmetic. Human gait has its fundamental around 1–2 Hz with discriminative harmonics mostly below ~6 Hz — and the UCI pipeline even low-pass filtered everything at 20 Hz before windowing. Sampling at 12.5 Hz (Nyquist 6.25 Hz) still covers essentially the whole band that distinguishes walking from climbing stairs; 50 Hz was oversampling the activity by an order of magnitude. Push to 6.25 Hz (Nyquist ≈ 3 Hz) and you finally start clipping real harmonics, which is precisely where the table's first significant loss appears. The improvement at 12.5 Hz plausibly stacks two effects I can't fully separate: the decimation filter throws away high-frequency sensor noise the activity never needed (a free denoiser), and the same fixed-size conv kernels now span 4x more of the gait cycle, seeing whole strides instead of fragments. Both point the same direction; I'll leave their shares unassigned.

Put today's table next to the window post's and the contrast is the real lesson. Thirty-two samples as a cropped window (0.64 s of context at 50 Hz): −0.87, worst subjects −1.98. The same thirty-two samples as a decimated window (full 2.56 s at 12.5 Hz): +0.47, worst subjects +1.93. Identical input tensors, shape-wise — a four-point swing depending on whether those samples cover the whole gait cycle or a fragment of it. The model doesn't need a fast clock; it needs the full sentence, even if every word is shorter.

For the config file: if you're shipping an IMU model, sweep the sample rate downward before you shrink anything else — my deployment numbers from the CPU serving post already showed this model class runs in 66 microseconds at batch 1, and a 4x shorter input only widens that margin while the sensor (the actual battery cost on a wearable) does 4x less work. Just don't skip the low-pass filter: it's one scipy.signal.decimate call at training time and a trivial IIR on-device, and it's the difference between a free lunch and a slow leak. Scope: one dataset whose signals were already cleanly filtered at capture, one small CNN, and subjects walking in a lab — a noisier consumer IMU may shift the cliff, which is an argument for running this same cheap sweep on your own data, not for trusting 50 Hz.

Originally published at b1ackjack12.github.io — an AI tech blog where every claim comes with a measurement. More experiments like this one live there.

Weight Decay 0.05: Catastrophe in Adam, Optimum in AdamW

b1ackjack12 — Tue, 03 Nov 2026 01:25:37 +0000

Every recipe on this blog since the optimizer face-off has carried weight_decay=5e-4 — a number I originally picked for SGD because every CIFAR repo on the internet uses it, then ported to Adam when Adam won, with exactly zero measurement of whether it belongs there. That's the precise shape of inherited config this dose-response series exists to audit. So: Adam at four doses spanning 2.5 orders of magnitude plus a zero control, AdamW at the matching doses, five seeds each on the usual 10-epoch CIFAR ResNet18 — forty-five runs, with each optimizer's arms compared against its own wd=0 baseline on identical seeds.

The reason to sweep both optimizers is that "weight decay" is secretly two different mechanisms sharing a keyword argument. In Adam, the classic implementation adds wd * w to the gradient, which then flows through the adaptive machinery and gets divided by the running RMS of the gradients. In AdamW, the decay is decoupled — subtracted directly from the weights, scaled only by the learning rate, never touching the adaptive denominator. Same config key, different algorithms. The question a sweep can answer is whether that difference is pedantry or points.

Two curves, two orders of magnitude apart

optimizer	wd	accuracy (5 seeds)	Δ vs own wd=0	t
Adam	0	89.32 ±0.29	—	—
Adam	1e-4	89.81 ±0.16	+0.49	+3.8
Adam	5e-4	89.77 ±0.33	+0.46	+3.4
Adam	5e-3	87.33 ±0.35	−1.99	−11.6
Adam	5e-2	64.97 ±5.39	−24.34	−10.1
AdamW	0	89.32 ±0.14	—	—
AdamW	5e-4	89.29 ±0.26	−0.03	−0.3
AdamW	5e-3	89.32 ±0.21	+0.00	+0.0
AdamW	5e-2	89.66 ±0.15	+0.34	+3.0

Read Adam's column first: it's the EMA-decay topology again — a shelf, a cliff, a crater. The benefit saturates at the smallest dose I tried (1e-4 already buys the full +0.49; 5e-4 adds nothing more), then 5e-3 falls off a cliff, and 5e-2 doesn't converge so much as thrash — 65% mean accuracy with a ±5.4 standard deviation, the largest variance any arm has posted on this blog. My inherited 5e-4, for once, sits safely mid-shelf; the audit acquits it.

Now read AdamW's column, which is the same curve shifted right by roughly two orders of magnitude. At the doses where Adam does all its living and dying — 5e-4, 5e-3 — AdamW registers nothing: deltas of −0.03 and +0.00, flat zeros. Its effect only begins at 5e-2, the exact dose that costs Adam twenty-four points. One number, two optimizers: catastrophe in one, optimum in the other. Copying a weight_decay between Adam and AdamW isn't tuning, it's copying a time constant between schedules — the same mistake this blog has now measured in its third costume.

The ~100x shift even has a plausible size, not just a direction. Decoupled decay shrinks weights by lr * wd per step — at lr 1e-3 and wd 5e-2, that's 5e-5 of each weight. Coupled decay rides the adaptive update, where the gradient (decay term included) gets divided by the running gradient RMS; if that RMS sits around 10⁻², the division inflates the decay's bite by roughly 100x, so Adam at wd 5e-4 lands at about the same effective shrinkage — 5e-5 — as AdamW at 5e-2. Two arms I measured as equally optimal, agreeing through arithmetic I didn't tune to make them agree. I'll flag it as consistency rather than proof (the RMS varies per weight and per epoch; that's the point of the coupling being a bug), but the curves' alignment is hard to read any other way.

The screening I owe a correction

One row above is a confession. September's face-off chose AdamW's dose by single-seed screening, and that screening said 0.005 beat 0.05. Today's five-seed sweep says the opposite: 0.05 is better by +0.34 (t = 3.0), and 0.005 is indistinguishable from no decay at all. The screening seed flipped a coin and I published the coin flip — exactly the failure mode the seed-noise post warned about and the nondeterminism post explained, now caught in my own pipeline. For the record, the correction doesn't change the face-off's verdict: AdamW at its true best dose reaches 89.66, which still only ties Adam's 89.77–89.81 at this budget. Adam survives as the house optimizer; its challenger just deserved a fairer fight than one seed gave it.

Config-file summary, in the series' usual shape. For Adam at lr 1e-3 on a run like mine: anything in 1e-4–5e-4 is on the shelf and the choice within it doesn't matter; 10x more is a cliff. For AdamW: the PyTorch default of 0.01 sits comfortably inside the active range (my 5e-2 arm is the shelf's measured edge in the good direction), and anything at Adam-ish magnitudes is a silent no-op — you think you're regularizing, and you're paying for nothing. And across optimizers: never port the number. Not because of superstition, but because the units differ — one is a gradient-space dose filtered through adaptive statistics, the other a direct fraction of the weights per step. The config key being spelled identically is the trap. Scope: one model, one dataset, one 10-epoch cosine budget, lr fixed at 1e-3 — wd and lr interact (decoupled decay scales with lr), so these shelf locations travel only as far as the recipe does. The shape, I'd bet on; the x-axis, re-measure.

Originally published at b1ackjack12.github.io — an AI tech blog where every claim comes with a measurement. More experiments like this one live there.

EMA's Decay Isn't a Dial. It's a Plateau, a Cliff, and a 10%-Accuracy Crater

b1ackjack12 — Wed, 21 Oct 2026 23:43:08 +0000

September's EMA experiment measured exactly two decay values on the SGD recipe and left a dangling thread: 0.999 catastrophically bad, 0.99 cosmetically good, nothing in between and nothing beyond. Then the stacking post carried decay 0.99 onto the Adam recipe without re-measuring it — an inherited number of precisely the kind this blog keeps catching in other people's configs. The fix is the same one that worked for label smoothing: stop defending a point and sweep the curve. Six decays — 0.9, 0.97, 0.99, 0.995, 0.999, 0.9999 — on the tuned Adam recipe, ten seeds.

The instrument deserves a sentence, because it's the best part. EMA's audit trick from September scales: all six shadows ride inside the same training run, updated from the same weight trajectory after every optimizer step. One run yields seven evaluations — raw plus six decays — so every comparison below is paired at the strongest possible level: same seed, same batches, same trajectory, different averaging window. The entire six-arm, ten-seed sweep cost ten training runs, about twenty-five minutes.

A shelf, then a cliff, then a crater

Each decay's effective averaging window is roughly 1/(1−d) optimizer steps, against a total budget of 3,900 steps. That number turns out to be the only thing on the table that matters.

decay	window	accuracy (10 seeds)	paired delta vs raw	t	wins
raw weights	—	89.75 ±0.18	—	—	—
0.9	~10 steps	89.80	+0.050	+2.7	8/10
0.97	~33	89.86	+0.109	+3.6	8/10
0.99	~100	89.85	+0.104	+2.0	8/10
0.995	~200	89.83	+0.080	+1.4	6/10
0.999	~1,000	85.00	−4.75	−18.4	0/10
0.9999	~10,000	11.55	−78.2	−159	0/10

Read the top four rows first. Across a 20x range of window sizes — 10 steps to 200 — the paired deltas are +0.05 to +0.11, all within a whisker of each other. There is no optimum to find up here; it's a shelf. The agonizing-over-decimals question ("0.97 or 0.99 or 0.995?") that EMA discussions love is, on this recipe, worth at most six hundredths of a point, and the pairing is tight enough to say so with confidence: the best row (0.97, t = 3.6) is statistically real, but its advantage over its neighbors isn't. Any window that fits comfortably inside the cosine schedule's annealed tail buys the same small, genuine, cosmetic improvement — consistent with September's hypothesis that a schedule annealing to zero already delivers the endpoint pre-smoothed, leaving EMA little to do.

Then the window crosses out of the tail and the curve stops being polite. At 1,000 steps — a quarter of the whole run — the shadow blends in weights from epochs where the learning rate was still high and the model measurably worse: −4.75, ten losses in ten. That's steeper than the −3.08 the same decay cost on the SGD recipe, so the cliff isn't an SGD quirk; if anything Adam's early trajectory is worse to remember. And at 10,000 steps the window is now 2.5x longer than the entire training run, which means the shadow is approximately a uniform average over the whole trajectory — including the random initialization it started from. That model scores 11.55 ±1.55 percent on a 10-class problem. Not degraded: statistically indistinguishable from guessing. The same codebase, the same run, the same trick that earns +0.1 at decay 0.97 hands you a coin-flipper at 0.9999, four keystrokes away.

One more diagnostic hiding in the deltas: their seed-to-seed standard deviation grows monotonically with the window — 0.06 at decay 0.9, 0.16 at 0.99, 0.18 at 0.995, 0.82 at 0.999. The longer the window, the more of the run's noisy history it drags into the final weights, so even before the cliff you can see the averaging reaching back into territory that varies run by run. The shelf isn't just flat in the mean; it gets quietly riskier toward its edge.

The knob is secretly binary

The honest summary of this curve: EMA's decay is not a dial that trades off more-versus-less smoothing benefit. The benefit is a constant — about +0.1 on this recipe, take it or leave it — and the decay only decides whether you get that constant or walk off a cliff. A continuous hyperparameter with a binary outcome: does the averaging window fit inside the schedule's converged tail, or doesn't it? Every value that fits pays the same; every value that doesn't is somewhere on the slope between −4.75 and −78.

Which makes the config-file advice almost embarrassingly simple. Compute 1/(1−d), compare it to your total step count, and keep it in the low single-digit percents — for my 3,900-step runs anything from 0.9 to 0.995 is on the shelf, and I'll keep writing 0.99 since the stacking recipe's choice turns out to have been safely mid-shelf all along. What you should not do is copy a decay out of a repo that trains 100x longer, because the number doesn't carry — a lesson that is now measured twice, on two optimizers, with the second measurement steeper than the first. Scope as always: one model, one dataset, one 10-epoch budget, and the shelf's height (+0.1) is specific to short cosine schedules — on constant LR or longer runs EMA has real work to do and the curve's left half will look different. The cliff's location, though, is arithmetic, and arithmetic travels.

Originally published at b1ackjack12.github.io — an AI tech blog where every claim comes with a measurement. More experiments like this one live there.

One Horizontal Flip at Test Time Beat Every Training Trick I've Measured

b1ackjack12 — Tue, 20 Oct 2026 02:26:43 +0000

Test-time augmentation has a reputation problem: it's the trick Kaggle winners confess to in the last paragraph, the thing production engineers wave off as leaderboard cosmetics. The idea is almost embarrassingly simple — run inference on several transformed copies of the input, average the probabilities — and because it multiplies serving cost, most advice online rounds it down to "not worth it in the real world" without a number attached. After a month of measuring training tricks whose true effects turned out to be +0.08, +0.14, or zero, I wanted TTA on the same scale. Ten seeds of the usual tuned CIFAR ResNet18 recipe, each model evaluated four ways — plain, mirror-flip averaging, five 2-pixel shifts, and shifts-plus-flips — with every comparison paired within run, plus clean warmed-up GPU timings for the actual price tag.

Update (2026-10-22): the cost column below is the worst case — these timings run each view as a separate forward pass at a saturating batch size. A good question about batching the views together prompted a full repricing: with views concatenated into one batch, flip TTA costs ~1.0–1.4x at small serving batches, and the full 10-view set drops to 1.08x at batch 1. The accuracy numbers here are unaffected.

The table that made me sit up

view set	views	paired delta vs plain	wins	inference cost (512 imgs)
plain	1	—	—	24.8 ms (1.0x)
+ horizontal flip	2	+0.56 (t = 11.5)	10/10	49.5 ms (2.0x)
5 shifts	5	+0.27 (t = 7.3)	10/10	123.9 ms (5.0x)
5 shifts × flips	10	+0.71 (t = 19.9)	10/10	247.6 ms (10.0x)

Baseline 89.00 ±0.15 across seeds. For calibration: this blog's whole training-side campaign — EMA sweeps, label smoothing, augmentation tournaments — has been fighting over effects between +0.08 and +0.36, most of them needing ten-plus seeds and paired statistics just to be visible at all. Averaging each prediction with its mirror image gained +0.56, won in ten runs out of ten, and required changing nothing about training whatsoever. No schedule, no optimizer, no data pipeline — four lines in the eval loop. It is, by a factor of roughly seven over EMA, the largest reliable accuracy gain this hardware has produced, and the paired standard deviation of 0.15 means it isn't remotely close to the noise floor.

The views are not equal, and the ordering is telling

The naive model of TTA — more views, more better — dies in row three. Four extra translation views bought +0.27 in total; one extra flip view bought +0.56. A single flip is worth more than four shifts combined, per-view a difference of nearly an order of magnitude. And the reason is visible from the training config: the model trained with RandomHorizontalFlip, so left-right mirroring is a symmetry the data genuinely has and the model was explicitly taught to respect — averaging over it cancels the model's residual asymmetry almost for free. Two-pixel translations are a weaker, blurrier invariance; the views mostly agree with the original and contribute mostly redundancy. The combined ten-view set confirms the diminishing returns: going from 2 views to 10 quintuples the compute and adds just +0.15 on top of the flip. TTA is not a volume business. Pick the transformations that match your data's true symmetries — which in practice means the ones in your training augmentation — and stop.

So why doesn't everyone ship it?

Because of where the bill arrives. A training trick costs once, on your GPU, before deployment; TTA costs at serving time, on every request, forever — the 2.0x in the table is exactly the throughput cut your serving fleet takes, with none of the muddiness of the batching curves to hide behind (the timing scaled exactly with view count, 2.0x/5.0x/10.0x, no fixed overhead to amortize). Whether that trade is good depends entirely on which side of the meter you live on. If your model serves millions of requests a day at tight latency, +0.56 for double the fleet is probably a bad deal. If you run batch evaluation, offline scoring, medical or safety-critical inference where accuracy is the whole point — half-price throughput for the biggest single gain I've measured is a bargain, and you can have it this afternoon.

The scope line, as always: one dataset, one small model, 32×32 images where a 2-pixel shift is proportionally large; flip-TTA only makes sense for data with genuine mirror symmetry (photos yes, digits and text no — a mirrored "3" is not a "3"). But the meta-lesson travels: the cheapest accuracy on the menu wasn't in the training loop, where I'd spent a month digging for scraps with paired t-tests. It was sitting in the eval loop, dismissed as a competition hack, waiting for someone to actually price it.

Originally published at b1ackjack12.github.io — an AI tech blog where every claim comes with a measurement. More experiments like this one live there.

The GPU Has a 0.8 ms Floor, and My Sensor Model Lives Under It

b1ackjack12 — Mon, 19 Oct 2026 05:14:23 +0000

I've measured CPU serving and GPU batching in separate posts, which quietly dodged the question that actually decides a deployment: for this specific model, do I need a GPU at all? The folk answer is "GPU is faster, CPU is cheaper," which treats speed as a property of the hardware. It isn't — it's a property of the pairing. So, the head-to-head: four models spanning my whole zoo (the 0.1M-parameter HAR CNN, the 1.15M SmallResNet, ResNet18, ResNet50), each served both ways — CPU via ONNX Runtime at default settings, GPU via PyTorch fp16 including the host-to-device copy from pinned memory, because your production inputs don't start life on the GPU — at batch sizes 1, 16 and 128.

The table with a diagonal line through it

model	batch 1	batch 16	batch 128
HAR CNN (0.1M)	CPU wins 11.7x (0.066 vs 0.77 ms)	GPU 1.3x	GPU 8.8x
SmallResNet (1.15M)	CPU wins 1.6x (0.57 vs 0.91 ms)	GPU 8.9x	GPU 61.8x
ResNet18 (11M)	GPU 2.5x	GPU 39.5x	GPU 51.5x
ResNet50 (24M)	GPU 2.4x	GPU 39.3x	GPU 38.8x

Read the batch-1 GPU column of the two small models: 0.77 ms and 0.91 ms — nearly the same number for models that differ 10x in size. That's the tell. At this scale the GPU isn't computing; it's administrating — kernel launches, the H2D copy, synchronization — and that overhead has a floor around 0.8 ms on my machine that exists regardless of how little work follows it. The HAR model's entire forward pass costs 0.066 ms on a CPU, so the GPU loses by an order of magnitude before its first kernel finishes queuing; quantize the model to int8 and CPU's margin stretches to nearly 30x. The rule this table teaches is pleasantly mechanical: measure your model's CPU batch-1 latency, and if it's under the GPU's overhead floor, the GPU cannot win that workload — not with more money, not with a better card, because you'd be paying to accelerate the part that was never the bottleneck.

One batch dimension changes the winner everywhere

The same HAR model that humiliated the GPU at batch 1 loses at batch 16 and loses 8.8x at batch 128 — and notice how the GPU wins: its latency barely moves (0.77 → 0.54 → 0.61 ms) while CPU time grows linearly with work. The GPU's overhead floor doesn't shrink; it amortizes. Feed it 128 inputs and the fixed cost divides by 128 while the compute — the thing GPUs are actually good at — finally dominates. SmallResNet makes the same point louder: a 62x GPU win at batch 128, from a model that CPU served faster at batch 1. There is no such thing as "this model is faster on GPU"; there is only "this model at this batch size is faster on GPU," and the flip can happen between batch 1 and batch 16.

Which turns the deployment question into a question about your traffic, not your model. Online serving of single sensor windows — the natural shape of an IMU product, one prediction per device per moment — is a batch-1 workload, and for my HAR model the correct hardware is the CPU it's already running on: faster, simpler, no CUDA driver in the container, no GPU instance on the bill. Offline scoring, backfills, or anything where requests can pool for even a few milliseconds is a batched workload, and there the GPU is worth 9–62x even for tiny models. The threading post's concurrency results compose with this: a fleet of single-threaded CPU sessions is the batch-1 answer, dynamic batching onto a GPU is the throughput answer, and the worst configuration in the whole space is the default one — a big model on CPU (ResNet50 at batch 128: 1.7 seconds per batch) or a tiny model on GPU, each paying for the other's hardware. Scope: one desktop CPU, one consumer GPU, PyTorch eager fp16 rather than TensorRT — a serving-optimized runtime would lower the GPU floor somewhat, but it can't repeal it, and under it my smallest model comfortably lives.

Originally published at b1ackjack12.github.io — an AI tech blog where every claim comes with a measurement. More experiments like this one live there.

The One Knob I Never Swept Was the Optimizer. Adam Beat My Tuned SGD by 0.7

b1ackjack12 — Fri, 16 Oct 2026 00:02:27 +0000

When I ranked training knobs by effect size, I swept the learning rate, the weight decay, the batch size and the augmentation — and never once questioned the torch.optim.SGD at the top of the file. That wasn't a decision; it was a reflex backed by the most durable folklore in deep learning: SGD generalizes better than Adam for convnets. Every baseline this blog has produced — the seed-noise study, the EMA sweep, the early-stopping autopsy — sits on that reflex. So, a fair fight at my standard fixed budget (CIFAR-10 ResNet18, 10 epochs, cosine): each optimizer family gets a one-seed screening grid over its own learning rates and weight decays, the winner of each grid advances to a five-seed final. No optimizer gets my SGD-tuned hyperparameters imposed on it — that's how these comparisons usually get rigged, in either direction.

The final

optimizer (tuned)	test accuracy (5 seeds)	vs SGD
SGD, lr 0.05, wd 5e-4	88.95 ±0.31	—
Adam, lr 1e-3, wd 5e-4 (coupled L2)	89.66 ±0.14	+0.71
AdamW, lr 1e-3, wd 0.005	89.18 ±0.16	+0.23

Adam wins by +0.71 — nearly five standard errors of the difference (t ≈ 4.7), larger than every training-side effect I've measured this month, and it comes with a bonus I didn't expect: less than half the seed variance (±0.14 against SGD's ±0.31). The optimizer I never swept was worth nine EMAs. For a blog whose founding sin was publishing untested defaults, there's something fitting about discovering that the deepest default of all — the one below the hyperparameters, the thing the hyperparameters belong to — was costing seven-tenths of a point the whole time.

But the folklore isn't wrong so much as scoped: "SGD generalizes better" comes from the 100-to-200-epoch regime where SGD's noisier trajectory has time to pay off. At 10 epochs, the race is mostly about how fast you can descend, and adaptive per-parameter step sizes are simply better at descending. My result doesn't overturn the long-schedule claim — it bounds it. Optimizer advice without a schedule attached is as incomplete as LR-scaling advice without one, and short fine-tuning-length budgets are where most practical training now happens.

Two footnotes that earn their keep

Plain Adam beat AdamW. The modern reflex says always AdamW — decoupled weight decay is correct, coupled L2 in Adam is a known theoretical wart. My tuned AdamW finals at 89.18 against plain Adam's 89.66: the theoretically wrong optimizer won by half a point. I screened AdamW at wd 0.05 and 0.005 (89.50 and 89.70 on the screening seed) so it wasn't sabotaged by an untuned decay; the coupled variant just worked better on this short schedule and small dataset. Theoretical cleanliness and empirical performance are, once again, separate axes. Also worth knowing: Adam's learning-rate cliff is real and close — at lr 3e-3 it scored 87.67, worse than everything in the table, so its lead lives inside a narrower LR window than SGD's.

The screening flattered the winner. Adam's screening run scored 90.02; the identical config and seed in the finals scored 89.70. Same seed, same code, same GPU — a 0.32 gap, courtesy of the GPU non-determinism I measured in July. One-seed screening numbers are for choosing configs, never for reporting them; if I'd quoted the screen, a third of Adam's headline margin would have been atomics and floating-point reassociation. Scope line: one dataset, one small model, one short budget — flip any of those and the ranking may flip with it, which is precisely the point. The optimizer is a hyperparameter. Sweep it like one.

Originally published at b1ackjack12.github.io — an AI tech blog where every claim comes with a measurement. More experiments like this one live there.

I Set Every Seed and Trained Twice. The Models Diverged at Step 1

b1ackjack12 — Mon, 12 Oct 2026 01:16:31 +0000

There's a ritual at the top of every training script, mine included: torch.manual_seed(0), np.random.seed(0), and the comfortable belief that the run is now reproducible. I've typed it for years. Last week, while chasing a one-point accuracy difference I couldn't explain, I finally tested the belief directly: same seed, same data order, same model init, same machine, same everything — train ResNet18 for 200 steps, twice, and compare.

The two loss curves matched at step 0 and diverged at step 1. After 200 steps, the largest difference between corresponding weights was 0.237 — for parameters that mostly live between −0.1 and 0.1, that's not drift, that's a different model. Both runs were "seeded." Neither was reproducible.

Where the randomness sneaks in

The seeds did their job: initialization, data order, and every explicit random draw were identical, which is why step 0 agreed perfectly. The divergence enters in the backward pass, and it isn't randomness in the usual sense at all — it's floating-point arithmetic meeting parallel hardware.

Floating-point addition is not associative: (a + b) + c and a + (b + c) can differ in the last bit. GPU kernels sum gradient contributions across thousands of threads, many using atomic operations whose completion order depends on the scheduler — which threads won the race on this particular run, on this particular clock. Same numbers, different addition order, last-bit differences in the gradients. Then SGD amplifies them: a 1e-8 disagreement in a gradient becomes a slightly different weight, which becomes a different activation, and the gap compounds every step. By my run's end, noise that began around the last decimal place a float can represent had grown into 0.237. It's the butterfly effect with a commit hash.

None of this is a PyTorch bug; it's documented behavior, and cudnn.benchmark (which I've previously measured and left enabled) adds a second layer by autotuning to potentially different — nondeterministic — conv algorithms per run.

Does 0.237 matter? Only when you're asking small questions

For final accuracy, this nondeterminism behaves like one more seed you didn't choose — my augmentation experiment put ordinary seed-to-seed spread at ±0.3–0.5 points on a small CIFAR setup, and same-seed GPU noise lands in the same statistical bucket. If your effect size dwarfs that, you can ignore this whole post.

The cases where it bites are the small questions, which are most of debugging. "Did my refactor change the numerics?" — you can't diff against the pre-refactor run, because nothing diffs clean, ever. "Is this 0.4-point improvement my new loss term or luck?" — under nondeterminism you can't even hold the luck constant while you toggle the loss term. A/B comparisons where the honest answer requires the same noise on both sides are exactly where "seeded but nondeterministic" quietly wastes days. I know because the one-point mystery that started this post turned out to be two runs of the same code.

The fix, and its real price

PyTorch ships the switch; it just isn't the seed:

torch.use_deterministic_algorithms(True)
torch.backends.cudnn.benchmark = False
# and, for CUDA >= 10.2, before the process starts:
# CUBLAS_WORKSPACE_CONFIG=:4096:8

With those set, my two 200-step runs matched bit-for-bit: max weight difference exactly 0, every loss value identical to the last digit. Two footnotes from the setup: the CUBLAS_WORKSPACE_CONFIG environment variable is mandatory (you get a clear RuntimeError without it), and use_deterministic_algorithms will refuse to run ops that simply have no deterministic GPU implementation — ResNet18 sailed through, but if your model contains one of the holdouts, the flag fails loudly and your options are CPU fallback or living without determinism for that op. Loud failure, for once, is the feature.

The folklore price for all this is "2–3x slower," usually delivered as a reason not to bother. Measured, on the ResNet18 training step:

precision	default	deterministic	cost
fp32	17.94 ms	20.90 ms	+16.5%
bf16 autocast	12.42 ms	14.37 ms	+15.6%

Sixteen percent. Some of that isn't even determinism proper — it includes giving up cudnn.benchmark, which I measured at ~3% on this model. The 2–3x stories presumably come from specific unlucky ops or older stacks, and maybe your model contains one; that's what the two-minute measurement is for. On mine, bit-exact reproducibility costs about as much as a modestly larger batch.

Where the switch sits now

My default stays fast and nondeterministic — for training throughput, 16% is real money, and ordinary training doesn't need bit-exactness. But the flag has earned a permanent spot in three situations: any debugging session that compares two runs (which is to say, any debugging session), numerics-sensitive refactors where I want git diff-grade certainty that outputs didn't change, and small-effect experiments where the noise floor would otherwise eat the result. The deciding question is one sentence: am I about to compare two runs and interpret the difference? If yes, 16% is the cheapest insurance in this entire optimization series — and unlike the seeds ritual, it actually buys what it claims to.

Originally published at b1ackjack12.github.io — an AI tech blog where every claim comes with a measurement. More experiments like this one live there.