DEV Community

realist
realist

Posted on

My A/B rig had 95% confidence intervals and was wrong in both directions

#rust #performance #benchmarking #hackathon

The change that gave me my biggest win of the event was scored 1.004x, "no clear difference" by my own A/B rig, with a 95% confidence interval of 0.978 to 1.019. On the real box it cut my parse phase from 28.5 ms to 19.9 ms.

This post is about that gap. I spent Speed Demon 2026 (a Hackathon Raptors performance hackathon) making two Rust programs about 1,200x and 360x faster than their Python references. But the thing I actually learned was not a trick for making code fast. It was that every instrument I used to decide "keep or revert" was blind to something specific, and I didn't find out which blind spot was which until I tested the instruments themselves.

Repo: https://github.com/FrankChinedu/speed-demon (team Deez-attacher; I did tracks A and C, a teammate did B and D).

The setup, briefly

  • Track A (ingest): aggregate 3.1M log lines into per-endpoint counts, p50/p95/p99 latency and error rates per hour. Python reference: 32.05 s.
  • Track C (join): parse a hostile 3.3M-record CSV (quoted commas, escaped quotes, embedded newlines, BOMs), join it to a regions table, write per-country totals. Python reference: 14.44 s.
  • The box: an i5-13500 with 12 threads, turbo off. Output must match the reference byte for byte.
  • The score: K = (Tref/T)^0.5 × min(Mref/M, 50)^0.2 × min(Sref/S, 1)^0.15 × (1/D)^0.15. T is wall time, M is peak memory, S is lines of code, D is dependencies. Time dominates.

I had three ways to measure a change:

  1. The live board. Push a commit, wait about 5 minutes, get one T and M from the real box. One sample per push.
  2. My A/B rig. A GitHub Actions job that builds two commits and runs them alternately, 21 times each, on a 2-CPU x86 runner, then prints a speedup with a 95% CI. Cheap, repeatable, statistically careful.
  3. My Mac. Which I stopped trusting on day one, because an M1 has a different cache hierarchy and several big local wins did nothing on the box.

By the numbers

Track A Track C
Reference time 32.05 s 14.44 s
My time (median of the final code) 0.0268 s 0.0404 s
Best K on the live board 40.16 (rank 37 of 134) 30.41 (rank 41 of 111)
Leader's K 62.50 56.60
Lines of code (S) / dependencies 522 / 0 529 / 0
Board runs logged, including probes 135 137
Experiments measured and reverted 14 14
Timing probes 13 9
A/B rig verdicts 14 15

I finished mid-pack. The leaders were two to three times as fast. So this is not a "how I won" post. It is about why it took me so long to know what was working.

Instrument 1: the A/B rig was confident in both directions

Across both tracks the rig gave 29 verdicts: 14 faster, 8 slower, 7 no difference. The intervals were tight. It felt like science.

It overstated compute wins. On track A, three changes to the per-line endpoint lookup scored 1.055x (an AVX2 space mask plus an XXH3-style hash), 1.034x (one-word table slots) and 1.029x (dropping the murmur finaliser). Stacked, that is about 1.12x on the rig. A timing probe on the box afterwards put the parse at 28.5 ms, against 29.9 ms before. About 1.05x. I wrote in my own log at the time: "the x86 compute gains barely reach the box."

It completely missed the win that mattered. My parser threads registered each new endpoint name in a shared registry behind a Mutex. Each thread kept its own lookup table and only took the lock on a first sighting, which still added up to about 22,000 lock acquisitions per run:

// before (851ead2): every first sighting per thread takes the lock
let mut r = self.registry.lock().unwrap();
Enter fullscreen mode Exit fullscreen mode

I replaced it with one lock-free table shared by every thread, where a new name is claimed with a single compare-exchange (1,842 inserts per run, one per distinct endpoint):

fn claim(&self, i: usize, tag: u64, name: &[u8]) -> Option<u32> {
    let slot = &self.slots[i];
    slot.compare_exchange(0, tag, Acquire, Relaxed).ok()?;
    let id = self.count.fetch_add(1, Relaxed);
    self.names[id as usize].set(name.to_vec()).unwrap();
    slot.store(tag | (id as u64 + 1), Release);
    Some(id)
}
Enter fullscreen mode Exit fullscreen mode

Rig: 1.004x, no clear difference. Box: parse 28.5 ms to 19.9 ms, and the whole run went from 0.0391 s to 0.0338 s and then 0.0329 s on a second sample. K went from 32.38 to 35.00.

At the time I wrote "contention the 2-CPU A/B runner cannot show" in the commit message and moved on. That was a hypothesis, not a measurement. Writing this post, I went back and tested it.

Testing the rig itself

I built the exact two commits the rig had compared (851ead2 and cc5a8a2) and ran them alternately in one long-lived Docker container on full seed-1 data: two warm-up rounds, then 15 timed pairs, twice per CPU setting, so 30 pairs each. The only thing I changed between settings was --cpuset-cpus.

CPUs Registry lock (median) Shared table (median) Speedup 95% CI (bootstrap, 30 pairs)
2 67.5 ms 69.3 ms 0.988x 0.924 to 1.003
8 35.5 ms 33.3 ms 1.096x 1.058 to 1.136

Same binaries, same output hash. At 2 CPUs the change is worth nothing, maybe slightly negative, which is exactly what the rig reported. At 8 CPUs it is a clear win, and the box has 12. This was on an M1 Pro, not the scoring box, so I don't trust the size of the number. The direction is the point: the benefit only exists when enough threads are fighting over the lock. My rig was not noisy here. It was structurally incapable of seeing this class of change, and it would have said "no difference" every time I asked.

The two errors have the same cause. A 2-CPU runner inflates anything that makes a single thread faster and hides anything that stops threads waiting on each other. That is a predictable bias, not random error, and I could have known it before I trusted a single verdict.

Instrument 2: the board's noise lives in the tail

My team's handoff doc said: "Board times are noisy by about ±5%. Do not keep a change for a gain smaller than that."

After the event I grouped every board run by the exact source tree it built. My final track A code was scored 21 times without a single change:

  • 18 of the 21 runs landed between 0.0262 s and 0.0280 s, within -2% and +4.5% of the 0.0268 s median. So ±5% was a fair description of a typical run.
  • 3 of the 21 were 8% to 22% slower: 0.0290, 0.0303 and 0.0327 s.
  • K for the same code ranged from 35.83 to 40.16. Eleven other teams' best scores fall inside that band, so unchanged code could have placed me anywhere from 37th to 48th.
  • Final track C code: 28 runs, 0.0397 s to 0.0456 s.

So the problem with one sample per push isn't that every number is a little off. It's that roughly one push in seven is badly off, and you can't tell which one from the inside.

That tail fooled me in the flattering direction too. On track C I sharded the writes to a shared "seen" bitset (rig: 1.073x, CI 1.04 to 1.11). The board came back at 0.0477 s against 0.0550 s and 0.0554 s for the two runs before, which looked like a 13% jump. A few pushes later, a deliberate recheck of the same code came back at 0.0526 s. My log says it plainly: "e542027 at 0.0477 was a lucky run." The change was real, but most of the gain I saw was luck.

My best K, the 40.16 at the top of this post, is the fastest of those 21 identical runs. The frozen commit, with byte-identical track A and C code, scored 38.73 and 29.91. The organisers' final scoring runs each entry 5 times on a holdout seed and takes the mean of the middle three. That is exactly the right defence against a tail like this, and I should have used it on myself from day one.

Instrument 3: a commit that deliberately gets the answer wrong

The instrument I ended up trusting most was a hack. The board reports T and M even when the output hash is wrong. So to time one phase on the real box, I committed a version that writes an empty file and exits right after that phase:

std::fs::write(out.join("summary.txt"), b"").unwrap(); // TIMING PROBE
std::process::exit(0);
Enter fullscreen mode Exit fullscreen mode

I pushed 22 of these across the two tracks. They answered questions no profiler on my Mac could:

  • Just reading the 239 MB track A input in parallel costs about 9 ms (9.9 ms with mmap, 8.6 ms with pread). Everything above that is mine.
  • The track A endpoint lookup cost about 10 ms of the parse, even though its table was small enough to sit in cache. That pointed me at the lookup path.
  • On track C, name interning plus the shared bitset cost about 21 ms of a run of about 58 ms.

A probe measures the real box, at 12 threads, with the real data. It doesn't model anything. Its weakness is that it is still one sample, so it inherits the board's tail, which is why I learned to push "code unchanged" reruns and treat any single number with suspicion.

The check I never ran: correctness on data I had never seen

Final scoring uses a holdout seed. Every one of the 5 runs must produce the right hash, or the track scores zero. Both of my tracks identify names by a 64-bit hash, with no byte comparison, because comparing bytes was a measurable cost.

That bet already burned me once. On the morning of November 1, about 30 hours before the freeze, I generated seed 6 to widen my test set, and track C undercounted distinct names. The FxHash-style step I was using collided in all 64 bits on near-duplicate names. The board's seed 1 never showed it. I replaced it with a folded-multiply step (commit 8c6f070), which cost no time.

After that fix I had checked full-scale data on only two seeds. The other seeds were tiny and sample scale, 10% of the size or less. So for this post I generated full-scale seeds 3, 4 and 5 with the official generator, ran the Python reference on each, and ran the frozen x86 build (AVX2 paths and the PGO profile compiled in) against them in an amd64 container:

Track Full seed 3 Full seed 4 Full seed 5
A match match match
C match match match

Six out of six, byte for byte. I'm relieved. But the bigger lesson is that this check took about ten minutes, and I only ran it seven days after the freeze.

What I'd tell someone entering next year

  1. Test your instrument against the target before you trust any verdict from it. Rerun one known change on your rig at 2 CPUs and at the target's core count. If the answers disagree, you have learned what your rig cannot see.
  2. Know which way your bias points. A small runner over-rewards single-thread speedups and under-rewards anything that removes waiting. That is predictable, so correct for it.
  3. One leaderboard sample is a lottery ticket. Push a "code unchanged" rerun before you keep or revert anything that matters, and compare medians, not your best run.
  4. A deliberately broken submission is a legitimate profiler when the scorer reports time even for wrong output. It is the only one that runs on the real hardware.
  5. Hash identity needs unseen data at full scale. A collision in all 64 bits sounds impossible until a weak mixing step makes it routine.

The zoom-out

I went into this thinking performance work meant finding the slow code. Most of my time actually went into finding out whether the code had changed at all. Every instrument answers a slightly different question from the one you're asking. The rig asked "is this faster on 2 CPUs?" The board asked "was this one run fast?" The Mac asked "is this faster on an M1?" None of them asked "is this faster on 12 threads, on average?" Until I knew which question each one was really answering, I was optimising whatever the instrument could see.

Next time I'll pick the instrument before I pick the optimisation.

References

Top comments (0)