DEV Community

Alok
Alok

Posted on

Our laptops said it was faster. The scoring machine disagreed.

Three changes to our Track C engine looked faster on our laptops and failed to hold up on the organisers' scoring machine:

  • 32-bit tag buckets in the name table: 9 to 14% faster locally, a tie or a loss on the scoring machine.
  • A read-ahead prefetch: 5 to 10% faster locally, no reliable gain on the scoring machine.
  • An inlined hash with unrolled extraction: 5 to 10% faster locally, and a 14% regression on the scoring machine, at 25.07 ms and 80.67 points (run-d4401d6162d7) against the 22 ms the unchanged engine runs there. We never isolated why the board rejected it.

We reverted all three. One of our profiling sessions had also been running 2.7 times slow, so its absolute timings were useless; the log does not record how we noticed. We cannot say how much of the local gain came from that session and how much came from the laptops being different hardware. For these three changes, the local numbers did not predict the board. The three experiments are logged in Track C's PERF.md. The rest of this post covers where that gap showed up, and the other places where our first reading of a result was wrong.

Speed Demon 2026 is a free online performance hackathon run by Hackathon Raptors. For each of four tracks the organisers provide a task spec, a dataset and a deliberately slow Python reference. Our replacement has to produce byte-identical output, including on data we cannot see. Every push to our branch is built and run by the organisers' harness on one fixed machine, an Intel Core i5-13500 with 12 threads and an 8 GB memory cap, in Falkenstein, and the results appear on a public live board. We call that machine the scoring machine and its results board runs. Our repo, with the per-step logs in each track's PERF.md, is here: https://github.com/Alok-codes21/speed-demon-2026

The four tracks
Track A: 3.1 million request log lines (239 MB) in, counts, percentiles and hourly error rates out. Reference 32.05 seconds and 228 MB. Our engine takes about 19 milliseconds and 15 MB, best board run 85.98 points.
Track B: 250 MB of text and 50,000 boolean and phrase queries, timed as one wall-clock number. Reference 56.35 seconds. Ours takes about 106 milliseconds, best board run about 76.6 points.
Track C: a messy CSV joined against a lookup table, sorted aggregates out. Reference 14.44 seconds and 367 MB. Ours takes 22 milliseconds and 11.1 MB, best board run 82.40 points.
Track D: exact shortest paths for 2,000 queries on a weighted graph with about 5 million edges. Our best board run scored 75.84.

These are the highest scores recorded on the public board, not final judging results. The final evaluation runs on a hidden holdout dataset, and the best of a set of noisy runs needs care, as the Track D section shows.

The formula and our two rules

K = (T_ref/T)^0.50 x (M_ref/M)^0.20 x (S_ref/S)^0.15 x (D_ref/D)^0.15

Wall time carries half the weight, peak memory a fifth, and source size and dependency count 15% each. Source size is code lines counted by scc after formatting. Points are 50 x log10(K). Wrong output on any gate scores zero for the track, however fast it ran.

Our first rule was that correctness is a gate: every change had to pass byte-exact checks against the Python reference on multiple datasets, multiple thread counts and hostile inputs we wrote ourselves before it went near the board. The second rule was that every board run went into PERF.md with its run ID, failures included, so we could always return to a known-good version.

Where the laptops lied

One late case is the v9 and v11 comparison at the end of Track B. The two engines differed by a hand-written integer formatter that replaced a library conversion in the serial tail of the run. On our laptops it measured about 1.5% faster. We submitted fresh runs of both and compared board medians: v9 at 0.1076 seconds over six runs, v11 at 0.1083 over five. Our log notes the board slowed through that afternoon for every team, so this was not a clean A/B test, and a gap that small is not one we would bet on in either direction. What the runs did show is that the formatter gave no clear gain on the machine that counts. v9 also cost fewer source lines (1,189 against 1,206), which the formula prices in, so we kept v9 and deleted the extra code. A single lucky run of v11 would have told us the opposite.

The three Track C changes from the top of this post are the same pattern at larger scale. Later Track C steps were timed as paired alternating runs inside one session, the comparison we should have been making from the start. Pairing controls for session drift, not for hardware differences, and the one change measured that way is the one that survived. The next section is that change.

A smaller case came from Track B's container, which reports the host's core count but limits CPU time through a lower quota. We read the cgroup CPU quota to choose the thread count instead of trusting the host count.

Contention, not instructions

After the first working version of Track C, the largest single gain we logged had nothing to do with instructions. Whenever a worker met a name for the first time, it needed a unique number for it, and all workers drew those numbers from one shared counter. That counter was the hottest line in the run: cores that should have been parsing independent records were queueing on one memory location. We changed it so each worker claims a block of 32 numbers and hands them out locally, touching the shared counter only when its block runs dry. The work per record barely changed, only the coordination did.

This is the one change where laptop and board agreed, and it is also the one we measured with paired alternating runs. Locally the drop showed up repeatedly. On the board, one run moved from 31.7 ms to 23.9 ms (run-0389516eba54). Unchanged code on that track varied by about 6% across rounds, so a drop from 31.7 to 23.9 ms sits far outside the noise. One board run is thin evidence on its own, but here it matched the local pairs and had an identified cause. We had been profiling for cycles, and the cost was in cores waiting on each other.

Track A: when memory is released matters

Track A's engine memory-maps the input, splits it into line-aligned blocks for parallel workers, interns endpoint strings and finishes each hour with a counting sort for the percentiles. Its memory experiment taught us more than its parsing did.

Our first attempt combined two ideas: prefaulting each worker's input range with MAP_POPULATE, and returning input pages to the kernel as chunks finished. Together they moved runtime from 0.033 to 0.042 seconds at that stage of the engine. The log offers two explanations, both inferred from timings and not from a kernel trace: remaps serialising on the kernel's memory-mapping lock, and the page returns interrupting other cores through TLB shootdowns. Because we tested the two ideas together, that run could not split the blame between them. A later step removed all the release machinery: time returned to 0.034 seconds, but memory went back from 33.7 to 34.7 MB. The releases had been the memory win, such as it was. Later experiments supported dropping eager prefaulting and moving reclamation away from the scan. The final engine still releases the input pages, but only after parsing is complete, so process exit has no populated page table to tear down and the release cannot interrupt a worker mid-scan. We had treated memory operations as free because they never appear in the code.

Track B: store what the queries need

The queries are part of the task's input, so a program can read them before it touches the corpus. They mention about 76,000 distinct terms, while the corpus has over 600,000. So we built postings only for the terms the queries ask about, and streamed the corpus in blocks instead of holding it in memory.

That first streaming variant cut board memory from 649 MB to 253 MB, but it dropped the positions needed to verify phrases. Re-reading candidate documents to check them made board wall time worse, from 0.882 to 0.992 seconds. We brought positions back only for the terms that phrase queries use, packed for binary search, while boolean terms kept deduplicated document IDs. The corpus stayed streamed. Positions cost memory: the runtime-first version measured 501 MB on the board, so the 253 MB figure belongs to the memory-first intermediate, not the final engine. From there it was 88 logged steps of smaller work, including an AVX2 tokenizer that classifies 32 bytes per instruction, down to roughly 0.107 seconds at the freeze, in line with the v9 median above.

Removing information saved memory until a later stage needed it, and then the cost came back as time.

Track D: one high score can be luck

Track D answers 2,000 exact shortest-path queries on a graph of about 5 million edges. Repeated board runs of identical source varied by about 20% in wall time, so no single run told us much. We tried changes to lookup behaviour (forward probe lengths from 128 to 512), to memory access (prepopulation, transparent huge pages), to data structures (heap variants, queue changes) and to the algorithm itself (contraction hierarchies). None beat our best-scored version clearly enough to survive that much noise.

A 6-worker cap scored 74.27 and 72.34 against our best of 75.84. Both runs landed below the best, and with that variation we could not attribute the gap to the cap itself. We never saw it help, so we reverted it.

Our fallback was to keep the best-scoring known-good version. That rule has a flaw: when identical code varies by 20%, the best score from a set of runs is partly luck, so best-scored is a tie-break and not proof, and 75.84 should not be read as a stable score for that engine. It also biases every comparison, since each alternative was judged against an inflated high-water mark. We kept it because nothing we tried showed a reproducible improvement.

The same caution applies to our smaller numbers. In Track A's engine, replacing two variable shifts with an 8-byte mask loaded from a 16-byte constant table was worth 0.21 points on one board run. That is one observed result, not evidence of a repeatable gain, so it stays a note in the log until it repeats.

Source lines are points

The source term counts code lines, so deleting code was an optimisation axis of its own. We cut the Track C source from 557 lines to 294 without touching the hot loops, checking the output hash and paired timings after every cut. By the formula's source-size term, that cut alone is worth about two points if everything else stays equal. For scale, the counter fix's time gain is worth about three by the same arithmetic.

What is in the final engines

The formatter that measured 1.5% faster on our laptops is not in the final Track B engine; the query-driven postings, the streaming and the selective phrase positions are. None of the three changes from the top of this post is in the final Track C engine; the block reservation is. Track A's final engine memory-maps the input, parses it in parallel and releases the input pages after parsing. Track D ships the best-scored known-good solver, without the unproven worker cap. The habit we would pass on is short: treat your laptop as a source of hypotheses, take the answers from the scoring machine, and repeat them unless the effect is too large to be noise. That still only gives a better estimate, since the final judging runs on data we have not seen.

The per-step logs for every engine, including the experiments we rejected and the run IDs behind the board numbers, are in the repo: https://github.com/Alok-codes21/speed-demon-2026

Thanks to Hackathon Raptors for publishing the formula, the correctness gates and the machine specification before the event started.

Top comments (0)