TL;DR: For Speed Demon 2026, a 72 hour performance hackathon run by Hackathon Raptors, I rewrote four deliberately slow Python programs in C++ with zero dependencies: a log aggregator, a search engine, a hostile CSV join and a shortest path solver. They ended between 375x and 2,100x faster than the references, byte for byte identical, and third overall on the live board when it closed. I developed them on a 2-core 2016 laptop and scored them on a 12-thread machine I could never log in to. This is the story of the disagreements between those two machines: a 12% win the laptop could not see, a 20% win the box punished, a memory number that was really a kernel accounting rule, a profile that never applied to the code it was profiling, and a hash that passed every dataset and was still wrong.
Repo: https://github.com/ikeshavvarshney/speed-is-life
The game
Each track ships a spec, a dataset and a slow, correct Python reference. You rewrite it in anything, the organisers build it offline on their box, run it, and hash the one output file. A single wrong byte scores zero. Otherwise:
K = (T_ref/T)^0.50 * (M_ref/M)^0.20 * (S_ref/S)^0.15 * (D_ref/D)^0.15
points = 50 * log10(K)
T is wall time, exec to exit. M is the cgroup's peak memory. S is scored
source lines after their formatter. D is one plus your dependency count. So
speed matters most, but memory, code size and dependencies all count, and a
change is only good if K goes up. Early on I made a change that was 6% faster and added 82 lines (run-83a72f8bdff1). K went down. That set the rule for the rest of the event: judge by K, never by T alone.
The box was an i5-13500: six P-cores, 12 threads, turbo off at 2.5 GHz, 24 MB of L3. My laptop is an i5-6300U: two cores, four threads, 3 MB of L3, running Docker inside WSL. Every push to the competition branch ran each changed track once on the box and posted T and M to a public board. That was the only window I had onto the machine that counted.
Where the two machines disagreed
| Change | Laptop said | Box said |
|---|---|---|
Four hash table lookups in one vpgatherqq (Track B) |
Almost nothing | 12% of the whole run, 0.0878 s to 0.0769 s |
| Prefetching the input 512 bytes ahead (Track B) | Parsing 20% faster | No gain. At 1024 bytes, 0.8 MB more memory |
Raw clone threads with spin locks (Track A) |
20% faster with 4 workers | 0.0414 s against 0.0162 s |
| Porting from Rust to C++ (all tracks) | Level | 30% slower on Track A until three build fixes |
| CLMUL quote masks in the CSV parser (Track C) | 16% less CPU | 0.0504 s to 0.0494 s, nothing |
| Turning transparent huge pages off (Track B) | Invisible | 57 to 65 MB down to 50 MB |
None of these is exotic. The gather result is probably the Downfall microcode mitigation, which makes gathers slow on my Skylake and is not needed on the 13500. The prefetch result is a better hardware prefetcher on the box. The spin locks were fine with four workers on four CPUs and terrible with twelve threads sharing six cores, where a spinning thread steals execution slots from the sibling doing real work. The CLMUL result told me something more useful than a speedup would have: a real CPU saving that did not move the board meant parsing was not the bottleneck at all.
The port is the one I would have got wrong without the board. All four tracks
started in Rust and moved to C++ for 21 to 30% fewer scored lines. On the
laptop the C++ was level. On the box it was slower until I fixed three things
that a 2-core machine does not punish: writable globals sharing cache lines
with hot read-only data, u64 being the same type as size_t (so the compiler
had to assume a store through one could change sizes and masks held in the
other, and reloaded them), and -mtune=alderlake to schedule for the box's
P-cores. After that, K was level with Rust in fewer lines.
Making the board report more than two numbers
The board publishes T and M and nothing else. On Track B I wanted the real
split of 154 ms across setup, parsing, list building, answering and exit,
measured on the box, before redesigning anything.
So I built diagnostic versions. They produce the correct output, pass every
gate, and then encode a counter in the two numbers the board does publish:
touch 128 MB + code * 256 KB of memory, release it, and sleep until
T = 3 s + code * 16 ms. Read T and M off the board, subtract, and you have a
phase time measured on the scoring machine. Those runs score about 20 points,
so they can never become a best run, and the real code went straight back
after each one. They are marked as diagnostics in the commit history.
| On the box | Before the redesign | After it |
|---|---|---|
| Parse | 88 ms | 54.75 ms |
| Build lists | 21 ms | 6.75 ms |
| Answer queries | 37.5 ms | 14.5 ms |
| Exec and exit | 7.2 ms | 1.7 ms |
Every row surprised me. List building and answering were far larger than the
laptop suggested. Exit was 7 ms, nearly all of it the kernel unmapping a
250 MB input page by page on one thread with the clock running, which is
waste you can remove by clearing the mapping on all threads before returning.
Track B went from K 28.70 and sixth place to K 41.67 and first, and Track C
used the same trick to find that its locked bitset write was only 16% of the
time, not the majority I had guessed.
I would ask the organisers before doing this again. It never improved a
score directly, but it does put deliberately shaped numbers on a public board,
and that deserves their say-so rather than mine.
Using another track as a control group
Board runs are noisy. Identical Track B code ran 0.0656, 0.0715 and 0.0659 s
on three pushes. Identical Track D code scored K from 20.15 to 21.33. On some
pushes every track ran slow together, because the box was busy.
That last observation is also the fix. Every push reruns every changed track,
so when I changed only Track A, Track B was an unchanged program running on
the same box minutes apart. Dividing A's time by B's in the same push cancels
the box's mood. On Track C, pinning workers to CPUs looked like a 4% win in
plain T. The C/B ratio said about 1%, inside the noise, and it cost 8 lines,
so it was rejected. Close calls got three runs each, compared by median.
Memory is a kernel accounting question
M is memory.peak of the cgroup, and two kernel rules decided more of it than
my data structures did.
Huge pages. The box has transparent huge pages set to always. A first
touch anywhere in a 2 MB aligned range of a large mapping, including a
thread's 8 MB stack, charges 2 MB. Threads racing on the same fault are each
charged until the race resolves. That explained memory wandering between 57
and 71 MB on unchanged code. One line turned it off for the process:
prctl(PR_SET_THP_DISABLE, 1, 0, 0, 0);
Track B then turns huge pages back on for exactly one hot table, faults it on
one thread so nothing can race, and turns them off again.
Per-CPU charge batches. The kernel charges memory to a cgroup in batches
of 64 pages per CPU, so the first charge on each CPU reserves up to 256 KB,
and memory.peak counts the reserve. Over 12 CPUs that is about 3 MB nobody
uses. On Track C, which peaks under 10 MB, that is a third of the score's
memory term. The fix was to make workers never charge memory at all: the main
thread faults the tables, the pages workers will write, their stacks and the
input's page tables before they need them. Track C went from 10.4 MB to 7.9 MB.
The same idea failed on Track A, and why it failed is the useful part. A
pool of pre-faulted regions big enough for the busiest moment cost more than
the reserve it saved: M went to 20.1 and 23.8 MB.
Two more rules from the contract mattered. The output file lives on tmpfs, so
Track A's 10.1 MB of output is 10.1 MB of its 17 MB peak. And the clock stops
when the process is reaped, so exit teardown is scored work.
The profile that never applied
Track A is built by clang 17 with profile-guided optimisation. My early
profiles were trained during the Docker build, and the run times varied by up
to 15% between builds of the same source. Final scoring builds once, so that
is a gamble. I trained a profile once on the laptop, committed it as data, and
the build became deterministic: the same binary, sha256 595f6323..., every
time. Seven runs on the board, all steady.
Then I compared two profiles side by side, and found out why it was so
steady. Clang names a static function in a profile by the source path it was
compiled from. The training compiled /w/main.cpp. The Dockerfile compiles
main.cpp. So the profile matched main and the standard library's sort, and
every hot function, from the parser to the percentile selection, was built
with no profile at all.
The obvious fix made it worse. Profiles trained under the right path ran
0.0169 s (trained with 12 threads) and 0.0193 s (trained with 4) against the
broken profile's 0.0156 s. A build with no profile at all ran 0.0158 s. So
the final Track A ships the profile that does not apply, on purpose, with a
comment in the README saying so. What it buys is not speed. It is a binary
that is the same every time it is built, which is exactly what you want when
someone else builds it once and scores it.
Tests that pass and prove little
The hash. Track B looks up tokens by a hash that stands in for their
bytes, so a collision involving a query term silently changes an answer. My
first version used two AES rounds, which Track A's notes suggested. Every
dataset matched the reference. Then I tested the hash as its own component on
real tokens, and wodeand and woweandvi from the seed 2 corpus came out
equal. Two AES rounds leave half the state colliding for inputs that differ
only along one of AES's diagonals: 28% of a targeted set collided. Three
rounds gave no collisions in 25 million targeted tokens and cost nothing
measurable. Tracks A and C had the same weakness on part of their input, and
got the third round too.
Byte-identical output on many seeds says very little about a collision that
needs two particular strings to exist. Final scoring runs on a hidden seed.
The answer that was too good. One change made Track B's answers 5x faster.
A limit had overflowed in list decoding, so every list decoded as empty. The
hash check caught it, but the timing is the part worth remembering: a result
that good is a reason to look for the bug first.
The check that checked the wrong binary. My fuzz script builds an
instrumented copy of the solution. When a code change broke the
instrumentation patch, the script carried on testing the previous binary and
reported PASS. Nothing wrong was pushed, but it now deletes old binaries first
and stops on any patch failure.
The crash 1 run in 150. Track C maps its tables at a fixed address. With
12 threads forced, about one run in 150 crashed. The kernel log showed the
thread pointer register inside my mapping: current kernels randomise the heap
start by up to 1 GB, so 0x40000000 could land on libc's thread block. It
moved to 0x60000000 with MAP_FIXED_NOREPLACE. I only found it because the
test plan said to repeat runs many times. An earlier Track A bug also failed
on only some runs, which is why the plan said so.
A wrong program as a measuring instrument
On Track C, every record sets one bit in a per-country bitset of names with a
locked instruction. I wanted to know what that lock cost before spending hours
removing it. So I pushed a version without the lock. It loses updates and gets
the wrong hash, which I expected, and its T was 0.0172 s against 0.0186 s.
That gave me a ceiling: 8% was the most any lock-free design could win. Three
correct designs (buffered writes with one lock per 64, finer partitions, a
lock per country) all came out level or slower, because each still had to
fetch the same cache lines. Without the ceiling I would not have known when to
stop.
Removing work that was already free
Track D is shortest paths on a 1.7 million node road-like graph, with
bidirectional Dijkstra and a bucket queue. Skipping the edge back to the
parent node removes a third of all relaxations. It changed nothing, because
those lookups were already cache hits. A cycle counter around each operation,
after three wrong guesses, showed the real cost: a random lookup in the
distance table at about 105 cycles. Prefetching the next node's neighbours'
labels as each node is popped was worth 11% on one core.
The layout came from counting the data, not from a textbook: 98.8% of nodes
have out-degree 4 or less, and only 3,478 of 5 million weights need more than
11 bits. So every node gets four 4-byte slots at a computed address, and its
edges can be prefetched from its id alone with no offset lookup.
Where I lost
Track D finished 33rd of 113. The leader ran 0.0614 s, and I ran 0.194 s.
The honest reasons: I never measured Track D's phase split on the box, so my
picture of it came from a laptop with an eighth of the cache, on the one track
that is mostly cache misses. Renumbering nodes for locality made queries 25 to
30% faster but cost 0.65 to 0.85 s of serial work, and I did not find a
parallel ordering in time. I did not try contraction hierarchies, partly
because the site warned that heavy indexes rarely repay themselves, and the
leaders' times say that warning did not hold for everyone. This was the track
where I applied my own lesson least.
By the numbers
| Track A, logs | Track B, search | Track C, CSV join | Track D, paths | |
|---|---|---|---|---|
| Reference time | 32.05 s | 56.35 s | 14.44 s | 72.69 s |
| Final time | 0.0152 s | 0.0640 s | 0.0186 s | 0.194 s |
| Speedup | 2,109x | 880x | 776x | 375x |
| Memory | 228 to 17.6 MB | 2,200 to 46.7 MB | 367 to 7.9 MB | 893 to 93.5 MB |
| Rank at close | 6 of 134 | 4 of 104 | 4 of 111 | 33 of 113 |
| Build window | 72 hours, 340 commits |
| Board pushes | 162 |
| Dependencies | 0 on every track |
| Overall at close | 3rd of 109 on machine score (judges' 35% not yet included) |
| Rejected experiments logged | More than 60, each with its run ID |
Every step and every rejected experiment is in each track's PERF.md, with
the board run ID that measured it.
What the time actually went on
The algorithms took the first day. The rest went on finding out which
measurement was about which machine. The laptop could tell me whether code was
correct and roughly whether it did less work. It could not tell me what a
gather cost, what a spin lock cost, what the kernel would charge for memory or
what a compiler flag was worth on a P-core. Each time I trusted it on one of
those, the board corrected me.
The question that sorted it out was the same every time: which machine is this
number about, and could it have come out the other way? A profile that cannot
apply cannot make a build slower. A dataset that passes cannot show a
collision between two strings it does not contain. A laptop with four threads
cannot show you what twelve threads spinning will do. When the answer was no,
the number was not evidence, however good it looked.
Built for Speed Demon 2026, the 72 hour performance engineering championship run by Hackathon Raptors. All four solutions, their performance logs and the tools used to check them are in the repo.
Top comments (0)