DEV Community: Keshav Varshney

My laptop was wrong about every optimisation that mattered

Keshav Varshney — Sat, 07 Nov 2026 14:57:46 +0000

TL;DR: For Speed Demon 2026, a 72 hour performance hackathon run by Hackathon Raptors, I rewrote four deliberately slow Python programs in C++ with zero dependencies: a log aggregator, a search engine, a hostile CSV join and a shortest path solver. They ended between 375x and 2,100x faster than the references, byte for byte identical, and third overall on the live board when it closed. I developed them on a 2-core 2016 laptop and scored them on a 12-thread machine I could never log in to. This is the story of the disagreements between those two machines: a 12% win the laptop could not see, a 20% win the box punished, a memory number that was really a kernel accounting rule, a profile that never applied to the code it was profiling, and a hash that passed every dataset and was still wrong.

Repo: https://github.com/ikeshavvarshney/speed-is-life

The game

Each track ships a spec, a dataset and a slow, correct Python reference. You rewrite it in anything, the organisers build it offline on their box, run it, and hash the one output file. A single wrong byte scores zero. Otherwise:

K = (T_ref/T)^0.50 * (M_ref/M)^0.20 * (S_ref/S)^0.15 * (D_ref/D)^0.15
points = 50 * log10(K)

T is wall time, exec to exit. M is the cgroup's peak memory. S is scored
source lines after their formatter. D is one plus your dependency count. So
speed matters most, but memory, code size and dependencies all count, and a
change is only good if K goes up. Early on I made a change that was 6% faster and added 82 lines (run-83a72f8bdff1). K went down. That set the rule for the rest of the event: judge by K, never by T alone.

The box was an i5-13500: six P-cores, 12 threads, turbo off at 2.5 GHz, 24 MB of L3. My laptop is an i5-6300U: two cores, four threads, 3 MB of L3, running Docker inside WSL. Every push to the competition branch ran each changed track once on the box and posted T and M to a public board. That was the only window I had onto the machine that counted.

Where the two machines disagreed

Change	Laptop said	Box said
Four hash table lookups in one `vpgatherqq` (Track B)	Almost nothing	12% of the whole run, 0.0878 s to 0.0769 s
Prefetching the input 512 bytes ahead (Track B)	Parsing 20% faster	No gain. At 1024 bytes, 0.8 MB more memory
Raw `clone` threads with spin locks (Track A)	20% faster with 4 workers	0.0414 s against 0.0162 s
Porting from Rust to C++ (all tracks)	Level	30% slower on Track A until three build fixes
CLMUL quote masks in the CSV parser (Track C)	16% less CPU	0.0504 s to 0.0494 s, nothing
Turning transparent huge pages off (Track B)	Invisible	57 to 65 MB down to 50 MB

None of these is exotic. The gather result is probably the Downfall microcode mitigation, which makes gathers slow on my Skylake and is not needed on the 13500. The prefetch result is a better hardware prefetcher on the box. The spin locks were fine with four workers on four CPUs and terrible with twelve threads sharing six cores, where a spinning thread steals execution slots from the sibling doing real work. The CLMUL result told me something more useful than a speedup would have: a real CPU saving that did not move the board meant parsing was not the bottleneck at all.

The port is the one I would have got wrong without the board. All four tracks
started in Rust and moved to C++ for 21 to 30% fewer scored lines. On the
laptop the C++ was level. On the box it was slower until I fixed three things
that a 2-core machine does not punish: writable globals sharing cache lines
with hot read-only data, u64 being the same type as size_t (so the compiler
had to assume a store through one could change sizes and masks held in the
other, and reloaded them), and -mtune=alderlake to schedule for the box's
P-cores. After that, K was level with Rust in fewer lines.

Making the board report more than two numbers

The board publishes T and M and nothing else. On Track B I wanted the real
split of 154 ms across setup, parsing, list building, answering and exit,
measured on the box, before redesigning anything.

So I built diagnostic versions. They produce the correct output, pass every
gate, and then encode a counter in the two numbers the board does publish:
touch 128 MB + code * 256 KB of memory, release it, and sleep until
T = 3 s + code * 16 ms. Read T and M off the board, subtract, and you have a
phase time measured on the scoring machine. Those runs score about 20 points,
so they can never become a best run, and the real code went straight back
after each one. They are marked as diagnostics in the commit history.

On the box	Before the redesign	After it
Parse	88 ms	54.75 ms
Build lists	21 ms	6.75 ms
Answer queries	37.5 ms	14.5 ms
Exec and exit	7.2 ms	1.7 ms

Every row surprised me. List building and answering were far larger than the
laptop suggested. Exit was 7 ms, nearly all of it the kernel unmapping a
250 MB input page by page on one thread with the clock running, which is
waste you can remove by clearing the mapping on all threads before returning.
Track B went from K 28.70 and sixth place to K 41.67 and first, and Track C
used the same trick to find that its locked bitset write was only 16% of the
time, not the majority I had guessed.

I would ask the organisers before doing this again. It never improved a
score directly, but it does put deliberately shaped numbers on a public board,
and that deserves their say-so rather than mine.

Using another track as a control group

Board runs are noisy. Identical Track B code ran 0.0656, 0.0715 and 0.0659 s
on three pushes. Identical Track D code scored K from 20.15 to 21.33. On some
pushes every track ran slow together, because the box was busy.

That last observation is also the fix. Every push reruns every changed track,
so when I changed only Track A, Track B was an unchanged program running on
the same box minutes apart. Dividing A's time by B's in the same push cancels
the box's mood. On Track C, pinning workers to CPUs looked like a 4% win in
plain T. The C/B ratio said about 1%, inside the noise, and it cost 8 lines,
so it was rejected. Close calls got three runs each, compared by median.

Memory is a kernel accounting question

M is memory.peak of the cgroup, and two kernel rules decided more of it than
my data structures did.

Huge pages. The box has transparent huge pages set to always. A first
touch anywhere in a 2 MB aligned range of a large mapping, including a
thread's 8 MB stack, charges 2 MB. Threads racing on the same fault are each
charged until the race resolves. That explained memory wandering between 57
and 71 MB on unchanged code. One line turned it off for the process:

prctl(PR_SET_THP_DISABLE, 1, 0, 0, 0);

Track B then turns huge pages back on for exactly one hot table, faults it on
one thread so nothing can race, and turns them off again.

Per-CPU charge batches. The kernel charges memory to a cgroup in batches
of 64 pages per CPU, so the first charge on each CPU reserves up to 256 KB,
and memory.peak counts the reserve. Over 12 CPUs that is about 3 MB nobody
uses. On Track C, which peaks under 10 MB, that is a third of the score's
memory term. The fix was to make workers never charge memory at all: the main
thread faults the tables, the pages workers will write, their stacks and the
input's page tables before they need them. Track C went from 10.4 MB to 7.9 MB.

The same idea failed on Track A, and why it failed is the useful part. A
pool of pre-faulted regions big enough for the busiest moment cost more than
the reserve it saved: M went to 20.1 and 23.8 MB.

Two more rules from the contract mattered. The output file lives on tmpfs, so
Track A's 10.1 MB of output is 10.1 MB of its 17 MB peak. And the clock stops
when the process is reaped, so exit teardown is scored work.

The profile that never applied

Track A is built by clang 17 with profile-guided optimisation. My early
profiles were trained during the Docker build, and the run times varied by up
to 15% between builds of the same source. Final scoring builds once, so that
is a gamble. I trained a profile once on the laptop, committed it as data, and
the build became deterministic: the same binary, sha256 595f6323..., every
time. Seven runs on the board, all steady.

Then I compared two profiles side by side, and found out why it was so
steady. Clang names a static function in a profile by the source path it was
compiled from. The training compiled /w/main.cpp. The Dockerfile compiles
main.cpp. So the profile matched main and the standard library's sort, and
every hot function, from the parser to the percentile selection, was built
with no profile at all.

The obvious fix made it worse. Profiles trained under the right path ran
0.0169 s (trained with 12 threads) and 0.0193 s (trained with 4) against the
broken profile's 0.0156 s. A build with no profile at all ran 0.0158 s. So
the final Track A ships the profile that does not apply, on purpose, with a
comment in the README saying so. What it buys is not speed. It is a binary
that is the same every time it is built, which is exactly what you want when
someone else builds it once and scores it.

Tests that pass and prove little

The hash. Track B looks up tokens by a hash that stands in for their
bytes, so a collision involving a query term silently changes an answer. My
first version used two AES rounds, which Track A's notes suggested. Every
dataset matched the reference. Then I tested the hash as its own component on
real tokens, and wodeand and woweandvi from the seed 2 corpus came out
equal. Two AES rounds leave half the state colliding for inputs that differ
only along one of AES's diagonals: 28% of a targeted set collided. Three
rounds gave no collisions in 25 million targeted tokens and cost nothing
measurable. Tracks A and C had the same weakness on part of their input, and
got the third round too.

Byte-identical output on many seeds says very little about a collision that
needs two particular strings to exist. Final scoring runs on a hidden seed.

The answer that was too good. One change made Track B's answers 5x faster.
A limit had overflowed in list decoding, so every list decoded as empty. The
hash check caught it, but the timing is the part worth remembering: a result
that good is a reason to look for the bug first.

The check that checked the wrong binary. My fuzz script builds an
instrumented copy of the solution. When a code change broke the
instrumentation patch, the script carried on testing the previous binary and
reported PASS. Nothing wrong was pushed, but it now deletes old binaries first
and stops on any patch failure.

The crash 1 run in 150. Track C maps its tables at a fixed address. With
12 threads forced, about one run in 150 crashed. The kernel log showed the
thread pointer register inside my mapping: current kernels randomise the heap
start by up to 1 GB, so 0x40000000 could land on libc's thread block. It
moved to 0x60000000 with MAP_FIXED_NOREPLACE. I only found it because the
test plan said to repeat runs many times. An earlier Track A bug also failed
on only some runs, which is why the plan said so.

A wrong program as a measuring instrument

On Track C, every record sets one bit in a per-country bitset of names with a
locked instruction. I wanted to know what that lock cost before spending hours
removing it. So I pushed a version without the lock. It loses updates and gets
the wrong hash, which I expected, and its T was 0.0172 s against 0.0186 s.

That gave me a ceiling: 8% was the most any lock-free design could win. Three
correct designs (buffered writes with one lock per 64, finer partitions, a
lock per country) all came out level or slower, because each still had to
fetch the same cache lines. Without the ceiling I would not have known when to
stop.

Removing work that was already free

Track D is shortest paths on a 1.7 million node road-like graph, with
bidirectional Dijkstra and a bucket queue. Skipping the edge back to the
parent node removes a third of all relaxations. It changed nothing, because
those lookups were already cache hits. A cycle counter around each operation,
after three wrong guesses, showed the real cost: a random lookup in the
distance table at about 105 cycles. Prefetching the next node's neighbours'
labels as each node is popped was worth 11% on one core.

The layout came from counting the data, not from a textbook: 98.8% of nodes
have out-degree 4 or less, and only 3,478 of 5 million weights need more than
11 bits. So every node gets four 4-byte slots at a computed address, and its
edges can be prefetched from its id alone with no offset lookup.

Where I lost

Track D finished 33rd of 113. The leader ran 0.0614 s, and I ran 0.194 s.

The honest reasons: I never measured Track D's phase split on the box, so my
picture of it came from a laptop with an eighth of the cache, on the one track
that is mostly cache misses. Renumbering nodes for locality made queries 25 to
30% faster but cost 0.65 to 0.85 s of serial work, and I did not find a
parallel ordering in time. I did not try contraction hierarchies, partly
because the site warned that heavy indexes rarely repay themselves, and the
leaders' times say that warning did not hold for everyone. This was the track
where I applied my own lesson least.

By the numbers

	Track A, logs	Track B, search	Track C, CSV join	Track D, paths
Reference time	32.05 s	56.35 s	14.44 s	72.69 s
Final time	0.0152 s	0.0640 s	0.0186 s	0.194 s
Speedup	2,109x	880x	776x	375x
Memory	228 to 17.6 MB	2,200 to 46.7 MB	367 to 7.9 MB	893 to 93.5 MB
Rank at close	6 of 134	4 of 104	4 of 111	33 of 113


Build window	72 hours, 340 commits
Board pushes	162
Dependencies	0 on every track
Overall at close	3rd of 109 on machine score (judges' 35% not yet included)
Rejected experiments logged	More than 60, each with its run ID

Every step and every rejected experiment is in each track's PERF.md, with
the board run ID that measured it.

What the time actually went on

The algorithms took the first day. The rest went on finding out which
measurement was about which machine. The laptop could tell me whether code was
correct and roughly whether it did less work. It could not tell me what a
gather cost, what a spin lock cost, what the kernel would charge for memory or
what a compiler flag was worth on a P-core. Each time I trusted it on one of
those, the board corrected me.

The question that sorted it out was the same every time: which machine is this
number about, and could it have come out the other way? A profile that cannot
apply cannot make a build slower. A dataset that passes cannot show a
collision between two strings it does not contain. A laptop with four threads
cannot show you what twelve threads spinning will do. When the answer was no,
the number was not evidence, however good it looked.

Built for Speed Demon 2026, the 72 hour performance engineering championship run by Hackathon Raptors. All four solutions, their performance logs and the tools used to check them are in the repo.

Repo: https://github.com/ikeshavvarshney/speed-is-life

I built a hackathon judging platform, and the test data had no winner in it

Keshav Varshney — Wed, 07 Oct 2026 18:13:58 +0000

TL;DR: I built podium, a self-hostable hackathon submission and judging
platform, in 72 hours: Express, Prisma and Postgres behind a Next.js client,
one docker compose up, 438 tests. The part of the brief that looked like a
formality was "a documented way to even out judges who score harshly or
generously". It turned out to be the whole project. This is the story of a
metric that reached a perfect score by construction and proved nothing, a
judge with two ballots who could move a project 23 places, a test data set
with no signal in it, and a feature I shipped on day one and deleted on day
three.

Repo: https://github.com/ikeshavvarshney/podium
Demo video: https://www.youtube.com/watch?v=3qqdSy8PHTQ
Screens and walkthrough: https://podium-demo.pages.dev/

The problem that looked easy

Judges do not share a scale. One gives 4 out of 5 because nothing is ever a

Another gives 5 because the demo worked. Average their raw scores and you are not ranking projects, you are ranking which judges each project happened to draw.

The textbook fix is one line. For each judge, subtract their mean and divide
by their standard deviation:

z(ballot) = (total - mean_j) / sd_j

A project's score is the mean of the z-scores it received. I wrote that on
the first day, the tests passed, and the results screen showed raw rank next
to normalized rank with little arrows between them. It looked finished.

The metric that scored 100% and proved nothing

The organizers ship a shared data set, fixtures.json: 40 projects, 30
judges, 126 score rows. After my importer refused one duplicate submission
and its four ballots, 122 ballots remained.

I wanted a number for the README, so I split the variance in those ballots
by which judge cast them:

Method	Variance explained by the judge
Raw weighted total	27.0%
Per-judge z-score	0.0%

27% down to zero. That is a fantastic table, and it is worthless.
Standardizing sets every judge's mean to zero by definition. The judge
effect did not go to zero because the method is good. It went to zero
because that is what subtraction does. Any method that subtracts the judge
mean scores 0.0% here, including a bad one.

The number I had was measuring my arithmetic, not my ranking. The real
question is whether the normalized order is closer to the truth, and for
that you need a truth.

The data set with no winner in it

So I looked at the other column: how much of the variance is explained by
which project a ballot was for.

Method	Explained by judge	Explained by project
Raw weighted total	27.0%	30.9%
Per-judge z-score, shrunk	3.5%	33.0%
Per-judge z-score, unshrunk	0.0%	33.3%

About a third looks like signal until you ask what chance alone gives. With
40 groups over 122 ballots, pure noise explains 32.2%. The fixture sits
right on it.

I checked a second way. Split each project's ballots at random into two
halves, rank the projects by each half, and correlate the two rankings. If
the scores carry any information about the projects, the halves agree. Over
400 random splits:

raw totals        -0.005
normalized         0.023

Zero. The fixture's scores carry judge effects and almost no project effect.
Which is fair: it is test data for checking that a portal works, not a
recorded hackathon. But it meant the benchmark I had planned to prove my
method on could not tell a good method from a random one. No method can
recover an order that is not there.

This was the benchmark that disappointed me, and I am glad it did before I
published a table built on it.

Building a truth to measure against

The fix was a seeded simulation where every project has a known latent
quality. 40 projects, 30 judges, 3 reviews each, 500 simulated events. Judges
differ in severity (mean 45 to 85) and in spread (4 to 16 points). One judge
in ten gives every project the same score. Then measure Spearman correlation
between each method's ranking and the true order:

Method	Mean Spearman with the true order	Beats raw in
Raw mean	0.749	n/a
Rank average	0.838	86% of events
Per-judge z-score, unshrunk	0.865	n/a
Per-judge z-score, shrunk	0.889	100% of events

That is evidence. It runs with npm run proof, it is seeded so it
reproduces exactly, and a unit test asserts the numbers so the document
cannot drift from the code.

The last two rows are where the maths fought back.

The judge with two ballots

Look at what plain z-scoring does to a judge who only scored a few projects:

One ballot. Their mean is that ballot. z = 0, whatever they gave. A 5 and a 1 are the same vote.
Two ballots. Their mean is the midpoint and their standard deviation is half the gap. The z-scores are always exactly +1 and -1. A judge who gave 80 and 79 has put the two projects two standard deviations apart.
Every ballot identical. Standard deviation zero. Division by zero.

The fixture has all three. 9 of its 30 judges cast fewer than three ballots,
and jdg_07 gave three projects exactly 75.

On the fixture, the unshrunk method moved one project, Dry Harbour, from
31st to 8th. Two of its ballots came from judges with fewer than three
ballots. It had not been scored well. It had been scored by people whose
scale was manufactured from one or two data points.

The fix is the standard empirical-Bayes one: do not trust a thin judge's own
statistics. Pull them toward the panel's.

mean_j* = (n_j * mean_j + 1 * m) / (n_j + 1)        m = mean of all ballots
sd_j*   = sqrt((n_j * var_j + 3 * v) / (n_j + 3))   v = pooled within-judge variance

z = (total - mean_j*) / sd_j*      and 0 for a flat judge

A judge with twenty ballots is read almost entirely on their own scale. A
judge with one is read mostly on the panel's. Dry Harbour now moves from
31st to 17th instead of 8th, and in the simulation agreement with the true
order goes from 0.865 to 0.889.

Three things I would say about this before a reviewer does:

The priors were tuned on the simulation. One ballot for the mean, three for the variance (variances are noisier). I chose them where the benchmark was best, so the 0.889 is an in-sample number.
The flat judge gets zero, not a penalty. Someone who scores everything 75 has told you nothing about order. Counting them as zero is the honest reading. They are also named on the results screen.
Shrinkage keeps 3.5% of the judge effect on purpose. With two ballots, a judge's mean is as much their projects as their severity. Removing all of it is the bug, not the goal.

Pairwise mode, and a modest win

The second judging mode asks judges to order small groups instead of scoring
a rubric. Borda count is the obvious tally. I also fitted a Bradley-Terry
model with the MM algorithm, which weighs a win by the strength of the
project beaten, so a project that drew an easy group is not rewarded for the
draw.

On 300 simulated events: Borda 0.827, Bradley-Terry 0.854, better in 82% of
events. That is a real gain and a small one. I publish the Bradley-Terry
ranking and show Borda beside it, because a ranking an organizer cannot
sanity-check against a simpler one is a ranking they will not trust.

The debugging story: a function's name was the documentation

The OpenAPI document is generated from the live Express routes. To know who
may call a route, the generator walks the middleware stack and reads each
function's name:

const mapped = ROLE_BY_MIDDLEWARE[h.name ?? ""];

requireAuth means signed in, requireEventAdmin means event admin. It
worked for two days.

On the last day I replaced the single admin check with area-scoped access:
an admin can be limited to Announcements, or Judging, or any of ten areas.
requireEventAdmin became a factory:

export function requirePermission(...areas: EventArea[]) {
  return (req, _res, next) => {
    assertPermission(eventContext(req), areas);
    next();
  };
}

Every route kept working. Every permission test passed. And 69 operations
in the published API document silently lost their access label, because an
arrow function returned from a factory has the name "". The API still
refused the requests. The documentation now said anyone could make them.

One test caught it: the one that fetches /api/openapi.json and checks that
admin routes are labelled. The fix is to name the inner function and attach
the areas to it, which the generator now publishes as x-required-area:

const requirePermission = (req, _res, next) => { ... };
return Object.assign(requirePermission, { areas });

The lesson I took is narrower than "test your docs". Reading fn.name is
reflection, and a refactor that changes nothing about behaviour can change
everything about what reflection sees. If documentation is derived from
code, the derivation needs its own assertion.

A second one from the same hour, shorter. My end-to-end check for "a limited
admin is refused on settings" reported that the limit did not work. The API
returned 200. The middleware was correct. The account I had picked as the
limited admin was the instance operator, who
bypasses area checks by design. The product was right and the test was
wrong, and a 200 where I expected 403 looks identical either way.

The design I abandoned

On day one I built judge track scope: an organizer could restrict a
judge to certain tracks, and mark a track as restricted. It had a column, a
dialog, a rule inside the assignment engine and its own tests.

On day three I dropped it. Migration 0026 is two lines:

ALTER TABLE "event_memberships" DROP COLUMN "track_scope";
ALTER TABLE "tracks" DROP COLUMN "restricted";

The reason is that the platform already had an answer to "which projects
does this judge see": the assignment. A judge reads exactly the projects an
organizer assigned and nothing else, and that boundary is enforced in the
API and tested with curl. Track scope was a second answer to the same
question. Two sources of truth meant every path had to consult both, the
auto-balancer had to explain why it left a project under-reviewed, and the
organizer had a setting that could quietly contradict what they had just
done by hand.

The same clean-up hit "reviews per project". It started as a hard cap that
blocked manual assignment past the number. It is now called "Judges per
project" and is only the target the auto-balancer aims for. An organizer who
wants a sixth opinion on a close call can simply assign one.

Both removals made the judging model smaller: one function decides whether a
judge may review a project, and it checks one thing, that it is not their
own team's.

The schema I would redo

Two decisions from the last day, both made because they were fast.

Admin areas are a TEXT[] with a magic string.

ALTER TABLE "event_memberships" ADD COLUMN "permissions" TEXT[] DEFAULT ARRAY[]::TEXT[];
UPDATE "event_memberships" SET "permissions" = ARRAY['ALL'] WHERE "role" = 'ADMIN';

The column sits on every membership row, including participants and judges,
where it means nothing. Postgres cannot check that an entry is a real area,
so validity lives in one Zod transform. And 'ALL' is a sentinel sharing a
namespace with real values. The rest of the schema leans on the database for
correctness: one ballot per judge per project is a unique index, the audit
log is append-only by trigger. This column is the exception. It should be an
enum array with a check that it is empty unless the role is ADMIN, or a
small join table.

Flagging a project reuses the submission status. An organizer can remove
a project from public view with a reason. I stored that as
status = DISQUALIFIED plus three flag columns, which was attractive
because every query that already filtered on SUBMITTED excluded flagged
projects for free: gallery, judge queues, ballots, results. The cost shows
up on restore, which has to put the status back and can only assume it was
SUBMITTED. Visibility and lifecycle are two different facts and I stored
them in one field.

The decision I would keep is the one everything else hangs on: a role is not
a property of an account, it is a row joining an account to an event. The
same person is a judge in one event and a participant in another, and no
request body ever gets to say who the caller is.

What I did not build

Stated here instead of left for someone to find.

Community voting counts accounts, not people. Registration needs no email verification, because the platform has to run with no mail server. The threat model marks Sybil accounts as reduced, not stopped.
Collusion checks flag, they do not prove. Lockstep judge pairs, outlier ballots, flat judges and shared organizations are surfaced before publishing. Two judges who vary their scores just enough stay under every threshold.
The acceptance checker verifies two of my four tiers. It has no probes for T3 or T4, so those claims rest on my own tests and a manual walkthrough.
A failed seed does not fail the boot. The entrypoint runs the seed with || echo "seed skipped". If the fixture import throws, the API still starts and reports healthy, minus the event. I found this after submitting, while reading my own entrypoint to answer a different question.
Uploads live in Postgres. Fine for a few hundred images and it keeps the install to one database. Not fine for a large archive.

By the numbers


Build window	72 hours, 140 commits
Server	about 13,400 lines of TypeScript
Client	about 19,500 lines, Next.js 15 and React 19
Tests	438 across 38 files, about 5,400 lines
Migrations	26, applied from an empty database on boot
API	159 operations on 121 paths, OpenAPI 3.1 generated from the routes
Acceptance checker	7 of 7 probes passing
Clean install	cloned from GitHub, healthy in 83 seconds
Raw mean against the true order	0.749
Shrunk z-score against the true order	0.889

What the time actually went on

The z-score took an afternoon. What took the rest was finding out that the
first three ways I tried to show it worked could not have shown it failing.
A variance table that reads zero for any method that subtracts a mean. A
data set where no ranking is recoverable. A two-ballot judge who looks like
a confident expert.

The question that sorted them out was the same each time: could this number
have come out badly? If the answer is no, it is not evidence, however good
it looks in a README.

The fixture had no winner in it. The useful thing the platform can do with
data like that is say so.

Built for DOGFOOD 2026, the 72 hour hackathon run
by Hackathon Raptors, where the brief is to build the platform a hackathon
is judged on. The normalization proof, the judging methodology, the threat
model and the full API document are in the repo.

Repo: https://github.com/ikeshavvarshney/podium
Demo video: https://www.youtube.com/watch?v=3qqdSy8PHTQ
Screens and walkthrough: https://podium-demo.pages.dev/

My Go tests never executed a line of the JavaScript they were testing

Keshav Varshney — Thu, 03 Sep 2026 14:40:42 +0000

TL;DR: I built a Real User Monitoring tool as one Go binary with an empty
dependency manifest, zero entries in go.mod. 245 test functions, go vet
clean, CI green on every push, a reproducible build verified on every commit.
Six bugs survived all of it. This is the story of a build that stamped a git
SHA into a binary I had called deterministic, an int64 that saturated into
six-year chart buckets, a demo page that was genuinely slow and reported
green, and a fix that was correct, deployed, and invisible for an hour.

None of them were findable from Go. That is the actual subject of this post.

The premise is a joke that turned out to be the product

RUM tells you how slow your site feels to real visitors. Every mainstream way
of collecting it ships 30 to 60KB of third-party JavaScript from someone
else's CDN, executing before your own code, on every page view. The tool that
measures page weight is itself page weight.

So the target was a beacon small enough that the size is the argument. It
landed at 942 bytes, against 7,226 for Google's web-vitals. Everything
else followed from refusing to make that number worse: no framework, because
a framework is a bundle; no chart library, because a chart library is 60KB to
draw six lines; no database driver, because the thing being stored is five
floats per page view.

Writing that was the easy half. The question that decides whether any of it
is real is: how do you know it works?

The gap: no Go test can execute JavaScript

Testing a browser beacon from a Go suite means testing everything except the
beacon. I could prove /b.js was served with the right MIME type and ETag,
that it stayed under its 1024-byte budget, that the ingest endpoint parsed a
payload I hand-wrote and the API read it back. All of that passed, and none
of it proves PerformanceObserver fires, that visibilitychange triggers
the flush, or that CLS session-window arithmetic produces a number a browser
would agree with.

Go tests embed the file, serve it, assert on its bytes, and never execute a
line of it. To a Go test, JavaScript is a byte slice.

So I drove real Chrome over the DevTools Protocol. Node 22 ships a global
WebSocket, so the harness needed nothing installed:

const { webSocketDebuggerUrl } = await (await fetch(`${CDP}/json/version`)).json();
ws = new WebSocket(webSocketDebuggerUrl);

tests/*.mjs (external harness)     drives real Chrome over CDP
        │  Page.navigate, Runtime.evaluate, Input.dispatchMouseEvent
        ▼
vitals binary                      serves dashboard, demo pages, beacon.min.js
        │  beacon executes for real, PerformanceObserver fires for real
        ▼
/api/collect  →  /api/series, /api/events (EventSource)
        │
        ▼
45 scripted assertions against what the browser actually measured

The harness lives outside the repository, because a browser automation
dependency inside a zero-dependency project would be absurd.

Two details from building it are worth passing on. Chrome 152 removed
Emulation.setPageVisibilityState, the usual way to fake a tab switch, so I
replaced it with what a visitor actually does: open a second tab and activate
it. And I was clicking with element.click(), which made INP appear only
intermittently, because synthetic clicks do not reliably produce Event Timing
entries. Switching to Input.dispatchMouseEvent made it deterministic and
produced the single best piece of evidence in the project:

metric inp collected by the browser  p75=600.00 ms (poor, n=1)

600.00ms against a handler deliberately blocking for exactly 600ms.

Six bugs, in increasing order of "that would have shipped"

1. The reproducible build that wasn't

make repro builds twice and compares hashes. CI does the same on a clean
runner and runs cmp. Both passed from the day I wrote them. The README
said the binary contains no build timestamp, git SHA, or injected version,
which is what makes the output byte-identical. False. I caught it only
because a hash changed between two commits that touched nothing but
Markdown.

$ go version -m vitals
build   vcs.revision=6acd604c28b79a3db52850287e061b98c0018d7e
build   vcs.time=2026-08-30T10:56:20Z

Since Go 1.18 the toolchain stamps the git revision, a timestamp, and a
module pseudo-version into any binary built inside a repository. On by
default. The fix is one flag, -buildvcs=false.

The fix is not the interesting part. My test checked a weaker property than
my claim. "Two builds at the same commit produce identical bytes" and "the
binary contains no commit metadata" are different sentences. Only the first
was under test. The second was what the README asserted, and a passing test
sat next to it the whole time looking like evidence.

I proved the fix the way I should have proved the claim: build, make an
empty commit so the SHA changes, build again, compare. Most
reproducible-build advice stops at -trimpath and -ldflags=-buildid=. The
VCS stamp is newer, quieter, and it undermines you without saying anything.

2. The int64 that saturated

I loaded the dashboard and read the caption under the chart:

1 of 48 buckets have samples, 3202560 min per bucket

Six years per bucket. On a chart labelled "last 24 hours".

The dashboard calls /api/series?from=24h and sends no to, because "now"
is the obvious default. An open range end was normalised to the year 9999.
Then:

span := rng.To.Sub(rng.From)
width := span / time.Duration(q.Buckets)

time.Duration is an int64 of nanoseconds, topping out near 292 years.
Subtracting 1970 from 9999 does not error, does not panic, does not warn. It
saturates. Divide by 48 and you get six-year buckets, rendered without
complaint.

No test caught it because I wrote the tests alongside the API, thinking
about the API, so every one passed both from and to like a well-behaved
caller. The dashboard, the only real client this API has, passes one. The
bug lived exactly in the gap between how I documented the endpoint and how
the only thing calling it actually calls it.

The regression test does not assert a bucket width. It asserts that no
combination of parameters can produce a span longer than a century, because
a span that long means an open end leaked through and the arithmetic
saturated.

3. The demo that demonstrated nothing

The project ships four deliberately broken demo pages so the dashboard shows
something other than a wall of green. One is "heavy image": a hero of 2,600
inline SVG shapes, meant to tank Largest Contentful Paint. It is genuinely
expensive to render.

In Chrome it measured 220ms. Good band. Green.

An inline <svg> element is not an LCP candidate. The spec is specific
about what counts: <img>, <image> inside an SVG, video poster frames,
background images, and block-level text. A root inline <svg> is not on
that list. The page burned real paint time while LCP quietly reported on a
paragraph above it.

This one stings because it is precisely the failure a performance dashboard
exists to prevent: the page was slow, the metric said fast, both were true,
and the gap was a spec detail I had not read carefully enough.

Fixed with an <img> and a data URI so it is actually a candidate, plus a
deliberate main-thread block so it is slow on any hardware rather than only
on a bad laptop. It reports 2852ms now. My first attempt used a 4200ms block
to force the poor band, which broke differently: the page was still blocked
when the tab hid, so the beacon flushed before any paint entry existed and
sent a record containing only TTFB. Worse demo, worse data. The page now
explains the trap in its own copy, because the trap is more interesting than
the demo.

4. I measured my own tooling and called it a result

The headline claim is a size comparison. That claim is the entire pitch, so
it had better be clean. My first numbers were not: I measured our beacon
with Go's gzip and theirs with Python's. Same algorithm, different
implementations, a few percent apart. Nobody would ever have caught it. It
would have been a property of my toolchain, presented as a property of my
code, in the one number the project rests on.

So I wrote a tool that runs every file through one compressor:

File	Raw	Gzipped
vitals beacon	942 B	571 B
vitals full beacon	2,656 B	1,415 B
`web-vitals.iife.js`	7,226 B	2,601 B
`web-vitals.attribution.iife.js`	12,505 B	4,172 B

7.7x smaller raw, 4.6x gzipped. And still not a fair fight, in both
directions.

Unfair to us: web-vitals only measures. It hands each metric to a
callback and leaves transport entirely to you, so a real deployment adds
reporting code on top of those bytes. Our 942 already include JSON
serialisation, sendBeacon, a fetch fallback, and flush-on-hide.

Unfair to them: web-vitals does more per metric: back-forward cache
restoration, prerendering and activationStart, soft navigations,
attribution to the element that caused the bad LCP, real INP grouped by
interactionId rather than the longest single event, and years of Safari
workarounds.

At 942 bytes I did none of that. So I wrote a second beacon that does five
of those six, at 2,656 bytes, and kept it as a separate file rather than
folding it into the first. Two reasons, one technical and one not. The
technical one: most sites want the small one, and paying 1.7KB for prerender
correction on a site that never prerenders is exactly the trade this project
exists to argue against. The other: the sub-1KB number is the headline
claim, and quietly redefining "the beacon" to mean the bigger file is the
move I would criticise in someone else's README.

Neither beacon does the Safari and Firefox work. That is years of
accumulated browser bug knowledge, not something you reproduce by reading a
spec. A comparison is more convincing with its caveats than without, because
a reader who spots an unstated one stops trusting the rest of your numbers
too.

5. The cache that outlived the bug

I broke dash.js with a one-character mistake, a literal newline inside a
string literal, which is a syntax error. The whole script failed to parse,
so the dashboard rendered nothing. Every Go test passed. Of course they did.
node --check is now in the pre-finish checklist, alongside make check.

The instructive part came next. I fixed it, rebuilt, restarted, and the page
was still blank:

Cache-Control: public, max-age=3600

The asset names carry no content hash. A browser that had already fetched
the broken script would not ask again for an hour. Server fixed, disk
fixed, user still broken, and no amount of restarting changes that, because
nothing on the server participates in the decision.

The fix is a policy split rather than a blanket. The dashboard's own scripts
are now no-cache, which still caches them but forces a revalidation that
answers 304 with an empty body. The beacon keeps the long max-age,
because it is fetched by every page view of an instrumented site, where a
conditional request per view is a real cost. Same file server, two
policies, chosen by who is asking and how often.

A long cache lifetime is a promise you cannot revoke. It is only safe when
the URL changes with the content. If you are not hashing names, you are
betting you will never ship a bug.

6. The half-open window that dropped the present

An end-to-end test started failing about half the time. Same code, same
machine, same command.

The dashboard's default window ends at "now". The store's range is
half-open, [from, to). A measurement recorded in the same clock tick as
the request that reads it lands exactly on the exclusive bound and falls
outside its own window. On Windows, where the wall clock is coarse enough
that two calls microseconds apart return the same value, that is a coin
flip.

if to.IsZero() {
    to = now            // excludes anything stamped at this instant
}

Every test passing an explicit to was fine. Only the default was wrong,
and the default is what the only real client sends: the same gap as bug 2,
in a different corner of the same function. The failure mode was "I just
recorded a measurement and it is not there", intermittently, which is the
worst kind of bug report to receive from yourself.

The fix is one millisecond, and the reasoning matters more than the line.
Records are stored at millisecond resolution, so ending the default window
one millisecond after now includes the present instant and nothing that has
not happened yet. An explicit to is left exactly where the caller put it.

A flaky test is a race you have already reproduced. I nearly re-ran it and
moved on.

The bug the whole suite was structurally blind to

This one arrived last and it is the one I would most want a reviewer to
see. The dashboard has a button that copies a report as a prompt for an AI
agent. Clicking it threw:

Cannot read properties of undefined (reading 'good')

The Go struct tagged one field json:"Distribution" with a capital D. The
dashboard, the API documentation, and the contract test all said
distribution.

Here is why nothing caught it: encoding/json matches field names
case-insensitively when decoding. The contract test unmarshals the response
into a struct tagged json:"distribution", and it bound happily to
"Distribution". It passed against either spelling. Every Go test in the
repository was structurally blind to this, and only a browser, which is not
case-insensitive about anything, could ever see it.

The regression test therefore does not use a struct. It decodes into
map[string]any and walks every key in the document:

if first := k[0]; first >= 'A' && first <= 'Z' {
    t.Errorf("key %s%s starts upper-case; a JavaScript client "+
        "reading the documented lower-case name gets undefined", path, k)
}

Decoding into a map is the entire point. It is the only way to assert the
key a non-Go client actually receives, rather than the key Go is willing to
accept.

What I did not build

Stated here rather than left for someone to notice.

The binary segment storage format was planned and cut. The design called
for compacting sealed day logs into a hand-written binary format with varint
delta timestamps and a string dictionary. It stayed JSONL. That decision now
has a number attached rather than a shrug: replaying 100,000 records takes
593ms, of which roughly 500ms is encoding/json on the read path. A
million records is about six seconds and 1.6GB to open. That 500ms is
exactly what the segment format would have removed, and it is the honest
size of what cutting it gave up.

There is no authentication. The dashboard and API are open to anyone who
can reach the port. Collection is rate limited per client address, but
forwarded headers are ignored because they are trivially spoofable, which
means behind a reverse proxy every visitor shares one bucket and one
derived session id.

reportAllChanges is not implemented. web-vitals can emit every LCP
candidate as it changes. This sends one record per page view on hide, which
is the whole reason it fits in 942 bytes.

The decision I would take back

The beacon exists as two files: beacon.src.js, readable and commented, and
beacon.min.js, minified by hand. There is no minifier in the project,
because a minifier is a build dependency and the project has no build step.

The cost is that a human keeps two files in sync, and humans drift. I
mitigated it rather than solved it. Tests assert that every metric key,
every observed entry type, and the collection endpoint appear in both
files, and that the minified file is actually minified rather than a copy
of the readable one under the wrong name. That catches the catastrophic
mistake. It does not catch a subtle logic change applied to one file and
not the other.

A real minifier would make the drift structurally impossible, and I would
take the build step. The zero-dependency rule was about runtime
dependencies, and I let it bleed into build tooling where it bought nothing
but risk.

By the numbers


Beacon	942 B raw, 571 B gzipped, 1024 B budget enforced by the build
Full beacon	2,656 B raw, 1,415 B gzipped, separate 2816 B budget
vs `web-vitals` 4.2.4	7.7x smaller raw, 4.6x gzipped
Dependencies	0. `go list -m all` prints one line
Go	~13,750 lines, of which ~8,100 are tests
Tests	245 functions, 10 benchmarks, 1 fuzz target, 20+ table-driven
Frontend	~3,400 lines, no framework, no bundler, no build step
Packages replaced	41, each with a note on where the original is better
Browser checks	45 scripted, in real Chrome 152
Commits	50, with the tree building and `make check` passing at every one

Two of the later features came from measuring rather than reasoning.
Benchmarking the store turned up a genuine quadratic path: an out-of-order
insert rebuilt both secondary indexes, 0.7 to 1.0 ms against 66 µs for
shifting them instead, and out-of-order arrival was not the rare event I had
assumed. The collector stamps the wall clock and takes the store lock as a
separate step, so concurrent page views land reversed constantly. And "you
lose at most two seconds on a crash" was still only a sentence, so I wrote a
test that builds the binary, runs it as a child process, feeds it 500
measurements, and kills it. A clean Close had never tested the thing the
sentence claimed.

Said out loud rather than buried: percentiles are read from histogram
buckets and carry up to 4.9% relative error on millisecond metrics and
0.0025 absolute on CLS, while band counts are exact. The small beacon
approximates INP as the longest event over 16ms, so it is pessimistic in
the tail, which is why the full beacon exists. Firefox and Safari are
untested, and so are the full beacon's own bfcache, soft-navigation, and
prerender paths: reviewed against the specs, never watched running in a
browser. That is the largest honest gap in the project, and it is why the
small beacon remains the default.

What the time actually went on

Anyone can write a PerformanceObserver that compiles and a Go server that
serves. The beacon is 942 bytes; it took an afternoon. What the rest of it
went on was building enough ways to catch myself being wrong: a dependency
checker that fails the build on a require block, a CDN reference, or a web
font; a size tool that measures every file with one compressor; a CI job
that builds twice and runs cmp; a fuzz target on the only place the
program reads untrusted input; a test that kills a real process; forty-five
scripted checks in a real browser, because my own suite could not execute
the file it was shipping.

Every one of those exists because a claim I had written down in English
turned out not to be a claim anything was checking. That translation step,
from the sentence in the README to the assertion in the test, is where all
six bugs lived.

Write the claim in plain English, then ask what would have to be true. It
is the cheapest tool in the list and it found more than any of the others.

The tool measures page weight and contributes 942 bytes of it. That was the
whole idea, and it survived contact with a real browser. Eventually.

Built for the Zero Dependency Hackathon 2026, the
72 hour dependency-free build hackathon run by Hackathon Raptors, Track D (Data & Storage). The full project,
STDLIB.md, the browser test harness, and the size comparison tool are all in
the repo.

Repo: https://github.com/ikeshavvarshney/vitals