DEV Community: Manjeet Singh

I misread my profiler, my rewrite made it slower, and two lines cut memory by 60%

Manjeet Singh — Tue, 03 Nov 2026 15:53:00 +0000

My laptop said the program peaked at 71 MB. The scoring machine said 175 MiB.

Same binary. Same data. Same Docker image. Neither number was wrong.

That was the first of many times during Speed Demon 2026, the 72-hour performance championship run by Hackathon Raptors, when a measurement I trusted turned out to be correct and useless at the same time. Over 106 logged steps I found that almost every mistake I made had the same shape: I read a true number that was answering a different question than the one I was asking. The profiler answered "where did the samples land", not "what would disappear if I removed this". The instruction counter answered "how much work", not "what is the core waiting for". My laptop answered "how fast is this on my laptop".

This post is the list of those questions, what each wrong reading cost me, and the measurement that finally answered the right one.

The task, and a machine you never touch

I'll stay on one of the four tracks, Track B. Answer 50,000 search queries over a 254 MB corpus of 250,000 documents. 62% are BOOL queries (AND, OR, NOT, applied strictly left to right), 38% are exact phrases, and every output line has to match the reference byte for byte. The reference Python takes 56.35 s and 2.2 GB.

Each submission is scored as one number:

K = (T_ref/T)^0.5 × (M_ref/M)^0.2 × (S_ref/S)^0.15 × (D_ref/D)^0.15
points = 50 × log10(K)

T is wall time, M the container's peak memory, S the line count after a formatter, D the dependency count. So halving the time multiplies K by 1.41, halving memory by 1.15, and doubling the code costs 10%. Keep those weights in mind: twice in this story they decided which "obvious" improvement was actually a loss.

The scoring machine is an Intel i5-13500 with turbo off. You push a commit, wait in a queue, and two numbers come back. Here is what that machine is, next to what I was developing on:

	Scoring box	My laptop
CPU	i5-13500, fixed 2.5 GHz	Ryzen 7 7840HS, boosting near 5 GHz
Threads used	12 on 6 P-cores (two per core)	12 of 16
L2	1.25 MB per core, shared by two threads	1 MB per core
Vectors	AVX2, no AVX-512	AVX-512
Environment	bare container	Docker inside WSL2, about 9 µs per page fault
What you get back	T and M	anything you want

Almost every lesson below is a row of this table biting me.

Belief 1: "Peak memory is what I allocate"

It isn't. Peak memory on the board is the cgroup's memory.peak, which charges every resident page. What becomes resident depends on the kernel, not just on you.

My scan wrote its output into many blocks per thread, each filled a little at a time. At any moment most of them were half written. On my laptop the WSL2 kernel ran transparent huge pages in madvise mode: a process gets 2 MB pages only where it asks. The scoring machine evidently ran always, so the first byte written into a fresh region could pull in a whole 2 MB page. Many half-written blocks per thread, times twelve threads, is roughly the 100 MiB I couldn't see.

The fix was two lines of C++:

#include <sys/prctl.h>
prctl(PR_SET_THP_DISABLE, 1, 0, 0, 0); // a block written in part stays small pages

Step	Change	Board run	T	M	Points
12	before	run-5c032275f3b6	0.301 s	175.2 MiB	60.51
13	the two lines	run-e6924d8063c9	0.292 s	69.6 MiB	64.85

Sixty percent less memory and 4.3 points, for two lines.

Then I made the same mistake again. When I rewrote the program in Rust I left the line out, and memory crept back to 69.1 MiB on a version that should have been under 50. This time I didn't guess. I switched my laptop VM to THP always, reran both builds, and watched the difference appear locally (about 5 MB), then vanish with PR_SET_THP_DISABLE. On the box the same line took that version from 69.1 to 48.8 MiB (run-174007cb966d).

The question I should have asked: not "how much do I allocate?" but "what will this kernel make resident when I touch it?" If your laptop and your target disagree about memory, look at the boring knobs first: transparent huge pages, the allocator's mmap threshold, how freed memory goes back.

Belief 2: "Shorter code will score higher"

On day two the C++ was 968 counted lines against a reference of 91, and lines are in the formula. I worked out that a version at 60% of the size and the same speed would score about 7% more, and I rewrote it from scratch in Rust: 612 lines, correct on every dataset on its first build.

Version	Board run	T	M	Lines	Points
C++ (step 23)	run-c2b149a34d40	0.232 s	52.7 MiB	968	67.95
Rust rewrite (step 24)	run-880032be5ecf	0.253 s	65.6 MiB	612	67.55

The arithmetic explains it exactly. Fewer lines are worth (968/612)^0.15 = +7.1%. But 9% more time costs (0.232/0.253)^0.5 = -4.2%, and 25% more memory costs (52.7/65.6)^0.2 = -4.3%. Net: -1.8% of K, which is the 0.40 points the board took away.

A rewrite throws away dozens of small decisions you made without writing them down (the THP line was one of them). I put the C++ back on the board and kept improving the Rust until it actually won, two steps later (0.241 s, 58.0 MiB, 68.86 points). The rewrite was right; shipping it on an estimate was not.

Hold on to that arithmetic, because near the end of the contest it pointed the other way.

Belief 3: "The profiler shows me where the time goes"

A sampled profile on the laptop said building the posting lists took 25% of all cycles. I spent an evening on that phase. Then I built a variant with its stores switched off, and the run got under 2% faster.

Sampling inside that VM attributed cycles to the wrong threads. The profile was answering "where did the timer interrupts land", and in a VM with twelve busy threads that is a different question from "what does this code cost".

From then on I used one method for cost: ablation. Build a variant with exactly one part disabled, count what disappears (user cycles, instructions, wall time), put it back. It is slower than reading a flame graph, and I have never been misled by it.

Belief 4: "Fewer instructions means faster"

This is the one I'm most glad I got wrong, because the way it broke taught me how the box actually works.

perf stat said the scan retired 136 instructions per token over 36.3M tokens. The disassembly showed why: the per-document loop kept 16 values live and reloaded most of them from the stack on every token, and the hash took 25 instructions. That is a deeply satisfying thing to find. I split the loop into short ones, made the hash one 128-bit multiply, and cut 10% of the whole program's instructions (6,516M to 5,881M).

The box's answer, run-152f40943619: 0.1407 s, against 0.1389 s before. Ten percent less work, zero time saved.

So I asked a different question: what is the core waiting for? Cache-miss counters, plus ablation, gave two answers I should have looked for first:

Phrase checks cost 144M instructions but 7.6M of the scan's 26 to 29M last-level cache misses. Each check read four scattered arrays (anchor starts, anchor records, query spans, query terms) to reject one window. Cheap in instructions, expensive in misses.
A 107 KB per-thread array existed to drop a term's repeats within a document. It was read and written at a random place for every list-term token. I counted what it actually did: 10.94M of 11.02M list-term occurrences were already new to their document. It removed 0.7% of the postings and cost a likely cache miss on most tokens.

The fixes were plain. Each phrase anchor got a 16-byte record holding everything needed to reject a window, prefetched for every anchor in a document before the other loops run. The dedupe array went away entirely; the few repeats are dropped later, when the lists are built from sequential data.

Step	Change	Board run	T
86	baseline	run-26e4b60b0e60	0.1389 s
87	10% fewer instructions	run-152f40943619	0.1407 s
88	phrase records prefetched	run-18961f21c438	0.1279 s
89	dedupe array deleted	run-1406c8e99675	0.1243 s

Later I tried the instruction route once more, on purpose: vectorizing the per-document loops cut 7% of instructions and bought 1% of time (run-807c1ffd2504). Same answer.

Two things to take from this. First, with two hyperthreads per core and a lot of memory traffic, the core's instruction throughput is not the limit; its wait for memory is, and the other hyperthread happily soaks up whatever instructions you remove. Second, and more general: count what a structure actually does before you optimize it. I had spent effort making that dedupe array fast. The right number to know was "0.7%".

Belief 5: "I can't measure the machine I can't touch"

By now I trusted the box over my laptop, but the box only tells you two numbers. So I made the program tell me more through those two numbers.

The trick: time a phase inside the program, then encode that duration into the outputs the board reports. For memory, touch one 4 KB page per 4 µs of the phase and keep the pages until exit, so peak memory rises by an amount you can decode. For wall time, sleep for ten times another phase before exiting, so the noise of a single run matters ten times less.

// Phase A's duration, written into peak memory: one touched page per 4 µs.
let pages = phase_a.as_micros() as usize / 4;
let p = mmap(null_mut(), pages * 4096 + 4096, 3, 0x22, -1, 0);
(0..pages).for_each(|k| write_volatile(p.add(k * 4096), 1));

// Phase B's duration, written into wall time, ten times over.
std::thread::sleep(phase_b * 10);

One such run (run-d59c352f1a2a) came back 8.4 MiB heavier and 64 ms slower than the normal run. Decoded: 8.6 ms parsing the queries, 6.4 ms of serial setup. Fifteen milliseconds of a 124 ms run on one thread while eleven threads waited. A later run of the same kind found the answer phase at 29 ms on the box, against 16 ms on my laptop: almost exactly the ratio of the two clocks.

The parse was the embarrassing part. I was splitting a 1.8 MB query file byte by byte, then word by word, about 28 instructions per byte, on one thread, while the corpus itself already went through a 64-byte AVX2 class mask. Reading the queries with the same mask halved the parse's instructions on the laptop (36.9M to 18.4M). On the box it was worth about 10 ms: 0.1243 s to 0.1144 s (run-fa63626812c0), an 8% step from code that was nowhere near the hot loop.

These measurement runs scored terribly on purpose (71.04 and 66.70 points). They were the best runs I made.

The general version: if a black box only reports scalars, make your program speak through them. Any benchmark harness that reports time and memory can be turned into a crude profiler of itself.

Belief 6: "If it's faster on my laptop, it's faster"

Not every disagreement went my way:

A smaller term table (1 MB instead of 2 MB) cut about 5M last-level misses on the laptop. On the box: 0.1243 s to 0.1335 s. Longer probe chains cost more than the misses saved.
Turning lists into bitmaps sooner made my laptop's answer phase about 10% faster. On the box: 0.1144 s to 0.1489 s (run-1aa9c7fc4435), a third slower. More 31 KB bitmaps were zeroed, filled and counted on a machine where two threads share 1.25 MB of L2.

My laptop had more cache per thread, twice the clock, AVX-512, and page faults that cost 9 µs because of the VM. Of course it disagreed. I stopped treating laptop results as measurements and started treating them as hypotheses that the box would accept or reject.

Belief 7: "The slow phase needs to be faster"

The biggest late gain came from asking a different kind of question: not "how do I make the answer phase faster", but "why does anyone have to wait for it?"

The old program scanned the whole corpus, built one index, then answered. The new design reads the corpus in phases of about 50 MB. Each phase gets its own small 16-bit postings and bitmaps, every query is answered against that phase into a running tally (count, id sum, ten smallest ids), and the next phase reuses the memory. Two things fall out of that. No full index ever exists, so memory drops. And the answers for phase k run on whichever threads have already finished scanning phase k+1, so the answer phase overlaps the scan instead of waiting for all of it, filling time when those threads would otherwise sit idle.

Step	Design	Board run	T	M	Lines	Points
99	one index, then answers	run-71af5d224eec	0.1102 s	47.8 MiB	739	77.32
102	phases with running tallies	run-be2f78572d3a	0.0907 s	41.4 MiB	1,103	78.70

Here is the arithmetic from belief 2, pointing the other way. 364 more lines cost (739/1103)^0.15 = -5.8%. But 18% less time is worth (0.1102/0.0907)^0.5 = +10.2%, and 13% less memory (47.8/41.4)^0.2 = +2.9%. Net +6.8% of K, about 1.4 points, which is what the board gave.

Same formula, same kind of trade, opposite verdict. Neither "shorter is better" nor "faster is better" is a rule. The weights are.

I then tried slicing the phases differently: 1 MiB slices gave 0.0973 s, 128 KiB slices 0.0953 s. The 256 KiB slices stayed.

Belief 8: "All my datasets pass, so it's correct"

While vectorizing a loop that reads eight ids at a time, I found a buffer that grew by 2 × n entries whenever fewer than 32 were left. A 64-byte window can add 32 tokens. Growing by 2n only guarantees that room once n reaches 24. So the first document each thread read could write past its buffer, if its first 64-byte windows held more than 16 one- or two-letter tokens.

All six datasets passed, including the hand-built edge cases. The bug had been on the board for ten steps. I generated 5,000 documents made only of single letters and digits, took the reference's answer, and ran mine: heap corruption, abort. The fix was one expression, 2 * n + 32, and that dense dataset now runs in every check before a push.

The spec's edge cases test the spec. They cannot test your invariants, because the spec doesn't know about your buffers. Whenever you write a bound, write the input that sits exactly on it.

What is still wrong

Track B ends at 0.0907 s, 41.4 MiB, 78.70 points, about 620 times faster than the reference. When I last checked, the leader was at 82.41 points and 0.064 s. I don't know how they get there, and that gap is real.
Query terms are matched by 47 bits of their hash with no byte comparison. The chance that some non-query token in a run collides with a query term is about 3 in 10,000. It has never happened on any dataset, but "never happened" is not "can't happen", and a second tag checked only on a hit would close it cheaply.
My laptop is still a bad model of the box. I've learned which of its answers to distrust, not how to make it agree.
The other tracks ended at 86.16 (log aggregation), 81.93 (CSV join) and 71.92 (shortest paths). Track D in particular never got the kind of breakthrough described here.

The rules I'd give myself on day one

Before you trust a number, say which question it answers. Most of my wasted days came from a correct number answering the wrong question.
Measure cost by removal. "What disappears when this is gone" is very hard to misread.
On shared cores, count misses before instructions. My 10% instruction cut bought nothing; deleting one array that did 0.7% of useful work bought 3%.
Count what a structure does before you speed it up. The answer may be "almost nothing".
Make the target measure itself. Two scalars are enough to carry a phase timing out of a machine you can't log into.
Treat the environment as code. Two lines about huge pages beat a day of tuning, twice.
Do the scoring arithmetic every time. The same trade of lines for speed lost me 0.4 points once and won me 1.4 later.

Check it yourself

The repo includes the organisers' checker, which builds a track offline and runs it in the same sandbox as the board. With Docker and Python, and the sample data generated as the repo's notes describe:

git clone https://github.com/manusingh090/speed-demon-2026-entry
cd speed-demon-2026-entry
python check.py b --data data/b-sample --expected eeeec1e9319da98a6f3c664c78f93e7078fd603c25c2bb3027174e81e3638a8f

Every row of track-b/PERF.md names the board run that measured it, including the failures and the deliberately terrible measurement runs, so any number in this post can be checked against the public board.

References

Speed Demon 2026, by Hackathon Raptors: rules, scoring and the live board
Transparent Hugepage Support, Linux kernel documentation on the always and madvise modes
prctl(2), for PR_SET_THP_DISABLE
The full Track B log: 106 steps, each with its board run

Thanks to Hackathon Raptors for a contest where the scoring machine had the last word on every claim, especially the ones I was sure about.

Seven green checks and a team page that named its judges

Manjeet Singh — Fri, 02 Oct 2026 06:29:50 +0000

Shipshape, the hackathon portal I built for DOGFOOD 2026, gives every team a history panel on its team page and submission page: who joined, who edited the project, when it went in. Once judging started, that panel would also have shown the team lines like these (the real log format; the names come from the test I wrote afterwards):

Jules Judge submitted a score for "Quiet Hours"

Jules Judge declared a conflict with "Quiet Hours": I mentored them

Only that team's members can open those pages. So the people being judged could read who was reviewing them, when each judge scored, whether a judge changed a submitted score, and a judge's private reason for stepping aside (its first 120 characters). Role isolation was the part of the portal I had tested hardest. None of those tests opened a team page, and none of the seven checks in the organizers' acceptance checker does either.

Teams submit projects, judges score them on a rubric, the scores are normalized so one harsh judge cannot sink a project, and the public can vote. It is Django 5.2 on SQLite. The brief came in four tiers (T1 teams and submissions, T2 judging, T3 community voting and comments, T4 an API, webhooks, signed records and more) under hard rules: 72 hours, and docker compose up must bring up a seeded portal with the network off. The organizers supply the acceptance checker and a fixtures.json every portal must load. The fixture has traps: 41 project records although the brief says "forty" (one a resubmission three minutes before the close), 30 judges, 126 scores, and a judge who gave everything a 4.

This is the list of things I believed while building it, grouped by where they broke rather than when.

I thought the features were the hard part

I read the brief as a list of features until I put "deadline actually stops submissions" next to the check that tests it. The checker POSTs {"title": "dogfood-late-submission-probe", "summary": "probe"} to a closed event and passes on any status from 400 to 499; the spec says it does not inspect why you refused. So a 404 would pass. So would a CSRF failure, or a complaint about the missing fields.

A portal could pass that check for the wrong reason, so the order of checks is the design, from the view's first version: signed in (401), event exists (404), JSON from this site (415, a 403 cross_origin, or 400), then the deadline (403 submissions_closed), and only then the team (403 no_team) and the fields (400).

body = read_json(request)
# The deadline comes before anything else about the payload: a late
# request is refused as late, whatever it contains.
try:
    assert_accepting_edits(event)
except WindowError as exc:
    if team is not None:
        services.record_refusal(event, request.user, team, exc, "save the project", "API")
    raise ApiError(403, exc.code, exc.message, deadline=iso_utc(event.submissions_close_at))
if team is None:
    raise ApiError(403, "no_team", "Join or start a team for this event before submitting.")

The test runs with Django's CSRF enforcement switched on, to prove the 403 is the deadline and not a missing token. Every write path re-reads the event row inside its transaction and checks the server clock. The deadline instant itself counts as closed. A refusal is logged after the rollback, so its audit line survives.

The other sentence that was harder than it read was "with the network off", which for two days I read as a rule about runtime, until another machine ran my FROM python:3.12-slim build and it went looking for downloads. The base image and the wheels now live in the repo, built FROM scratch and installed --no-index. I picked bzip2 even though its archive is 36.9 MB to xz's 27.1 MB, because Docker unpacks xz by calling an external xz program wherever the Docker engine runs, but unpacks bzip2 itself, with Go's standard library. An offline Docker-in-Docker box then caught compose trying to pull the image from Docker Hub before building it; pull_policy: never fixed that.

I thought a z-score would fix harsh judges

In the reference example that came with the judging brief, a judge called J-harsh averages 1.71. Fixture judge jdg_02 averages 4.22. A 3 from the first is praise; from the second, a complaint. The usual fix is a z-score: how far a mark sits from that judge's own average, in units of that judge's own spread. A project's score is the mean of its judges' z-scores.

On the fixture, that formula breaks in two places. With exactly two scores, a plain z-score is always plus or minus 0.7071, because the gap between the marks cancels out of the formula. A judge who gave 3.0 and 3.1 speaks as loudly as one who gave 1 and 5. Seven fixture judges have exactly two counted scores, so plain z-scores would turn all 14 of their marks into full-strength votes. And jdg_07, the judge who gave everything a 4, has zero spread: a division by zero.

The fix is shrinkage. Each judge starts with kappa imaginary projects of population-typical spread, and their own marks pull them away from it as they pile up:

sigma_j^2 = ((n_j - 1) * s_j^2 + kappa * sigma_pop^2) / (n_j - 1 + kappa)

With kappa at its default of 5, the two-score judges separate by their actual gap: 0.277 for a third of a point, 0.534 for two thirds, 0.757 for a full point. That last one is above 0.707, so shrinkage can push a z-score up as well as down: when a judge's own spread is wider than the population's (0.65), shrinkage pulls it toward 0.65 and the z-score rises. And every z that jdg_07 produces comes out exactly 0, a neutral vote instead of a crash.

My first test of this used kappa 0.001 as "plain" and failed with AssertionError: 0.6214804438775124 not greater than 0.7. The test was wrong, not the engine: even a thousandth of an imaginary project pulls a nearly flat judge toward the population (0.7071 at kappa 0, 0.6215 at 0.001). The test now pins 0.7071 at kappa 0.

I thought plain z-scores were close enough

The reference picked kappa 5 with no evidence, so I measured: Spearman rank correlation with a hidden true quality (1.0 means the same order), 1,000 simulated events per row, seed 2026, three reviews per project. Condensed from python src/judging/engine.py --trials 1000, which gives the same numbers when rerun:

                               raw    plain z  kappa 5  kappa 25  kappa 5 beats raw
reference judges, 12 projects  0.766  0.838    0.841    0.841     75% of events
reference judges, 40 projects  0.809  0.891    0.890    0.889     98%
fixture-like, 30 judges        0.802  0.812    0.830    0.832     69%

The "plain z" column is computed at kappa 0.01, so a judge with no spread does not divide by zero; kappa 0 gives the same column to three decimals.

With the reference's judges, most of the win is just centring each judge on their own mean, and any kappa from 2 to 25 lands within about 0.003 of plain z. The third row is shaped like the real fixture, 30 judges with about four whole-number scores each, and there plain z barely moves: 0.802 to 0.812. Rerun at true plain z (kappa 0), it beats the raw average in only 55% of those events (56% at the kappa 0.01 the table uses), close to a coin flip, where kappa 5 wins 69%. So plain z-scores went.

Kappa 5 still fails to beat the raw average in about one fixture-like event in three, so the results page counts informative reviews (a review counts only if its judge has two or more scores with some spread) and flags projects ranked on fewer than two. Three fixture judges carry no signal, 5 of the 122 counted reviews, and one project is flagged: Small Relay. Normalization moves 33 of 40 projects, median 2 places. Dry Harbour climbs from 29th to 8th: the judge who dragged it down scored nothing else, so that lone mark normalizes to neutral, and most of its other judges marked it above their own averages (jdg_26 gave 4.67 against a 3.70 average).

I thought my rankings were deterministic

JUDGING.md once promised that ties were broken by raw average, then project id, "so the order is always deterministic". It quoted the fixture's biggest moves from my Windows machine: Glass Beacon up 10 places, Paper Anchor down 10. Before handing judging over, I rechecked every documented figure against a fresh seed, and that was the first time the proof ran inside a Linux container. It said up 8 and down 9.

Three projects average exactly 3.50, and the old code handed out positions, not ranks, so the tie went to whichever float came out a hair larger:

# before: positions 1, 2, 3, even for equal averages
for position, row in _rank(reviewed, key=lambda r: (-r.raw_mean, -r.z_mean, r.project.pk)):
    row.raw_rank = position

# after: competition ranking (1, 2, 2, 4), values compared at 9 decimal places
ordered = sorted(rows, key=lambda r: -round(value(r), 9))

Raw average 3.50	Raw rank, Windows	Raw rank, container	Raw rank, fixed	Normalized rank
Glass Beacon	21	19	19	11
Open Kiln	19	20	19	21
Paper Anchor	20	21	19	30

The fix was competition ranking at nine decimals, a fixed score read order and test_equal_scores_share_a_rank_whatever_order_they_were_summed_in, which took the suite from 125 tests to 126. The rebuilt container and a fresh local seed then printed the proof identically.

My tests could not see it because they all ran on one machine. Working the arithmetic again, I think my docs also blame the wrong thing: they say "summation order" between two databases, but the weighted sum always walks the criteria in a fixed order. The likely culprit is Python:

Python 3.11, each criterion weighted 1/3
  marks (3, 4, 4) sum to 3.666666666666666
  marks (3, 3, 5) sum to 3.6666666666666665
  Glass Beacon's raw mean: 3.4999999999999996

The Windows venv runs 3.11, so Glass Beacon sorts last of the three despite the best z-score. The container runs 3.12.14, whose sum() switched to a more accurate algorithm: all three are exactly 3.5, the old key falls through to the z-score, and Glass Beacon comes first. Both orders I saw follow from that.

A day later another test failed only in the Linux image, with AssertionError: 3.0 == 3.0. It compared whichever of two tied projects came first, both at 3.0 before and after the change it checked, so it had never tested what it claimed.

I thought quadratic voting would stop a loud minority

The community vote was going to be quadratic, the textbook answer to a loud minority: n votes on one project cost n squared credits. My first simulation disagreed. A fifth of the voters backing one weak project won 100% of events, worse than one person, one vote at 96%, because a bloc member puts 5 votes on the target while a sincere voter's favourite gets 2 or 3. So I added a ceiling of 3 votes per project per ballot and reran with 1,000 events per row, 40 projects and 300 voters. "Seen" is how many projects a voter actually looks at:

Method	Best project wins, nobody gaming, 10 seen	Same, 20 seen	20% bloc wins, 10 seen	10% bloc, 3 identities each, 10 seen
One person, one vote	79%	90%	95%	100%
Quadratic, no ceiling	64%	84%	100%	100%
Quadratic, ceiling of 3	48%	82%	57%	100%

At 20 projects seen, the ceiling held a 20% bloc to 0% wins where the uncapped version let it win 40%. The cost is in the first column: when nobody games the vote, one person, one vote crowns the best project (the highest hidden quality) 79% of the time at 10 seen; my capped method manages 48%. It orders the whole field better (rank correlation 0.984 against 0.926), but the winner is the number people remember.

The last column is the same for every method: a bloc whose members each hold three identities wins every event. Past that point the work is in who gets a ballot: three access modes (an open link, email links stored only as hashes, or signed-in accounts) and a detector that flags bursts, shared devices and aliased addresses.

Ballot order mattered too. With one shared order and voters looking at about 10 projects, 94% of the top three came from the first ten listed, and the best project won 13% of the time. A stable shuffle per voter, seeded by an HMAC of who the voter is and never stored, lifted that to 51%.

I thought isolation meant guarding the judges' endpoints

By the time the T3 brief repeated the isolation requirement, judge queries started from the signed-in judge, anything outside a judge's assignments answered 404 like a missing id, and a request for a peer's scores was refused with 403 before any lookup, tested with three spellings of another judge and one id that belongs to nobody.

This time the question was not what a judge can reach, but where judging data ends up. A grep for history in the templates, then a list of judging log calls that pass team=, found it. In T1, the team history panel read Activity.objects.filter(event=event, team=team) with no filter on the kind of entry. In T2, five judging actions (assigning by hand, unassigning, submitting a score, changing one, declaring a conflict) started tagging their audit lines with the team. Neither change was wrong alone, but together they sent T2's judging lines into a panel T1 had built to show teammates who joined and who edited.

Every isolation test probed judging endpoints, and so do the checker's; this leak ran from judges to the team through a generic audit log. The seed imports the fixture's 126 scores without logging anything, which is likely why walkthroughs on seeded data showed nothing odd. The fix is one helper both pages must go through:

# Entries a team must never see in its own history: who judges their project,
# when a judge scored it, a judge's conflict and their reason, and anything
# about the community vote. They stay in the organizers' activity log.
STAFF_ONLY_VERBS = ("score.", "judge.", "judging.", "voting.")

def team_history(event, team, limit):
    entries = Activity.objects.filter(event=event, team=team)
    for prefix in STAFF_ONLY_VERBS:
        entries = entries.exclude(verb__startswith=prefix)
    return entries.select_related("actor")[:limit]

Its test plants a conflict reason in capitals, I MENTORED THEM, and asserts it reaches neither team page but still reaches the organizers' log. The weakness was built in from the start: a deny list keyed on how verbs are spelled.

"Hidden looks exactly like missing" failed once more the next day, in a threat-model test comparing the API's answer for a draft with its answer for an id that does not exist:

AssertionError: {'error': 'not_found', 'detail': 'No such project, or nothing you can see.'} != {'error': 'not_found', 'detail': 'Nothing here, or nothing you can see.'}

Two polite 404s, one sentence apart: enough for anyone counting up through project ids to tell a hidden draft from an empty slot. The hidden case now raises the same Http404 as a missing one.

Some of my tests pass because the attack still works

When the tests moved out of the Django project, running the suite from inside src/ printed Ran 0 tests in 0.000s OK: a pass that touched nothing. The test runner now defaults to the tests/ folder, so it cannot quietly find zero tests.

The threat model, one of the two bonus challenges I took on, lists 43 attacks: 28 stopped, 3 capped or flagged, 1 logged only and 11 open. The doc and the tests share attack ids, and four of the open attacks have tests with _gap_ in the name that pass because the attack works:

def test_SY5_gap_a_fresh_browser_gets_a_fresh_ballot_on_an_open_link(self):
    url = reverse("voting:ballot_link", args=[self.event.slug, self.config.link_token])
    for _ in range(2):
        browser = Client()  # a private window, or cookies cleared
        browser.get(url)
        browser.post(reverse("voting:api_ballot", args=[self.event.slug]) + f"?link={self.config.link_token}",
                     data=json.dumps({"votes": {str(self.other.pk): 1}}), content_type="application/json")
    self.assertEqual(Ballot.objects.filter(event=self.event).count(), 2)

Close that hole and the test goes red, which forces a change to the doc that admits it. A suite that records only what works cannot tell you when your list of weaknesses has gone stale.

Writing the threat model turned up four more holes besides the 404 wording. A passed deadline could be moved later under scores already given (now refused once a score or ballot exists). Submitted projects were public before the deadline, handing early ideas to teams still working (now private by default). One inbox could cast several ballots (more on that below). And nothing looked for a judge boosting a friend.

For that last one I flag a project when one judge's normalized score sits far from the rest of its panel. I first picked a threshold of 2.0, because only 4 of 106 honest gaps on the fixture reached it. Then the simulation ran: at 2.0 the flag caught only 25% of single-judge boosts. The sweep, over 1,000 simulated events:

threshold   1 colluder caught   2 colluders caught   honest projects flagged
1.0         86%                 84%                  29.7%
1.25        75%                 70%                  14.5%
1.5         58%                 56%                  5.8%
1.75        40%                 40%                  2.1%
2.0         25%                 28%                  0.6%

I settled on 1.5: it catches 58% of single boosts and flags 5.8% of honest projects, two or three in every 40, a list short enough for an organizer to read. On the real fixture the flag fires far more often: on 8 of the 32 projects it can check, against the simulated 5.8%, and the 8 projects with only two reviews can never be checked. Two colluders on a three-judge panel still get a bottom-half project into the top three in 24% of events. The cheapest defence was already there: random assignment puts a given judge on a given friend's panel about one time in ten.

On its first run, my HTTP attack probe reported two attacks as working. Both were bugs in the probe, and the doc still lists them. Once fixed, the probe ran 14 attacks and the portal stopped all 14.

I thought an email address was a voter

Until 29 hours into the build (for the first 20 hours that email ballots existed), an email ballot belonged to the address as typed, under a unique constraint on (event, email). My own anti-abuse detector knew better: its canonical_email drops +tags and Gmail dots, so ada+2@x.org is ada@x.org. But the ballot, the per-address link limit and the own-team check all used the typed address, so one inbox could hold one ballot per spelling, merely flagged. My docs even claimed +tags were dropped everywhere.

The fix touched five code paths and zero migrations. Ballot.email started holding a different kind of value under the same constraint name, one_ballot_per_email, with a compatibility clause in place of a data migration:

if voter.kind == Ballot.Kind.EMAIL:
    # Ballots from before inboxes were used keep the address as typed.
    return qs.filter(email__in={inbox(voter.email), voter.email}).order_by("pk").first()

Replaying the change shows what that shortcut costs:

A ballot saved before the change as kim+1@x.org is missed when the inbox returns as kim+2@x.org, so one inbox gets two ballots. Only databases that held email ballots before the change are exposed; a fresh seed has none.
The archive importer writes addresses raw, so lee+1@x.org and lee+2@x.org import as two ballots.
With no stored inbox, the own-team check runs a leading-wildcard LIKE and filters in Python. With 50 Gmail accounts in the table, all 50 were loaded for one check.
The typed spelling is lost from the ballot (Mal.Lory@googlemail.com is stored as mallory@gmail.com; only the email-link row keeps it).

What I would write now: keep email_as_typed, add an indexed inbox column computed at write time, put the unique constraint on (event, inbox), and give EmailPass and User the same column. The importer could not get around a rule the database enforces. The RunPython backfill would fail on any inbox already holding two ballots, and that failure is useful: it forces the "which ballot wins" decision my compatibility clause skips. One cost stays: on a provider where a+b@ and a@ are different people, they share a ballot.

The second redo is the audit log. Activity serves three audiences (the team, organizers, platform admins), and who may read a row is decided by its verb's spelling: of 25 verbs logged with a team, 5 are hidden only by their prefix. I would add an audience column, set when the line is written and defaulting to staff, so a verb nobody thought about stays hidden instead of showing up on a team page.

One choice I would keep: nothing derived is stored. The whole fixture ranking is recomputed per request in about 25 to 30 ms and 6 queries on my machine. So when I noticed, about 32 hours in, that organizers had no button to publish the judges' results, the fix was three columns on the judging settings (results_published_at, results_published_by, share_feedback) and no stored ranking to freeze: the public page runs the same calculation as the organizers' one.

One rubric per event, on purpose

The reference design allowed a rubric per track. Shipshape has one per event, and I would make that cut again. A z-score compares a judge's mark with that judge's own mean and spread, which only means something on one scale. In the fixture, 7 of the 30 judges scored in two tracks, giving 50 of the 126 scores, including jdg_24 and jdg_26, the two judges whose session cookies .dogfood.toml hands the checker. Per-track rubrics would have put 40% of the scores on two scales inside one judge's z-score. Organizers keep weights that stay editable after scoring starts, which is cheap because totals are recomputed on every request.

I also skipped reliability weighting of judges, because early scores would decide how much later ones count. I did not attempt the pairwise judging bonus, and I will not dress that up as a principle: I took on the threat model and API-first instead.

I thought I should claim every tier I built

The checker has seven checks, three for T1 and four for T2, and its rule for a verified tier fits in three lines (plus a loop that drops any tier above one that failed):

verified = [t for t in TIERS
            if any(c.tier == t for c in checks)
            and all(c.ok for c in checks if c.tier == t)]

There are no T3 or T4 checks, so any is false for them and no portal can get them verified. The spec says overclaiming is the one thing that costs points. I built all four tiers, and for about a day and a half my report ended claimed T1 T2 T3 T4, verified T1 T2; my first commit carried it. Then I split the claims: .dogfood.toml claims what a program can confirm, and the README claims T3, T4 and the two bonuses next to the tests behind them. All seven checks pass, and the report now ends:

T2  judge cannot see peer scores ...... PASS
T2  participant blocked ............... PASS
T2  csv export works .................. PASS

claimed T1 T2, verified T1 T2

I held the API docs to the same rule. Checking every /api/v1/ answer against its OpenAPI schema during the tests found five places the docs were wrong, and showed my 307 tests reached only 73 of 134 endpoints, so a run now fails if any endpoint goes unexercised. A Schemathesis run reached about 1,000 requests with no server error before I stopped it, and the docs say that rather than claiming a pass.

Where the failures actually were

The formulas were the part I could prove fastest, because a simulation has no stake in my being right. The method I shipped beats the raw average in 69% of fixture-like events; plain z-scores do it in 55%. I would rather print both numbers than print a ranking and call it fair. Every serious failure here happened somewhere else: the wording of a 404, which Python summed the marks, which column a ballot was keyed on, where an audit line ended up.

Checking facts for this post, I seeded a fresh fixture, set the results date three days ahead, gave "Best in show" to Glass Signal and opened the winning team's page as one of its members. Awards stay private until the results date, so organizers can decide winners before the ceremony, and the public awards list correctly returned 0 rows. The team page returned 200 with "Best in show" in its history, from this line:

log_activity(event, actor, "award.given", f"gave “{prize.name}” to “{project.title}”",
             team=project.team, project=project)

award. is not on the deny list. No test covers it, and it is still there as I write this: the opening scene again, with a different verb. Adding award. to the list would close this one. The audience column would have kept it closed without anyone having to remember.

Try it

git clone https://github.com/manusingh090/Shipshape.git
cd Shipshape
docker compose up

No network is needed. Open http://localhost:8080, choose Sign in and press one of the Be buttons: Be Rosa is the organizer, Be Priya a team captain, Be Tom a participant with no team, Be Ada and Be Diego are judges. The tables come from python src/judging/engine.py --trials 1000 (add --collusion for the collusion simulation) and python src/voting/method.py --trials 1000 (add --seen 20 for the 20-seen column and --order for ballot order), which need only plain Python.

References

The DOGFOOD 2026 brief and acceptance checker, in the repo as tests/acceptance/spec.md and tests/acceptance/run.py
JUDGING.md in the repo: section 4 (normalization and the Monte Carlo proof), section 10 (the vote simulations), section 11 (the threat model)
Schemathesis: https://schemathesis.readthedocs.io/
Shipshape: https://github.com/manusingh090/Shipshape
Google Slides : https://docs.google.com/presentation/d/1z8AnsudKyrrzO9azSSoI51wdYGTVh7hNFrpsKxTI_Eg/edit?usp=sharing