How Not To Rank a Hackathon

Mihir Rabari — Thu, 08 Oct 2026 16:11:06 +0000

Every hackathon I've entered, somebody else decided my fate. This one made me build the judge first.

DogFood Hackatho, run by Hackathon Raptors, gives every team the same 72-hour job: build a submission-and-judging platform against one published spec, one anonymized dataset, and one acceptance suite. Then the twist, straight from their brief: "the winning project is the one we run." Your code accepts your own submission, scores it, proves the score wasn't forged, and prints your certificate. The platform that judges you is the thing you shipped.

We were two people: Manas on backend, data model, and the judging math. Me on frontend, voting, the embeddable widget, the threat model, and the demo video. At the deadline we had all four spec tiers plus all four bonuses, a single docker compose up to a seeded portal, 548 passing tests, a 7/7 acceptance report, and a collection of scars worth more than any of it. The best ones:

1. The architecture decision was deleting most of the architecture

Day one we drew the stack that production instinct tells you to draw: seven Docker services. Postgres, Django, Next.js, nginx, then Prometheus, Alertmanager and Grafana on top. That's what a mature system looks like, right?

Then we reread the acceptance checker. It's vendored byte-for-byte from the spec's appendix, we diffed it to be sure (8,855 characters, identical), and it does exactly seven HTTP checks against a portal that must be up in 30 seconds. Grafana is not involved. Nobody asks for your dashboard. No panel has ever mattered less.

Mature systems have observability. Acceptance judges don't. We deleted the observability layer entirely, then went further and folded the remaining four services, Postgres, Django, Next.js and nginx, into one container with supervisord as PID 1.

If you self-host this in production, split it back out. That's not what it's for. It's for the organizer who clones the repo on a laptop five minutes before judging and needs a seeded portal, not an orchestration lesson. We traded some debuggability for an install story with zero moving parts. The customer is a stranger with a timer. Design for the stranger.

2. The scoring algorithm we almost shipped

The spec says judges score projects against a weighted rubric, then asks how you rank the results. The obvious implementation is the arithmetic mean. One line. We had it working.

Then Manas asked the question that killed it: what happens when one judge is systematically generous and another grades like a machine? Raw averages mix project quality with judge calibration, and the lenient judge drags every project they touched up the table. "We averaged the scores" is an answer. It rewards whoever was kindest.

So we fit a two-way additive model instead:

y_ij = μ + b_j + q_i + ε_ij

b_j is judge j's bias, q_i is project i's quality. Fitted by alternating means, Gauss-Seidel style: start with every bias at zero, compute qualities as the mean of de-biased scores, compute biases as the mean of de-qualified scores, recenter so the biases sum to zero, repeat until it settles. Every mean is taken only over cells that actually exist, because real score matrices are full of holes.

The edge cases were the real work. A judge who scores everything identically carries zero information: their bias reenters as zero and we flag their leverage instead of silently dropping them. A disconnected score graph (judges A saw projects 1-3, judges B saw 4-6, no overlap) gets is_connected: false and the API refuses to publish a ranking, 422, rather than invent one. Refusing felt like failure for about an hour. Then we realized you can't un-publish a winners list.

And we committed the proof: normalization-proof.txt in the repo, a rank-movement table on the fixture dataset showing bias shrinks and winners don't shuffle. For the pairwise bonus, a Bradley-Terry MM estimator with a phantom prior and a recovered-ranking test. Here's the industry punchline: Devfolio advertises "automatic score normalization" as a headline feature and publishes nothing about how it works. If the math decides who wins, the math is the product. Show it.

3. Three Docker gotchas

Postgres identifiers and the rebrand. We renamed the project mid-build, and a bulk rename turned POSTGRES_USER=dogfood into POSTGRES_USER=hack-hamster. Postgres refuses: SQL identifiers can't have hyphens. Identifiers stayed underscored (hack_hamster), the brand stayed hyphenated. SQL syntax outlives brand decisions.

The nginx Host header. With proxy_pass http://django_backend, nginx forwards the upstream name as the Host header by default. Django's ALLOWED_HOSTS check rejected Host: django_backend and the entire API 400'd behind the proxy while working perfectly when hit directly. One proxy_set_header Host $host; per location. Defaults are upstream-name-shaped, not request-shaped.

Webhook delivery threads. First version delivered webhooks from a hidden thread pool: a queue, two daemon workers, visible to no one. Worked great in dev. But a self-hoster inherits a worker process they didn't ask for and can't see, and "it works on my machine" is not a delivery guarantee. We replaced it with a synchronous single attempt plus a manage.py flush_webhooks batch retry. Explicit beats invisible.

4. The spec sentence that wasn't simple

"CSV export at every stage." We read it as one endpoint with an event_slug query parameter, built it, moved on. The spec means one CSV per stage: we ended up at six endpoints (scores, roster, assignments, submissions, audit, rankings). Annoying at the time, correct in hindsight. Six is more cacheable, more testable, and more honest about what an organizer is actually downloading.

The sneakier trap was the demo config. Our first seeding flow printed random session tokens and told you to paste them into the platform config file. Every fresh Docker volume invalidated it, which means the config we shipped worked on exactly one machine on Earth: ours. The fix was deterministic demo tokens, HMAC-SHA256(DJANGO_SECRET_KEY, "{label}:{email}"), derived from things that don't change across volumes. Never a database primary key; fresh volumes renumber those. New rule we live by: never bake runtime-ephemeral values into committed config, and never derive committed values from row order.

5. Role isolation is a backend property

Three roles plus anonymous, and it's very easy to enforce the boundaries in the UI and feel done. We treated it as a backend property and walked the whole thing over raw HTTP: 48 cells of role-by-endpoint matrix. Organizer hitting judge endpoints: 403. A judge asking for another judge's scores: 403 (that's the graded cell in the spec, peer scores stay sealed). Participants can't touch the CSV export. Anonymous: 401 everywhere except the public gallery. Zero mismatches, committed as a generated matrix file so it can't rot. Sessions are HMAC-signed, so a tampered cookie fails verification instead of failing open. Enforcement lives in the permission layer, not in components. If it's enforced in the UI, it's a suggestion.

One bug from the final hours belongs to the same family: demo account password hashes were getting clobbered by the boot-time fixture re-import, so the committed demo logins died on any fresh volume. Anything that must survive a fresh volume can't depend on insertion order.

6. How the endgame actually ran

The rules say AI tools are expected, so here's the honest version. The last night we ran about ten coding agents in parallel against the issue tracker: small, single-purpose issues, one branch each, a human merge queue into dev, no agent ever merged its own PR. They were good at exactly what you'd hope (pinning tests, docs conformance, port mismatches, small endpoint bugs) and useless at spec judgment, which stayed with us.

The part that made it survivable had nothing to do with AI. From hour zero we ran a conflict-zero contract: no file has two owners. I never touched config/, Manas never touched web/. Disjoint edits, zero merge conflicts, which means no tired person at hour 60 resolving a conflict badly and quietly breaking something that was green an hour earlier. We also froze the seam, the .hack-hamster.toml file the acceptance checker reads, at hour 20. After that, neither half of the team could break the other.

At hour 68 I was recording the demo video while the last fixes landed behind me. One full event lifecycle, submission to signed certificate, 2 minutes 27 seconds.

7. The receipts

My favorite line in the repo is in acceptance-report.txt. The spec's seven checks only machine-verify tiers 1 and 2. We built all four tiers. So the report we committed for judging reads, in full honesty: claimed but not verified: T3 T4.

We could have written "all tiers verified." Almost nobody would have checked. The organizers' own rule 06 is "claim your tiers honestly."

Here's what that line bought us: once a report admits what it can't prove, everything else in the repo reads differently. The 548 tests. The role matrix. The normalization proof. The conformance check that refuses to ship if a single route is undocumented. A repo that admits one gap is a repo you can trust on everything else.

If you want to check our work

Everything is in the open repo: choksi2212/dogfood-hackathon. Ten-minute reading order: .hack-hamster.toml (what we claim), acceptance-report.txt (what the checker verified), ARCHITECTURE.md (why the shape is what it is), JUDGING.md (the math, defended), THREAT-MODEL.md (the attacks we assumed).

The platform that judged us forced a kind of honesty a normal side project never demands: you can't bluff code that has to run against itself. If somebody forks it and runs a real event on it, that's the actual win.

Built at DogFood 2026 by Hackathon Raptors. Team Hack Hamster: Manas Choksi (backend, data model, judging math) and me, Mihir Rabari (frontend, voting, widget, threat model, demo video).

DEV Community: Mihir Rabari