Rankings with receipts: what building a hackathon judging platform taught us

DOGFOOD 2026 had a theme that sounds like a joke until you try it: build the platform that will judge you. Every team builds a hackathon portal. The organisers then run their checker against it, and the portals themselves become the thing being judged.

We built Evenhand: a self-hosted portal where teams submit to a public gallery, judges score against a weighted rubric, and organisers publish rankings that are corrected for harsh and generous judges. Each ranking comes with a receipt.

The spec had a few hard rules:

• docker compose up has to bring up a working, seeded portal with the network off. No cloud accounts, no hosted database, no external API.
• The organisers ship a fixtures.json of 41 projects, 29 judges and 121 reviews. It has traps planted in it.
• A checker (run.py) makes curl-style requests against your portal. The check the spec calls "the one that matters most" asks: can judge B read judge A's scores? The answer has to be no.

We used NestJS, Next.js, Prisma 7 and PostgreSQL 16, in three containers with one public port. Below are four things we learned. Each one is something we only understood by building it.

---

1. The architecture decision that shaped everything: the API is the only authority, and the network enforces it

Early on we wrote one sentence on the whiteboard, and it settled most later arguments:

If I can curl another judge's scores, it is not isolation.

Hackathon portals usually hide things in the UI. The judge page doesn't render other judges' scores, so they look private. But the checker never opens a browser. It sends HTTP requests. So we decided that every rule lives in the API, and the Next.js app is just one more client with no more trust than curl.

           browser                     curl / run.py / scripts
              │ cookie: session              │ Authorization: Bearer <token>
              ▼                              ▼
 ┌──────────────────────────── edge network ────────────────────────────┐
 │  web  :8080   Next.js  ── the only published port                     │
 │               /api/* is proxied to the api                            │
 └───────────────┬───────────────────────────────────────────────────────┘
                 │  backend network (internal: no route to the internet)
 ┌───────────────▼───────────────────────────────────────────────────────┐
 │  api  :3001   NestJS                                                   │
 │    ThrottlerGuard → SessionGuard → RoleGuard → route guards → DTOs     │
 │    @evenhand/judging-engine (pure TS, zero dependencies)               │
 │  db   :5432   PostgreSQL 16                                            │
 └────────────────────────────────────────────────────────────────────────┘

That one decision drove four others.

Deny first, before touching the database. If a judge asks for /api/judges/jdg_24/scores and they aren't jdg_24, the API refuses before it checks whether jdg_24 exists. A real judge and a made-up one get the same 403. If you check existence first and return 404 for unknown judges, an attacker can list your judges from the outside. The whole permission matrix (visitor, participant, judge, organiser, admin, against own scores, peer scores, another event, CSV, audit log) is one end-to-end test that runs on every change.

One origin. Next.js rewrites /api/* to the NestJS container, so the browser, curl and the checker all use http://localhost:8080. Cookies stay same-origin, there's no CORS to configure, and the checker needs only one base_url. Remember this choice, because it caused section 2.

"No network" is built into the topology instead of left to discipline. db and api sit only on an internal Docker network:

networks:
  backend:
    internal: true   # no route to the internet
  edge: {}

services:
  api:
    networks: [backend]
  web:
    networks: [backend, edge]   # the only container that can be reached
    ports: ['8080:8080']

If someone adds an npm package that phones home, it fails immediately with EAI_AGAIN, in development as well as in CI, so we never have to remember to test offline. It also caught something we didn't know about. The Prisma CLI runs migrate deploy on every container start, and it was trying to report its version to checkpoint.prisma.io. The internal network blocked it, and we set CHECKPOINT_DISABLE=1 so it stops trying. CI also runs an offline drill: clone the repo, build with the network on, then start on empty volumes with every network internal and run the organisers' checker and our browser tests.

The maths that decides who wins is a pure library. src/judging-engine has no dependencies, no database and no clock. It takes reviews as input and returns rankings. Rows are summed in a fixed order, so results are identical bit for bit whatever order the reviews arrive in. That made section 3 possible: we could test the algorithm against thousands of synthetic events in milliseconds, and we could generate a proof document from the code and fail the build if the committed proof differs from what the code computes today.

---

2. The Docker gotcha: the proxy that made every user the same person

This bug wasn't in a Dockerfile. It came from the network layout.

We had rate limits on everything: logins, exports, review saves, comments, votes. They were tested and they worked. NestJS's throttler keys limits by client address by default, which is the obvious choice.

Then we ran a live test against the full Docker stack instead of the API alone, and saw this:

• 21 bad bearer tokens from anyone locked out every user for the rest of the minute, admins included.
• Eleven different people logging in within one minute: the 10th and 11th got 429 Too Many Requests.
• Exports, review saves, comments and votes were each one allowance shared by the whole event.

The cause was the "one origin" decision from section 1. Every browser request reaches the API through the Next.js container, so the API sees one IP address for every user. Keying limits by IP behind your own proxy turns "per user" into "for everyone." A limit meant to stop one person brute-forcing tokens became a way for one person to take the event offline.

It got worse. The API trusted X-Forwarded-For from the web container, and the web container passed on whatever the client sent. So an attacker could also choose the address the API saw, and get past the per-address login limit by changing a header.

The fix was to key limits by who is calling instead of where the request comes from:

export function rateLimitTracker(req: TrackedRequest): string {
  const auth = req.headers?.authorization;
  if (typeof auth === 'string' && auth.startsWith('Bearer ')) {
    return `token:${hashToken(auth.slice('Bearer '.length).trim())}`;
  }
  const session = req.cookies?.session;
  if (typeof session === 'string' && session) return `session:${hashToken(session)}`;
  // ...personal voting links get their own bucket too
  return `ip:${req.ip ?? 'unknown'}`; // only anonymous requests share the address
}

Two more rules came with it:

• Login and registration are limited per email, not per address. Guessing one account's password stays at 10 tries a minute however the attacker varies X-Forwarded-For, and people signing in at the same moment don't use up each other's attempts.
• The failed-credential limiter refuses only failing requests. After 20 failures in a minute from an address, further bad tokens get 429 instead of 401, but a valid credential always gets through. Tokens are random 256-bit values, so letting a correct one through tells an attacker nothing. The limiter slows guessing down; the token entropy is what actually protects accounts.

The trade-off is written down: someone who knows your email can use up your logins for a minute. Sessions that are already open are unaffected. That's a much smaller problem than one person locking out a whole hackathon during judging.

Lesson: if your app sits behind a reverse proxy you ship yourself, every IP-keyed control in it is suspect. Test rate limits through the real network path, not against the bare service.

Smaller Docker problems that also cost us time

• CRLF in the entrypoint. Git on Windows checks files out with CRLF line endings by default. docker/api-entrypoint.sh gets copied into the image with #!/bin/sh\r as its first line, and the container fails with exec /usr/local/bin/api-entrypoint.sh: no such file or directory. The file is there; the error is really about the interpreter /bin/sh\r, which doesn't exist. We fixed it twice: * text=auto eol=lf in .gitattributes, and a sed -i 's/\r$//' in the Dockerfile for copies that arrive from a zip or an editor. LF everywhere also keeps Prisma's migration checksums identical across machines.
• Prisma chooses its engine binary by the OpenSSL version it finds at install time. If the build stage and the runtime stage see different OpenSSL versions, the runtime tries to download a matching engine on startup. Online you'd never notice. Offline it fails. Every stage now shares one base image with OpenSSL installed, and the build fails early if the OpenSSL 3 engine is missing:

  RUN ls node_modules/@prisma/engines/schema-engine-*openssl-3.0.x >/dev/null \
   || { echo "Prisma schema engine for OpenSSL 3 missing: runtime would try to download it" >&2; exit 1; }

• Next.js rewrites are fixed at build time. Setting the API address as a runtime environment variable does nothing for rewrites(); it has to be a build ARG.
• A fresh CI runner has no cached images. The offline step starts with --pull never, so the drill has to pull the pinned Postgres image while the network is still on.

---

3. The scoring algorithm we almost shipped

The textbook answer

The spec asks for "cross-judge normalization, with the method documented." The usual method is per-judge z-scores: subtract each judge's mean, divide by their standard deviation, then average per project. It's in every "how to run a hackathon" guide, it's about 10 lines of code, and it's broken in two ways that the fixtures happen to expose.

Break 1: it divides by zero. The fixtures include jdg_07, who gave 4/4/4 to every project they reviewed: a flat judge. Their standard deviation is 0, so their z-score is undefined. Two more judges have only one review each, so their SD is also 0. And jdg_19's three reviews happen to average exactly the same. With z-scores, four judges' reviews silently disappear.

Break 2: it assumes every judge saw projects of average quality. That's only true if judges are assigned at random. Ours aren't: judges are matched to tracks. Suppose one track is stronger than another. The judge who sees only strong projects gives high scores because the projects are good, and z-scoring takes that away from them. The z-score can't tell "this judge is generous" from "this judge saw good projects."

What we built instead

We estimate every project's quality and every judge's leniency jointly, from the full record of who reviewed what:

s_pj = μ + q_p + b_j + ε

minimise  Σ (s_pj − μ − q_p − b_j)²  +  λ_b Σ b_j²  +  λ_q Σ q_p²

q_p is project quality and b_j is judge leniency. The ridge penalties pull both towards zero when there's little evidence. (This is equivalent to Gaussian priors on both, the approach from Roos, Rothe & Scheuermann, AAAI 2011.) It's solved exactly with a hand-written Cholesky factorisation inside the pure engine.

This model doesn't need either z-score assumption:

• A flat judge's offset is absorbed by b̂, and nothing divides by their spread.
• A judge with one review is shrunk towards zero, because one score says little about a judge.
• A project with two reviews is shrunk towards the mean and shown with a wider uncertainty.
• A judge who reviews in two tracks acts as a bridge, so the model can compare those tracks.

We don't choose the λs; the data does. They're picked by leave-one-review-out prediction error, computed exactly from the hat matrix (e_i / (1 − h_ii)), and a test checks that against real refits. The grid we planned (0.25 to 8) turned out too narrow: on the fixtures, the error was still falling at 8. We widened it to 0.125–64, and the result now carries an atEdge flag so a choice clipped by the grid is reported instead of passing silently.

On 20 seeded synthetic events with a known true ranking (40 projects, 5 tracks of different strength, 30 judges):

Method	Mean Spearman ρ vs truth	With harsher judges
Joint ridge model	0.92	0.90
Raw mean	0.90	0.77
Per-judge z-score	0.74	0.74

On the synthetic data, z-scores did worse than doing nothing.

The finding we didn't expect: the fixtures contain almost no signal

On the organisers' real fixture data, the model predicts a held-out review only 0.9% better than predicting the overall mean. Who judged a project, and which project it was, explain very little of any single score.

We have direct evidence for this. One team submitted the same project twice (more on that in section 4), and three judges scored both copies. The same judges, scoring the same project, disagreed with themselves by 1.11 points on average on a 1–5 scale.

We could have printed 40 confident ranks. Instead the portal shows tie groups: projects the data can't separate share a group. On the fixtures there are 2 groups, not 40 ranks. A ranking that claims more precision than the data has misleads people, and the people it misleads are the ones who didn't win.

The tie-grouping has its own trap. Group "projects within one SD of their neighbour" and a long run of small gaps chains everyone into a single tie. So a project joins a group only if it's within one pooled SD of the group's leader:

export function tieGroups(ranked: readonly { normalized: number; sd: number }[]): number[] {
  const groups: number[] = [];
  let group = 0;
  let leader: { normalized: number; sd: number } | undefined;
  for (const p of ranked) {
    const pooled = leader ? Math.sqrt((leader.sd ** 2 + p.sd ** 2) / 2) : 0;
    if (!leader || leader.normalized - p.normalized >= pooled) {
      group++;
      leader = p;
    }
    groups.push(group);
  }
  return groups;
}

The bug we actually wrote: flagging flat judges

The organiser dashboard flags judges who "gave every project the same score," so jdg_07 can be spotted before results are computed. Our first version flagged judges whose weighted scores were identical across reviews.

The fixtures proved it wrong. jdg_19 gave 3/5/3, 3/4/4 and 5/4/2. All three average 3.67. That judge clearly told the projects apart; the averages just matched by chance. Our flag put them in the same group as jdg_07.

The fix: a judge is flat only when they give exactly the same marks, criterion by criterion, across at least three final reviews. Two reviews are too few to tell either way. On the fixtures, that flags jdg_07 and nobody else.

A related rule: a missing or out-of-range criterion is an error, not a zero. Treating a blank as 0 is the kind of default that quietly changes who wins. The engine throws a RubricError, and the database rejects out-of-range values as well.

---

4. The spec line we thought was simple: duplicates

The spec says "forty projects." The fixture file has 41. The extra one is "Dry Harbour", submitted twice by team tm_07 as prj_07 and prj_41. Both copies have reviews.

We planned for it with two rules that each looked obvious:

1. One live submission per team. It's a business rule, so enforce it in the database with a partial unique index.
2. Show both copies of a suspected duplicate until an organiser decides. Never silently drop someone's work.

The two rules contradict each other on the first import. Loading the fixtures as they are violates rule 1. Enforcing rule 1 throws away data, which breaks rule 2.

The fix was a third state. The older copy goes on hold, and the index ignores held rows:

CREATE UNIQUE INDEX "submissions_one_live_per_team"
  ON "submissions" ("event_id", "team_id")
  WHERE "superseded_by_id" IS NULL AND NOT "duplicate_hold";

That only covered importing. Resolving the duplicate turned out to be most of the work.

Which reviews count after a merge? "The later score wins" sounds reasonable, but the fixture scores have no timestamps, so "later" can't be computed. Dropping the older copy's reviews would delete jdg_01's only score in the whole file. The rule we settled on:

• If a judge reviewed both copies, the kept (newer) copy's review counts, since they judged the final version. Their older review is kept but marked superseded.
• If a judge reviewed only the older copy, the review moves to the kept entry and records where it came from.
• On the fixtures, prj_41 ends up with 6 counted reviews.

It has to be undoable exactly. Organisers make mistakes. "Reopen" reverses a decision row by row, and a test compares every assignment and review against a snapshot taken before the decision.

Decisions can't chain. If A merges into B and B later merges into C, undoing the first decision becomes ambiguous. So confirming is refused (409 duplicate_chain) when either entry is already part of a confirmed decision. Two organisers can click two flags that share an entry at the same moment, so both rows are locked with SELECT … FOR UPDATE before the check, and the flag leaves PENDING through a conditional update so it can't be decided twice.

"In judging" had three different definitions. Assignment, the judge console, the progress dashboard and rankings each had their own idea of which entries count. Duplicates, holds and disqualifications made the copies disagree. We replaced them with one function: submitted, eligible, not held, not replaced. Everything that needs the definition calls it.

Rankings had to notice on their own. Every ranking run stores SHA-256 hashes of its inputs and its result. A duplicate decision changes the reviews a run reads, so the inputs hash no longer matches, the run goes stale, and the portal refuses to publish it. It also refuses to publish while any duplicate is still undecided. We didn't need a special case for this; the receipts handled it.

The duplicate also gave us something useful. A team submitting the same work twice, scored by the same judges, is a free test–retest experiment. That's where the 1.11-point self-disagreement in section 3 comes from. The hardest part of the spec ended up supporting our main argument about the scoring.

Runner-up for "simple until it wasn't": "edit until the deadline." Does submitting lock the entry? (No: a team that finds a typo at 17:55 should be able to fix it.) Does the deadline apply to the first write or to every write? (Every write: create, edit and submit, checked against the server clock in UTC.) Can organisers edit a team's entry? (No: they can read it, but the content belongs to the team.)

---

What we'd tell the next team

1. Write down where authority lives before you write any code. "The API is the only authority" settled more arguments than any other sentence in the project.
2. Use topology to enforce constraints. An internal: true network caught a dependency calling home that no code review would have spotted.
3. Any IP-keyed control behind your own proxy is probably wrong. Test through the real network path.
4. Test the textbook algorithm against your actual data before you ship it. Z-scores looked fine until we counted the judges they silently dropped.
5. Report uncertainty instead of hiding it. "These 16 projects can't be told apart" is a more honest result than a confident rank 7.
6. A spec line with two "obvious" rules in it probably has a conflict between them. Look for it before the import fails.
7. Write a decision record for each choice, at the time you make it. We have 32 of them. When something broke on the last day, we could see why it had been built that way.

The code, the 32 decision records and the generated normalization proof are all in the repo: github.com/Len3hq/evenhand. It's MIT-licensed, and docker compose up gets you a seeded portal at localhost:8080, network or no network.

Built with Claude Code. The humans on the team made the design decisions, reviewed every change, and can defend the schema and the maths in writing.