I built a hackathon judging platform, and the test data had no winner in it

TL;DR: I built podium, a self-hostable hackathon submission and judging
platform, in 72 hours: Express, Prisma and Postgres behind a Next.js client,
one docker compose up, 438 tests. The part of the brief that looked like a
formality was "a documented way to even out judges who score harshly or
generously". It turned out to be the whole project. This is the story of a
metric that reached a perfect score by construction and proved nothing, a
judge with two ballots who could move a project 23 places, a test data set
with no signal in it, and a feature I shipped on day one and deleted on day
three.

• Repo: https://github.com/ikeshavvarshney/podium
• Demo video: https://www.youtube.com/watch?v=3qqdSy8PHTQ
• Screens and walkthrough: https://podium-demo.pages.dev/

The problem that looked easy

Judges do not share a scale. One gives 4 out of 5 because nothing is ever a

1. Another gives 5 because the demo worked. Average their raw scores and you are not ranking projects, you are ranking which judges each project happened to draw.

The textbook fix is one line. For each judge, subtract their mean and divide
by their standard deviation:

z(ballot) = (total - mean_j) / sd_j

A project's score is the mean of the z-scores it received. I wrote that on
the first day, the tests passed, and the results screen showed raw rank next
to normalized rank with little arrows between them. It looked finished.

The metric that scored 100% and proved nothing

The organizers ship a shared data set, fixtures.json: 40 projects, 30
judges, 126 score rows. After my importer refused one duplicate submission
and its four ballots, 122 ballots remained.

I wanted a number for the README, so I split the variance in those ballots
by which judge cast them:

Method	Variance explained by the judge
Raw weighted total	27.0%
Per-judge z-score	0.0%

27% down to zero. That is a fantastic table, and it is worthless.
Standardizing sets every judge's mean to zero by definition. The judge
effect did not go to zero because the method is good. It went to zero
because that is what subtraction does. Any method that subtracts the judge
mean scores 0.0% here, including a bad one.

The number I had was measuring my arithmetic, not my ranking. The real
question is whether the normalized order is closer to the truth, and for
that you need a truth.

The data set with no winner in it

So I looked at the other column: how much of the variance is explained by
which project a ballot was for.

Method	Explained by judge	Explained by project
Raw weighted total	27.0%	30.9%
Per-judge z-score, shrunk	3.5%	33.0%
Per-judge z-score, unshrunk	0.0%	33.3%

About a third looks like signal until you ask what chance alone gives. With
40 groups over 122 ballots, pure noise explains 32.2%. The fixture sits
right on it.

I checked a second way. Split each project's ballots at random into two
halves, rank the projects by each half, and correlate the two rankings. If
the scores carry any information about the projects, the halves agree. Over
400 random splits:

raw totals        -0.005
normalized         0.023

Zero. The fixture's scores carry judge effects and almost no project effect.
Which is fair: it is test data for checking that a portal works, not a
recorded hackathon. But it meant the benchmark I had planned to prove my
method on could not tell a good method from a random one. No method can
recover an order that is not there.

This was the benchmark that disappointed me, and I am glad it did before I
published a table built on it.

Building a truth to measure against

The fix was a seeded simulation where every project has a known latent
quality. 40 projects, 30 judges, 3 reviews each, 500 simulated events. Judges
differ in severity (mean 45 to 85) and in spread (4 to 16 points). One judge
in ten gives every project the same score. Then measure Spearman correlation
between each method's ranking and the true order:

Method	Mean Spearman with the true order	Beats raw in
Raw mean	0.749	n/a
Rank average	0.838	86% of events
Per-judge z-score, unshrunk	0.865	n/a
Per-judge z-score, shrunk	0.889	100% of events

That is evidence. It runs with npm run proof, it is seeded so it
reproduces exactly, and a unit test asserts the numbers so the document
cannot drift from the code.

The last two rows are where the maths fought back.

The judge with two ballots

Look at what plain z-scoring does to a judge who only scored a few projects:

• One ballot. Their mean is that ballot. z = 0, whatever they gave. A 5 and a 1 are the same vote.
• Two ballots. Their mean is the midpoint and their standard deviation is half the gap. The z-scores are always exactly +1 and -1. A judge who gave 80 and 79 has put the two projects two standard deviations apart.
• Every ballot identical. Standard deviation zero. Division by zero.

The fixture has all three. 9 of its 30 judges cast fewer than three ballots,
and jdg_07 gave three projects exactly 75.

On the fixture, the unshrunk method moved one project, Dry Harbour, from
31st to 8th. Two of its ballots came from judges with fewer than three
ballots. It had not been scored well. It had been scored by people whose
scale was manufactured from one or two data points.

The fix is the standard empirical-Bayes one: do not trust a thin judge's own
statistics. Pull them toward the panel's.

mean_j* = (n_j * mean_j + 1 * m) / (n_j + 1)        m = mean of all ballots
sd_j*   = sqrt((n_j * var_j + 3 * v) / (n_j + 3))   v = pooled within-judge variance

z = (total - mean_j*) / sd_j*      and 0 for a flat judge

A judge with twenty ballots is read almost entirely on their own scale. A
judge with one is read mostly on the panel's. Dry Harbour now moves from
31st to 17th instead of 8th, and in the simulation agreement with the true
order goes from 0.865 to 0.889.

Three things I would say about this before a reviewer does:

1. The priors were tuned on the simulation. One ballot for the mean, three for the variance (variances are noisier). I chose them where the benchmark was best, so the 0.889 is an in-sample number.
2. The flat judge gets zero, not a penalty. Someone who scores everything 75 has told you nothing about order. Counting them as zero is the honest reading. They are also named on the results screen.
3. Shrinkage keeps 3.5% of the judge effect on purpose. With two ballots, a judge's mean is as much their projects as their severity. Removing all of it is the bug, not the goal.

Pairwise mode, and a modest win

The second judging mode asks judges to order small groups instead of scoring
a rubric. Borda count is the obvious tally. I also fitted a Bradley-Terry
model with the MM algorithm, which weighs a win by the strength of the
project beaten, so a project that drew an easy group is not rewarded for the
draw.

On 300 simulated events: Borda 0.827, Bradley-Terry 0.854, better in 82% of
events. That is a real gain and a small one. I publish the Bradley-Terry
ranking and show Borda beside it, because a ranking an organizer cannot
sanity-check against a simpler one is a ranking they will not trust.

The debugging story: a function's name was the documentation

The OpenAPI document is generated from the live Express routes. To know who
may call a route, the generator walks the middleware stack and reads each
function's name:

const mapped = ROLE_BY_MIDDLEWARE[h.name ?? ""];

requireAuth means signed in, requireEventAdmin means event admin. It
worked for two days.

On the last day I replaced the single admin check with area-scoped access:
an admin can be limited to Announcements, or Judging, or any of ten areas.
requireEventAdmin became a factory:

export function requirePermission(...areas: EventArea[]) {
  return (req, _res, next) => {
    assertPermission(eventContext(req), areas);
    next();
  };
}

Every route kept working. Every permission test passed. And 69 operations
in the published API document silently lost their access label, because an
arrow function returned from a factory has the name "". The API still
refused the requests. The documentation now said anyone could make them.

One test caught it: the one that fetches /api/openapi.json and checks that
admin routes are labelled. The fix is to name the inner function and attach
the areas to it, which the generator now publishes as x-required-area:

const requirePermission = (req, _res, next) => { ... };
return Object.assign(requirePermission, { areas });

The lesson I took is narrower than "test your docs". Reading fn.name is
reflection, and a refactor that changes nothing about behaviour can change
everything about what reflection sees. If documentation is derived from
code, the derivation needs its own assertion.

A second one from the same hour, shorter. My end-to-end check for "a limited
admin is refused on settings" reported that the limit did not work. The API
returned 200. The middleware was correct. The account I had picked as the
limited admin was the instance operator, who
bypasses area checks by design. The product was right and the test was
wrong, and a 200 where I expected 403 looks identical either way.

The design I abandoned

On day one I built judge track scope: an organizer could restrict a
judge to certain tracks, and mark a track as restricted. It had a column, a
dialog, a rule inside the assignment engine and its own tests.

On day three I dropped it. Migration 0026 is two lines:

ALTER TABLE "event_memberships" DROP COLUMN "track_scope";
ALTER TABLE "tracks" DROP COLUMN "restricted";

The reason is that the platform already had an answer to "which projects
does this judge see": the assignment. A judge reads exactly the projects an
organizer assigned and nothing else, and that boundary is enforced in the
API and tested with curl. Track scope was a second answer to the same
question. Two sources of truth meant every path had to consult both, the
auto-balancer had to explain why it left a project under-reviewed, and the
organizer had a setting that could quietly contradict what they had just
done by hand.

The same clean-up hit "reviews per project". It started as a hard cap that
blocked manual assignment past the number. It is now called "Judges per
project" and is only the target the auto-balancer aims for. An organizer who
wants a sixth opinion on a close call can simply assign one.

Both removals made the judging model smaller: one function decides whether a
judge may review a project, and it checks one thing, that it is not their
own team's.

The schema I would redo

Two decisions from the last day, both made because they were fast.

Admin areas are a TEXT[] with a magic string.

ALTER TABLE "event_memberships" ADD COLUMN "permissions" TEXT[] DEFAULT ARRAY[]::TEXT[];
UPDATE "event_memberships" SET "permissions" = ARRAY['ALL'] WHERE "role" = 'ADMIN';

The column sits on every membership row, including participants and judges,
where it means nothing. Postgres cannot check that an entry is a real area,
so validity lives in one Zod transform. And 'ALL' is a sentinel sharing a
namespace with real values. The rest of the schema leans on the database for
correctness: one ballot per judge per project is a unique index, the audit
log is append-only by trigger. This column is the exception. It should be an
enum array with a check that it is empty unless the role is ADMIN, or a
small join table.

Flagging a project reuses the submission status. An organizer can remove
a project from public view with a reason. I stored that as
status = DISQUALIFIED plus three flag columns, which was attractive
because every query that already filtered on SUBMITTED excluded flagged
projects for free: gallery, judge queues, ballots, results. The cost shows
up on restore, which has to put the status back and can only assume it was
SUBMITTED. Visibility and lifecycle are two different facts and I stored
them in one field.

The decision I would keep is the one everything else hangs on: a role is not
a property of an account, it is a row joining an account to an event. The
same person is a judge in one event and a participant in another, and no
request body ever gets to say who the caller is.

What I did not build

Stated here instead of left for someone to find.

• Community voting counts accounts, not people. Registration needs no email verification, because the platform has to run with no mail server. The threat model marks Sybil accounts as reduced, not stopped.
• Collusion checks flag, they do not prove. Lockstep judge pairs, outlier ballots, flat judges and shared organizations are surfaced before publishing. Two judges who vary their scores just enough stay under every threshold.
• The acceptance checker verifies two of my four tiers. It has no probes for T3 or T4, so those claims rest on my own tests and a manual walkthrough.
• A failed seed does not fail the boot. The entrypoint runs the seed with || echo "seed skipped". If the fixture import throws, the API still starts and reports healthy, minus the event. I found this after submitting, while reading my own entrypoint to answer a different question.
• Uploads live in Postgres. Fine for a few hundred images and it keeps the install to one database. Not fine for a large archive.

By the numbers

Build window	72 hours, 140 commits
Server	about 13,400 lines of TypeScript
Client	about 19,500 lines, Next.js 15 and React 19
Tests	438 across 38 files, about 5,400 lines
Migrations	26, applied from an empty database on boot
API	159 operations on 121 paths, OpenAPI 3.1 generated from the routes
Acceptance checker	7 of 7 probes passing
Clean install	cloned from GitHub, healthy in 83 seconds
Raw mean against the true order	0.749
Shrunk z-score against the true order	0.889

What the time actually went on

The z-score took an afternoon. What took the rest was finding out that the
first three ways I tried to show it worked could not have shown it failing.
A variance table that reads zero for any method that subtracts a mean. A
data set where no ranking is recoverable. A two-ballot judge who looks like
a confident expert.

The question that sorted them out was the same each time: could this number
have come out badly? If the answer is no, it is not evidence, however good
it looks in a README.

The fixture had no winner in it. The useful thing the platform can do with
data like that is say so.

---

Built for DOGFOOD 2026, the 72 hour hackathon run
by Hackathon Raptors, where the brief is to build the platform a hackathon
is judged on. The normalization proof, the judging methodology, the threat
model and the full API document are in the repo.

Repo: https://github.com/ikeshavvarshney/podium
Demo video: https://www.youtube.com/watch?v=3qqdSy8PHTQ
Screens and walkthrough: https://podium-demo.pages.dev/

Comments

[Arhan Canli]

The 27.0% "explained by judge" baseline is closer to chance than it looks. With 30 judges over 122 ballots (about 4 each), pure noise already gives (30-1)/(122-1) = 24.0%, the same way your 40-project figure gives 39/121 = 32.2%. So raw sits about 3 points above its null on judge, and 1.3 points below its null on project (30.9%). That fits your shrinkage point: with ~4 ballots a judge's mean is partly the quality of the projects they happened to draw, so subtracting all of it removes signal as well as severity.

A check that doesn't need a simulated truth: for each judge, compare their mean against the mean of the other judges' scores on the same projects. If severity is real, that gap should repeat when you split a judge's ballots in half.

How did you pick the 3 ballots of prior for the variance, and did the 0.889 hold when you re-ran the simulation with a different severity range?

[Muhammed Rasin O M]

If the data has no planted truth, every method looks equally fine. The split-half check is a habit worth copying before trusting any metric. I build Misata, a synthetic data generator that plants declared patterns in business data and ships an answer key, so I'm working on the same idea for tables. I'd be curious how you chose the distribution for the hidden quality.