The first honest run of my judging model ended with two numbers I didn't want to look at.
The best of forty projects scored 3.72 out of 5. The worst scored 3.46. The entire leaderboard was 0.26 points wide, and every score in it was uncertain by about 0.15.
A normal leaderboard would have printed those forty projects as ranks 1 to 40 anyway. That's the problem with the format: a bare rank claims that each project is meaningfully ahead of the one below it, and that claim is far more confident than anything the judges actually told us. (I haven't surveyed every platform, so read "every" in the title as "any leaderboard that prints bare ranks".)
I checked whether my model knew anything the dumbest baseline didn't. The baseline predicts each review with the average of every other score. On a held out review, the model beat it by 0.9%.
So I built the results page to say so. It prints ties and uncertainty instead of forty confident ranks. If you read one line of this post, make it this one: a ranking has to say how little it knows.
What I built
Evenhand is my team's entry for DOGFOOD 2026, a 72 hour hackathon run by Hackathon Raptors. Every team builds the same thing: a submission and judging portal that must start with docker compose up, with the network off, seeded from the organisers' fixtures.json (one event, 30 judges, 40 teams, 41 projects, 126 scores). A checker, run.py, makes seven HTTP requests and prints PASS or FAIL. Evenhand passes 7 of 7 and implements all three tiers, up to community voting.
The stack is a NestJS API, a Next.js web app, Postgres, and a judging engine in pure TypeScript with no I/O, so the maths can be tested on its own. The part the brief itself calls the failure surface is stage seven, normalization: what to do when some judges are harsh and some are generous.
A judge who never varies breaks the textbook fix
Some judges are harsh and some are generous, so a project's raw average depends on who it drew. The textbook fix is a z-score for each judge: subtract their mean, divide by their standard deviation, and every judge is on one scale. I had it half written before I looked properly at the fixtures and found jdg_07.
jdg_07 scored three projects. Every criterion, every project: 4, 4, 4. Their standard deviation is zero, and a z-score divides by it.
The organisers planted that judge on purpose, and it wasn't the only one. On the fixtures, z-scores drop four judges: jdg_07, jdg_12 and jdg_23 for identical scores, and jdg_19 for something subtler. jdg_19 gave 3/5/3, 3/4/4 and 5/4/2, clearly different opinions about three projects, and all three average exactly 3.67. At the level z-scores work, the review's overall score, jdg_19 is flat too.
That case killed a design I had already written. My first warning for flat judges on the progress dashboard flagged judges whose averages never varied, which would have flagged jdg_19 for doing the job properly. It now fires only when a judge gives exactly the same marks on every criterion to at least three projects. On the fixtures, that's jdg_07 and nobody else.
z-scores also assume every judge saw projects of average quality. When judging is split by track, that's false: a judge on the strongest track looks harsh because their projects are good. Raw averages punish projects that drew harsh judges, and z-scores punish projects whose judges saw strong work. I needed something that could tell a harsh judge from a strong project.
Harshness and quality have to be estimated together
Every score depends on two unknowns, the project's quality and the judge's leniency, and the only way to separate them is to look at who reviewed what:
s_pj = μ + q_p + b_j + ε
minimise Σ (s_pj − μ − q_p − b_j)² + λ_b Σ b_j² + λ_q Σ q_p²
A score is the overall average μ, plus the project's quality q, plus the judge's leniency b, plus noise. If two judges reviewed the same projects, the difference between them is leniency. If a project scores highly with judges who are harsh everywhere else, that's quality.
The penalty terms make it ridge regression. The idea isn't mine: Roos, Rothe and Scheuermann proposed estimating reviewer bias and submission quality together in their AAAI 2011 paper on calibrating biased reviewers, and Yehuda Koren's Netflix Prize era paper uses the same equation for recommendations, b_ui = μ + b_u + b_i, with the shrinkage chosen by cross validation. Replace viewers with judges and films with projects, and it's this model.
The penalties do real work. A judge with one review has their leniency pulled towards zero, because one score says almost nothing about a person. A project with two reviews is pulled towards the average and shown with a wider uncertainty. And jdg_07 stops being a problem: nothing divides by their spread, and the model estimates them at +0.04 points, almost exactly average.
I solved it exactly, with a handwritten Cholesky factorisation, and a test checks that the answer is the same whatever order the reviews arrive in. If shuffling the input changes the ranking, it isn't a ranking.
The two λ values decide how hard to pull towards zero, and I didn't want to choose them: a number I choose is a number I can be accused of tuning. So the model picks them by held out error, dropping each review, refitting and predicting it, with an exact shortcut from the hat matrix that a unit test checks against real refits. I planned a grid from 0.25 to 8, and it was hit at both edges: synthetic events chose 0.25, and on the fixtures the error was still falling at 8. A choice on the edge of your search isn't a choice, it's clipping. I widened the grid to run from 0.125 to 64, and the engine now reports when a chosen λ sits on an edge. On the fixtures the data chose λ_b = 32 and λ_q = 16: pull everything hard towards the average.
The number I did not want to publish
With λ that large, I checked how much the model had learned. The baseline's held out error is 0.3996. The fitted model's is 0.3961. That's 0.9%, from 121 reviews, and I should say plainly that I haven't put an interval on it. It would be wide, and I'd rather say so than imply precision.
The model isn't broken. On 20 seeded synthetic events with a known true quality for every project, it recovers the true ranking with a Spearman correlation of 0.920, against 0.896 for raw means and 0.736 for z-scores. That gain over raw means is small at 0.024, and it beats them on 18 of the 20 seeds. When judges differ more, the gap widens to 0.897 against 0.772, and it wins on all 20. But those events are generated from the model's own assumptions, so they prove it recovers what it assumes, not that real judges obey. The honest reading is that the model works and the fixtures carry little signal.
The judges were less sure than any leaderboard would have said
The organisers left the evidence in the data. The same team submitted the same project twice, as prj_07 and prj_41, and three judges scored both copies:
| Judge | prj_07 | prj_41 | Difference |
|---|---|---|---|
| jdg_19 | 2.33 | 3.67 | +1.33 |
| jdg_21 | 3.67 | 4.67 | +1.00 |
| jdg_26 | 4.67 | 3.67 | −1.00 |
The same judge, looking at identical work, disagrees with themselves by 1.11 points on average. If each review carried only the noise the model estimates (σ̂ = 0.60), two reviews of the same work would differ by about 0.68, since the expected gap is 2σ̂ divided by √π. Three pairs prove nothing about any single judge, but the direction is hard to ignore, and it isn't a quirk of invented data. When Pier and colleagues had 43 reviewers score 25 real NIH grant applications, agreement was effectively zero (an intraclass correlation of 0, Krippendorff's α of 0.024), and they concluded that the outcome "depended more on the reviewer to whom the grant was assigned than the research proposed."
That's the gap in the title. The judges contradicted themselves, and a leaderboard built on their scores would still have printed 1 to 40 without a flicker of doubt.
Mine admits it: the whole leaderboard was 0.26 points wide
So I had a choice. I could print forty ranks, clean and confident, and nobody would question it. Or I could print what the data supports.
Evenhand ranks projects, and it also puts them into tie groups: projects within one pooled standard deviation of their group's leader form a group, and the page says to read ranks inside a group as equal. Every row shows the raw mean, the normalized score with its uncertainty drawn as a bar, how far it moved, and a one line reason. On the fixtures, forty projects fall into two groups.
That threshold is a choice, so I tested how much it matters by recomputing the groups from the published table, which is rounded to two decimals. At half the width, the same data gives four groups. At one and a half times the width, it still gives two. At twice the width, it gives one. I'm not claiming the line is sacred.
Then I checked what the correction had actually done. 32 of 40 projects change rank, by at most five places: prj_29 Flat Relay rises five, prj_35 Warm Beacon falls five. Thirty of those moves stay on their own side of the line between the two groups. The only two projects that cross it, prj_31 and prj_17, swap ranks 16 and 17, and both sit right on the line, so even that change belongs to the threshold as much as to the data.
What doesn't depend on the threshold is the width. Forty projects spread across 0.26 points, each uncertain by about 0.15, is not a leaderboard. It's a tie with a few small tilts. A page that announced 32 rank changes would have been announcing news the data can't support.
This isn't only a hackathon problem. Any small panel with rotating reviewers has the same shape: grant committees, conference programmes, hiring loops.
The part I'm proudest of is a refusal
The brief offers a bonus for pairwise judging: judges compare two projects, and a Bradley–Terry model turns the comparisons into a ranking. It's an elegant option, and it has been used at hackathons. Anish Athalye built Gavel for HackMIT on Crowd-BT, a version of Bradley–Terry from Chen and colleagues that also learns how reliable each judge is, so careless or contrary votes count for less. Gavel ranked 195 projects from 1,351 votes by 108 judges. His reason was that "people are much better at making comparative judgements than determining absolute scores".
I cut it and don't regret it, because Gavel works on comparisons its judges actually made. The fixtures contain ratings, not comparisons. To run Bradley–Terry on them, I'd have to turn "judge A gave X a 4 and Y a 3" into "X beat Y": inventing data the judges never gave. The result would look rigorous and mean less than the averages it came from. If a future event collects real comparisons, Crowd-BT is the right tool.
The same logic applies to a project with only one final review. It's listed on the results page, but not ranked. One opinion isn't a measurement.
Three checks that were green for the wrong reason
The maths taught me to distrust confident numbers. The next three bugs taught me the same about confident checks: each was green in one situation and wrong in another.
A refusal is part of the answer. I realised the spec was harder than it looked on the rule the organisers stress most: "if I can curl another judge's scores, it is not isolation." The obvious implementation scopes the query to the caller, finds nothing and returns 404. But then a 404 means "that judge exists and isn't you", and a different answer means they don't exist. The refusal becomes an oracle for who's judging. So every permission check runs before the database is touched. Judge B asking for judge A's scores, and judge B asking for a judge id I made up, get byte for byte the same 403, and a test compares the two bodies. Every service follows one order: permission, then deadline, then the write and its audit row in one transaction.
Seven hours before the freeze, a judge still kept a track they'd lost. The isolation matrix covered 87 of the API's 93 operations across 7 roles, 90 rows and 630 expected status codes, and it was green. We still had a hole on the last morning. Judges are invited per track, and organisers can change a judge's tracks later. The matrix checked every role against every route, but each route at one moment, not across a change. An organiser runs assignment, so a judge gets projects in one track. Then the organiser moves that judge to another. The assignments were still there, and the judge's queue, review page, saves and submits all asked "do you have an assignment for this project?" The answer was still yes, for a track they no longer covered. The fix changed the question, so a judge reaches a project only while their role covers its track now:
export const coveredBy = (userId: string): Prisma.SubmissionWhereInput => ({
OR: [
{ trackId: null },
{ track: { judgeTracks: { some: { judgeRole: { userId, role: 'JUDGE' } } } } },
],
});
Unstarted assignments outside the new tracks are removed with an audit entry. Started reviews stay on the record but are locked, and they return if the track is given back. A permission isn't a fact you store when someone is assigned. It's a question you ask every time, about now.
The duplicate detector that flagged everyone. T3 asks for community voting with "anti abuse that means something". One mode is a shared link, where anyone who opens it gets a ballot, which is the easiest mode to stuff. So I stored the address that took each ballot and showed organisers how many came from an address that took more than one. The tests passed. Then, testing by hand, I looked at the addresses in the audit log: 243 of 246 rows came from 172.19.0.4, and the other three from the API container. 172.19.0.4 is the web container. In the bundled setup, every browser reaches the API through Next.js, so my detector would have told an organiser that their honest voters were cheating. I checked whether the idea could ever work by sending one request with a made up forwarded address, X-Forwarded-For: 203.0.113.9, and the API recorded it. So behind a proxy that reports real addresses, it works, and a ballot's address is now stored only when one was reported:
const reportedAddress = (req: Request): string | null =>
req.headers['x-forwarded-for'] ? ip(req) : null;
Without a proxy, nothing is stored or flagged, and the organiser's page says so. The decision I'd take back is designing the detector before looking at a single real input. My tests used supertest, where every request comes from the same address, exactly the condition that makes the detector meaningless, and they happily asserted that it flagged repeats.
The same lesson cost me an evening in CI. My offline drill starts the stack on empty volumes with every Docker network cut, to prove the portal never needs the internet. It passed on my laptop and failed on GitHub's fresh runners, because the pinned Postgres image had been on my machine since day one and was never downloaded on a clean one. The fix was one line, pulling it before the cut. A test that only passes on the machine you wrote it on is testing your machine.
The schema I'd redo: every review scored a moving target
Two rules that are each right turned out to conflict. Teams can edit their submission until the deadline, because a deadline that applies only to the first save isn't a deadline. And organisers can run assignment while submissions are still open, so judges can start early. Together, a judge could score a project on Saturday, the team could rewrite it on Sunday, and the review would sit attached to a project it had never seen. My schema made that invisible: a review pointed at the live submission row, and the row had no memory.
We shipped a patch before the freeze. A review now stores a SHA-256 of the entry's content when it's submitted, and the entries table and every ranking run mark reviews of an earlier version. If I built it again, I'd model submissions as immutable versions, with a review pointing at the version it scored. The hash tells you that something changed. A version would tell you what the judge saw.
The numbers, and what it doesn't do
| Measure | Result |
|---|---|
Organisers' checker (run.py) |
7 / 7 PASS (T1 and T2) |
| End to end tests (real app, real Postgres) | 937 |
| Unit tests (engine and API) | 125 |
| Role isolation matrix | 87 of 93 operations, 90 rows, 630 expected status codes |
| Browser steps (real Chromium) | 54 |
| Offline drill | Fresh clone, empty database, network cut: all green |
| Width of the leaderboard / typical uncertainty | 0.26 points / about 0.15 |
| Held out prediction, model vs overall mean | 0.9% better |
| Synthetic recovery (Spearman ρ) | 0.920 model, 0.896 raw, 0.736 z-score |
The checker has no T3 checks, so its report ends with "claimed but not verified: T3", and the README maps each T3 requirement to the test that checks it.
Evenhand doesn't model a judge's scale: to the model, a judge using 1 to 5 and one using only 3 to 4 differ in leniency, not range, because one to eleven reviews per judge are too few to estimate both. Comparisons across tracks rest on judges who review in more than one, and the engine reports how many connected groups of judges and projects exist, because two groups with no shared judge can't be compared. It doesn't version submissions. It sends no email, because it runs offline. A shared voting link can't tell people apart, and accounts and emailed links can. And its audit log is append only and hash chained, so a direct edit in Postgres is detected. Detected isn't prevented: a superuser can still rewrite history, just not quietly.
What I'd tell someone building a judging system
Look at the data's worst judge before you choose a method. jdg_07 broke z-scores in one line of arithmetic.
Measure your model against the simplest baseline, publish the result even when it's embarrassing, and print the uncertainty next to every rank. Then check which of your changes survive it. Most of mine didn't.
Ask a permission on every request, not once at assignment. And before building a detector, look at one real input.
To check the main claim yourself:
git clone https://github.com/Len3hq/evenhand
cd evenhand
docker compose up -d --wait # seeded, offline, on http://localhost:8080
npm run acceptance # the organisers' checker: 7 of 7 PASS
npm run proof -w @evenhand/judging-engine # regenerate the Normalization Proof
Open docs/proof/normalization.md and read the table in section 4: the normalized scores run from 3.72 down to 3.46, each with an uncertainty of about 0.15, and the tie groups are marked beside them. A test fails if the committed copy differs from what the code computes, and every design decision has its own record in DECISIONS.md.
A leaderboard is a claim about how much its judges know. Mine says how sure it is.
References
Magnus Roos, Jörg Rothe and Björn Scheuermann, How to Calibrate the Scores of Biased Reviewers by Quadratic Programming, AAAI 2011.
Yehuda Koren, Factor in the Neighbors: Scalable and Accurate Collaborative Filtering, ACM Transactions on Knowledge Discovery from Data 4(1), 2010.
Elizabeth L. Pier et al., Low agreement among reviewers evaluating the same NIH grant applications, PNAS, 2018.
Xi Chen, Paul N. Bennett, Kevyn Collins-Thompson and Eric Horvitz, Pairwise Ranking Aggregation in a Crowdsourced Setting, WSDM 2013.
Anish Athalye, Designing a Better Judging System and Implementing a Scalable Judging System, 2015.
Built for DOGFOOD 2026 Hackathon by Hackathon Raptors #hackathonraptors
Top comments (0)