DEV Community: Daniel

Every hackathon leaderboard is more confident than its judges. Mine admits it.

Daniel — Wed, 07 Oct 2026 13:17:48 +0000

The first honest run of my judging model ended with two numbers I didn't want to look at.

The best of forty projects scored 3.72 out of 5. The worst scored 3.46. The entire leaderboard was 0.26 points wide, and every score in it was uncertain by about 0.15.

A normal leaderboard would have printed those forty projects as ranks 1 to 40 anyway. That's the problem with the format: a bare rank claims that each project is meaningfully ahead of the one below it, and that claim is far more confident than anything the judges actually told us. (I haven't surveyed every platform, so read "every" in the title as "any leaderboard that prints bare ranks".)

I checked whether my model knew anything the dumbest baseline didn't. The baseline predicts each review with the average of every other score. On a held out review, the model beat it by 0.9%.

So I built the results page to say so. It prints ties and uncertainty instead of forty confident ranks. If you read one line of this post, make it this one: a ranking has to say how little it knows.

What I built

Evenhand is my team's entry for DOGFOOD 2026, a 72 hour hackathon run by Hackathon Raptors. Every team builds the same thing: a submission and judging portal that must start with docker compose up, with the network off, seeded from the organisers' fixtures.json (one event, 30 judges, 40 teams, 41 projects, 126 scores). A checker, run.py, makes seven HTTP requests and prints PASS or FAIL. Evenhand passes 7 of 7 and implements all three tiers, up to community voting.

The stack is a NestJS API, a Next.js web app, Postgres, and a judging engine in pure TypeScript with no I/O, so the maths can be tested on its own. The part the brief itself calls the failure surface is stage seven, normalization: what to do when some judges are harsh and some are generous.

A judge who never varies breaks the textbook fix

Some judges are harsh and some are generous, so a project's raw average depends on who it drew. The textbook fix is a z-score for each judge: subtract their mean, divide by their standard deviation, and every judge is on one scale. I had it half written before I looked properly at the fixtures and found jdg_07.

jdg_07 scored three projects. Every criterion, every project: 4, 4, 4. Their standard deviation is zero, and a z-score divides by it.

The organisers planted that judge on purpose, and it wasn't the only one. On the fixtures, z-scores drop four judges: jdg_07, jdg_12 and jdg_23 for identical scores, and jdg_19 for something subtler. jdg_19 gave 3/5/3, 3/4/4 and 5/4/2, clearly different opinions about three projects, and all three average exactly 3.67. At the level z-scores work, the review's overall score, jdg_19 is flat too.

That case killed a design I had already written. My first warning for flat judges on the progress dashboard flagged judges whose averages never varied, which would have flagged jdg_19 for doing the job properly. It now fires only when a judge gives exactly the same marks on every criterion to at least three projects. On the fixtures, that's jdg_07 and nobody else.

z-scores also assume every judge saw projects of average quality. When judging is split by track, that's false: a judge on the strongest track looks harsh because their projects are good. Raw averages punish projects that drew harsh judges, and z-scores punish projects whose judges saw strong work. I needed something that could tell a harsh judge from a strong project.

Harshness and quality have to be estimated together

Every score depends on two unknowns, the project's quality and the judge's leniency, and the only way to separate them is to look at who reviewed what:

s_pj = μ + q_p + b_j + ε

minimise  Σ (s_pj − μ − q_p − b_j)²  +  λ_b Σ b_j²  +  λ_q Σ q_p²

A score is the overall average μ, plus the project's quality q, plus the judge's leniency b, plus noise. If two judges reviewed the same projects, the difference between them is leniency. If a project scores highly with judges who are harsh everywhere else, that's quality.

The penalty terms make it ridge regression. The idea isn't mine: Roos, Rothe and Scheuermann proposed estimating reviewer bias and submission quality together in their AAAI 2011 paper on calibrating biased reviewers, and Yehuda Koren's Netflix Prize era paper uses the same equation for recommendations, b_ui = μ + b_u + b_i, with the shrinkage chosen by cross validation. Replace viewers with judges and films with projects, and it's this model.

The penalties do real work. A judge with one review has their leniency pulled towards zero, because one score says almost nothing about a person. A project with two reviews is pulled towards the average and shown with a wider uncertainty. And jdg_07 stops being a problem: nothing divides by their spread, and the model estimates them at +0.04 points, almost exactly average.

I solved it exactly, with a handwritten Cholesky factorisation, and a test checks that the answer is the same whatever order the reviews arrive in. If shuffling the input changes the ranking, it isn't a ranking.

The two λ values decide how hard to pull towards zero, and I didn't want to choose them: a number I choose is a number I can be accused of tuning. So the model picks them by held out error, dropping each review, refitting and predicting it, with an exact shortcut from the hat matrix that a unit test checks against real refits. I planned a grid from 0.25 to 8, and it was hit at both edges: synthetic events chose 0.25, and on the fixtures the error was still falling at 8. A choice on the edge of your search isn't a choice, it's clipping. I widened the grid to run from 0.125 to 64, and the engine now reports when a chosen λ sits on an edge. On the fixtures the data chose λ_b = 32 and λ_q = 16: pull everything hard towards the average.

The number I did not want to publish

With λ that large, I checked how much the model had learned. The baseline's held out error is 0.3996. The fitted model's is 0.3961. That's 0.9%, from 121 reviews, and I should say plainly that I haven't put an interval on it. It would be wide, and I'd rather say so than imply precision.

The model isn't broken. On 20 seeded synthetic events with a known true quality for every project, it recovers the true ranking with a Spearman correlation of 0.920, against 0.896 for raw means and 0.736 for z-scores. That gain over raw means is small at 0.024, and it beats them on 18 of the 20 seeds. When judges differ more, the gap widens to 0.897 against 0.772, and it wins on all 20. But those events are generated from the model's own assumptions, so they prove it recovers what it assumes, not that real judges obey. The honest reading is that the model works and the fixtures carry little signal.

The judges were less sure than any leaderboard would have said

The organisers left the evidence in the data. The same team submitted the same project twice, as prj_07 and prj_41, and three judges scored both copies:

Judge	prj_07	prj_41	Difference
jdg_19	2.33	3.67	+1.33
jdg_21	3.67	4.67	+1.00
jdg_26	4.67	3.67	−1.00

The same judge, looking at identical work, disagrees with themselves by 1.11 points on average. If each review carried only the noise the model estimates (σ̂ = 0.60), two reviews of the same work would differ by about 0.68, since the expected gap is 2σ̂ divided by √π. Three pairs prove nothing about any single judge, but the direction is hard to ignore, and it isn't a quirk of invented data. When Pier and colleagues had 43 reviewers score 25 real NIH grant applications, agreement was effectively zero (an intraclass correlation of 0, Krippendorff's α of 0.024), and they concluded that the outcome "depended more on the reviewer to whom the grant was assigned than the research proposed."

That's the gap in the title. The judges contradicted themselves, and a leaderboard built on their scores would still have printed 1 to 40 without a flicker of doubt.

Mine admits it: the whole leaderboard was 0.26 points wide

So I had a choice. I could print forty ranks, clean and confident, and nobody would question it. Or I could print what the data supports.

Evenhand ranks projects, and it also puts them into tie groups: projects within one pooled standard deviation of their group's leader form a group, and the page says to read ranks inside a group as equal. Every row shows the raw mean, the normalized score with its uncertainty drawn as a bar, how far it moved, and a one line reason. On the fixtures, forty projects fall into two groups.

That threshold is a choice, so I tested how much it matters by recomputing the groups from the published table, which is rounded to two decimals. At half the width, the same data gives four groups. At one and a half times the width, it still gives two. At twice the width, it gives one. I'm not claiming the line is sacred.

Then I checked what the correction had actually done. 32 of 40 projects change rank, by at most five places: prj_29 Flat Relay rises five, prj_35 Warm Beacon falls five. Thirty of those moves stay on their own side of the line between the two groups. The only two projects that cross it, prj_31 and prj_17, swap ranks 16 and 17, and both sit right on the line, so even that change belongs to the threshold as much as to the data.

What doesn't depend on the threshold is the width. Forty projects spread across 0.26 points, each uncertain by about 0.15, is not a leaderboard. It's a tie with a few small tilts. A page that announced 32 rank changes would have been announcing news the data can't support.

This isn't only a hackathon problem. Any small panel with rotating reviewers has the same shape: grant committees, conference programmes, hiring loops.

The part I'm proudest of is a refusal

The brief offers a bonus for pairwise judging: judges compare two projects, and a Bradley–Terry model turns the comparisons into a ranking. It's an elegant option, and it has been used at hackathons. Anish Athalye built Gavel for HackMIT on Crowd-BT, a version of Bradley–Terry from Chen and colleagues that also learns how reliable each judge is, so careless or contrary votes count for less. Gavel ranked 195 projects from 1,351 votes by 108 judges. His reason was that "people are much better at making comparative judgements than determining absolute scores".

I cut it and don't regret it, because Gavel works on comparisons its judges actually made. The fixtures contain ratings, not comparisons. To run Bradley–Terry on them, I'd have to turn "judge A gave X a 4 and Y a 3" into "X beat Y": inventing data the judges never gave. The result would look rigorous and mean less than the averages it came from. If a future event collects real comparisons, Crowd-BT is the right tool.

The same logic applies to a project with only one final review. It's listed on the results page, but not ranked. One opinion isn't a measurement.

Three checks that were green for the wrong reason

The maths taught me to distrust confident numbers. The next three bugs taught me the same about confident checks: each was green in one situation and wrong in another.

A refusal is part of the answer. I realised the spec was harder than it looked on the rule the organisers stress most: "if I can curl another judge's scores, it is not isolation." The obvious implementation scopes the query to the caller, finds nothing and returns 404. But then a 404 means "that judge exists and isn't you", and a different answer means they don't exist. The refusal becomes an oracle for who's judging. So every permission check runs before the database is touched. Judge B asking for judge A's scores, and judge B asking for a judge id I made up, get byte for byte the same 403, and a test compares the two bodies. Every service follows one order: permission, then deadline, then the write and its audit row in one transaction.

Seven hours before the freeze, a judge still kept a track they'd lost. The isolation matrix covered 87 of the API's 93 operations across 7 roles, 90 rows and 630 expected status codes, and it was green. We still had a hole on the last morning. Judges are invited per track, and organisers can change a judge's tracks later. The matrix checked every role against every route, but each route at one moment, not across a change. An organiser runs assignment, so a judge gets projects in one track. Then the organiser moves that judge to another. The assignments were still there, and the judge's queue, review page, saves and submits all asked "do you have an assignment for this project?" The answer was still yes, for a track they no longer covered. The fix changed the question, so a judge reaches a project only while their role covers its track now:

export const coveredBy = (userId: string): Prisma.SubmissionWhereInput => ({
  OR: [
    { trackId: null },
    { track: { judgeTracks: { some: { judgeRole: { userId, role: 'JUDGE' } } } } },
  ],
});

Unstarted assignments outside the new tracks are removed with an audit entry. Started reviews stay on the record but are locked, and they return if the track is given back. A permission isn't a fact you store when someone is assigned. It's a question you ask every time, about now.

The duplicate detector that flagged everyone. T3 asks for community voting with "anti abuse that means something". One mode is a shared link, where anyone who opens it gets a ballot, which is the easiest mode to stuff. So I stored the address that took each ballot and showed organisers how many came from an address that took more than one. The tests passed. Then, testing by hand, I looked at the addresses in the audit log: 243 of 246 rows came from 172.19.0.4, and the other three from the API container. 172.19.0.4 is the web container. In the bundled setup, every browser reaches the API through Next.js, so my detector would have told an organiser that their honest voters were cheating. I checked whether the idea could ever work by sending one request with a made up forwarded address, X-Forwarded-For: 203.0.113.9, and the API recorded it. So behind a proxy that reports real addresses, it works, and a ballot's address is now stored only when one was reported:

const reportedAddress = (req: Request): string | null =>
  req.headers['x-forwarded-for'] ? ip(req) : null;

Without a proxy, nothing is stored or flagged, and the organiser's page says so. The decision I'd take back is designing the detector before looking at a single real input. My tests used supertest, where every request comes from the same address, exactly the condition that makes the detector meaningless, and they happily asserted that it flagged repeats.

The same lesson cost me an evening in CI. My offline drill starts the stack on empty volumes with every Docker network cut, to prove the portal never needs the internet. It passed on my laptop and failed on GitHub's fresh runners, because the pinned Postgres image had been on my machine since day one and was never downloaded on a clean one. The fix was one line, pulling it before the cut. A test that only passes on the machine you wrote it on is testing your machine.

The schema I'd redo: every review scored a moving target

Two rules that are each right turned out to conflict. Teams can edit their submission until the deadline, because a deadline that applies only to the first save isn't a deadline. And organisers can run assignment while submissions are still open, so judges can start early. Together, a judge could score a project on Saturday, the team could rewrite it on Sunday, and the review would sit attached to a project it had never seen. My schema made that invisible: a review pointed at the live submission row, and the row had no memory.

We shipped a patch before the freeze. A review now stores a SHA-256 of the entry's content when it's submitted, and the entries table and every ranking run mark reviews of an earlier version. If I built it again, I'd model submissions as immutable versions, with a review pointing at the version it scored. The hash tells you that something changed. A version would tell you what the judge saw.

The numbers, and what it doesn't do

Measure	Result
Organisers' checker (`run.py`)	7 / 7 PASS (T1 and T2)
End to end tests (real app, real Postgres)	937
Unit tests (engine and API)	125
Role isolation matrix	87 of 93 operations, 90 rows, 630 expected status codes
Browser steps (real Chromium)	54
Offline drill	Fresh clone, empty database, network cut: all green
Width of the leaderboard / typical uncertainty	0.26 points / about 0.15
Held out prediction, model vs overall mean	0.9% better
Synthetic recovery (Spearman ρ)	0.920 model, 0.896 raw, 0.736 z-score

The checker has no T3 checks, so its report ends with "claimed but not verified: T3", and the README maps each T3 requirement to the test that checks it.

Evenhand doesn't model a judge's scale: to the model, a judge using 1 to 5 and one using only 3 to 4 differ in leniency, not range, because one to eleven reviews per judge are too few to estimate both. Comparisons across tracks rest on judges who review in more than one, and the engine reports how many connected groups of judges and projects exist, because two groups with no shared judge can't be compared. It doesn't version submissions. It sends no email, because it runs offline. A shared voting link can't tell people apart, and accounts and emailed links can. And its audit log is append only and hash chained, so a direct edit in Postgres is detected. Detected isn't prevented: a superuser can still rewrite history, just not quietly.

What I'd tell someone building a judging system

Look at the data's worst judge before you choose a method. jdg_07 broke z-scores in one line of arithmetic.

Measure your model against the simplest baseline, publish the result even when it's embarrassing, and print the uncertainty next to every rank. Then check which of your changes survive it. Most of mine didn't.

Ask a permission on every request, not once at assignment. And before building a detector, look at one real input.

To check the main claim yourself:

git clone https://github.com/Len3hq/evenhand
cd evenhand
docker compose up -d --wait                  # seeded, offline, on http://localhost:8080
npm run acceptance                           # the organisers' checker: 7 of 7 PASS
npm run proof -w @evenhand/judging-engine    # regenerate the Normalization Proof

Open docs/proof/normalization.md and read the table in section 4: the normalized scores run from 3.72 down to 3.46, each with an uncertainty of about 0.15, and the tie groups are marked beside them. A test fails if the committed copy differs from what the code computes, and every design decision has its own record in DECISIONS.md.

A leaderboard is a claim about how much its judges know. Mine says how sure it is.

References

Magnus Roos, Jörg Rothe and Björn Scheuermann, How to Calibrate the Scores of Biased Reviewers by Quadratic Programming, AAAI 2011.
Yehuda Koren, Factor in the Neighbors: Scalable and Accurate Collaborative Filtering, ACM Transactions on Knowledge Discovery from Data 4(1), 2010.
Elizabeth L. Pier et al., Low agreement among reviewers evaluating the same NIH grant applications, PNAS, 2018.
Xi Chen, Paul N. Bennett, Kevyn Collins-Thompson and Eric Horvitz, Pairwise Ranking Aggregation in a Crowdsourced Setting, WSDM 2013.
Anish Athalye, Designing a Better Judging System and Implementing a Scalable Judging System, 2015.

Built for DOGFOOD 2026 Hackathon by Hackathon Raptors #hackathonraptors

Fifty Green Tests, Two Real Bugs, and the Oracle One `npm install` Away

Daniel — Mon, 31 Aug 2026 16:27:34 +0000

A QR code cannot hide a bug from you, because checking it costs nothing. You point a phone at it. A URL pops up or it doesn't.

That's why I picked a QR encoder for a hackathon whose entire rule is standard library only: no packages, an empty dependency manifest, verified at submission. I rebuilt qrcode, the library nearly every JavaScript project reaches for, in 844 lines of Node and an empty dependency manifest. By the end I had fifty tests, all green, and two bugs that not one of them could see. A phone found the first in four seconds, on day three, the first time I bothered to try. A tool one command away would have found both on the first afternoon. I just wouldn't run it, because running it felt like cheating.

The encoder was the easy part. This is a write-up about the hard part, which was verification, which I also got wrong.

What `npm install qrcode` actually installs

qrcode gets 24 million downloads a week. Before rebuilding it I looked at what it was.

npm install qrcode writes 29 packages to disk, 2.4 MB. Three of them do the actual work: qrcode itself, pngjs for image output, and dijkstrajs, a shortest-path graph algorithm it uses to choose the most compact way to split text across QR's encoding modes. The other 26 are yargs and its dependency tree, there for a command-line interface: cliui, string-width, wrap-ansi, emoji-regex, y18n, camelcase, and down. They install on your server whether or not you ever run qrcode from a terminal. Call require('qrcode') and you have shipped an internationalized argument parser and everything under it.

This isn't a complaint about qrcode. It's that "one dependency" was a fiction. The number was 29, and I hadn't read a line of any of them, which is the normal condition of every Node service I've worked on: a few thousand packages of strangers' code, unread, running as me. The polite word is "dependency." The accurate word is borrowed trust. I trust these people because auditing them is more work than the feature is worth.

The hackathon's premise is that we borrow far more than we need. So how much of the 29 is load-bearing?

The parts you build, and the one you don't

Most of it. The specification (ISO/IEC 18004) is a stack of small closed problems: Reed-Solomon error correction over GF(2⁸), which is two lookup tables and polynomial division; the module matrix with its finder squares and timing lines; eight mask patterns and a penalty function to choose one; fifteen and eighteen bits of metadata wrapped in BCH codes. None of it needs anything outside std.

The one place I stopped was DEFLATE. PNG compresses its pixels with zlib, and a DEFLATE encoder is a hackathon project by itself. node:zlib is standard library; I called deflateSync and wrote the rest of the PNG by hand: signature, chunks, CRCs. That's the single asterisk on "zero dependencies." It's the opening line of my README and the last time I'll mention it, because it isn't where this goes.

Fifty green tests and a phone that said no

By the end of day two I had a test suite I trusted. It wasn't lazy.

Galois-field arithmetic, checked against the values printed in the standard. Reed-Solomon output, checked byte for byte against the standard's worked example: encode "01234567" at version 1, level M, and Annex I gives you the exact codewords. The matrix, checked by an invariant: total modules minus fixed-pattern modules must equal the data-bit count, for every version. Format and version bits, checked against the standard's BCH tables. And a round-trip: a small decoder in the test file that walked the data path and pulled the bytes back, asserting decode(encode(text)) === text for every text, every error level, every mask.

Fifty tests. Green. I rendered "01234567", held up my phone, and got nothing. Bigger, more light, a printout, a different scanner app. Nothing scanned.

The bug a scanner reads first

QR format information is fifteen bits (mask, error level, ten of BCH check) written twice for redundancy, one copy in an L around the top-left finder square. A scanner reads it before it reads anything else.

This placed the first six:

for (let i = 0; i <= 5; i++) modules[8][i] = bit(i);

modules[row][col]. That runs along row 8. It should run down column 8:

for (let i = 0; i <= 5; i++) modules[i][8] = bit(i);

Row and column, swapped, in both copies. Every bit a scanner needs to begin was on the wrong axis.

The round-trip test didn't catch it because it never read the format bits. They're fixed-function modules, and the data-path walk skips them by definition. The payload round-tripped perfectly, and the payload was never where the bug was.

The BCH test didn't catch it because it checked the value, not the address. My fifteen bits were the correct fifteen bits. I had verified what they were and assumed that told me where they went.

Fifty tests, one blind spot

The tests were not weak. They were the wrong shape.

Every one was rigorous about computation: fields, polynomials, BCH. Computation is the part of a QR code that looks hard, and it's the part I'd worked to understand. None checked placement, because by the time I wrote tests, placement felt beneath checking. Put the bits where the figure shows them. I wasn't going to get that wrong.

A test suite is built from its author's model of where the danger is. Mine faithfully encoded mine. The bug lived in the exact gap between "I checked this closely" and "this didn't need checking," and fifty green marks correctly reported that everything I'd thought to test was fine.

That generalizes past QR codes. A passing suite is evidence about the parts you thought to check. Its silence is not evidence.

The oracle I refused to install

qrcode was one npm install away the whole time. I could have generated the same code with both, in a scratch directory that never touched my submission, and diffed the matrices. The transposed bits would have shown immediately.

I didn't, because on a zero-dependency project, installing the thing I was replacing felt like cheating. Not against a rule (there is no such rule), but against the point. I was doing this properly. From the spec. Myself.

That reflex is the same one the hackathon criticizes, pointed inward. "Don't blindly trust a stranger's package" and "don't blindly trust your own code" are one statement. Both kinds of trust need an outside check, and I had argued myself out of the best one on the table.

Writing this, I finally ran the diff. The broken build differs from qrcode on "01234567" by twelve modules, all in the format region, all visible in the first comparison anyone would run. A phone in four seconds, or one diff on day one. I took neither.

The diff also found a second bug.

The second bug, the one that scanned fine

With the format bits fixed, my codes scanned, and I thought I was done. Then I diffed qrkit against qrcode properly: every byte-mode input I could think of, versions 1 through 10, all four error levels. 47 of 48 came back byte-identical. The one that didn't was the string "zero dependency": same version, same mask, same mode, 124 modules different.

It was the padding. When your data doesn't fill the symbol, the spec tops it up with two alternating codewords, 0xEC and 0x11, and it says the first is always 0xEC. Mine:

while (bytes.length < capacity) bytes.push(pad[bytes.length % 2 === 0 ? 0 : 1]);

I picked the pad byte from its absolute position in the stream. When the real data ended on an even byte count the first pad came out 0xEC, correct by accident. When it ended odd, like "zero dependency" does, the first pad came out 0x11 and the entire tail inverted.

This bug cannot break a scan. Pad codewords are discarded before the decoder reads your data. The round-trip test walked over these exact bytes on every run and could not see it, because "correct" meant "the text comes back," and the text always came back. Wrong, and fully functional. Only a byte comparison against an implementation I didn't write could surface it, and that was the comparison I'd spent three days avoiding.

The fix counts the pad bytes instead of the stream bytes, plus a fifty-first test so it can't come back:

for (let p = 0; bytes.length < capacity; p++) bytes.push(pad[p % 2]);

The spec that disagrees with itself

There's a third source of truth here, and it's also unreliable.

Mask selection is meant to be deterministic: apply eight masks, score each, keep the lowest. But the rule that penalizes finder-lookalike patterns is loose enough that implementations disagree on what counts. qrkit, whose scoring I ported from Nayuki's reference implementation, and qrcode, which rolls its own, choose different masks on three of every eight inputs I tested. Both scan.

And a 2008 note by Johan Persson works through the standard's own example, the "01234567" I'd been testing against, and shows the mask it picks is not the lowest-scoring one by the standard's own rules. The reference fails the reference.

So the spec is an oracle you cross-check too. There's no document that hands you the answer. There's a quorum: the spec, its examples, other implementations, a physical scanner. Where they disagree, you decide. I lost an evening treating my mask disagreement with qrcode as a defect before I understood that.

The scorecard, told honestly

Measure	Result
`npm install qrcode`	29 packages, 2.4 MB
`yargs` and its subtree	26 of the 29
`qrkit`	0 dependencies, one 844-line file
Format matrix vs `qrcode`, all 4 EC levels by 8 masks	32 / 32 byte-identical
Byte-mode output vs `qrcode`, versions 1 to 10 by 4 EC levels, after the fix	48 / 48 byte-identical
Mask choice matching `qrcode` when both choose freely	5 / 8
Encode speed, `qrkit` vs `qrcode`	677 µs vs 271 µs
Bugs shipped	2
found by 50 unit tests	0
found by a phone	1
found by a diff available on day one	2

The speed number held at 2.5x across four runs. That's not measurement noise, qrkit is genuinely slower. It re-scores all eight masks with the full penalty function on every build; qrcode has had years to not do that. The value here is the third row of the table, not the seventh.

What I didn't do

I wrote this list before I wrote the README, because the failure mode of a solo project is finding out your limits were load-bearing.

Byte and numeric encoding modes only, no alphanumeric, no kanji, so an all-caps string that qrcode packs tighter comes out larger from qrkit. Versions 1 through 10, not 40. No ECI, no structured append. No decoder: reading a code back out of a photo is thresholding, perspective correction, and error location, and it's the same large, fragile surface that would have sunk the project if I'd tried it in a weekend.

What I'd tell someone building one of these

Reach for the oracle you didn't write first, before the first unit test. A phone, a reference implementation, a printed example. If you're rebuilding X, npm install X in a scratch directory and diff. That's the control group.
Treat a green suite as a map of the ground you thought to cover. Ask what class of bug it structurally cannot see.
"Different function" is not "independent implementation." An encoder and a decoder written by one person from one reading of one spec agree with each other, not with the world.
Some bugs never fail a functional test, because the function still works. Conformance is a separate question, and only a foreign implementation answers it.
Write down what you didn't do.

Where trust actually comes from

I expected the lesson to be about dependencies, that I'd finish having shown you don't need the tree.

That isn't it. Deleting the dependencies bought me nothing; an empty manifest is just a different set of things to be wrong about. Writing it myself bought me less than I thought, because every check I built came from the same head as the bug. My test file is 769 lines against 844 of source. I spent nearly as long proving the thing as building it, and I still shipped two defects, because the proof and the code had the same author and the same blind spot.

The bugs died when the code met something with no stake in whether I was right: a phone camera, a committee's standard, a stranger's hand arithmetic from 2008, and the library I was too proud to install.

References

ISO/IEC 18004, QR Code bar code symbology specification. Current revision 2024; the 01234567 worked example and codeword tables cited here are in Annex I of the 2015 revision. Paid standard.
Johan Persson, "A note on minor errors in the International QR Barcode standard" (2008). Source for the claim that the standard's own worked example does not select the lowest-penalty mask under its own rules, and for the hand-computed matrix qrkit is tested against. https://www.coastalmonitoring.org/resources/jpclass/QR/qr-comment.pdf
Project Nayuki, "QR Code generator library." The reference implementation qrkit's mask-penalty scoring follows. https://www.nayuki.io/page/qr-code-generator-library
qrcode (npm), MIT. The package qrkit replaces and is diffed against. https://github.com/soldair/node-qrcode. Download figure for the week of 23 to 29 August 2026, from https://www.npmjs.com/package/qrcode.
qrkit, MIT. https://github.com/Dannyamah/qrkit

Built for the Zero Dependency Hackathon by Hackathon Raptors. #hackathonraptors

140 Bugs Were Hiding in One Function, and My Tests Couldn't See Any of Them

Daniel — Mon, 10 Aug 2026 09:57:25 +0000

Anyone can port a library. Point a translator at the source, clean up the output, get it to compile, and you have something that looks like a port. The actual engineering problem is different and much harder: proving that the new code means the same thing as the old code, across thirty algorithms, hundreds of edge cases, and a test suite written by people who were not thinking about you.

This is the story of porting textdistance, a Python library for measuring string similarity, to Rust. The result is textdistance-rs. The porting took a fraction of the time. Everything else: the differential fuzzing, the 140 divergences, the 35 year old threshold I violated, the floating-point drift at the 15th decimal place, is what this writeup is actually about.

Thirty algorithms and one architectural bet

The original textdistance covers a lot of ground: edit-based distances (Levenshtein, Damerau-Levenshtein, Hamming, Jaro-Winkler), token-based measures (Jaccard, Sørensen-Dice, cosine, Tversky), sequence-based methods (LCS, Ratcliff-Obershelp), phonetic algorithms (MRA, Editex), and compression-based distances built on normalized compression distance. Over thirty algorithms in total, all reimplemented in Rust.

But before writing a single algorithm, I had to make the decision that shaped everything downstream: how does the existing Python test suite (397 tests I did not write) talk to the Rust code?

The obvious answer is PyO3: wrap every algorithm in a #[pyclass], build a native extension, and the Python tests import Rust directly. The answer I chose instead was a subprocess CLI. The Rust core is a standalone binary that speaks JSON over stdin/stdout, and a thin Python adapter shells out to it:

// The entire cross-language surface is one struct.
#[derive(Deserialize)]
struct Request {
    algorithm: String,
    s1: String,
    s2: String,
    qval: Option<usize>,
    external: Option<bool>,
}

fn dispatch(req: &Request) -> Response {
    match req.algorithm.as_str() {
        "hamming" => hamming::distance(&req.s1, &req.s2),
        "levenshtein" => levenshtein::distance(&req.s1, &req.s2),
        "jaro_winkler" => jaro::winkler_similarity(&req.s1, &req.s2),
        // …thirty more arms
        _ => Response::error("unknown algorithm"),
    }
}

The reasoning: PyO3 binds you to Python's ABI, complicates the build for anyone who just wants the Rust crate, and turns every signature mismatch into a compile-and-link problem. A JSON pipe is crude, but it is debuggable with your eyes. You can replay any failing case by piping a JSON string into the binary from your shell. When the goal is verifying behavioral equivalence against a foreign test suite, that inspectability is worth a lot.

It was a speed decision, and I knew it. What I did not know was that this bet had quietly closed a door I would walk straight into later on.

The bug that was older than the port

The single worst debugging session of the project traced back to Jaro-Winkler, and specifically to a paper published in 1990.

Jaro similarity comes from Matthew Jaro's 1989 work on record linkage for census data. William Winkler's 1990 refinement adds a bonus for strings that share a common prefix, on the empirical observation that clerical typos rarely occur in the first few characters of a name. My first implementation applied that prefix boost the way the formula usually gets quoted:

// What I wrote first: boost applied unconditionally.
let jaro = jaro_similarity(s1, s2);
jaro + (prefix_len as f64) * P * (1.0 - jaro)

It looks right. It passes the examples people paste into blog posts. And it is wrong, because Winkler's actual method only applies the boost when the base Jaro score exceeds a threshold of 0.7 in his published work. Below that, the strings are too dissimilar for a shared prefix to mean anything, and boosting them inflates scores for garbage matches:

// What Winkler actually specified in 1990.
let jaro = jaro_similarity(s1, s2);
if jaro > BOOST_THRESHOLD { // 0.7
    jaro + (prefix_len as f64) * P * (1.0 - jaro)
} else {
    jaro
}

Combined with a second, smaller mistake: the original library treats two empty strings as perfectly similar (1.0), and my port returned 0.0. This one function accounted for 140 divergences in fuzz testing. One commit fixed all of them.

The lesson that stuck with me: I was not diverging from the Python library. I was diverging from a 35-year-old paper that the Python library had implemented faithfully and I had implemented from memory. When you port a library, you inherit its citations, not just its code.

Proof by ten thousand disagreements

That raises the obvious question: how did I even know there were 140 divergences? The unit tests for Jaro-Winkler passed. Example-based tests check the inputs somebody thought of, and nobody thinks of "ab" vs "aaab" with a coincidental prefix and a sub-threshold Jaro score.

The answer was differential fuzzing — running both implementations on the same randomized inputs and diffing the outputs. The technique goes back to McKeeman's work on differential testing for compilers: when you have two implementations that claim to compute the same function, disagreement between them is a test oracle you get for free. You don't need to know the correct answer; you only need to know they should match.

The harness is embarrassingly simple, which is the point:

from hypothesis import given, strategies as st
import textdistance          # the Python original
import textdistance_rs       # the port, via the JSON pipe

@given(st.text(), st.text())
def test_jaro_winkler_parity(s1, s2):
    expected = textdistance.jaro_winkler(s1, s2)
    actual = textdistance_rs.jaro_winkler(s1, s2)
    assert abs(expected - actual) < 1e-9, (s1, s2, expected, actual)

Hypothesis, Python's property-based testing library, descended from Claessen and Hughes' QuickCheck, generates the adversarial inputs: empty strings, single characters, Unicode combining marks, strings that are prefixes of each other. Every property-based failure gets shrunk to a minimal reproducing case, which is how a six-hour bug hunt eventually collapsed into a two-character counterexample I could reason about on paper.

After the Jaro-Winkler fix and a handful of smaller ones, the fuzzer reached zero divergences across every algorithm and input class it could generate. That number, not the passing unit tests, is the claim I actually stand behind. But notice the caveat hiding in that sentence: zero divergences on everything that could cross the pipe. Two things could not.

The door I closed on myself

Several of textdistance's alignment algorithms — Needleman-Wunsch, Smith-Waterman, Gotoh — accept a sim_func parameter: an arbitrary Python callable that scores the similarity of two characters. The test suite exercises it:

# From the original test suite. This cannot be serialized.
alg = NeedlemanWunsch(sim_func=lambda a, b: 1 if a == b else -1)

A lambda cannot be serialized into JSON and sent down a pipe to a Rust subprocess. It is a closure over a live Python interpreter. There is no workaround that isn't a horror, no eval-strings-in-Rust, no RPC-callback-per-character scheme that would be slower than just running the Python.

So those tests fail. 22 of them, permanently, by architecture rather than by bug. This is the direct, foreseeable cost of the subprocess decision from the beginning of this article. PyO3 would have handled it natively: a #[pyfunction] can hold a reference to a Python callable and invoke it from Rust mid-algorithm. I traded that capability for build simplicity and debuggability, and for 28 algorithms the trade was clearly right. For these 4, I built a wall and then walked into it.

If I ran this project again, I would keep the subprocess architecture for development and verification, it earned it's keep during fuzzing, and add a PyO3 layer at the end specifically for callable-taking algorithms. That hybrid was always possible. I just didn't see the need until the tests told me.

The 15th decimal place

The other place the port compromised was subtler, and it's the result I'm least happy reporting: the "pure Rust" story does not fully hold for the compression-based algorithms.

Normalized compression distance, from Cilibrasi and Vitányi's work on clustering by compression, approximates the information distance between strings using a real compressor:

NCD(x, y) = (C(xy) − min(C(x), C(y))) / max(C(x), C(y))

By construction, the output is exquisitely sensitive to the compressor's exact byte counts and the arithmetic done on them.

The original library computes arithmetic-coding probabilities using Python's Fraction, exact rational arithmetic, no rounding. My Rust implementation used f64. The fuzzer caught divergences on the order of 1e-15: not wrong in any way a human would notice, but not equal, and for NCD "close" is doing real work in the formula, because those values get subtracted and divided in ways that can amplify the drift.

I tried rational arithmetic in Rust. I tried reordering operations to match Python's evaluation order. What shipped is the honest defeat: for the arithmetic-coding NCD path, the final distance computation moved back into the Python adapter, where Fraction guarantees exact parity. Rust still does the heavy lifting; Python does the last arithmetic mile. Eight algorithms carry that asterisk, and I'd rather document the asterisk than pretend it isn't there.

What the port actually proved

None of these four things — the JSON pipe, the Winkler threshold, the fuzzing harness, the sim_func wall were about writing Rust. They were about the much slower work of finding every place a fast, plausible looking reimplementation quietly disagrees with the thing it's replacing, and then deciding, for each one, whether to fix it, work around it, or write it down and ship anyway.

That's the actual deliverable here, more than the crate itself: not thirty algorithms in Rust, but a documented account of where equivalence held, where it took a 35 year old paper to restore, and where it simply didn't. A port you can't interrogate isn't really verified, it's just translated. This one, at zero fuzz divergences and two known, named exceptions, is the former.

References

life4/textdistance — the original Python library.
Len3hq/textdistance-rs — the Rust port discussed throughout.
Jaro, M. A. (1989). Advances in record-linking methodology as applied to matching the 1985 census of Tampa, Florida. Journal of the American Statistical Association, 84(406), 414–420.
Winkler, W. E. (1990). String Comparator Metrics and Enhanced Decision Rules in the Fellegi-Sunter Model of Record Linkage. Proceedings of the Section on Survey Research Methods, American Statistical Association.
McKeeman, W. M. (1998). Differential Testing for Software. Digital Technical Journal, 10(1), 100–107.
Claessen, K., & Hughes, J. (2000). QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. Proceedings of the Fifth ACM SIGPLAN International Conference on Functional Programming (ICFP '00).
Cilibrasi, R., & Vitányi, P. M. B. (2005). Clustering by Compression. IEEE Transactions on Information Theory, 51(4), 1523–1545.

The full port and the DECISIONS log this writeup is pulled from is up at Len3hq/textdistance-rs if you want to dig in yourself.
Written for Hackathon Raptors, worth a follow if you're into this kind of writeup.

DEV Community: Daniel

Every hackathon leaderboard is more confident than its judges. Mine admits it.

What I built

A judge who never varies breaks the textbook fix

Harshness and quality have to be estimated together

The number I did not want to publish

The judges were less sure than any leaderboard would have said

Mine admits it: the whole leaderboard was 0.26 points wide

The part I'm proudest of is a refusal

Three checks that were green for the wrong reason

The schema I'd redo: every review scored a moving target

The numbers, and what it doesn't do

What I'd tell someone building a judging system

Fifty Green Tests, Two Real Bugs, and the Oracle One `npm install` Away

What npm install qrcode actually installs

The parts you build, and the one you don't

Fifty green tests and a phone that said no

The bug a scanner reads first

Fifty tests, one blind spot

The oracle I refused to install

The second bug, the one that scanned fine

The spec that disagrees with itself

The scorecard, told honestly

What I didn't do

What I'd tell someone building one of these

Where trust actually comes from

References

140 Bugs Were Hiding in One Function, and My Tests Couldn't See Any of Them

Thirty algorithms and one architectural bet

The bug that was older than the port

Proof by ten thousand disagreements

The door I closed on myself

The 15th decimal place

What the port actually proved

References

What `npm install qrcode` actually installs