DEV Community: Gaurav

I built a hackathon judging app and it was too confident

Gaurav — Fri, 02 Oct 2026 06:48:25 +0000

DOGFOOD 2026 had a funny premise. Everyone builds the same thing: the portal that will judge the hackathon. The winner gets forked and used for real events. So you are building the thing that decides who gets the prize money, and then it gets used on you.

My entry is called Raptor Desk. Django, HTMX, SQLite, one docker compose up. This post isn't a feature tour. It's about the part that actually kept me busy: getting a fair ranking out of messy judge scores, then finding out my app was more sure of itself than it had any right to be.

Repo: https://github.com/GauravS13/raptor-desk

the official data is a trap

The organizers ship a fixtures file with the brief: 40 projects, 30 judges, 126 score sheets, each scoring functionality, quality and innovation. It looks boring. It isn't.

The plain average has a tie for first place. prj_11 and prj_34 both score 4.333.
prj_10 is 3rd on the average. It had two reviews, and one came from jdg_15, the second most generous judge in the set. Correct for that and it drops to 7th. That's prize money moving because of who happened to review it.
One judge (jdg_07) gives the exact same score on every criterion for every project.
prj_41 is prj_07 submitted twice.
Three different teams are called "StillTrail".
49 reviews have no written comment at all.

The one that changed how I thought about the whole problem: judges differ more from each other than projects do. Judge leniency in this data runs from about −0.82 to +0.52. The gap between neighbouring projects is 0.02 to 0.10. So "who reviewed you" matters more than "how good you are" unless you correct for it.

the textbook fix breaks on this data

The first thing everyone reaches for is a per-judge z-score. Take each judge's scores, subtract their mean, divide by their standard deviation. Harsh and generous judges end up on the same scale.

On the official fixtures it gives NaN for 7 projects. Dividing by the standard deviation is the problem. The flat-lining judge has a spread of zero. So does any judge who only reviewed one project. Divide by zero, and those projects can't be ranked at all.

What I used instead is an additive model. Every score is the project's quality, plus the judge's leniency, plus noise:

score(judge, project) = quality(project) + leniency(judge) + noise

Fit it with a little ridge regularisation so a judge with one review can't get a wild leniency estimate. Then alternate: estimate qualities given leniencies, then leniencies given qualities, until it settles.

q = (sum over judges of (s - b[judge]) + lam * mu) / (n_reviews + lam)
b = (sum over projects of (s - q[project])) / (n_reviews_by_judge + lam)

It never divides by anyone's spread, so the flat-liner can't break it. That judge's scores carry no ranking information anyway, so the app detects it (identical scores on every criterion, every time) and leaves it out of the fit.

How do you know it's better? The fixtures have no right answer, so I made data that does. 200 fake hackathons where I know each project's real quality, with harsh and generous judges, noise, missing reviews and one flat-liner. Then I compare how well each method gets the true order back.

method	how well it recovers the true order (Spearman)	beats the plain average in
plain average	0.697	–
z-score with shrinkage	0.778	96% of runs
additive model	0.813	96% of runs

Good. Bias correction works. That part went fine.

then i made it say how sure it was

A ranking with no uncertainty is a lie with decimals. So the app resamples each project's reviews 300 times, refits, and counts how often each project lands in the top 1, 2, 3 and 5. That gives every project a "chance of a top-5 finish". Anything between 20% and 80% is flagged as a close call.

On the fixtures this says what you'd hope: places 1 and 2 look solid, places 3 to 5 are close calls. Then the app suggests which judge should review which close-call project to settle it, within a budget the organizer sets. Whatever is still close after that goes to a deliberation page where a person has to write down why they decided.

I was pretty happy with this. Then I tested it.

then i checked if it was telling the truth

If the app says "90% interval", the true value should land inside it about 90% of the time. If it says a project has a 70% chance of a top-5 finish, projects like that should make the top 5 about 70% of the time. I had the 200 fake hackathons with known answers, so I could just check. 7,973 projects in total.

It was too confident.

The "90% intervals" contained the true quality 52% of the time. Not 90.
For projects with one useful review it was 22%. With two, 48%. With three or more, 59%.
When the app said 95% or more, the project actually finished top 5 about 80% of the time.

The reason is kind of obvious once you see it. The resampling only shuffles the reviews a project already has. With two or three reviews there just aren't many ways to shuffle, so the spread looks small. It can't imagine a fourth judge who disagrees. With one review there's nothing to shuffle at all, so the interval has zero width. Very confident, about nothing.

It wasn't useless though. Two numbers made me keep it:

As a forecast it still clearly beats guessing. Brier score 0.073, against 0.109 for just saying the base rate every time (lower is better).
Of all the projects the ranking put on the wrong side of the prize line, half had already been flagged as close calls. The doubt was pointing at the right places, it just wasn't strong enough.

So I didn't hide it. The numbers are in the repo's JUDGING.md, generated by a script that CI reruns, so they can't quietly drift. And the app's job changed a bit in my head: chances are for pointing people at the doubt, never for deciding. A model never settles a prize place on its own.

The fix is known. You resample the judges' leniencies too, not just the reviews. I didn't ship it. It would have changed every chance the app shows, and I'd want to rerun the same calibration check on it first. Shipping a fix for overconfidence without measuring it would be the same mistake again.

one judge can pick the winner

The calibration result made me wonder what else the resampling couldn't see. It shuffles reviews but always keeps every judge. So what happens if you take a judge out?

I added a "kingmaker check" (the code is short). Refit the whole ranking once per judge, leaving that judge out. If removing one judge alone changes who's in the top 5, or who's first, flag it.

On the official fixtures:

without judge	their reviews	what changes
jdg_04	4	first place goes from prj_34 to prj_37
jdg_15	6	first place goes from prj_34 to prj_11
jdg_24	11	prj_16 and prj_18 move into the top 5, prj_33 and prj_37 drop out
six others	5 to 9	one project in or out of the top 5

9 of 30 judges each move a top-5 place on their own. And two of them flip first place, the one place the resampling called fairly safe. jdg_04 did four reviews. Four.

The reason is the same as before. Each judge's leniency is estimated from a handful of reviews. Take a judge out and the corrections for everyone they overlapped with shift a little, and at the top the gaps are tiny.

I thought about automatically down-weighting these judges and decided not to. A kingmaker isn't a bad judge. It just means a prize is resting on one person's opinion. So the app shows the list next to the close calls, and the organizer can ask for more reviews or decide on the record. There's also an API for it: GET /api/events/{id}/kingmakers.

three bugs i actually learned something from

1. signed by itself. Results get frozen into a snapshot signed with Ed25519, and anyone can verify it. My first verifier checked the signature using the public key stored inside the document. Spot it? Anyone can change the results, sign them with their own key, put their key in the document, and it verifies perfectly. It's self-consistent, it's just not mine. Fixed on Sunday night:

# before: trusts whatever key the document brings
return signing.verify(payload, signature["signature"], signature["public_key"])

# after: the key itself has to be this deployment's
if signature.get("public_key") != public_key_hex():
    return "signed with a different key, not by this deployment"

The offline verifier now takes the portal's key from outside the document too.

2. the audit log lost exactly the attacks. Every vote attempt is audited, accepted or refused. Refusals (voting twice, voting for your own team) raised an error inside the database transaction. The error rolled back the transaction, and the audit entry with it. So the log had every honest vote and none of the cheating attempts. Simplified:

# before
with transaction.atomic():
    if refused:
        audit.record("vote.refused", ...)   # rolled back with everything else
        raise refused_error

# after
with transaction.atomic():
    refusal = check(token, project)
    if refusal is None:
        ...create the vote and audit it...
if refusal is not None:
    raise refused(request, token, *refusal)  # audited outside the rolled-back block

3. same data, different numbers. The resampling uses a fixed seed, so I assumed it was reproducible. It wasn't. The reviews came out of the database in whatever order the database felt like, and with a seeded random generator a different input order means different resamples. Same scores, different results: prj_37's chance of a top-5 finish was 0.64 with the reviews in one order and 0.58 in another. The docs computed from the fixtures file and the app read the same scores from the database in a different order, so they disagreed. Fix was one line, sort the input before doing anything. Then CI regenerates the docs from the engine and fails if they don't match.

the spec was harder than it looked

The organizers' acceptance checker only tests tiers 1 and 2: public gallery, judge isolation, CSV export. Tiers 3 and 4 (public voting, webhooks, an embed, import and export) aren't checked by anything.

So I wrote my own checker for those, same report format, and said in the README that it's mine. The thing I learned writing it: a portal that refuses everything passes every "should be refused" test. If you only check that a participant gets 403, a broken app that 403s everyone looks perfect. So every "refused" check has an "allowed" twin right next to it. A vote for your own team is refused, and someone from another team voting for the same project goes through. If both don't hold, the check fails. 32 checks in total.

Smaller ones that bit me:

My own security headers broke the API docs. A strict Content-Security-Policy blocked the Swagger UI from a CDN, so /api/docs was a blank page. Serve it locally.
One route ate another. /api/events/import got matched as /api/events/{event_id} with an event id of "import". Moved it to /api/event-bundles.
Imports went straight to public. An exported event that was published came back in as published, so importing an archive made it public instantly. Imports now land as drafts.
My CI was never green. I'd pinned astral-sh/setup-uv@v10. That version doesn't exist. Meanwhile the README was going on about proof. Now CI also reruns the official checker against a fresh docker compose up and compares its output with the committed report byte for byte.

what i cut, and what i'd redo

Cut, and fine with it (every one of these, with the reasons, is in the decision log):

PostgreSQL as the default. SQLite in WAL mode with immediate transactions is plenty for a hackathon and keeps it one command.
Celery and Redis. A small outbox table and a worker container do the emails and webhooks.
Alpine.js. Its normal build needs unsafe-eval, which my CSP refuses.
Pairwise "A or B?" judging replacing the scores. It's there, but beside the scores, not instead of them. A rubric with weights and a pass/fail gate can't be expressed as pairs.
Blocking votes per IP address. A room full of people on the venue Wi-Fi looks exactly like ballot stuffing. Organizers can mark trusted networks instead, and suspicious bursts get held for review, never thrown away.
Sealed commit-reveal voting. Didn't have time to do it properly.

Would redo:

The score ledger from the first commit. I added a hash-chained, signed ledger of every score after reviews already existed, so I needed a backfill migration for the old ones. It works, but "verifiable from the start" beats "verifiable after a migration".
The resampling. See above. Resample judges, not just reviews, and check calibration again before shipping it.

try it

git clone https://github.com/GauravS13/raptor-desk
cd raptor-desk
docker compose up
# http://localhost:8080 , demo accounts are on the sign-in page

python3 tools/run.py .dogfood.toml --fixtures data/fixtures.json   # official checker, 7/7
uv run python tools/simulate_proof.py --check                     # every number in this post

The calibration and kingmaker numbers come from tools/simulate_proof.py, and the method is in JUDGING.md. If you think my maths is wrong somewhere, I'd honestly like to know.

Built for #DogfoodHackathon by Hackathon Raptors.

We Built a Feature We Never Shipped — And That Was the Whole Point

Gaurav — Fri, 04 Sep 2026 05:01:09 +0000

Somewhere in the middle of building KilnForge, we wrote a fully working feature, tested it, confirmed it was correct — and then didn't ship it. On purpose. That's not a typo. It's the most honest answer we have to what this post is actually about.

Key Takeaways

We hand-rolled a byte-level EXIF orientation parser, then discovered Bun.Image already does that internally — shipping ours would have double-rotated every tagged photo

Three other "obvious" sharp features don't exist natively in Bun.Image: a BMP/GIF encoder, a fit: cover resize mode, and arbitrary-angle rotation — each needed a real, hand-rolled fallback

We also found a bug in our own reproducible-build proof: 3 compiles produced 3 different hashes, traced to 2 differing bytes out of ~89MB — our own filename, not the build

Measured, not estimated: KilnForge runs 1.81x–10.22x faster than sharp on resize/rotate/convert, and 41.73x faster than tar at unpacking archives — but 1.22x slower at packing them, published honestly either way

263 tests, 6,463 assertions, all passing — including differential tests run against real sharp and real tar output, not synthetic stand-ins

How We Verified All of This

Every claim in this post comes from one of three concrete checks, not from reading a changelog:

Direct API probing — calling the actual method on the actual installed Bun binary (1.4.0) and recording what happens, success or exception, before writing a single line of pipeline code around it.
Pixel-level differential comparison — comparing our output against Bun.Image's own native output, byte for byte, on identical source images (used to catch the rotation off-by-half-pixel bug and to confirm the EXIF auto-rotation behavior).
Real reference-implementation testing — running our output through the same test suite against real sharp and real tar output on identical inputs, not synthetic stand-ins, via golden-corpus tests that ship in the repo.

All of it is reproducible: bun test runs all 263 tests, including the golden-corpus suite, from a clean checkout.

The Package We'd Normally Install

If you're processing images on a server, you reach for sharp. It's fast, it's mature, and underneath it's a native addon wrapping libvips — which means a real C build toolchain has to exist on whatever machine installs it. If you're bundling files, you reach for tar. Both are exactly the kind of dependency a zero-dependency hackathon exists to prove you don't strictly need.

So we set out to replace both, using Bun's own native Bun.Image and Bun.Archive APIs — no sharp, no tar, no native build step. dependencies: {} in package.json, checked mechanically on every build via a script that scans every import in src/** and refuses to let either package sneak in. This is our entry for a zero-dependency hackathon that scores exactly that kind of mechanical proof over a README claim.

That part sounds simple in a pitch. It wasn't simple in practice, because Bun.Image and Bun.Archive are both very new APIs — days old at the point we started. The write-ups we found while planning disagreed with each other on basic questions, color-profile survival being the clearest example: some sources claimed ICC profiles survive a transcode, others claimed they get stripped. We didn't pick a side. We built a minimal PNG carrying a real color-profile chunk, ran it through Bun.Image's own encoder, and checked the output bytes directly. The chunk was gone. One test resolved a contradiction that no amount of re-reading either source would have.

We decided early that we wouldn't build anything on top of an assumption we hadn't personally verified against the real, installed Bun binary. That decision is the reason this post has a story to tell instead of just a features list.

What We Built By Hand

Before writing any pipeline code, we wrote a Foundation Verification Harness — a script that runs real probes against the actual Bun runtime and records what it finds, separate from whether the probe itself crashed:

async function probe(
  results: ProbeResult[],
  name: string,
  fn: () => Promise<{ finding: boolean; detail: string }>,
): Promise<boolean> {
  try {
    const { finding, detail } = await fn();
    results.push({ name, ok: true, finding, detail });
    return finding;
  } catch (err) {
    results.push({
      name, ok: false, finding: false,
      detail: `threw unexpectedly: ${err}`,
    });
    return false;
  }
}

ok answers "did this probe run without crashing." finding answers "what did it actually discover." Those are different questions — a probe can run perfectly cleanly and still report a negative finding, and that's a successful, informative run, not a failure. Every architectural decision downstream got pinned to what this harness actually measured, not to what we assumed going in.

It's a good thing we did, because several of those assumptions were wrong.

`Bun.Image` has no BMP or GIF encoder

Our original plan was to choose between a PNG channel and a BMP channel for reading and writing raw pixels. Real inspection of Bun.Image.prototype showed no .bmp() and no .gif() method at all — both formats are decode-only. PNG wasn't the preferred channel. It was the only one. That meant hand-rolling a real PNG encoder/decoder from scratch: chunk parsing, node:zlib for the inflate/deflate, scanline filter/unfilter, our own CRC32 — just to get a working raw-pixel round trip.

There's no native `fit: cover` resize mode

Bun.Image.resize() only exposes fill (stretch to exact dimensions) and inside (aspect-preserving, fits within the box). There's no crop-to-fill. We built it ourselves — compute the overscale factor that would cover the target box while preserving aspect ratio, resize with the native fill mode, then center-crop the raw pixel buffer down to size:

async function resizeCover(
  image: Bun.Image,
  targetWidth: number,
  targetHeight: number,
): Promise<Bun.Image> {
  const meta = await image.metadata();
  const scale = Math.max(
    targetWidth / meta.width,
    targetHeight / meta.height,
  );
  const overW = Math.round(meta.width * scale);
  const overH = Math.round(meta.height * scale);

  const resized = image.resize(overW, overH, { fit: "fill" });
  const rgba = await toRGBA(resized);
  const cropped = cropCenter(rgba, targetWidth, targetHeight);
  return loadImage(fromRGBA(cropped));
}

Arbitrary-angle rotation isn't native either

Bun.Image.rotate(45) genuinely throws — "only multiples of 90 are supported" — confirmed directly, not assumed from a changelog. We wrote our own rotation via an inverse coordinate transform, and it had a bug worth mentioning: the first version was wrong by exactly half a pixel, because it treated a pixel's raw integer index as its coordinate instead of the center of the cell that pixel occupies:

// Pixel index i occupies the continuous interval [i, i+1) with its
// CENTER at i+0.5 — sampling must rotate from that center, not from
// the raw integer index, or the whole mapping is off by half a pixel
// (verified by hand against Bun.Image's native rotate(90), which has
// an exact, unambiguous discrete answer to check against).
const relY = dy + 0.5 - destCenterY;
const relX = dx + 0.5 - destCenterX;
const srcX = relX * cos + relY * sin + srcCenterX;
const srcY = -relX * sin + relY * cos + srcCenterY;

That one-line fix (+ 0.5) took a 58% pixel mismatch against Bun.Image's own native rotate(90) — used as a cross-check oracle for the 90° case, which is native — down to zero mismatches across every test we threw at it.

The Feature We Never Shipped

Here's the part we're actually proudest of, and it's not a feature that shipped — it's one that didn't.

We built a complete, tested, byte-level EXIF orientation parser by hand: reads the JPEG APP1 marker, walks the TIFF structure, handles both byte orders (Intel and Motorola), covers all eight orientation values, and converts the result into the correct rotate-and-flip transform. It worked. It passed its own test suite standalone. Every image-processing library we've ever used needs this, so naturally we assumed Bun.Image would need it wired in too.

Before wiring it into the live request pipeline, we ran one more check — because by this point "test it before you trust it" was the whole method, not just something we did once at the start. We fed a real EXIF-tagged photo straight into Bun.Image with no correction of our own, and checked what came out.

Bun.Image had already rotated it. Correctly. On its own, during decode.

We confirmed it two ways: metadata() reported swapped width and height immediately after decode, and a direct pixel comparison against a manual native .rotate(90) on the same source came back with zero mismatches. Bun.Image auto-applies EXIF orientation internally — nobody advertised that clearly anywhere we'd read, but the runtime doesn't lie the way documentation sometimes does.

If we'd shipped the original plan — wiring our own parser into the request pipeline on top of that — every single EXIF-tagged photo uploaded to the service would have been rotated twice. Once by Bun.Image, invisibly, during decode. Once by us, right after. The bug wouldn't have been subtle. It would have been every photo, every time. And it would have passed casual testing anyway — anyone checking with an un-rotated test image would have seen nothing wrong.

The parser still exists in the codebase today, fully tested, completely real — just deliberately disconnected from the request pipeline, with a comment explaining exactly why:

// NOTE: this module deliberately does NOT auto-apply EXIF orientation
// before handing bytes to Bun.Image. Empirical testing (see
// src/image/exif.ts's module comment) found Bun.Image already applies
// EXIF orientation correction internally during decode — confirmed via
// metadata() reporting swapped dimensions immediately and 0 pixel
// mismatches against a manual native .rotate(90) on the same source.
// Calling our own applyOrientation() on top of that would double-rotate
// every EXIF-tagged upload.

That's the whole point of this post, if it has one: the win here wasn't building the parser. It was catching, before a single real user ever touched it, that building it into the pipeline would have been the bug.

The Bug in Our Own Proof

We found one more bug the same way, and it wasn't in the product — it was in our own evidence.

One of the bonuses this hackathon offers is a reproducible build: compile the same source three separate times, hash each binary, prove they're identical. We ran it. We got three different SHA-256 hashes. By the letter of the check, that's a fail.

Instead of writing that down as an honest limitation and moving on, we diffed the three binaries byte for byte. Out of roughly 89 megabytes, exactly two bytes were different. One of them was plain, readable text — our own --outfile filename, which Bun's compiler embeds into the binary as an internal module path. We'd given each of the three test builds a different output filename, so the test introduced the only variance it then measured.

Fixed it by using an identical filename across three separate temp directories instead, and reran. Byte-for-byte identical. Verified with cmp, zero differences. The build had been reproducible the entire time — our test of it hadn't been.

The Real Numbers

Bun's own release coverage cites roughly 1.2x–1.4x faster resize/convert performance for Bun.Image versus sharp. Our numbers below are our own independent run, on our own fixtures — not a substitute for that public figure, cited alongside it rather than instead of it.

Operation	KilnForge (mean)	sharp (mean)	Result
resize 64×48 → 20×20 (fill)	0.23ms	2.35ms	10.22x faster
rotate 90° (20×40)	0.11ms	0.99ms	9.00x faster
convert JPEG → WebP (q85)	0.30ms	1.15ms	3.83x faster
resize 256×256 → 30×30 (cover)	1.87ms	3.39ms	1.81x faster

Operation (archive)	KilnForge (mean)	tar (mean)	Result
unpack	0.22ms	9.18ms	41.73x faster
pack (5 files × 5KB)	0.11ms	0.09ms	1.22x slower

We didn't publish only the numbers that flatter us. On the archive side, unpacking a tarball through Bun.Archive runs 41.73x faster than tar — mostly because tar's own API extracts to real disk I/O per file while ours stays in memory, a genuine architectural difference. But packing a tarball is 1.22x slower with our implementation than with tar. That's a small, real margin on a handful of small files, and we're stating it instead of quietly leaving it out of the chart.

None of these numbers are estimates. They come from real differential tests — our output compared directly against real sharp and real tar output on the same inputs, both kept as devDependencies used only by the benchmark and test scripts, never imported by anything that ships. A script that scans every import in the actual service code confirms that split mechanically, not by us promising it. All 263 tests pass, 6,463 assertions total, across 34 files.

None of this is worth much without seeing it actually run. Here's the real thing — server up from a clean checkout, resize/watermark/convert//batch all live, the Foundation Verification Harness's own output on camera:

What Going Zero-Dependency Actually Costs

The honest answer to "what did it take to replace sharp and tar" isn't "we're faster" — though on most operations, measured fairly, we are. The honest answer is that a mature library like sharp has quietly handled a decade of edge cases you never think about until you're the one who has to handle them: EXIF orientation, crop-to-fill resizing, arbitrary rotation angles, alpha-safe format conversion, decompression-bomb protection. None of that goes away when you drop the dependency. It just moves onto your desk.

The only way we found to know we'd actually covered those edge cases — instead of just assuming our replacement code was equivalent — was to keep testing against the real runtime and the real reference implementation, the whole way through. That's what caught the EXIF bug before it shipped. That's what caught the reproducibility bug in our own proof. Neither of those would show up in a features list. Both of them are the actual work.

We're The Vighnahartas — a team entry for Track F of the Zero Dependency Hackathon (Aug 28–31, 2026). KilnForge is the project this post is about. Everything referenced here — the code, the tests, the benchmark scripts — is in the public repo, built on Bun 1.4.0.

I Ported an 830-Star Python Library to Rust and the Fuzzer Found 12 Bugs — All of Them Mine 🦀

Gaurav — Sat, 08 Aug 2026 04:45:54 +0000

9,549 differential fuzz cases against the real Python original. Zero divergences. Twelve bugs found — every single one was something I wrote, not something Python did.

tinytag is an 830-star, zero-dependency Python library that reads tags and audio properties out of WAV, AIFF, FLAC, MP3, Ogg, MP4, and WMA files. I picked it on purpose, not at random: six binary formats in one small, well-tested codebase means six different byte-layout quirks, six different chances to get "close enough" instead of exactly right — a good stress test for what "behavioral equivalence" actually costs, not just what it means in theory. For Port Mortem / Code Resurrection 2026 (Track D, Python → Rust), I rewrote six of those seven formats from scratch — no libpython, no PyO3, no shortcuts — and checked every claim below against the real source files in this repo, not memory.

Key Takeaways

9,549 differential fuzz cases against the real Python original, 0 divergences, in the last qualifying 65-second run (fuzz/log.txt)

3 real bugs in upstream tinytag, found by fuzzing, reproduced faithfully — not fixed, because matching Python's actual output is the scoring target here

12 bugs the fuzzer found in this port during development, all fixed before this was written

0 unsafe blocks, 0 dependencies, 1.65x average speedup, honestly measured

👀 What This Thing Actually Does, in 10 Seconds

Before any of the bug stories, here's the whole point of the project in one command. Point the CLI at a real MP3, and it reads the tags straight out of the file — no metadata database, no network call, just parsing bytes:

$ ./target/release/tinytag tests/samples/cbr.mp3

{
  "filename": "cbr.mp3",
  "filesize": 8186,
  "mime_type": "audio/mpeg",
  "is_lossless": false,
  "duration": 0.47020408163265304,
  "channels": 2,
  "bitrate": 127.99548611111112,
  "samplerate": 44100,
  "artist": ["Basshunter"],
  "album": ["I Can Walk On Water I Can Fly"],
  "title": ["I Can Walk On Water I Can Fly"],
  "track": ["1"],
  "genre": ["Dance"],
  "year": ["2007"],
  "comment": ["Ripped by THSLIVE"]
}

Real output, from a real fixture file in this repo. That's the entire contract: same file in, same JSON out, whether you run the Python original or this Rust port. Everything below is the story of how hard that "same" turned out to be to actually prove — and where it wasn't quite the case yet.

🎯 9,549 Cases, Zero Divergences — Here's How I Checked

The scoring target isn't "does it work" — it's "does it produce byte-for-byte identical output to the Python original, on the same inputs, including the malformed ones." That bar is what surfaced almost everything interesting in this project.

To check it, I built a differential fuzz harness (src/bin/fuzz_harness.rs) that mutates real audio fixtures and runs both the vendored Python original and the Rust port against the exact same bytes, then diffs the structured output field by field. It doesn't link Python into the Rust binary — the rules explicitly disallow that — it spawns a real CPython subprocess and compares against it, which is the honest shape of differential testing.

Last qualifying run, quoted directly from fuzz/log.txt:

differential fuzz run started (rust-orchestrated), seed=1785680270143253800
seed corpus: 206 fixtures (full in-scope set, see fuzz_harness.rs)


9549 cases run over 65.0s
0 divergences found
ZERO DIVERGENCES

177 unit tests, each traced 1:1 to an expected value in upstream's own test_all.py — not hand-guessed — plus that fuzz run, are the actual evidence of correctness here, not "I read the code carefully."

🐛 The Fuzzer Found 12 Bugs. All of Them Were Mine.

The single biggest source of divergence across this whole project was my own instinct to write more defensive Rust than the Python original actually is. Twelve times, the fuzzer caught it. A representative few, documented in full in DECISIONS.md:

Bug one: AIFF's sample rate flipped across a rounding boundary, because of a function name one letter off from the right one. Here's the actual line that shipped first:

let sr = (mantissa as f64) * 2f64.powi(exp as i32 - 0x3FFF - 63);

powi and powf both raise a number to a power — the difference is powi takes an integer exponent and uses a fast approximation, while powf takes a float exponent and routes through the platform's real pow(). Rust is explicit that powi is not guaranteed to be correctly rounded. Python's 2 ** negative_int always goes the pow() route — the powf route. Nine times out of ten that difference is invisible. But right at the edge of u32::MAX, that tiny rounding error was enough to flip a truncated integer by exactly one — and "one over the line" made Rust reject a sample rate Python happily accepted. The fix ended up needing exact integer arithmetic instead of floats at all, once fuzzing pushed on it hard enough (full diff in the commit history; the one-letter version above is the part that actually mattered). Found by fuzzing mutated exponent bytes in no_audio.aiff (DECISIONS.md #14e).

Bug two, and the one that never seemed to fully die: MP4's atom-tree traversal. More on that below — it's worth its own section, because "found it, fixed it" undersells what actually happened.

Bug three: a too-short WAV fact chunk returned an empty value instead of raising an error, like Python does. My first instinct was to add a safety check — "if the chunk is at least 4 bytes, read it; otherwise skip." That felt like good defensive programming. It was also wrong. Python's own unpack_from('I', chunk) has no such check: a chunk shorter than 4 bytes just raises struct.error, which upstream converts into a real parse error. The fixed version reads exactly like that — no polite skip, just propagate the error:

// Matches upstream's unguarded `unpack_from('I', chunk)` — a too-short
// buffer is a real parse error there (struct.error -> ParseError), not
// something to skip gracefully.
num_samples = u32_le(&chunk, 0)? as u64;

That ? at the end is doing the actual work: it means "if this read fails, stop and hand the error up" — no fallback, no default value, exactly what Python does when struct.unpack_from blows up (DECISIONS.md #14a).

The pattern underneath all twelve: Rust's standard tools (read_exact, bounds-checked slicing, length guards) nudge you toward code that's more correct than the reference by default. For a behavioral-equivalence port, that's a bug class of its own — every one of these had to be found by actually running the same corrupted bytes through both implementations, not by code review.

🕳️ The Edge Case That Took the Longest to Actually Nail

Every other bug in this post, I fixed once and it stayed fixed. MP4's atom-tree traversal took multiple separate rounds, across seven different fuzzed files, before it actually held — and each time I thought it was done, a new mutated .m4a would find another angle on the same underlying mistake.

The root problem: MP4 files are nested trees of "atoms" (moov → udta → meta → ilst, and so on), and Python's traversal tracks two numbers per level — the file handle's real read position, and a logical position it's supposed to be at. On well-formed files those two numbers never disagree, so the distinction is invisible. On a corrupted file, they can legitimately drift apart: a header read that fails immediately still consumes bytes from the stream, even though the code that failed never got to use them. Python just keeps reading from wherever the handle actually sits after that — it never tries to force the two numbers back in sync.

My first version did try to force them back in sync. It looked like the responsible thing to do — reset to each atom's declared end after every step, like a checkpoint. It was exactly backwards: forcing a resync meant every parent atom kept re-reading the same corrupted bytes instead of drifting past them the way Python does, so a corruption Python shrugs off and recovers from (going on to find real tags further in the file) made my port just... stop, silently returning less than it should have.

The fix that actually held is one line of discipline, applied consistently everywhere a position gets threaded through:

curr_pos = on_atom(reader, t, size, curr_pos)?;

Always take the actual resulting position back from whatever just ran — never assume it matches what you expected going in. skip_to, the function that skips a genuinely unrecognized atom, follows the same rule: a relative seek from wherever the reader currently is, never an absolute jump to a precomputed value:

fn skip_to<R: Seek>(reader: &mut R, atom_size: u64) -> Result<u64, ParseError> {
    let pos = reader.seek(SeekFrom::Current(atom_size as i64))?;
    Ok(pos)
}

Simple once it's written down. What made it take multiple rounds is that the bug doesn't announce itself as one bug — it shows up as a different symptom depending on which atom happens to be corrupted and which sibling atom comes after it: sometimes a missing tag, sometimes a whole covr (cover art) atom's image data getting mis-skipped because a child atom inside it was independently corrupted (DECISIONS.md #14g). Each new fuzzed file (multi_value.m4a, classical.m4a, mixed_case_atoms.m4a, xmp_empty.m4a, test2.m4a, mvhd_version_1.m4a, and finally mpeg4_with_image.m4a) looked like a new bug the first time it failed. It wasn't — it was the same root cause, wearing a different fixture each time, until the "always return the real position" rule got applied everywhere instead of just where the first failure pointed (DECISIONS.md #14f, #14g).

🔬 The Bug I Found in an 830-Star Library (and Didn't Fix)

Three real bugs in upstream tinytag, all found by fuzzing, all left exactly as-is in the Rust port. That's deliberate: fixing any of them would make the port diverge from Python's real output on that input, which is the opposite of what's being measured.

The one worth reading in full: MP4's gnre atom has a genuine off-by-one bug, and it's the kind every programmer has written at 2am. Upstream reads a 2-byte genre index, subtracts 1 (MP4 genre codes are 1-indexed), and looks it up in a genre table. It checks that the index isn't too big — but never checks that it isn't negative:

let raw = u16_be(data, 8)?;
let len = crate::id3v1::GENRES.len() as i64;
let idx = raw as i64 - 1;
if idx < len {
    let real_idx = if idx < 0 { (len + idx) as usize } else { idx as usize };
    if let Some(genre) = crate::id3v1::GENRES.get(real_idx) {
        tag.set_str_field("genre", (*genre).to_string());
    }
}

Walk through it with a stored value of 0, meaning "no genre set": idx = 0 - 1 = -1. The bounds check idx < len passes fine (-1 is less than the table length). Then real_idx = len + idx — Python's negative-indexing behavior, deliberately reproduced here — lands on the last entry in the genre table, "Anime" in this pinned version. So a file that explicitly says "no genre" ends up tagged as Anime. Nothing in the original code marks 0 as meaning "give me the last genre" — it reads exactly like a missing if idx < 0: return that nobody caught, on either side, for years. This port keeps it exactly as-is, on purpose (DECISIONS.md #18a).

The other two, same treatment: an ID3v2 extended-header seek hardcoded to skip - 6 bytes regardless of how many were actually available (wrong offset on truncation, #17), and track/disc "X/Y" splitting that silently discards a third slash-segment via Python's split('/')[:2] (#18).

⚡ 1.65x Faster, and I'm Not Going to Oversell It

Per-file speedup, Rust vs Python, 200 iterations per file. Overall average: 1.65x. Source: bench/results.json, methodology in bench/methodology.md.

File	Python ns/call	Rust ns/call	Speedup
vbr_xing_header.mp3	75,714	37,786	2.00x
flac453sStereo.flac	78,752	40,715	1.93x
vbri.mp3	133,237	72,671	1.83x
M1F1-mulawC-AFsp.afc	71,670	40,502	1.77x
test.wav	59,566	34,668	1.72x
test3sMono.wav	62,809	36,550	1.72x
flac1sMono.flac	84,086	60,926	1.38x
UTF16.mp3	151,569	110,596	1.37x

Overall: 1.65x faster than the Python original, averaged across an 8-file workload spanning WAV/AIFF/FLAC/MP3. That's not dramatic, and I'm not going to pretend it is. Both implementations do the same fundamentally small, byte-shuffling work — parsing a WAV chunk header or an ID3v2 frame is a handful of comparisons and a slice index, not a hot numeric loop where Rust routinely wins by 10-50x over CPython. The pattern that does show up consistently: files with more tag frames to walk (UTF16.mp3, 1.37x) show a smaller speedup than files that are mostly audio-property parsing with few or no frames (vbr_xing_header.mp3, 2.00x) — consistent with per-frame Python interpreter overhead being the more meaningful cost driver here, not some inherent parsing-algorithm gap. Real, reproducible, not cherry-picked, and the honest headline is "correctly and safely faster," not "wildly faster."

🚧 The One Thing I Couldn't Fix

One disclosed, un-fixed edge case: AIFF's extended-precision sample rate can, on sufficiently corrupted input, exceed u32::MAX. Python's arbitrary-precision int represents that value exactly (values like 10³³ Hz have shown up under fuzzing — obviously not a real sample rate). Rust's samplerate: u32 genuinely cannot. Rather than wrapping to garbage via a lossy cast, this port declines the value (None) when it would overflow. It's a real language-capability limit, not a bug — and it's rare enough that a typical 65-second fuzz run doesn't always hit it, but no real-world AIFF file comes anywhere near this range either (DECISIONS.md #15).

You might expect an "unsafe block I couldn't remove" story here — there isn't one. This port has zero unsafe blocks; byte-level tag parsing doesn't need raw pointers, and the format-tree traversal work stayed entirely inside safe, bounds-checked slices the whole way through. The equivalent trade-off isn't a memory-safety compromise, it's the one directly above: a case where "faithful to Python's exact output" and "representable in Rust's type system" genuinely conflict, and something had to give. That's the actual cost of this port, not an unsafe block.

🔄 The Decision I'd Take Back

If I'm honest about one call I got wrong: I started the fuzz and benchmark harnesses in Python.

That felt reasonable at the time — the harness needs to spawn a reference Python process anyway, so writing the orchestration logic (mutate a fixture, run both sides, diff the output, report) in Python too meant one language for that whole layer, ~369 lines split across fuzz/diff_fuzz.py and bench/run_bench.py. It worked. It just wasn't fast, because every single fuzz case meant spawning a fresh Python subprocess — and that per-case process-spawn cost dominated the actual work being measured.

Partway through, I rewrote both as Rust binaries (src/bin/fuzz_harness.rs, src/bin/bench.rs) that spawn Python exactly once and feed it an embedded reference-oracle script, instead of once per case. Same total amount of "real Python running real Python code" — just amortized across the whole run instead of paid per-case. The result wasn't just cleaner: the Rust-orchestrated harness ran roughly 2x more cases in the same wall-clock time, and that extra throughput is directly how two of the twelve bugs above (#14a, #14b) got found at all — the original Python-orchestrated version's slower cases-per-second never reached them inside a qualifying run's time budget.

If I started over, the harness would be a Rust binary from the first commit, not a mid-build rewrite. Not because the Python version was wrong — it worked, it just quietly capped how many bugs I was going to find, and I didn't realize that was the cost until I'd already paid it.

📼 Watch It Live

Full video walkthrough: youtu.be/Hg3GoT8XJr0

Plus six ~30-second recorded terminal demos (one per format) and the full test-suite-and-fuzz-run, captured for real, not staged:

The other six — WAV, AIFF, FLAC, MP3, Ogg, MP4 — are in public/demo/.

📊 All the Numbers in One Place

177/177 tests passing, traced 1:1 from upstream's own test data
9,549 cases, 65 seconds, 0 divergences on the last differential fuzz run
34 documented architectural decisions (DECISIONS.md)
0 unsafe blocks, 0 dependencies
3 real bugs found in the original, reproduced faithfully
12 bugs the fuzzer found in this port during development, all fixed
1.65x average speedup, honestly measured, honestly reported

Every decision, including the ones that were wrong on the first attempt, is in DECISIONS.md — 34 of them, not a highlight reel.

🔧 Go Poke at It Yourself

git clone https://github.com/GauravS13/tinytag-rs.git
cd tinytag-rs
cargo test                                              # 177 tests, traced 1:1 from upstream
cargo build --release --bin fuzz_harness && ./target/release/fuzz_harness 65   # run the fuzz yourself

Full source, tests, DECISIONS.md, and the fuzz/bench harnesses: github.com/GauravS13/tinytag-rs.

Submission by Siddhivinayk for Port Mortem / Code Resurrection 2026, Track D (Python → Rust).Tagging @partnerships_raptors