DEV Community: Pal

Your database is not an audit log: 72 hours of verifiable hackathon judging

Pal — Tue, 29 Sep 2026 18:32:47 +0000

Most hackathon portals answer one question — who won? — with a row in a database. The row renders, everyone moves on, and nobody asks the harder question that no row can answer on its own: how would you know if it were wrong?

Dogfood Portal is a self-hostable platform for running a hackathon's submissions and judging: teams submit, judges score, and it publishes a final leaderboard. I gave myself two rules for the 72 hours. It has to come up from a single docker compose up with the network unplugged — no Redis, no message broker, no background worker, no CDN, two services in the compose file and one of them is Postgres. And the results it publishes have to be verifiable by anyone, offline, without trusting my server. The second rule is the whole post.

Here is the thing that rule forces you to admit:

A hackathon portal's "final results" are a SELECT over a table. Anyone who can reach the database can UPDATE a score after the fact, and the results page will render the new number exactly as confidently as the old one. Nothing on the page knows it changed.

That is not a Django problem and it is not a Postgres problem. A database stores current state; it was never a record of what happened. Treating "it's in the database" as "it's the result" is a quiet category error — reaching for a tool whose contract is close to the one you want, reading storage as if it were proof, and shipping the gap without noticing it is there. So the results here carry their own proof, and there is one command that re-checks all of it with the database container stopped.

What I built

Dogfood Portal is a single Django 5.2 + PostgreSQL 16 monolith — one web service, one db service, nothing else. It boots offline, runs the migrations, seeds a demo event (an organizer, judges, teams, and scored submissions) on first start, and serves a public project gallery, a judging flow, and a normalized leaderboard. The interesting half is underneath: the final results are exported as a signed, reproducible bundle, and there is an offline verifier that re-derives and re-checks the whole thing from that bundle alone.

$ docker compose exec -T web python src/manage.py release_bundle /state/release
$ docker compose exec -T -w /app/src web python -m normalize.release /state/release
[PASS] ranking.csv matches the signed run
[PASS] signed run reproduces from the pinned ballots
[PASS] normalization.published event is on the audit chain
[PASS] audit checkpoint signature verifies
[PASS] every artifact traces to one operator public key
RESULT: VERIFIED

That second command needs no database and no network. It is the point of the project, so the rest of this post is what "verifiable" had to actually mean, and what generalizes past one Django app.

What a database won't do for you

The naive design is one table: results, with a rank and a score per project, written when the organizer clicks publish. It renders fine. It is also unfalsifiable — there is nothing to check it against, because the row is simultaneously the claim and the only evidence for the claim.

So publishing does not write a row you have to trust. It writes a chain you can follow:

the normalized ranking is computed and frozen as a signed run — the pinned input ballots, the model settings, the resulting scores, and an Ed25519 signature over all of it;
that publication is recorded as a normalization.published event on an append-only audit hash-chain, where each event carries the hash of the one before it;
the chain head is sealed with a signed checkpoint;
the exported ranking.csv is bound back to the signed run;
and every signature in the bundle traces to one operator public key.

The verifier walks that chain backwards. It re-runs the estimator from the pinned ballots and checks the result it gets matches the signed one; it re-checks every signature; it confirms the published event actually sits on the chain and that the checkpoint over the chain verifies; it confirms one key signed all of it. Any broken link fails the run and names itself — signed run does NOT reproduce, not a generic false. With the db container stopped, from the bundle alone, on a machine that has never seen my server.

Everything that verifier needs travels in one exported bundle: the ranking CSV, the signed run with its pinned inputs and result, the audit checkpoint and the slice of chain leading up to it, and the single public key the whole thing is signed under. The export self-verifies before it will finish writing, so a bundle that does not check out is never produced in the first place. For the everyday case — "prove to me this leaderboard is the one you actually published" — there is a smaller dogfood.results-certificate.v1 you can hand to a single team, and it verifies through the same path. The bundle is the audit; the certificate is the receipt; both are checked by code, not taken on faith.

The transferable version is small: sign the result; don't ask people to trust the server. If the only evidence for a number is the system that produced it, you have not published a result, you have published an assertion.

The measurement: a leaderboard implies a precision the ballots don't have

Raw judge scores are not comparable. One judge's 7 is another's 9, and if two projects were seen by different judges, their averages are measuring the judges as much as the work. So the leaderboard is not a mean — it is a ridge / partial-pooling estimate that models each ballot as quality + judge_bias + noise, fits it with a gauge that pins each connected group of judges-and-projects to a common zero, and picks the regularization strength by cross-validation rather than by taste.

That gauge is the subtle part, and it is where most naive score-normalization quietly goes wrong. Judge biases and project qualities are only defined relative to each other, so the model has a free parameter you have to pin down or the numbers wander: add a constant to every judge's bias and subtract it from every quality, and nothing observable changes. The fix is to require each connected group of judges-and-projects to sum to zero. "Connected" is meant literally — two projects are in the same group if a chain of shared judges links them — and it carries an honest limit worth stating out loud: two projects that share no judge, even transitively, live in different groups with no common zero, so the model refuses to rank them against each other rather than inventing a bridge. On this event everything was one connected component, so the limit never bit; the code enforces it regardless, and a self-check confirms the gauge sums return to zero within floating-point noise — about 1e-17 — before a run is ever signed.

On the seed event — 126 ballots, 41 projects, 30 judges across 8 tracks, all one connected component — this is not cosmetic: cross-validation lands on λ = 10, and correcting for judge bias is enough to break a tie that the raw averages leave at the top.

Then it measures its own confidence, and this is the number I did not get to choose:

Across all 40 adjacent pairs on the leaderboard, not one is statistically resolved. For the closest, the model's probability that the higher-ranked project is actually better than the one just below it is about 0.518 — a coin flip. Correcting for judge bias tightened within-submission spread by about 1.08×, not the dramatic factor a demo would prefer.

The honest reading is that these judges were fairly consistent and the field was close, so a strict ranking would be inventing separation the ballots do not contain. The tool therefore publishes the pairwise probabilities and intervals next to the ranking, so "3rd vs 4th" reads as the coin flip it is. Publish the uncertainty, not just the order — a ranking that hides it is the same flattering lie as a benchmark that reports its fastest run.

The bug that only appears after a round-trip through Postgres

The signed run is supposed to be reproducible: sign it once, and anyone re-running the estimator on the same ballots should land on the same bytes and the same signature. It worked in memory and failed the moment the run was reloaded to verify it.

The cause was -0.0. The gauge that pins each group to a common zero legitimately produces negative-zero components; in memory that is what got signed. But the run is stored as jsonb, and the reload path normalizes -0.0 to 0.0. So the verifier hashed a projection that differed from the signed one in a byte that is numerically invisible — -0.0 == 0.0 is true — and the signature check failed for a reason no arithmetic could see.

The fix was not to special-case the comparison. It was to normalize negative zero inside the projection that gets hashed, before signing, so the run signs exactly what the database will hand back. Hash what you'll read back, not what you happen to be holding — a signature over an in-memory object is a signature over a serialization decision you forgot you were making.

Append-only, not delete

A score is never edited in place. Each revision is an append-only row keyed UNIQUE(ballot, version), with a database CHECK keeping every score in 1–5, and the current ballot is just a pointer at the latest version. A withdrawal is a soft state flip, not a DELETE, and the admin path refuses to cascade-delete anything a scored ballot depends on. History survives the people who would rather it didn't.

Underneath the ballots is the log that makes them evidence rather than just rows: an append-only audit chain on a schema I froze on purpose, so its meaning cannot drift out from under old events. There is a single head row and a stream of events, and each event stores its own payload, the hash of that payload, the hash of the event before it, and a row hash over the whole record — the ordinary hash-chain shape, where altering any past event breaks every hash that follows it. Writing one event takes a select_for_update lock on the head so two concurrent writers cannot fork the chain, links the new event to the current head, and advances the head — all inside a single database transaction.

The load-bearing claim there is that the data write and its audit event are one atomic thing — that you can't get the score without the chain entry, or vice-versa. The way I believe that is a test that can fail: it monkeypatches the audit chain head's save() to raise mid-transaction, then writes a ballot. Afterwards, the ballot, the judge assignment, and the audit event are all absent, and the chain head is still at sequence 0. A test that cannot fail is decoration; this one holds a real invariant, because it forces the exact failure the invariant is supposed to survive.

Permission is a grant on an event, not a role

Judging platforms tend to leak along their authorization seams: a judge from one event nudges a score in another, an organizer glimpses a draft they should not. The tempting shortcut is a global role — is_judge, is_organizer — checked in a view. It is the same category error as the results table, convenient and wrong at exactly the edges that matter. So permission here is a grant scoped to a specific event, and it is enforced in the service layer where the write actually happens, not only in a view decorator that some second code path could quietly bypass. To keep myself honest about that, a mutation test deletes the check and expects a test to go red — if removing an authorization guard changes nothing, the guard was never load-bearing in the first place. Invitations are cut from the same cloth: each is signed and single-use, and "single-use" is enforced by taking a row lock and marking the invite spent inside the accept transaction, so two clicks on the same link cannot both succeed. The honest boundary is that this enforcement lives in the application's service layer, not yet in the database as row-level security — a deferral I wrote down on purpose rather than a thing I hope nobody notices.

Living without a broker

The "no moving parts" rule means the usual answers were off the table, so a few things are hand-built that a bigger stack would outsource. Rate limiting and caching run on Django's database-backed cache — a table created at boot, not a Redis. Static files are served in-process by WhiteNoise, vendored into the image, because there is no CDN to reach on a machine with the cable pulled. And the entrypoint does in sequence what an orchestrator usually spreads across services: wait for the database, generate the secret key race-safely with O_EXCL on first boot only, migrate, create the cache table, seed the demo event if the database is empty, ensure the admin exists, then hand off to gunicorn. It is less clever than a broker, and it has the property I wanted: one command, offline, and nothing to stand up first.

The contract is replayed, not assumed

A grader checks the portal by hitting a fixed set of routes and asserting on the responses. That contract is easy to break by accident — a refactor renames a path, a template grows a nav link, and the grade drops for a reason unrelated to the feature you were building. So the routes the grader depends on are frozen, feature routes are appended after them and never linked from the base navigation, and a stdlib-only tools/replay.py re-runs the same seven checks the grader runs, from the same contract file the grader reads. It is stricter in exactly one way — it does not follow redirects, so a route that only reaches 200 after an append-slash bounce is a visible failure rather than a silent pass — and it ends on a green 7/7 checks green. The compatibility claim is a thing the build re-measures, not a thing I remember being true.

Proof that it runs on a machine that isn't mine

"Works on my machine" is the oldest lie in software, and a portal that only boots on the laptop that built it has proven nothing about whether you can run it. Two things guard against that. The first is that single docker compose up: a cold clone, an empty volume, the network off, and the container brings itself all the way up — waits for the database, generates its secret, migrates, seeds the demo event, creates the admin — with no manual step I could have forgotten to document, because there is nowhere to hide one. The second is continuous integration that does exactly this on hardware that has never seen my working tree: it clones fresh and runs the suite against a real Postgres on every push, so "it boots and it passes" is a fact re-established by a stranger rather than a thing I merely remember working. The image build is gated as well — Django's own system checks at --fail-level WARNING, plus a makemigrations --check that fails the build when a model changed without its migration, which is precisely the mistake that turns a green local run into a broken clone. And the tests come in two deliberate layers: the pure-Python core — the hash-chain, the signing format, the estimator's math — has tests that need no database and run anywhere, while the full suite runs against Postgres in CI. The proof that something reproduces is someone else reproducing it, so the whole build is arranged to make that the cheap and default path rather than a favor I have to ask.

A signature is evidence about a result, not about my character

It is worth being precise about what the cryptography does and does not buy, because the failure mode of a security feature is someone trusting it past its contract. A valid signature proves two things: the bytes have not changed since the key-holder signed them, and anyone can re-derive the ranking from the pinned ballots and get the same answer. It does not prove the judging was fair, that the ballots were honest, or that I did not sign a bad result on purpose. A determined operator with raw database access can still change data — the chain does not make that impossible, it makes it detectable: the re-derivation or a hash link fails, and the verifier tells you which one. Detectable is a weaker and far more honest promise than tamper-proof, and it is the one the design can actually keep.

The same restraint applies to the review diagnostics — leave-one-out residuals, judge influence, coverage gaps. They ship behind a banner that says, in the product, that they are descriptive and not fraud detection, because a residual is a question worth a human's attention, not a verdict. Describe what the proof covers, and put the limit where the reader is, not in your own head.

I made the documentation fail the build

This is the practice that makes any number in this post safe to write. A README figure goes stale in the commit after the one that earned it, and prose does not fail a build, so a handful of tests do nothing but read a document and compare it to the code — the threat-model table has to match the controls that actually exist, and a doc that names a route or a setting has to name one the code really defines. For each claim, ask what file would contradict it, then read both in one test.

The honest-claims rule is the same tool pointed at myself: every capability the docs describe has to map to a feature that actually shipped in src/. The week of submission that rule cost me two lines — a demo script called the replay checker a "byte-for-byte" comparison when it runs status-and-content checks and follows no redirects, and a config comment claimed a test asserted something no test asserted. Both were true-ish and both were wrong, so both got reworded before the freeze. A capability you can describe but not point at is not a feature, it is a plan.

What it does not do

Multi-event support is partial, by a decision recorded in the threat model rather than an oversight. The diagnostics are descriptive and not fraud detection. Row-level security is deferred, so raw-database tampering is detectable, not prevented. The normalizer will tell you two adjacent ranks are a coin flip and will not pretend otherwise to make a cleaner slide. And it is one person's 72 hours — the seam is real, it is written down, and the verifier is exactly the thing that keeps the seam from being load-bearing.

Try it

git clone https://github.com/pal-123456789/dogfood-portal
cd dogfood-portal
docker compose up --build --wait

Open the gallery at /projects and the signed leaderboard at /normalize/results, publish the results, then export a bundle and — with the db container stopped — run python -m normalize.release over it and watch it re-derive the ranking and check every signature offline.

Prefer to watch first? Here's the walkthrough:

Repository: pal-123456789/dogfood-portal
Docs (architecture, data model, judging, ablation, threat model): pal-123456789.github.io/dogfood-portal
Runbook / 60-second demo: run.md
Interactive walkthrough site: dogfood-site-eta.vercel.app
Video walkthrough: youtu.be/-NThDgcTt-8
On X: the thread

If you take one thing from this, let it be the move rather than the app: don't ask anyone to trust that your results are right — hand them the thing that lets them check, offline, without you in the room. Judging is a place people are right to be a little suspicious, and the honest answer to suspicion is not a louder claim, it is a smaller thing they can verify for themselves. The database will always tell them whatever it currently says. Whether that matches what actually happened is a different question — and the whole point of the 72 hours was to make that question answerable by someone who has no reason to take my word for anything.

The standard library is not a validator: 72 hours of zero-dependency JSON in Rust

Pal — Tue, 01 Sep 2026 09:32:16 +0000

I spent last weekend building a JSON toolkit in Rust under one rule: no third-party dependencies. Not "few". None. The [dependencies] table in Cargo.toml is present and empty, and Cargo.lock holds exactly one package — the project itself. No serde, No serde_json, No clap, No itoa, No ryu.

That constraint is the premise of the Zero Dependency hackathon,
and it is a good premise, because it forces you to find out what the standard library actually promises. Here is the thing I did not expect to find:

About 10% of the JSON documents that RFC 8259 says a parser must reject are accepted by Rust's own number parser.

Not a subtle 10%. NaN, Infinity, .5, 5., +1 and 012 are all invalid JSON, and
f64::from_str and i64::from_str take every one of them. If you write a JSON parser the obvious way — scan to the end of the number token, hand the slice to from_str — you ship a parser that is silently non-conformant, with no warning anywhere.

I have the number because I counted it against a real corpus before writing the parser. The rest of this post is what that measurement did to the design, and what generalizes to languages that are not Rust.

What I built

jaq-lite is a hand-rolled RFC 8259 parser, a serializer, and a jq-style query CLI with rustc-style caret diagnostics — 4,670 lines under src/ and 4,432 lines of tests, standard library only.

$ echo '{"users":[{"name":"ada","age":36},{"name":"linus","age":54}]}' | jaq-lite '.users[] | .name'
"ada"
"linus"

It supports identity, field access, quoted fields, indexes, iteration, pipes, commas, parentheses, the optional operator ?, and eleven builtins (length, keys, keys_unsorted, type, to_entries, from_entries, flatten, first, last, reverse, not). Exit codes follow jq: 2 for a bad flag, 3 for a filter that does not compile, 5 for input that is not JSON, 0 otherwise.

The measurement

JSONTestSuite is the standard conformance corpus:
318 files in test_parsing/, named by what a parser is supposed to do with them. 95 y_ files must parse, 188 n_ files must be rejected, and 35 i_ files are cases the standard leaves to the implementation. I vendored it at a pinned commit before writing a line of parser code.

51 of those files are n_number_*. Four contain invalid UTF-8, leaving 47 that are a single well-formed literal. I tested each one against Rust's documented from_str grammars — the float grammar being Sign? (inf | infinity | nan | Number), case insensitive, and the integer grammar being Sign? Digit+ with leading zeros permitted.

19 of the 47 are accepted. Fifteen by f64::from_str:

.123    -.123   .2e-3        <- float grammar allows Digit* '.' Digit+
1.      -2.     0.e1         <- and Digit+ '.' Digit*
2.e3    2.e+3   2.e-3
Inf     +Inf    Infinity     <- the inf / infinity keywords, case-insensitive
-Infinity       NaN   -NaN   <- and nan, with or without a sign

And four more by i64::from_str, which permits a leading + and leading zeros: +1, 012, -01, -012.

Nineteen fixtures is about a tenth of the entire must-reject set. Delegating validation to from_str fails all nineteen with no error message, no warning, and no test failure anywhere except a conformance harness you had to build first in order to notice.

This is not a bug in Rust, and that is the part worth internalizing. from_str is a conversion, and its contract is Rust's literal syntax, not JSON's. It is permissive about exactly the forms Rust's own grammar permits, which is why it takes inf and +1.
Nothing in its signature or its name claims to be a validator. I reached for it because it was the obvious tool, and the obvious tool answers a different question.

So the number grammar is hand-written: a small state machine over bytes that enforces RFC 8259's actual production — optional minus, an integer part that is either 0 alone or a nonzero digit followed by digits, an optional fraction that must have at least one digit after the dot, an optional exponent that must have at least one digit after e and its optional sign — and then, critically, checks the terminator. 1 000.0 is in the corpus precisely because a grammar that stops at the space and does not ask what follows will accept 1 and then either choke or silently succeed.

The state machine is a few dozen lines. It was the highest-value few dozen lines in the project, and I only knew that because I had counted the fixtures first.

This is not a Rust problem

Before publishing that finding I checked whether it was a quirk of one standard library. It is not. Every language ships a lenient string-to-number conversion, because a conversion's job is to be generous about its own literal forms. Paste these and watch:

>>> float("Infinity"), float("NaN"), float(".5"), float("5."), float("+1"), float("012")
(inf, nan, 0.5, 5.0, 1.0, 12.0)
>>> float("1_0"), float("1_0.5")     # digit separators, since 3.6
(10.0, 10.5)
>>> float("  12  ")                  # surrounding whitespace is stripped
12.0

Python goes one better, and this one genuinely surprised me — the standard library's JSON module itself is non-conformant by design:

>>> import json
>>> json.loads("NaN"), json.loads("Infinity"), json.loads("-Infinity")
(nan, inf, -inf)

That is documented behaviour, not a bug, and it is why a payload written by Python's json.dumps can be rejected by a strict reader in another language. If you exchange JSON with a Python service, allow_nan=False is worth knowing about.

JavaScript's Number() is looser still, because it accepts every numeric literal form the language has, including bases JSON does not have at all:

Number(".5")   // 0.5      Number("0x1")  // 1
Number("5.")   // 5        Number("0b11") // 3
Number("+1")   // 1        Number("0o7")  // 7
Number("012")  // 12       Number(" 12 ") // 12
Number("Infinity") // Infinity     Number("") // 0  <- the empty string is zero

JSON.parse is strict about all of them, which is the point: V8 ships a separate grammar for JSON rather than reusing Number(). Somebody there had this same realization and paid for the second implementation.

The transferable rule is small enough to remember: if a function's job is to convert, it is not your validator. When a wire format specifies a grammar, implement the grammar. The convenience helper will agree with it most of the time, and the disagreements are exactly the inputs an attacker or a buggy producer will send you.

The same trap, three more times

Once you are looking for it, the pattern is everywhere in a parser. Three more instances I hit in the same weekend, all from the same root cause — a helper whose contract is nearly the one you want:

char::is_whitespace() follows Unicode White_Space, which is about a dozen characters including the no-break space U+00A0 and the vertical tab. JSON's whitespace is exactly four bytes: space, tab, line feed, carriage return. A lexer that skips is_whitespace() accepts documents it must reject.

char::is_numeric() accepts １, U+FF11 FULLWIDTH DIGIT ONE — which is in the corpus as a must-reject fixture. The funny part is that from_str gets this one right, because Rust's integer parser only takes ASCII digits. So a lexer that uses is_numeric() to find where a number token ends fails the fixture even with a perfectly correct grammar behind it. The answer is is_ascii_digit(), and the general answer is that "is this a digit" is a different question from "is this one of the ten characters my grammar permits".

read_to_string() panics on 25 of the 318 fixtures, because they are not valid UTF-8 — 12 of those are must-reject cases, which means the naive harness dies before it can print a score for the very inputs it exists to check. The parser therefore takes &[u8] and gates UTF-8 explicitly with Utf8Error::valid_up_to(), so "not valid UTF-8 after 3 bytes" is a diagnostic rather than a panic. The conformance harness uses fs::read, never read_to_string.

And two fixtures nest 100,000 levels deep. Recursive descent on those is a stack overflow, which is a crash and not a rejection — the corpus scores it as a failure either way, but your users experience the difference. Both recursive descents carry an explicit u32 counter and refuse before recursing: 128 levels for a document, 64 for filter parentheses. Both limits are readable back out of the error variant that reports them, rather than being buried in an English message, because a number in prose is not available to a caller.

Diagnostics are worth the afternoon

The one feature I would keep if I had to cut everything else is the error format. This is rustc's shape, and it takes maybe two hours to implement:

$ jaq-lite . bad.json
jaq-lite: bad.json: line 1, column 2: expected a string as the object key
  |
1 | {1:2}
  |  ^
[exit 5]

The renderer works on raw bytes rather than on a str, for the reason above: input that is not valid UTF-8 is one of the failures it has to draw a caret under. A position is stored as a byte offset and lines are only counted when a message is actually printed, so nothing pays for a line table it never uses.

The 35 files where the standard declines to decide

The i_ fixtures are the interesting third of the corpus: lone surrogates, a UTF-8 BOM before an empty object, integers that exceed 64 bits, duplicate keys. RFC 8259 permits a parser to accept or reject each of these, so a score against them means nothing — you can claim any number you like by changing your mind.

What I did instead was write four policies before looking at the files, and derive all 35 decisions from them. Policy 1 is the UTF-8 gate and settles 14 of the cases. The other three handle the numeric range, duplicate keys, and structural edge cases, covering 10, 10 and 1 respectively, with no residue: 10 accepted, 25 rejected, every one traceable to a rule rather than to a judgement call made at 2am.

That is the transferable move for implementation-defined behaviour anywhere: publish the policy, not the tally. A reader can disagree with a policy. A tally offers nothing to disagree with.

A number is the bytes it was written with

The design decision I am happiest about started as a compatibility question. jq does not round-trip numbers through a float: give it 1.0 and you get 1.0 back, -0.0 stays -0.0, and a 30-digit integer comes back intact. It only canonicalizes exponent form,
turning 1e2 into 1E+2.

So a number here is not an f64. It is:

pub struct Number {
    raw: Box<str>,   // the exact span the RFC 8259 grammar validated
    val: f64,        // for comparisons and arithmetic-free semantics
}

Construction is crate-private, which makes "raw was validated" a structural invariant rather than a comment. Three things fall out of it. Round-tripping becomes byte-exact over all 95 documents that must parse, rather than merely semantically equal. -0.0 != 0.0 comes free from comparing the text, while PartialEq on val still gives jq's 1.0 == 1. And it deletes code from the serializer: no appending .0 to integral floats, no scientific-notation threshold, no non-finite branch, because a literal the grammar accepted can never be inf or NaN.

The cost is real and I state it rather than hiding it: roughly 16 bytes plus the digits against 8 for a bare f64, which a number-dense document pays for.

Numbers the tool synthesizes — length, indices — are the only remaining place text gets generated, and they go through core::fmt::NumBuffer with format_into. That is also the itoa replacement, and it is measurably faster than the format! most people would reach for: 1.68 times, measured. One consequence is a named divergence from jq, in the README rather than in a footnote: 1e2 stays 1e2 here.

Compatibility is compared, not remembered

"jq-compatible" is the kind of claim that rots quietly. So CI runs 62 comparisons against a real jq 1.8.1 binary on every push: eight identity cases, six path expressions, 29 builtins over the corpus, and 19 over shapes the script writes into its own temporary directory. It reads its builtin roster out of the code's own match arms and fails if a builtin exists that no comparison reaches, which is a different and much more useful failure than a disagreement — it catches the feature you added and forgot to compare.

An external binary is not a dependency: nothing about jq ships in the artifact, and the comparison exists so that "jq-compatible" is a measurement rather than an adjective.

One trap worth passing on. jq's Windows build writes CRLF, because its stdout is in text mode; Rust writes LF. A naive byte diff therefore fails on every single line for a reason that has nothing to do with correctness, and Git Bash does not save you. The differential runs under WSL, which is also where the reproducibility check lives.

Then I made my documents fail the build

This is the practice I would take to any codebase, and it came out of being burned. A README number goes stale in the commit after the one that wrote it, and prose does not fail a build, so nothing tells you. In this repo, 14 of the 191 tests do nothing but read documents and compare them to the code:

every hash recorded in BUILD_LOG.md must agree with the others that describe the same build
the README must quote the reproducible-build script's assertion phrases verbatim, and the CI job the README names by name must exist in ci.yml
the README must contain no 64-hex-character token, so the decision not to publish a digest as if it were a constant cannot quietly reverse
the test count stated in CLAIMS.md is grepped out of src/ and tests/ rather than typed, because it had already gone stale twice
every limit the code enforces must be stated in the README, and every builtin the code has must be named there

The one that convinced me was a row in the claims ledger that said thirteen while the file it described held fourteen tests. Every gate passed while it was false, because no program was reading it. Now one is. A rewording fails in the commit that reworded it.

The general shape: for each claim in your docs, ask what file would contradict it, then read both in a test. It costs about twenty lines each and it converts documentation from a liability into an asset that CI defends.

A digest is evidence about a tree, not about a project

The build is byte-for-byte reproducible, and the mechanism is three lines in Cargo.toml:
codegen-units = 1 removes parallel-codegen nondeterminism, and debug = 0 with strip = "symbols" removes the absolute paths and symbol tables that otherwise record where the build happened.

Two details make the check honest. The script builds in two directories of different lengths, because equal-length paths give a false pass — a path baked into the binary would still compare equal. And it runs a control build with debug info left in, which yields a different digest every time; that is the sensitivity check that proves the comparison can fail at all. A test that cannot fail is not evidence.

The published digests are dated samples tied to the commit that produced them, not constants.
The binary was 468,704 bytes at commit 34 and 490,336 at commit 65, because 31 commits of source landed in between. Publish the recipe; date the digest.

Reporting speed without lying

The README prints 26.5 MiB/s for parsing and 197.9 for serializing, over a 1,196,671-byte document the test generates for itself. Then it spends a paragraph explaining why you should not trust that figure: five runs of one unchanged binary, across two boots of the same laptop, span 17.7 to 41.8 MiB/s. A factor of 2.4, from source that did not change.

So the suite asserts a floor and never the figure — 5 MiB/s release, 1 debug, raisable through an environment variable and lowerable by nothing. A regression that changes the shape of the algorithm fails a test; an unlucky sample on a busy machine does not. And what gets printed is the mean over the timing window rather than the fastest round in it, because interference only ever slows a round down, which makes the fastest round the flattering one to report and not the one a caller experiences.

What it does not do

No arithmetic, no variables, no user-defined functions. JQ_COLORS is not parsed. There is no --max-depth for either nesting limit. 1e2 does not become 1E+2. All of that is in the README, and one of those 14 tests fails the build if the code enforces a limit the README does not state — which is the only reason I trust the list to still be complete.

Try it

git clone https://github.com/pal-123456789/jaq-lite
cd jaq-lite
cargo test --release

191 tests, no network access after the clone, and one package in the lockfile.

Repository: pal-123456789/jaq-lite ·
v0.1.0 release ·
five-minute walkthrough

If you take one thing from this: the corpus came first, and it is the reason I know a number instead of an opinion. Whatever format you are parsing, find the conformance suite before you write the parser. Nineteen files is not a lot to fix on day one, and it is a very unpleasant thing to discover in production.

How We Ported Python’s natsort to Rust — And the 133,000 Fuzz Inputs That Proved Us Wrong

Pal — Wed, 05 Aug 2026 15:26:51 +0000

A post-mortem on building natsort-rs, fixing a 45x performance bug, and catching subtle edge cases with a custom Pytest monkeypatch bridge.

Why natsort?
Porting Python code to Rust using the "Astral playbook" sounds straightforward on paper: rewrite the hot path, compile to native binary, profit. We chose Seth M. Morton’s popular natsort library—a ubiquitous utility for natural sorting (item2 before item10).
In Python, natsort relies heavily on dynamic typing, tuple-based comparisons, and fallback numeric parsing (try_int, try_float). Translating Python’s dynamic Union[str, int, float] return types into Rust required introducing explicit tagged enums (ParsedComponent) and a hand-crafted Ord trait implementation in key.rs.
Zero unsafe blocks. Pure memory safety. But getting it to build was only 10% of the battle.
The Benchmark That Humbling Us (And the Fix)
When we ran our first full 10,000-item workload benchmark against Python's natsort, the result was embarrassing:
- Python natsort: ~0.24s mean
- Our Compiled Rust Port: ~0.90s mean
An native binary running nearly 4x slower than interpreted Python was unacceptable.

The Root Cause: In our initial implementation, natsorted() recompiled the number-matching regular expression from scratch for every single item in the input array. For 10,000 items, we were recompiling regex 10,000 times.

The Fix: We refactored split.rs to compile the regex once per natsorted() invocation and pass a reference down through the key generation pipeline.

The Real Result: Latency dropped from ~900ms to ~20ms—shifting our port from 4x slower than Python to 11.8x faster on identical workloads. Disclosing this mistake is important: benchmarking without profiling is just guessing.
The Edge Case That Ate Hours: Bare "nan" Tokens
To prove behavioral equivalence, we didn't just write unit tests—we built a differential fuzz harness (fuzz/differential_fuzz.py) that generated randomized strings, floats, and signs, feeding them simultaneously into real Python natsort and our Rust CLI.
At iteration 125,000 (REAL mode, Seed 2), the fuzzer flagged a mismatch (1 out of 500 batches / 0.2% failure rate).
- The Input: A bare text token "nAn" without any accompanying digits.
- Python's Behavior: Python's parse_string_factory applied try_float() to every split component—even ones that didn't match the numeric regex. Because Python’s float("nan") parses as NaN, Python treated "nAn" as a float and sorted it first under default NaN ordering.
- Our Rust Port: Our regex splitter only attempted float parsing on substrings containing digits or adjacent signs. It treated "nAn" as plain text, placing it alphabetically amidst text strings.
The Fix: We rewrote split_into_components_with_regex to split into all pieces (matched and unmatched), uniformly apply float parsing, and insert NUMAFTER-aware separators between adjacent numeric components. We re-ran 133,000+ randomized inputs through the fuzzer across DEFAULT and REAL modes: 0 mismatches.
The Pytest Bridge: What Fuzzing Missed
Fuzzing is great for random input distribution, but structured logic bugs require deterministic tests.
Instead of re-writing natsort’s test suite in Rust (which risks recreating our own biases), we built a Pytest monkeypatch bridge in tests/original/conftest.py. It intercept calls to natsort.natsorted at module load time and shells out directly to our compiled Rust binary using raw bitmask CLI flags (--alg=).
This bridge caught two major bugs that 133,000 fuzz items completely missed:
- The NUMAFTER Leading Placeholder Bug: When an input string started with a number (e.g., "100_apples"), our sequence-alternation logic inserted a hardcoded empty string as a leading placeholder rather than a NUMAFTER-aware sentinel value. This masked NUMAFTER's effect entirely whenever the first component was numeric.
- Missing PRESORT: ns.PRESORT had zero implementation in our Rust port—an outright missing feature.
Once both were fixed, running pytest test_natsorted.py directly against the unmodified upstream Python test file yielded 16 passed, 35 honestly skipped (unsupported locale/path flags), 0 failed, and 0 errors.
What We Walked Back & Honest Disclosures
If you want honest numbers over confident claims, here is what natsort-rs does not do:
- The Unicode Claim Correction: We initially documented that Rust's char::to_digit(10) supported non-ASCII Unicode decimal digits (like Devanagari or Arabic-Indic numerals). A clippy review revealed char::to_digit is strictly ASCII-only at any radix. We updated our documentation to explicitly state ASCII-only numeral support rather than leaving an overclaim standing.
- Numeric Bounds: We bounded integers to i64 and floats to f64. Numbers exceeding 9.2 × 10¹⁸ fall back gracefully to text comparison rather than arbitrary-precision integer comparisons.
- Legacy OS Locales: We explicitly skipped ns.LOCALE and os_sorted. Replicating 30-entry legacy OSX/BSD lookup tables or linking against heavy ICU C-libraries added build complexity for non-deterministic OS features.
Summary
- Codebase: 0 unsafe blocks, clean clippy & fmt.
- Testing: 42 Rust-native tests (including 7 proptest property-based tests verifying Ord transitivity and reflexivity).
- Parity: 133,000+ fuzz items (0 mismatches) + Upstream Pytest suite bridge (16 passed, 0 failed).
- Performance: 9–12x speedup over Python.

Check out the code, benchmark methodology, and full 12-section decision log on GitHub:
🔗
GitHub Link
X Link
YouTube Link

Submitted for Hackathon Raptors: Port Mortem 2026 (Track D: Python -> Rust) 🦖