DEV Community: codewitharyan29

Same score, 23 places apart: the normalization maths behind a hackathon judging platform

codewitharyan29 — Fri, 02 Oct 2026 07:11:31 +0000

Two projects in the official DOGFOOD data set both had a raw average score of 3.44, from three judges each.

After I normalized the scores, one finished #37. The other finished #14.

Same number. 23 places apart. This is the story of why, and of the other technical problems I hit while building Verdict, a platform that runs a hackathon end to end: events, teams, deadline-enforced submissions, rubric judging, public voting, certificates, webhooks and an embeddable gallery. It was built for the DOGFOOD hackathon by @partnerships_raptors.

Stack: FastAPI, SQLAlchemy 2.0 async, PostgreSQL, Redis, Next.js.
Repo: https://github.com/Lakshya-Varshney/Verdict
Demo_Video: https://youtu.be/bmdsrYy_dy4

1. The problem: a raw average cannot tell who was asked

The obvious way to rank projects is to average judge scores. It silently assumes every judge uses the scale the same way.

They do not. On the official data (40 projects, 30 judges, 126 review rows), judges with at least 5 reviews had personal averages from 3.11 to 4.22, with the same spread (standard deviation 0.81 for both). That is a full point of pure leniency.

Now look at the two projects:

Project	Raw mean (rank)	Its reviewers' own average	Normalized rank
Flat Meadow	3.44 (#24)	3.97 (generous panel: 4.22, 3.61, 4.08)	#37
Glass Signal	3.44 (#26)	3.48 (close to the pooled mean 3.57)	#14

Flat Meadow got a below-par verdict from each of its reviewers. Glass Signal got a normal one from each of its reviewers. The raw average sees two identical numbers.

2. The fix: per-judge z-scores

For each judge, compute their mean and standard deviation over everything they scored, then convert each score into "how far above or below this judge's own usual":

z(judge, project) = (score - judge_mean) / judge_stddev
normalized(project) = mean of z over the judges who reviewed it

Why z-scores and not min-max or rank averaging? They correct both leniency (the mean) and spread (the standard deviation), and they stay well-defined when projects have different numbers of reviews. Two reviews on one project and five on another is fine, because each project is just the mean over whoever reviewed it.

Result on the real data: 38 of 40 projects change rank, Spearman correlation 0.864 against the raw ranking. I also recomputed everything independently with plain statistics and no application code, and the two agree to floating-point precision.

3. The edge case that decides whether the maths is correct

A z-score divides by the judge's standard deviation. What if a judge gave 4/4/4 on every project? Or reviewed only one project? Then the standard deviation is zero and there is nothing to divide by.

The tempting shortcut is to use that judge's raw score. I rejected it, because it mixes two scales. Raw scores sit around 3.0. Z-scores sit around 0. If you average them together, every project a flat judge touched jumps by roughly that judge's raw score, and the judge with no opinion ends up as the loudest voice in the ranking.

The design that works: standardize the flat judge's raw score against the pooled mean and standard deviation of every score in the event.

if not flat:
    z = (score - mu) / sigma                 # judge's own scale
elif pooled_sd > 1e-12:
    z = (score - pooled_mu) / pooled_sd      # same scale as everyone else
else:
    z = 0.0

The flat judge still contributes a real signal (higher raw score means higher z), it lands on the same scale as everyone else, and it is never silently trusted: they appear in a zero_variance_judges list and in the audit log. The results payload also separates "identical scores everywhere" from "only one review", so the UI never mislabels a judge who simply had too little data. There is a regression test for exactly this: test_zero_variance_judge_does_not_swamp_z_scores.

On the official data, four judges take this path.

4. Real data fights back: the duplicate project row

The official fixtures.json has 41 project rows for 40 teams. prj_41 repeats prj_07: same team, same title ("Dry Harbour"), same repo, different id. Deduplicating by id would never catch it, so I merge by team plus title.

The interesting part is the consequence. Three judges had scored both ids, so their criterion rows collided after the merge: 9 rows, taking 378 score rows down to 369. One of those judges had scored the original at 2.33 and the duplicate at 3.67. Under the documented rule (the later review wins), all three of her remaining projects score 3.67, and she correctly lands in the zero-variance path from section 3.

Data cleaning is not neutral. It changes the maths downstream. So I wrote the rule down (later review wins, averaging is the alternative) and surfaced every flagged judge in the results payload.

5. Role isolation, tested by attacking the running system

Reading the code tells you what you intended. Probing the live API tells you what you built. I ran adversarial probes against the running system and found 12 real problems, each now covered by a regression test. The ones that taught me the most:

An organizer could grant themselves admin. Every permission check accepts admin from any event, so one self-grant meant site-wide control. Only an admin can grant admin now.
Any organizer could read every other event's audit log. 98 of the first 100 rows returned belonged to other events. Audit access is scoped to the caller's own events.
Judges could change scores after results were published. The judging window was enforced in the UI, not the API. A UI button is not an access control.
The vote count was hidden in one field and leaked in another. While voting was open count was null, but vote_count still carried the real number.
alice+1@gmail.com counted as a new voter. Email voting now uses a canonical mailbox: lowercase, drop the +tag, ignore Gmail dots.

Roles are read from the database on every request and scoped per event, so cross-event isolation is structural rather than a convention.

6. A 13 second gallery, and `lazy="raise"`

With the real 40-project data the gallery took 13 s and /roles took 18 s, because SQLAlchemy was eager-loading every relationship.

I set lazy="raise" on every relationship. Any accidental lazy load now throws instead of quietly running extra queries, so every query has to be written explicitly. It forces more typing and gives predictable performance. test_real_scale_endpoints_stay_fast keeps it that way.

7. An audit log you can verify, not just trust

An append-only audit log is normally append-only by convention. Anyone with database access can still edit history.

In Verdict, every audit row contains the hash of the previous row. GET /admin/audit/verify and an offline script both detect tampering. To keep the chain linear, appends take a Postgres advisory lock (pg_advisory_xact_lock), so two concurrent transactions can never read the same tip hash and fork the history.

The same principle runs through the certificates: Ed25519 signatures instead of HMAC, so anyone can verify a certificate offline with only a public key, with no shared secret. Webhooks use a transactional outbox, so events cannot be lost or invented, and no message broker is needed to run offline.

8. The scope decision: normalized absolute scores

Judging can be built two ways: judges score each project against a rubric, or judges compare two projects at a time (pairwise, Bradley-Terry style). I shipped normalized absolute scoring.

It works directly with the weighted rubric, the result is recomputed from raw scores on demand (never trusted from storage), and I could check it independently against the real data. That made the correctness work possible: the normalization proof, the role-isolation tests and the verifiable audit trail.

What I would carry into the next build

Test the maths on messy real data, not a clean example. My clean example proved the idea. The duplicate row and the flat judges decided the actual design.
Attack the running system early. Role-isolation bugs hide in code that reads correctly.
Make failures loud. lazy="raise" and server-side checks turn "slow and silently wrong" into "fails immediately".

The full normalization proof, the threat model and all the tests are in the repo: https://github.com/Lakshya-Varshney/Verdict

Built for DOGFOOD. Tagging @partnerships_raptors for the Write Up Quest.

Killed My Own Process 300 Times to Prove ChronoVault Doesn't Lie About Durability

codewitharyan29 — Thu, 03 Sep 2026 15:27:27 +0000

TL;DR: I built ChronoVault v2 for Hackathon Raptors' Zero Dependency Hackathon (Track D — Data & Storage) — a content-addressable snapshot and recovery engine, zero runtime dependencies, backed by a hand-rolled pack-file format. A comment saying "this is atomic" is worth nothing to a judge, so instead of asserting crash-safety, I hard-killed my own writer process mid-write, over and over, and made the vault prove it survived. This is that story, plus a benchmark number I didn't want to publish and three Windows bugs that only showed up on real hardware.

279 tests. 20 CLI commands. 0 runtime dependencies. judge_mode.py VERIFIED on Python 3.11 and 3.14.

Proving crash-safety instead of asserting it

Anyone can write "atomic writes, crash-safe" in a README. It costs nothing and proves nothing. So recover-check is tested against a real SIGKILL, not a simulated one: a writer subprocess runs a snapshot, gets hard-killed at a randomized point mid-write, and the vault is reopened cold — no manual repair step allowed.

The test checks three things, not one:

Every snapshot acknowledged before the kill is intact.
The in-flight snapshot at the moment of the kill is either cleanly absent or a discarded partial write — never garbage returned from a read.
The vault is immediately usable afterward, with no repair command required.

This is also where the pack-quarantine logic earned its place instead of being a "nice to have." Early in testing, a kill landing mid-pack-write left a truncated pack that the recovery scan initially tried to read as valid — and returned a wrong length from. Quarantine — isolate anything that fails its checksum before it ever reaches the index — closed that hole. The fix wasn't "trust the file less," it was "never let an unverified pack answer a read at all."

Run it enough times and the interesting failures stop being about the crash itself and start being about the code paths a crash exposes that a clean shutdown never does. That's the actual value of hard-killing your own process 300 times instead of once: the first ten runs prove the happy path works, and the next 290 are what find the pack-quarantine gap.

Architecture at a glance

flowchart TD
    subgraph L1["CLI"]
        CLI["cli.py — 20 subcommands"]
    end

    subgraph L2["Core Engine"]
        Snap["snapshot.py<br/>atomic rename + fsync"]
        CAS["Content-Addressed Store<br/>hashlib SHA-256, pack files"]
        Idx["path_history.py<br/>rename-aware lineage"]
        Rec["recover.py<br/>recover-check, SIGKILL-tested replay"]
    end

    subgraph L3["Persistence"]
        Pack[("Pack files on disk")]
        Quar["Pack quarantine<br/>corrupt/truncated packs isolated"]
    end

    CLI --> Snap
    CLI --> CAS
    CLI --> Idx
    CLI --> Rec
    Snap --> CAS
    Snap --> Idx
    CAS --> Pack
    Idx --> Pack
    Rec -.->|verified via SIGKILL injection| Snap
    Rec --> Pack
    Pack --> Quar

    style L3 fill:#2d2d2d,stroke:#f5a623,stroke-width:2px,color:#fff
    style Rec fill:#2d2d2d,stroke:#f5a623,stroke-width:2px,color:#fff

The shape that matters is where recovery sits: it reads pack files directly, never through the CLI or the snapshot writer, the same way a verify-style tool deliberately bypasses the normal open path. That's deliberate — if the write path itself is what crashed, recovery can't depend on any code the crash might have left half-updated.

The number I didn't want to publish

Every other number in this post makes ChronoVault look good. This one doesn't.

I benchmarked vault snapshot against diskcache — the closest real-world comparison for local content-addressed storage:

Metric	ChronoVault	diskcache
Write throughput	baseline	9–13x faster
On-disk size, repeated / near-duplicate snapshots	5–47x smaller	baseline

Why it loses on writes: every vault snapshot call pays for a full content hash of each file plus a pack-index lookup before a single byte is committed, because deduplication has to happen before the write, not after. diskcache skips that entirely. That's the whole trade in one sentence: I pay a hashing-and-lookup tax on every write so a hundred near-identical snapshots don't cost a hundred times the disk.

If your workload is write-latency-bound, diskcache is the right tool and I'm not pretending otherwise. ChronoVault wins when the workload is storage-bound — long retention windows, many near-duplicate snapshots, disk-constrained environments. A real loser column next to a real winner column, not a benchmark that only shows its best number.

Where the standard library actually made me suffer

Three places, all invisible until I tested on real Windows hardware instead of assuming POSIX behavior generalizes.

multiprocessing's spawn context raised a KeyboardInterrupt that wasn't one. Worker processes died mid-operation with nobody near Ctrl+C. Windows' spawn-based multiprocessing ties process teardown to _winapi.WaitForSingleObject, which can surface as a spurious interrupt with no signal ever sent — a path fork-based Unix multiprocessing never touches. I misdiagnosed this three separate times as my own bug before proving otherwise, using a detached Start-Job test that ran the worker fully isolated from the parent's console.

WinError 32 — Windows refusing to delete a file it considers still open. POSIX lets you unlink() an open file; the inode survives until the last handle closes. Windows refuses the delete outright. Handling it meant retrying past the handle-release window instead of importing psutil to hide it.

A join-before-drain ordering bug deadlocked the IPC queue. worker.join() was called before the result queue was fully drained. If the OS-level pipe buffer filled before the parent drained it, the child blocked on a full pipe and the parent blocked on join() — a deadlock that only appears under load. Fixed by draining before joining.

None of these are exotic. They're the specific price of touching multiprocessing and cross-process queues without a battle-tested wrapper library between you and the OS.

The edge case that ate an afternoon

Rename-aware file lineage in the path-history index. The question sounds simple — "was this file renamed, or deleted and recreated?" — until content, path, and inode can each change independently, sometimes in the same tick, with zero heuristics borrowed from someone else's diffing library to lean on.

The ambiguous case that actually happened: a file renamed and edited in the same snapshot cycle. "Renamed A→B, then edited" and "A deleted, unrelated B created" are both defensible reads of the same delta, and guessing wrong either invents a false history or silently drops a real one.

I didn't solve it by getting cleverer. I wrote down, explicitly, which cases the lineage tracker resolves correctly, and which ones it honestly doesn't try to disambiguate. A documented limitation beats a confident guess that's wrong on the exact case a judge tests.

What I'd do differently

The Windows CI matrix went in after most of the Windows-specific bugs had already been found manually, one laptop test at a time. Every area I later ran through real Windows hardware for the first time produced at least one genuine finding. If Ubuntu + Windows CI had existed from the start, at least two of those bugs would have surfaced in an automated run instead of on my own machine, days closer to the deadline than I'd have liked.

Where it landed

279 tests passing on Python 3.11 and 3.14, judge_mode.py VERIFIED
20 CLI commands, zero third-party runtime dependencies
Crash-safety proven, not asserted — recover-check verified against real SIGKILL injection, with pack quarantine closing the gap testing found
CI green across Python 3.10–3.14 × Ubuntu + Windows
STDLIB.md documents 14 real substitutions, each cross-referenced to the file that uses it
All four bonus categories claimed (+16) — Single File, Reproducible Build, Package Killer re-earned via the diskcache numbers above, STDLIB Log

python judge_mode.py

runs the full verification — test suite, single-file hash check, isolated-mode dependency audit, and the content-addressing proof's 8 invariants — as one command, so a judge doesn't have to trust this post; they can re-derive it.

Closing thought

Anyone with an AI coding agent can produce a CLI that hashes files and calls it content-addressed storage. The hard part was building enough ways to catch myself being wrong — a crash-safety claim verified with a real SIGKILL instead of a comment asserting atomicity, a benchmark table honest enough to put a 9–13x loss next to a 5–47x win, and a judge_mode.py that re-derives every claim in this post instead of asking a judge to trust it. The vault is the artifact. The willingness to kill it 300 times and publish what survived is the actual submission.

Repo: github.com/codewitharyan29/ChronoVault-v2
Track: D — Data & Storage

Built for Zero Dependency 2026, run by @partnerships_raptors.

ZeroDependencyHack #Python #OpenSource #DeveloperTools #CLI

PythonStandardLibrary #BuildInPublic

Porting natsort to Rust taught me that "it compiles" is the easy 5%

codewitharyan29 — Thu, 06 Aug 2026 14:44:40 +0000

TL;DR: I ported Python's natsort library to Rust for Port Mortem 2026.

Writing the sorting algorithm turned out to be the easy part.

The difficult part was proving the Rust implementation behaved identically to the original Python library across thousands of edge cases without quietly changing behavior.

That meant treating verification as the real project.

Why I chose natsort

Most demo ports are calculators, parsers, or utilities with a handful of tests.

I wanted something that would actually stress the verification process.

natsort sorts strings the way humans expect:

file2
file10
file20

instead of lexicographic order.

That sounds simple until you discover it supports:

seven algorithm variants
signed and floating-point numbers
scientific notation
full Unicode numeric characters
locale-aware behavior
Roman numerals
fractions
a real upstream test suite covering years of edge cases

A port that looks correct on five examples proves almost nothing.

A port that survives the library's own test suite, differential fuzzing, CLI comparisons, property testing, and mutation testing starts becoming evidence.

The architecture

The verification pipeline ended up looking like this:

             Python Test Suite
                    │
                    ▼
          Thin Python Adapter
                    │
                    ▼
             Compiled Rust Binary
                    │
                    ▼
          Rust natsort Implementation
                    │
                    ▼
 Compare:
 • stdout
 • stderr
 • exit codes
 • ordering
 • fuzz results
 • properties

The adapter intentionally contains almost no logic.

Its only job is forwarding Python test inputs to the Rust implementation.

That design choice turned out to matter more than I expected.

What broke (and why those bugs mattered)

1. Differential fuzzing found a bug in the original library

Differential fuzzing generates thousands of random inputs and compares both implementations.

Most mismatches were my mistakes.

One wasn't.

>>> natsorted(["1e400", "1e500"], alg=ns.REAL)
['1e400', '1e500']

>>> natsorted(["1e500", "1e400"], alg=ns.REAL)
['1e500', '1e400']

Both calls succeed.

Both disagree.

The reason is surprisingly simple.

Both values overflow to floating-point infinity.

Since both become inf, the comparison reports them equal, and Python's stable sort simply preserves whichever order the inputs arrived in.

Rust's f64 behaves the same way.

That means the correct port is not one that "fixes" the behavior.

The correct port reproduces it.

I reported the issue upstream:

natsort#192

Finding a real bug in a mature library simply by holding another implementation beside it was one of the most satisfying moments of the project.

2. My adapter was lying to me

I assumed the adapter was a thin wrapper.

It wasn't.

One function calculated the correct Rust-backed answer...

...and then ignored it.

Instead it silently fell back to a handwritten Python implementation that didn't understand signed numbers or Unicode.

index_natsorted(["-5", "-1", "3"], alg=ns.REAL)

returned:
[2, 1, 0]

correct:
[0, 1, 2]

Differential testing doesn't automatically catch bugs in the adapter.

You have to verify the adapter itself.

That lesson probably saved me more time than any optimization.

3. Windows disagreed with Linux

Everything looked perfect.

Until I ran it on Windows.

Five bugs immediately appeared.

executable needed .exe
python3 didn't exist (py does)
subprocess output defaulted to CP-1252 instead of UTF-8
file reads had the same encoding problem
Unix-only resource crashed benchmarks entirely

None of those appeared on Linux.

None appeared in my development environment.

Every one appeared on a real Windows machine.

"Works on my machine" turned out to be the weakest verification strategy of all.

4. GitHub Actions lied with a green checkmark

The deployment workflow looked successful.

The live website returned 404.

The culprit?

A multi-line shell block:

curl ... | sh
build
deploy

The installation silently failed.

The following commands still exited successfully.

GitHub reported the entire step as green.

The fix wasn't clever.

It was boring.

I switched to:

cargo install wasm-pack

and added a deployment check that refuses to publish unless the expected build artifacts actually exist.

A green checkmark only matters if it's checking the thing you care about.

How I proved behavioral equivalence

The implementation wasn't the deliverable.

Evidence was.

I ended up with six independent layers of verification.

1. Upstream tests

Run the original natsort test suite without modifying it.

Tests excluded only when Rust genuinely cannot reproduce Python-specific behavior, and every exclusion is documented.

2. Differential fuzzing

27,000 seeded comparisons across all algorithm variants.

Same inputs.

Same outputs.

Reproducible failures.

3. CLI differential testing

2,800 CLI invocations comparing:

stdout
stderr
exit codes

Matching output alone isn't enough.

Programs communicate failure through exit codes too.

4. Property testing

Randomized verification of:

idempotence
antisymmetry
transitivity

These find classes of bugs that handwritten examples never will.

5. Mutation testing

I deliberately broke my implementation.

reversed comparisons
removed signs
altered parsing logic

If the verification pipeline couldn't detect injected bugs, it wasn't trustworthy.

Every injected mutation was caught.

6. Honest benchmarking

Instead of publishing a single "10× faster" headline, I reported:

mean
median
p95
p99
peak memory

The real result was 6–8× faster, depending on workload.

Honest numbers build more confidence than inflated ones.

The decision I'd take back

I spent a huge amount of time chasing perfect Unicode parity.

It was technically rewarding.

It pushed differential matching from roughly 43% to 100% across the full Unicode character set.

I'm proud of that work.

But I crossed the point of diminishing returns.

One adapter-level discrepancy survived until the deadline on a single CI runner.

I excluded that specific test rather than claim support I couldn't fully defend.

Looking back, I'd stop polishing Unicode earlier and spend those hours investigating the remaining verification discrepancy instead.

Verification gets exponentially more expensive near the finish line.

Knowing when to stop is part of engineering.

Final numbers

By the end of the project:

Original upstream tests running against Rust
27,000 differential fuzz comparisons
2,800 CLI differential runs
Property testing
Mutation testing
Honest benchmark suite
6–8× speedup
One upstream bug reported
Cross-platform verification on Linux and Windows

Reproduce everything

Everything needed to verify the project is public.

Demo

https://youtu.be/jVz_2AP4U4w

Live WASM demo + verification dashboard

https://codewitharyan29.github.io/Port-Mortem/

Repository

https://github.com/codewitharyan29/Port-Mortem

Upstream issue

https://github.com/SethMMorton/natsort/issues/192

**Hackathon rapters : https://www.raptors.dev/project/code-resurrection-2026-port-mortem

Closing thoughts

Anyone with an AI coding agent can produce something that compiles.

Compilation is the easy part.

The difficult part is proving—without hand-waving—that another implementation behaves the same as the original across thousands of edge cases, different operating systems, different execution paths, and even the original project's own bugs.

By the end of this project, I stopped measuring success by whether the Rust code compiled.

I measured success by how much evidence I had that it behaved like Python.

Built for Port Mortem / Code Resurrection by Hackathon Raptors

#PortMortem2026 #HackathonRaptors #Rust #Python #OpenSource #SystemsProgramming #Testing #Verification #Fuzzing
**