DEV Community: Ashutosh Mishra

7/7 PASS: The Bugs My Acceptance Checker Could Never See

Ashutosh Mishra — Tue, 06 Oct 2026 06:11:51 +0000

My portal answers all seven of the acceptance checker's questions correctly. The report is committed in the repo: seven PASS lines and a tidy last line that says claimed T1 T2, verified T1 T2.

After the freeze I did the kind of thing that only makes sense once a hackathon is over. I copied the project, put one old bug back in (the exact lines from my own commit history), and ran the checker again.

T2  judge sees own scores ............. PASS
T2  judge cannot see peer scores ...... PASS
T2  participant blocked ............... PASS
T2  csv export works .................. PASS

claimed T1 T2, verified T1 T2

Same tidy last line. Meanwhile, in that same running copy, a judge who owns exactly one ballot asked for her own scores and received two.

Nadia asks for her OWN scores and receives 2 ballot(s):
    Nadia's private note: loved it
    Omar's private note: meh

That is the first half of this post: what a green report is allowed to mean. The second half is about bugs of a different species, the kind you only find by running the shipped thing like a stranger would. I did not do that until it was far too late, and it is my biggest regret of the weekend.

TL;DR

Rubric is a self-hostable hackathon submission and judging portal (Django, PostgreSQL, one docker compose up), built for DogFood 2026, whose brief is "build the platform that will judge you". It claims tiers T1 and T2.

The acceptance checker asks seven questions. I broke a scratch copy of my own code five ways: the checker noticed 2 of 5, my own tests noticed 5 of 5.

Bugs and their lifespans: an isolation leak (about an hour), a misspelled field that hid the live link from every judge (well over a day), a deliberately leaky debug route (around 45 hours), and a container that served no CSS or JavaScript while CI and the checker stayed green.

The front door of my own app was undocumented, with a README pointing at a password that did not exist, because nothing I wrote ever tried it.

Everything reproduces from one commit, 777bbb5, plus two small scripts.

Here is the thing being judged. Rubric is Django 5.2 and PostgreSQL 16 with server-rendered pages and HTMX; one docker compose up gives a seeded portal on localhost:8080 with the network off. It has five roles, events, teams by invite link, submissions with a deadline that holds, a public gallery, expiring judge invitations, an organizer-weighted rubric, seeded judge assignment, cross-judge normalization and a live organizer dashboard, with every state change written to a hash-chained audit log. The voting tier is built but cannot be machine-verified, so I claim T1 and T2 only.

This is the judge's ballot: one project, three weighted criteria, keyboard shortcuts, autosave on every click.

72 hours in one picture

My plan had a lovely shape. The original window was the 25th to the 28th in my time: both weekend days in full and exactly one weekday. Then the dates shifted by a day (the 26th to the 29th), and the same 72 hours now held two weekdays and one fewer weekend day. Classes did not get the memo.

Disrupted schedule, though: is that not the job description? Engineers build inside constraints they did not choose, and the constraint is half the fun. I had a gate-by-gate plan written before kickoff (the grey strip below), and my commit history is a rather honest record of what happened to it once the calendar changed.

Reading the chart like a detective:

Ahead of plan early. Gate G5 was due around hours 42 to 50. It landed around hour 39.
Then a weekday happened. From Monday afternoon to early Tuesday morning, my time, there is not a single commit, about 15 hours. College does not pause for hackathons.
The cheap stretch items never got their slot. G7 (around hours 58 to 64) was reserved for API parity, import and export, and pairwise mode "only if genuinely ahead". It never started, because the gate before it, voting plus hardening, ran until around hour 63 and ate it.
Tests nearly matched the application. Roughly 7,600 lines of tests against 9,500 of application code, plus 4,500 of templates and CSS, about 90% of which landed in the first 39 hours or so. Then the UI stopped changing around hour 63, with roughly nine hours still on the clock. (Lines added, not net; treat the proportions as a mood.)

So the screens existed early. What I kept postponing was looking at them. Later is a word that does not appear in a 72-hour timetable, and I will come back to what it cost.

Seven questions, and what a PASS is allowed to mean

DogFood gives every team the same small program, run.py. It never logs in (you hand it one header per role) and it makes exactly seven requests against your running portal:

#	Tier	Request	It passes when
1	T1	`GET /projects`, no auth	200
2	T1	same page	one of the first three fixture project titles appears in the body
3	T1	`POST /projects/new` as a participant	any 4xx (the fixture event closed in the past)
4	T2	judge A asks for judge scores	200
5	T2	judge B asks for judge A's scores	401 or 403
6	T2	a participant asks for judge scores	401 or 403
7	T2	organizer asks for the CSV	200 and a comma in the first line

Now read that table like an adversary. Every judge in it is a fixture judge. Check 7 is satisfied by a single comma. Neither is a flaw in the checker: the brief says plainly that judges also read the repo, the docs and the demo video, and that the checker is not the whole assessment. It is a floor. My job, as I saw it, was to find out how far that floor sits from the walls.

The bug that lived for about an hour

The spec has a favourite sentence, paraphrased: hiding another judge's scores in your template is not refusing, because the check has to live in the backend, where curl arrives. I believed it. Before writing real code I wrote the one function that decides who may see judge scores, and put my paper drafts through a stress test. The first draft never checked that the caller was a judge: a participant asking for judge scores computed a target of None, watched None != None come out false, skipped the permission check and got an empty 200, which is exactly what the checker's "participant blocked" question is waiting for. The second draft fixed that so enthusiastically that it locked out organizers too. Both wrong, in different ways. That felt like a triumph, and it was a trap, because afterwards the function was declared authoritative, and authoritative things stop getting looked at. Keep an eye on None. It is about to come back.

It went into the code around hour 25, in the commit that introduced judging. This is the function as it was committed:

# commit d62c0d8, around hour 25
target = judge_external_id or actor.judge_external_id
if target != actor.judge_external_id:
    raise PermissionDenied
return Ballot.objects.filter(
    assignment__judge__external_id=target,
    assignment__event=actor.event,
)

Then the adversarial rounds, which I ran with Gemini and Claude and pointed at the isolation function and nothing else, caught it. One of them asked the question I had not: what about judges who are not in the fixture file?

Fixture judges carry an id like jdg_07. A judge invited through an organizer's link in a real event does not. Their external_id is NULL, and when you hand Django None it writes IS NULL. So every id-less judge matched every other id-less judge, and each could read the others' ballots. About an hour after it landed, this went in:

# commit b410a0a, about an hour later
if judge_external_id and judge_external_id != actor.judge_external_id:
    raise PermissionDenied
return Ballot.objects.filter(
    assignment__judge=actor.user,
    assignment__event=actor.event,
)

Django documents this behaviour openly: an exact lookup with None is interpreted as SQL NULL (docs). It is sensible, and it is exactly the wrong thing to have underneath a security decision. The external_id is still used for one thing, validating an id a caller explicitly asks for. It is never used to decide who the caller is.

The same fix commit repaired two neighbours that the review turned up. Autosave could still edit a ballot after it had been submitted. And scores had no range check at all, so a 9,999 or a minus 3 went straight into the database (the column only limits digits). One review, one commit, three defects. Bugs travel in groups.

Why the checker could never see it

The checker's judges are judge_a and judge_b, both fixture judges with ids, and its peer probe asks for jdg_07, also a fixture judge. The bug only exists in a population the demo data does not contain: real judges, invited the way a real organizer invites them. It would have passed every acceptance run and leaked on the first real event. This is the unglamorous shape of the number-one item on OWASP's list, broken access control: the check exists, and is even correct, for the inputs somebody thought of.

The regression test builds the missing population: two judges with external_id=None on the same project, each with a completed ballot, asserting that judge one sees only their own. It is test_judges_without_external_ids_only_see_their_own_ballots, and it is the one that goes red in the next section.

Five ways to break a portal on purpose

Once I had one bug the checker could not see, the obvious question was how many more kinds there could be. So I wrote a small harness. For each experiment it copies the submitted code to a temp directory, changes one thing, seeds a throwaway database, starts the server, runs the real run.py, and then runs the repo's own tests for that area. This is mutation testing done by hand, at the level of "somebody deleted the security check".

The historic bug (row 2): seven of seven PASS, as promised in the opening, and the regression test goes red alone. The control (row 3): I deleted the peer-score check entirely, and both the checker and the repo test fail. Every experiment needs a control that is supposed to be caught, or you cannot tell a blind checker from a broken harness.

The deadline (row 4) has history. During the build, "closed event refuses submissions" turned green for the wrong reason: the checker's POST was refused by CSRF protection, so the 4xx happened whether or not any deadline logic existed. The fix was precision about who needs CSRF protection. Requests that authenticate with an Authorization: Bearer header skip it, because a browser never attaches that header on its own and a cross-site attacker cannot forge it; cookie-session requests keep full enforcement. Now removing the deadline turns the report to verified nothing, because T1 is the floor.

The CSV (row 5): I replaced the export with placeholder,rows: seven of seven PASS, while the real CSV test (per-project rows, raw and normalized means, ranks, review counts, flags) goes red. The tallies (row 6): showing vote tallies during the voting window is a T3 leak. The checker covers T1 and T2 only, so it says nothing, and two repo tests catch it.

None of this says the checker is bad. One of the three misses is outside anything it claims to check, and the other two slipped through for reasons visible in its own source. A floor tells you the floor holds. "Two of five versus five of five" tells you far more than "seven of seven".

The alarm that stayed for about 45 hours

My favourite piece of boring engineering is the route-by-role matrix. tests/authz_expectations.yaml declares, for every named route, who may call it and what each kind of caller must get back, and a test walks the URLconf and fails the build if any route has no declaration. Forty-seven routes, forty-eight declarations (one extra covers the checker's peer-scores probe).

An alarm nobody has heard go off is a rumour. So around hour 18, in the commit that built the auth and policy layer, I planted a deliberately unprotected route, debug/_leaky_test_only/<judge_id>/scores, and watched the coverage test scream. Satisfying. The docstring said, in capitals, to remove it before the freeze.

The part I am less proud of: it was a plain route in the URLconf, with no debug flag around it. It returned an empty list, so it never actually leaked anything, but it sat in the main branch for around 45 hours, until around hour 63. A backdoor built to test your backdoor detector is still a backdoor, and "I will remove it later" is the most expensive sentence in software.

It finally went in the commit that added the safer drill: a synthetic URLconf holding one undeclared route, and a test asserting the alarm fires and names that exact route.

@override_settings(ROOT_URLCONF='tests.urls_synthetic_undeclared')
def test_coverage_alarm_fails_on_undeclared_route(self):
    coverage_test = AuthzCoverageTest()
    with self.assertRaises(AssertionError) as ctx:
        coverage_test.test_every_route_has_declared_expectation()

    self.assertIn('synthetic_control_case_undeclared', str(ctx.exception))

A second test asserts that the production URLconf has no such route and that the old path returns a plain 404. The alarm is still proven, and nothing leaky ships. I should have done it in that order from the start.

What the reviewers found, and what they missed

The adversarial rounds did not stop at isolation. I pointed them at the voting code too, and a few of their findings deserve a table, because each row is a place where my own tests had been quietly agreeing with me.

Finding	What would have happened	The fix
Project links were stored raw	A `javascript:` URL typed into a project link reached an `href`. Forms validate URLs, but my services bypassed forms.	Validation in the service layer (absolute http or https, a host, no whitespace or control characters, 1,024 characters at most), plus a render-time `safe_link` filter so bad data written straight to the database still renders no link.
Organizer role was checked against the current event	An organizer of one event could edit a different event.	One helper, `require_event_role(actor, event, role)`, checks the event being edited, with named isolation tests.
Voter identity in the audit log	Signed-in votes stored the raw user id, and the audit entry carried actor and project together, so an organizer could connect a voter to a choice.	A secret ballot: an HMAC pseudonym keyed by the server secret and an event seed. Voting audit entries carry no user.
Open-link fingerprint included the User-Agent	Cycling the header from one machine counted as a new voter every time.	The fingerprint is the IP address only.

That last row is easy to check yourself. Five different User-Agent strings from one address, against the submitted code:

UA A/1 -> 302   (accepted)
UA B/2 -> 409
UA C/3 -> 409
UA D/4 -> 409
UA E/5 -> 409

The one nobody found. On the judge's ballot there is a button that opens the project's live site. I wrote the template against a field called live_site_url. The model's field is live_url. A Django template treats a missing attribute as empty, so the button simply never rendered: no error, no log line, no failing test, for every judge on every project. It went in with the same commit as the isolation function, around hour 25, and stayed invisible for well over a day, until the link-handling fix was read line by line. The regression test is called test_judge_ballot_renders_live_url_from_the_model_field.

I would like to say the reviewers should have caught it. A reviewer reads code for what is wrong, though, and a misspelled attribute is not wrong in any way a linter can see. It is merely absent. Absences are what people find, because people open the page.

Green CI, unstyled app

Here is a cousin of that typo, one layer further out. The container runs with DEBUG=false. My settings declared a STATIC_URL, but nothing in a production configuration actually served static files: no STATIC_ROOT, no static-serving middleware, no collectstatic. Every page loads /static/css/rubric.css and /static/js/htmx.min.js. In the container both would have been 404s, which means an unstyled app with every HTMX interaction dead. Django's development server serves static files whenever debug is on, so every local run looked fine, and the acceptance checker only looks at status codes on data routes, so it stayed happy too.

I reproduced it on the commit just before the fix (7632286 is the fix; check out its parent) with DJANGO_DEBUG=false, which is how the container runs. Pages return 200, both static files return 404, and the checker prints 7/7 PASS.

The fix is boring in the best way: WhiteNoise in the middleware, a STATIC_ROOT, and collectstatic as a build step.

RUN cd /app/src && \
    DJANGO_SECRET_KEY=build-only-secret \
    DJANGO_DEBUG=false \
    DATABASE_URL=sqlite:////tmp/collectstatic.sqlite3 \
    python manage.py collectstatic --noinput

CI now fetches the CSS and the JavaScript from the running container and fails if either is missing. It landed around hour 53, three quarters of the way in, which is late for a bug that makes your product look like a 1996 homepage.

Proving "runs offline" instead of asserting it. The first rule of the event is that the portal starts with the network off. I did not want to write that sentence, I wanted a job that fails when it is false. The design: build and pull the images while online, add a firewall rule that drops public egress from the Compose network, check that the rule really blocks things (a control request from inside the container must fail and the rule's drop counter must go up), run the acceptance checker, and require the offline report to be identical to the online one. Merely detaching a container from a network proves nothing, because another attached network or a runtime path could still reach outside, so the check watches the actual boundary.

It failed on its first run in CI. The script read the network's ID off a container that had been created but never started, and the ID came back empty. The fix reads the network's name instead. The same commit added a rule that lets already-established connections through, so only new outbound connections are dropped, and a diagnostics step that runs only on failure. That was around hour 60.

The bug you only find by running it

Now the confession. Every check in this post is a machine asking a machine a question. For three days that was the whole development loop, on purpose: I wanted the logic solid and fully tested first, and I told myself the polish, and the looking, would get their turn. I wanted the interface to be clean, quietly sophisticated and easy to use, because a judging portal is a UX problem before it is anything else. I did not run the project by hand until the last day, and by then it was far too late to do anything about what I saw.

So after the freeze I did it properly. I wrote a crawler (a page of Playwright) that logs in as each seeded persona, follows every internal link, and records errors and sideways scrolling at five screen widths.

SIDEWAYS = "document.documentElement.scrollWidth > document.documentElement.clientWidth + 1"

It visited 218 pages at each width, 1,090 page loads in all. (The organizer crawl hit my 70-page cap, so treat that persona as a sample.) The good news is real: no server errors and no tracebacks anywhere. Here is the rest.

Finding	What happens
The organizer's nav does not fit a laptop.	At 1600 px wide it is fine. At 1440 px every organizer page scrolls sideways by 32 px, at 1280 px by 112 px, and "Log Out" slides off the right edge.
There is no phone layout.	At 390 px, nearly every page for every persona scrolls sideways, and the nav is cut off after "Results".
"My Submissions" is a trap for visitors.	Logged out, the nav link returns a raw JSON `authentication_required` error. "Create Team" next to it politely redirects to login. Same visitor, two behaviours.
The front door had no handle.	See below.

The front door

This is the one that stings. Run docker compose up, open the portal, and click Log In. Who are you? The seed script prints four logins, but as Authorization: Bearer headers for curl and for the checker. No seeded account has a password. My README, at the submitted commit, told you to sign in as an organizer with a password that did not exist. A judge who wanted to click around my portal had to know how to create an organizer on the command line, and nothing told them.

The funny part is that a working door existed all along. The auth middleware accepts the same token from a cookie named session, so pasting the seeded token into your browser as that cookie logs you in as that persona. I only found it while writing the crawler, because that is how the crawler gets in. It was in none of my docs.

# log in as the seeded organizer in your browser: add a cookie
#   name:  session
#   value: rubric_seed_organizer_tok_9f8e7d6c5b4a
curl -s -o /dev/null -w "%{http_code}\n" \
  -H "Cookie: session=rubric_seed_organizer_tok_9f8e7d6c5b4a" http://localhost:8080/organizer
# 200

The root cause is a neat little irony. The checker never logs in, by design. And, as it turns out, neither did my seed script, on behalf of a human. Every layer of evidence I built was shaped like that checker. None of them was shaped like a person arriving at the door. The shape was set on day one: my authentication is a standalone token system, chosen because the checker attaches a header and never logs in.

I fixed the README's login instructions after the freeze, in a docs-only commit, and left the code exactly as submitted. The nav and the phone layout stay as they were. I would rather show you the scar than repaint it.

If you take one habit from this section: put a human-shaped check in by hour 36. Ten minutes of "log in, click everything, shrink the window" would have found every row in that table while there was still time to fix them. It costs a page of code, and mine took a page.

The maths that fought back

The brief asks for cross-judge normalization "documented and defended". The problem in one sentence: some judges are stingy, some are generous, and some give everything the same number, so a raw average partly measures who happened to be assigned to you.

A judge who gave everything a 4

Judge jdg_07 scored three projects and every criterion on every one is a 4. Divide by their spread and you divide by zero, which is a fine way to crash a hackathon.

My method is shrinkage in the spirit of empirical Bayes (deliberately not a full Bayesian posterior, which a statistician would poke). Distrust each judge's own average and spread in proportion to how few ballots you have seen from them, by blending in four imaginary ballots from the average judge. The James-Stein estimator is the famous ancestor.

mu_s  = (n * ybar + 4 * mu0)  / (n + 4)      # shrunk mean
s2_s  = (n * s2   + 4 * var0) / (n + 4)      # shrunk variance
z     = (y - mu_s) / sqrt(s2_s)              # ballot value as a z-score

mu0 and var0 are the pool's own mean and variance over all counted ballots. For jdg_07: n = 3, ybar = 4, raw variance 0. The shrunk variance is 0.2246, strictly positive, so the judge gets a finite z-score of 0.5115 and nothing divides by zero. Average a project's z-scores, map back to the 1 to 5 scale, done.

The sentence I had to take back

My first write-up said a constant judge's influence "converges toward zero". It read well, and it was a claim about a formula I had not run on real data. When I ran it, jdg_07 contributed 0.5115 to each of their three projects: a fixed offset that does not differentiate between them. I corrected the docs and logged the correction.

Then the temptation. Remove jdg_07, recompute, and their three projects move 8, 9 and 13 places. Damning? I ran the same experiment on an ordinary judge, jdg_06:

Judge removed	Project	Rank before	Rank after	Places moved
`jdg_07` (constant)	`prj_09`	16	24	8
	`prj_17`	19	28	9
	`prj_19`	18	31	13
`jdg_06` (ordinary control)	`prj_13`	31	26	5
	`prj_26`	32	34	2
	`prj_35`	26	31	5

The ordinary judge's projects move too; rank is brittle in a tightly bunched field. The biggest jump, prj_19, is the best story in the data: it has only two reviews, one of them jdg_07's flat 4, 4, 4, so its only relative signal is the other reviewer, jdg_29 (3, 2, 5). Take the constant judge away and the project is judged by one person. So I did not assert "constant judges cannot move ranks" in the tests. I wrote the sensitivity down as a limitation.

A second constant judge the brief never mentioned

The spec promises one awkward judge. My dashboard flags two. Judge jdg_19 reviewed four projects, one of which (prj_07) is the earlier half of a duplicate submission, superseded by the team's later prj_41, so its ballot is excluded. The three that remain:

Project	Functionality, quality, innovation	Ballot mean
`prj_03`	3, 5, 3	3.667
`prj_24`	3, 4, 4	3.667
`prj_41`	5, 4, 2	3.667

Three different score sets, one identical average: zero variance across the ballots the method sees. The flag fires on the weighted ballot value, not on repetitive-looking raw numbers. The duplicate decision moves the headline numbers too: dropping the superseded copy's five ballots changes the pooled mean from 3.5661 to 3.5758 and the variance from 0.4202 to 0.3930. Those two numbers have a story of their own. Around hour 62 I corrected them: an early draft of my judging document said 3.6190 and 0.6019, the real run said otherwise, and prose does not fail a build. Now a test recomputes the statistics from a seeded run and asserts that the document's digits match.

What the pictures show, and what they do not

On the real fixture, 40 projects are ranked from 121 ballots by 29 judges: seven ranks unchanged, fifteen up, eighteen down. A two-way tie for first at 4.3333 raw is broken in favour of prj_34, and the biggest single move is prj_28, down seven places. None of that proves the new ranking is right. A fixture has no ground truth.

The spread of judge averages narrows from 0.3235 to 0.2023, about 37.5 percent. That is largely by construction: shrinkage pulls things toward the middle whether or not the middle is correct. To test closeness to a truth you have to plant one: 20 projects with known quality, 10 judges with leniency offsets from -1.5 to +1.5, a little noise, three judges per project, and a Spearman correlation against the truth.

Over 100 seeds the correlation rises from 0.7347 to 0.8949, and normalization wins in 100 of 100. I did not tune anything, and it is still a simulation whose judges follow the additive model the method assumes, which is why the proof page labels it SYNTHETIC. One more unflattering fact: eight of the 41 projects have fewer than three reviews, and the dashboard says so out loud.

What I planned, what I shipped

My written cut order, made before kickoff, started with pairwise mode and ended with T3 itself. Reality followed it almost to the letter, which is the only reason none of this feels like a disaster.

Most of T4 did not ship: no webhooks, certificates, embeddable widget or OpenAPI spec. A small JSON and CSV surface does exist (judge scores, ballot submission, organizer progress, audit log and verification, a voting summary, comment flagging, the CSV export). The commit history has a small joke about it: django-ninja, picked to generate the OpenAPI schema, sat in requirements.txt from the first hour to the very last, imported by nothing, and left in the commit literally named final.

Pairwise mode and email-gated voting are not built. Comparing two projects at a time and recovering a ranking with a Bradley-Terry style estimator, as Gavel does for HackMIT, is the elegant way around judge calibration, and also a second judging engine. I would rather ship one hard bonus with a simulation behind it than two with excuses. Email-gated voting either fails offline or makes an identity claim I cannot back.

The audit chain has a stated ceiling. Every state change is hash-chained and a database trigger refuses updates and deletes. That makes tampering visible against a trusted head hash. It does not stop someone with full write access from rebuilding the chain, so the honest advice is to publish the head hash somewhere the database cannot reach. "Tamper-evident" and "tamper-proof" are different words, and I only claim the first.

Docs that outlive their folder. My design notes lived in a folder that was always going to be deleted at submission. A late sweep found nearly a hundred references to them in code comments, the stylesheet header and the subtitle of the organizer dashboard. A test now fails the build if any file cites a document that does not ship.

The regret I would fix first is the UI. I planned something clean and easy. What shipped works, has a ballot screen I am happy with, and has a navigation bar that does not fit a laptop. With another day I would not add a feature; I would spend the first hour clicking through it.

If you are building one of these tomorrow

Ask which population your demo data cannot contain. Fixture data is a tidy little world. My bug lived in the untidy one.
Never scope access by a column that is allowed to be empty. Scope by the row that logged in.
Break your own thing on purpose, and keep the drills in the test suite. A check you have never seen fail is a rumour. A deliberately broken route is a debt with a due date, so pay it early.
Measure before you assert, and write the correction down. "Converges toward zero" felt true and was not.
Score your checkers. List what each layer cannot see. "Two of five and five of five" is information; "seven of seven" is a mood.
Run the shipped artifact like a stranger by hour 36. Log in the way a visitor would, click everything, shrink the window, and test the container, not just the dev server. The cheapest bug report in the world is a person arriving at your door.
Look for absences. A reviewer reads code for what is wrong. A misspelled template attribute is not wrong, it is missing, and it can hide for a day.

The scoreboard

Thing	Number
Acceptance checker	7 of 7 `PASS`; T1 and T2 claimed and verified
Tests	271 (270 pass; 1 PostgreSQL-only test skipped on SQLite; the PostgreSQL CI job is green on the evaluated commit)
Routes with a declared access policy	47 (48 declarations)
Deliberate breakages caught: checker / repo tests	2 of 5 / 5 of 5
Isolation leak, committed to fixed	about an hour
Hidden live-link typo	well over a day
Deliberately leaky debug route, planted to purged	around 45 hours
Container before the static fix	pages 200, CSS and JS 404, checker 7 of 7 `PASS`
Crawl, five widths	218 pages each, no server errors; organizer nav overflows by 32 px at 1440 wide and 112 px at 1280
Real fixture	40 of 41 ranked, 121 ballots, 29 judges
Judge-mean spread, before to after	0.3235 to 0.2023 (by construction)
Simulated recovery of a planted truth	0.7347 to 0.8949, better in 100 of 100 seeds (synthetic)
Bonuses claimed / not built	Normalization Proof, Threat Model / pairwise mode, email-gated voting, most of T4

Run it yourself

git clone https://github.com/Dev-Am12/Rubric.git && cd Rubric
git checkout 777bbb5              # the evaluated commit
docker compose up                 # seeded portal on http://localhost:8080
python3 run.py .dogfood.toml      # the seven questions

pip install -r requirements.txt   # once, for the two scripts below
python mutation_check.py .        # five breakages, two kinds of evidence
pip install playwright && playwright install chromium
python crawl_check.py http://localhost:8080   # a stranger's walkthrough

Both scripts only touch temporary copies or issue GET requests. I ran them on Linux, and they need nothing beyond the standard library, Playwright and the project's own dependencies. Grab them here: https://gist.github.com/Dev-Am12/9d9782e47cfa9fadff1fd5a6b3f90b9a

The repository is at github.com/Dev-Am12/Rubric. The design decisions, including every correction above, are in DECISIONS.md, and the normalization method is defended in JUDGING.md. Demo: https://drive.google.com/drive/folders/1sVArFhpSnswrgRyhIGcacO7rwGqx6_lW

The line I keep coming back to is small. A green report is a receipt for the questions somebody thought to ask. The useful thing you can do the day after a hackathon is ask a few more, and one of them should be: can a stranger get in?

Worth every hour

I will end on the part no table can hold. I enjoyed this. Seventy-two hours with a shifting calendar, a navigation bar that does not fit a laptop and a checker that was far too polite about my bugs, and I had fun for all of it. Hackathon Raptors events are always a good time to take part in, and DogFood was no exception.

Whatever the result turns out to be, these 72 hours were not a waste. I built something I am proud of, broke it on purpose, learned where my own blind spots live, and wrote it all down so the next person gets to skip a few mistakes. That sounds like a pretty good weekend and a half to me.

References

DogFood 2026 brief and spec: dogfoodhack.com and dogfoodhack.com/spec
Rubric, evaluated commit: Dev-Am12/Rubric@777bbb5
OWASP Top 10 2021, A01 Broken Access Control: owasp.org
Django exact lookup and None: docs.djangoproject.com
Mutation testing: Wikipedia
Empirical Bayes and the James-Stein estimator: Empirical Bayes method, James-Stein estimator
Spearman's rank correlation: Wikipedia
Gavel, HackMIT's pairwise judging system: github.com/anishathalye/gavel

Built solo for DogFood 2026 by Hackathon Raptors. #hackathonraptors #DogfoodHackathon

BlitzBroker by Team CodeBlitz. A from-scratch MQTT Broker in Rust.

Ashutosh Mishra — Tue, 08 Sep 2026 18:33:07 +0000

118 Green Tests. One Client Still Hanging.

mosquitto_pub -q 1 doesn't print much when it works. Connect, publish, get an ack, exit. Half a second, if that.

Ours took four seconds. Then it timed out.

Nothing had crashed. Nothing had printed an error. Our broker was up, accepting connections, parsing every byte correctly. 62 tests passing at that exact commit, every one of them green. A real MQTT client just sat there, waiting for an acknowledgment that was never going to come, because we had built half of a feature and tested only the half we'd built.

That gap between "the tests pass" and "the thing works" turned out to be the actual subject of this project, more than the protocol was.

What we were avoiding

Zero Dependency's rule is simple to state and expensive to keep: an empty manifest. No crates. We picked Track C, that is, Web & Network, and built BlitzBroker, an MQTT 3.1.1-subset pub/sub broker in Rust, std only.

MQTT, if you haven't touched it: clients connect to a broker, subscribe to named topics, publish messages, and the broker fans each message out to every current subscriber of that topic. It's the protocol under most "smart home" and IoT stacks: lightweight, binary, built for flaky connections. Normally you get it by running Mosquitto or EMQX, or by pulling in a client library and pointing it at someone else's broker. We wrote the broker.

The honest reason to pick Rust for this specific event, rather than for its own sake: Rust's std gives you almost nothing for the "Web & Network" track. No async runtime. No HTTP. No JSON. No MQTT, obviously. Real Rust network code leans on tokio and serde almost by reflex, avoiding them here isn't a small flex, it's undoing the thing most Rust web code assumes on day one.

The shape of the thing

One broker thread, later four, each owning a disjoint slice of the topic registry, talking to connection threads only through channels. No shared-state locking on the registry, a topic either belongs to a thread or it doesn't, so there's nothing to race. Register and Disconnect broadcast to every shard, since a single client might end up with subscriptions scattered across more than one; everything else routes to exactly one shard, chosen by hashing the topic.

That routing decision is also where the more interesting bug lived, but first: the queue.

Every subscriber gets a bounded outbound queue. When it fills, a slow client, a burst of traffic, something has to give. std::sync::mpsc::sync_channel(N) looked like the obvious tool here, until we actually read what it does under pressure: it blocks the sender once full. That's exactly backwards for a broker. A stalled subscriber blocking the thread that's trying to serve every other subscriber isn't backpressure, it's a single client taking the whole broker hostage. We wanted drop-oldest — discard the stalest buffered message, keep moving and std doesn't hand you that. So it's a Mutex<VecDeque<T>> and a Condvar, maybe forty lines, and it's the one piece of this project I'd point to as "yes, this is genuinely what the stdlib made painful": not that the primitives were missing, but that the one primitive that was there had the wrong policy baked in, and there was no way to ask it for a different one.

The bug that actually ate an afternoon

Wildcard subscriptions in MQTT let a client ask for sensors/+/temp and receive publishes to sensors/kitchen/temp, sensors/garage/temp, anything matching that shape. We wrote the matcher iterative, deliberately not recursive, since both the topic and the filter are attacker-controlled strings arriving off the wire and a recursive matcher is a stack-exhaustion attack waiting to happen, tested it thoroughly against the spec's own worked examples, wired it into the broker's publish path. Subscribed with a wildcard, published to a matching topic, watched it arrive. Shipped it.

Then we added sharding, and it broke, the same way the QoS1 gap did. No error, no panic, just a wildcard subscriber that stopped receiving anything.

The routing rule hashes the topic string to pick a shard. That's correct for exact matches: a subscription to "sensors/temp" and a publish to "sensors/temp" are the same string, same hash, same shard, every time. It's wrong for wildcards, and it took an embarrassingly long time to see why: "sensors/+/temp" and "sensors/kitchen/temp" are different strings. They hash to different shards, almost certainly. The subscription succeeds. The shard holding it just never sees the publish that should have matched it, because that publish landed on a different shard entirely, one that has no idea the wildcard filter exists.

The fix, once we saw it clearly, was one sentence: treat a wildcard subscription like Register or Disconnect, broadcast it to every shard, not just one. Exact-match subscriptions keep routing by hash; wildcard ones go everywhere, so whichever shard eventually gets the matching publish already has the filter to check it against. We verified it the only way that felt honest after the first miss: real mosquitto_pub/mosquitto_sub, eight different retained topics deliberately spread across shards, one wildcard subscriber, all eight arriving. Not because a unit test said so, because we watched it happen.

The half-built feature

Which brings back the four-second hang. MQTT's QoS 1 means "the broker must acknowledge receipt of this publish." We built the wire format for that acknowledgment: encode it, decode it, round-trip it and wrote tests proving the bytes were exactly right. What we hadn't built was the broker actually sending one when a client published at QoS 1. Every test we'd written was protocol-layer: bytes in, struct out, struct in, bytes out. None of them ever ran the connection-handling code that's supposed to notice "this was QoS 1, better say something back" because that logic didn't exist yet, and a test can't fail to cover code that isn't there.

mosquitto_pub -q 1 was the first thing that ever actually waited for that acknowledgment instead of just checking whether the bytes we produced were well-formed. It hung, because we'd never asked anything to wait before.

Same lesson, twice, in one project: encode/decode tests prove your wire format is correct in isolation. They prove nothing about whether the broker behaves correctly end to end. The only way we found either gap was pointing a real, independent MQTT client at the compiled binary and watching what it actually did not what our own code claimed or the AI agent claimed it would do.

The bonus we didn't take

Zero Dependency offers a "Single File" bonus, that is, the whole submission as one source file someone could read top to bottom. We tried. A branch exists, ~3,885 lines, compiles, passes the full test suite.

It's not a single-file program. It's a single file. Every mod boundary from the multi-file layout got preserved wholesale, the module walls are all still there, just copy-pasted into one document instead of split across several. That's a stapled concatenation, not the thing the bonus is actually asking for, and we knew the difference the moment we looked at it honestly.

We didn't claim it. It's disclosed in the README as attempted and not achieved, the branch stayed public, and the bonus tally doesn't include it. It would have been trivial to point at "one file, compiles, tests pass" and call it done and nobody skimming a submission list would have caught the difference on the first pass. But this whole event is built around the idea that a naive, honestly-disclosed gap is worth more than a confident claim that doesn't survive a second look, and once you've internalized that watching your own QoS1 hang, it's hard to selectively unlearn it for the bonus you'd rather have.

What actually got killed

No tokio, thread-per-connection plus a channel-based actor model instead of an async runtime. No serde, the entire MQTT codec, every packet type, hand-rolled against the spec. No crossbeam, the drop-oldest bounded queue above, built because the stdlib's one bounded-channel primitive had a backpressure policy we didn't want, not because a policy was missing entirely. No mqtt/aedes, the packages you'd actually reach for to do exactly this locally, without thinking about it, in about four lines and one npm install.

The decision I'd take back

Small, and it cost real time rather than teaching us anything: our decision log kept colliding. More than once, someone edited it from a copy that predated the last person's changes, and a decision we'd already written down quietly vanished, replaced by someone else's entry claiming the same slot. Not a code bug but rather a coordination one, which is anyways understandable for a team of 4. The fix was never clever, just "pull the actual current version of a shared file before you touch it," and we should have said that out loud on day one instead of relearning it three separate times.

The numbers, honestly

118 automated tests, all passing, none of them the reason we found either real gap, live testing against actual clients was. A four-shard broker pushing roughly 495k msg/s against a single-shard broker's 402k at high concurrency across 64 topics. A real improvement, and a modest one, because at this scale the network syscalls dominate over how many threads are splitting the registry, and we'd rather say that plainly than let a comparison table imply more than it earned. Thirteen documented stdlib substitutions. Zero entries in cargo tree.

The bug that mattered most wasn't in the MQTT spec, and it wasn't really in our code either. It was in trusting a test suite to tell us something it was never actually testing.

Finally

Win or not, this was again one of the most fun 72 hours I've spent building something on a Hackathon Raptor's Hackthon. I cannot thank enough to my teammates at Team CodeBlitz for the experience and knowledge we shared together. We're eagerly waiting for results and feedback from the esteemed judges.

Repo: https://github.com/Dev-Am12/BlitzBroker
Demo: https://drive.google.com/drive/folders/1ScvHPNuxs_JTPaasKMEvzKrL9Wo5OLah?usp=sharing
Built for Zero Dependency 2026, run by Hackathon Raptors.