DEV Community: Bro Force

I wrote a justification in my docs, then tested it. It was mostly wrong.

Bro Force — Fri, 02 Oct 2026 07:41:45 +0000

Built for the @partnerships_raptors Hackathon Raptors DOGFOOD hackathon: 72 hours, build the submission and judging platform that will judge you. FastAPI, PostgreSQL 16, Next.js, one docker compose up. Code: github.com/broforce6909-cmd/DogFood.

My judging docs contained a table explaining why you must not put an epsilon in the denominator when you normalize a judge who gave every project the same score. It said the judge would "become the most influential voice in the event."

After the build, I tested that sentence on the organizers' own data. It is false. The thing that actually mattered was something I had filed under "small detail." This post is about that, and about three other places where the problem was harder than the spec made it sound.

1. Normalizing judges: what actually moves the ranking

The setup. Judges do not share a scale. In the organizers' fixture, judge averages run from 2.0 to 4.22. I standardise each judge (z = (score - judge_mean) / judge_sd), average z-scores per project, and rescale to 1-5. Two edge cases needed rules: a judge with zero spread, and a judge with very few ballots.

My rules: a zero-spread judge maps to z = 0 ("I am not distinguishing between these"), and every judge's mean and sd are shrunk toward the field's with a constant k = 3.

The test. I ran the real scoring.py on the organizers' fixtures.json (30 judges, 126 ballots, 41 projects before I dedupe a duplicate team; equal rubric weights assumed) and compared alternatives. Ranking agreement with what I shipped, as Kendall tau:

Method	tau	Top-5 overlap
Raw mean, no normalization	0.80	4/5
Plain z-score, epsilon in the denominator	0.80	4/5
Shrinkage on, but a flat judge borrows the global sd	0.92	5/5
Shipped: flat judge = z 0, shrinkage k = 3	1.00	5/5

Finding 1: normalization matters. 13 of 41 projects move three or more places versus the raw ranking, and raw third place is not normalized third place. Averaging raw scores across judges who use different scales gives a measurably different leaderboard.

Finding 2: my epsilon argument was wrong. An exactly flat judge has score - mean = 0, so 0 / (0 + 1e-9) = 0. The epsilon version produced the same ranking as no shrinkage at all. It did not blow up. And the "flat judge" rule versus borrowing the global sd moved one project in the top ten. The rule is still the principled choice, but I had sold it as a disaster-prevention measure and it is not one on this data. Only 3 of 30 judges are flat, and two of those have a single ballot, which is flat by definition.

Finding 3: the real hazard was the small-n judge. For a judge with two ballots, population sd gives z = +1 for the higher and z = -1 for the lower, always, whether the two scores differ by 0.33 or by 1.0. In the fixture, 8 of 30 judges have two or fewer ballots, and the six two-ballot judges' gaps range from 0.33 to 1.0. Without shrinkage each of them hands out the same-sized push regardless of how much they actually discriminated. That, not the flat judge, is what the plain z-score gets wrong (tau 0.80 again), and shrinkage is what fixes it.

Finding 4, the disappointing one: k is a hand-picked constant and the top five is sensitive to it.

k	Kendall tau vs k = 3	Top 5
~0 (no shrinkage)	0.80	one project in, one out
1-5	0.94-1.00	identical
10	0.94	third place drops out of the top five
20	0.92	further reshuffle
∞ (everyone is average)	0.89	...

There is a stable plateau around k = 1 to 5, and I picked k = 3 inside it. But there is no ground truth here, so I cannot tell you k = 3 is right, only that it is not fragile nearby. It is published in the API response as shrinkage_k. The honest claim is "inspectable and stable in a range," not "optimal." I chose a fixed constant over a hierarchical model because with a dozen judges and a few ballots each, the between-judge variance would itself be estimated from almost nothing. You would get a more impressive number with no more information in it.

Reproduce all of it with normalization_sensitivity.py in the repo (no database needed).

A related catch. My first version z-scored against one global pool. A judge restricted to one track was being compared against a mean that included tracks they never saw, so a track that simply drew stronger projects looked like a harsh judge, and normalization cannot tell those apart. normalize_by_track() now partitions by the submission's track (not the judge's, since an unrestricted judge spans tracks). A normalized 4.1 in one track and a 4.1 in another are still not the same claim, and I wrote that into the docs rather than hide it.

2. Role isolation: make the bad request unexpressible

The rule is one sentence: organizers may read every ballot and may not write one. Everywhere else, organizer and admin can do anything to an event. Scoring is the single exception, because anyone who can author a judge's score can forge a result, and nothing downstream (export, normalization, audit log) could tell a forgery from a judgement.

So PUT /api/judging/assignments/{id}/scores has no "on behalf of" parameter at all. Refusing a request is a check. Not being able to express it is a property. Checks drift; properties do not.

The unit of isolation is the ballot (a judge_assignments row), not the user or the submission. Three requirements collapse into one ownership check: a judge cannot see a peer's scores, a track judge cannot see another track, and scoring closes outside the judging window. A Score has no rules of its own, so it is readable exactly when its ballot is.

One small decision worth copying: a peer's ballot returns 403, not 404. That a peer holds a ballot is not a secret (the organizer grid shows it), so pretending otherwise protects nothing. Another team's draft returns 404, because there the existence is the secret.

3. Check every feature's read side against its write side

After all four tiers worked, I audited with one question per feature: can a real user produce the state this code reads? Every one of these passed its own tests:

Voting configuration had no write path. voting_method, vote_credits, votes_per_voter and voting_access were read by every voter route. Neither EventCreate nor EventUpdate had a field for them. A real organizer could not turn on quadratic voting; every event that used it was seeded.
Six of ten webhook topics never fired. GET /webhooks/topics advertised all ten. Six were never passed to hooks.schedule() anywhere.
Certificate lookup had no auth check. Any participant could list any project's codes.
CSV import skipped the stored-XSS validation that the JSON API had. Two write paths to one column, one validator.

The pattern is that I had tested what I built, not what a user could reach. Demo data hid all four.

4. The benchmark that lied

My first load pass (200 submissions, 50 judges, ~1,000 ballots) found nothing alarming. I wrote "fine at hackathon scale." Then I seeded 1,200 submissions, 520 judges, 3,600 ballots, 54,600 votes and 110,000 chained audit entries, and checked in both the seeding and measuring scripts. Three real bottlenecks:

The connection pool was the ceiling, not any query. create_engine() had no pool_size, so SQLAlchemy's default of 5 + 10 = 15 connections applied to the whole process, while Starlette runs sync routes on a 40-thread pool. Request 16 was not CPU-starved; it was queued behind 14 holding connections. That is multi-second tail latency with zero errors, invisible below real concurrency, which is why pass one missed it. pool_size=20, max_overflow=20 took the public gallery's p99 from 4.8s to 1.2s.
verify_chain() hydrated 110,000 ORM objects to read twelve plain fields each. A column-only select with yield_per cut verification from about 13s to about 8s. The rest is 110,000 SHA-256 calls, which is what a hash chain costs.
The vote tally built a full ORM set on every poll to produce a filter that narrowed nothing for most events. Skipping it cut concurrent p50 from roughly 11-13s to 4-7s.

Caveats I printed rather than rounded away: one laptop, and identical code varied about 2x between runs, so I report ranges. I did not cache the tally. Caching would be faster but would break the "computed on every request, never stale" property results depend on, which is a product decision, not a perf tweak.

What I cut, and what I would redo

Cut and do not regret:

Expertise matching in assignment. Z-scoring assumes judges see comparable slices of quality, so assignment is balanced, shuffled and conflict-free. Smarter matching would have made assignment better and normalization wrong.
Breaking ties. Equal scores share a rank (1, 1, 3). Deciding first place on a float tie-break is worse than reporting a tie.
A job scheduler. So webhooks are not retried: five failures disable a hook. A queue is a dependency this stack does not need yet.

Would redo:

A judge-to-tracks table. The organizers' file gives nine judges two tracks each; my judge record holds one or all, so they load as all-track judges. No loaded ballot changes, but a fresh assignment would treat them as unrestricted.
Cached, invalidated results. GET /results takes about 1.7s at 3,600 ballots because I compute it on every request.
Secret ballots. Organizers can see who voted for what. Fixing that makes one-vote-per-person unenforceable without blind signatures, so I named it as an open problem instead of faking it.

The repo has a section called "What it does not do yet," and in places it is longer than the feature list. I think that is the most useful part of any hackathon project: not what works, but exactly where it stops, and which of your own claims survived when you went back and tested them.

Thanks to Hackathon Raptors (@partnerships_raptors) for a brief worth building. This is my entry for the Write Up Quest.

I Gave My Secret Scanner a Confidence Score — Then Found a Number I Didn't Want to Publish

Bro Force — Mon, 31 Aug 2026 19:12:40 +0000

TL;DR: I built forge for raptors.dev's Zero Dependency Hackathon
(Track F) — a single-binary secret scanner that ranks its own findings by
confidence instead of just flagging them, on top of a hand-rolled
log-structured store. Nothing outside Python's standard library. This is
the story of a 327-false-positive run I almost buried, a build hash that
changed on a machine I hadn't touched, and a benchmark number I genuinely
did not want to put in the README — and put in anyway.

For raptors.dev's Zero Dependency Hackathon (Track F, Open/Wildcard), I
built Forge: one forge binary that does secret scanning, file
search, dedup, repo stats, a terminal dashboard, run-to-run diffing, a
risk timeline, and per-finding confidence explanations. Underneath all
of it sits a hand-rolled log-structured key-value store. Nothing outside
Python's standard library.

5,502 lines in src/forge/. 313 tests. 0 runtime dependencies. ~77 KB
single-file build.

I'm not going to walk the whole feature list — the README does that.
This is the honest version: what I actually rebuilt, a real bug that
testing against live repos actually caught, where the standard library
fought back, which real package I think I made pointless, and the one
build bug that cost me an entire afternoon and taught me more than
anything else in the project.

The number I didn't want to publish

Every one of the numbers above makes Forge look good, so here's the one
that doesn't. I benchmarked forge scan against detect-secrets on two
targets — a small tree (42 files, 291 KB) and a large one (480 files,
5.1 MB) — 7 timed runs each, cold scans, no result cache, matching
detect-secrets' no-cache behavior:

	Forge	`detect-secrets` 1.5.0
Startup baseline (`--version`)	0.52s	1.14s
Small tree (42 files, 291 KB)	0.79s	1.61s
Large tree (480 files, 5.1 MB)	4.66s	1.81s
Scan-only throughput, large tree	~1.3 MB/s	~7 MB/s

Forge starts twice as fast and wins the small-tree case, because
detect-secrets pays import cost for pyyaml, requests, urllib3,
and its full plugin registry before it scans a single byte. On the large
tree, that startup advantage stops mattering and Forge's scan loop
runs 5–6× slower per byte. Twelve regexes plus entropy tokenization on
every line, no real thread parallelism because of the GIL, versus a
library that's had years to get its throughput right. I checked whether
the confidence scorer was the cause before writing this — it's under 4%
of scan time, so ranking isn't the tax. Raw regex throughput is.

I don't have a fix for this that doesn't compromise something else I
actually care about (zero deps, zero install, ranked output), so instead
of hiding it in a footnote I put it in the README next to the win. The
trade is real: Forge gives up large-tree throughput for zero
installation and confidence-ranked findings; detect-secrets gives up
both of those for raw speed. Different tools optimizing for different
things, and a judge should get to see the actual trade instead of a
cherry-picked number.

Architecture at a glance

One CLI entrypoint, ten subcommands, everything fanning down into a
single storage layer. This is the actual internal import graph, not an
idealized version of it — grouped into three layers so the shape reads
at a glance:

graph TD
    subgraph L1["CLI"]
        CLI["cli.py — 10 subcommands"]
    end

    subgraph L2["Feature modules"]
        Scanner["scanner.py<br/>12 regexes + entropy"]
        Dedup["dedup.py"]
        Stats["stats.py<br/>hand-parsed .git objects"]
        Dashboard["dashboard.py<br/>curses / ANSI fallback"]
        Watcher["watcher.py<br/>polling watch"]
        Baseline["baseline.py<br/>new-vs-existing gate"]
        Sarif["sarif.py<br/>SARIF 2.1.0"]
        History["history.py<br/>diff + risk timeline"]
    end

    subgraph L2b["Scoring"]
        Confidence["confidence.py<br/>naive-Bayes log-odds scorer"]
        Entropy["entropy.py"]
    end

    subgraph L3["Persistence"]
        Cache["cache.py"]
        Storage[("storage.py — LogStore<br/>CRC32 · crash recovery")]
    end

    CLI --> Scanner & Dedup & Stats & Dashboard & Watcher & Baseline & Sarif & History

    Scanner --> Cache
    Scanner --> Confidence
    Dashboard -.->|reuses| Scanner
    Watcher -.->|reuses| Scanner
    Confidence --> Entropy

    Cache --> Storage
    History --> Storage

    style Storage fill:#2d2d2d,stroke:#f5a623,stroke-width:2px,color:#fff
    style Confidence fill:#2d2d2d,stroke:#f5a623,stroke-width:2px,color:#fff

Solid arrows are "needs this to function"; dashed arrows labeled reuses
are "calls this subcommand's logic directly" (dashboard and watcher
both call into the scanner rather than re-implementing scan logic). Two
things worth noticing about the shape: everything that needs to
persist between runs — the cache and the risk/diff history — funnels
through one storage engine, so scan, dedup, stats, and diff all
get crash-safe persistence for free instead of each subcommand rolling
its own file format. And confidence scoring gets its own layer, not
tucked inside the scanner as an afterthought — ranking isn't
post-processing, it's a first-class step between "regex matched" and
"finding reported," which is why forge explain can reconcile its
per-term breakdown back to the exact score forge scan printed.

What I reimplemented

The obvious piece is the scanner itself: twelve regexes for AWS keys,
GitHub tokens, Stripe secret keys, GitLab PATs, JWTs, PEM blocks, plus a
Shannon-entropy fallback for anything that doesn't match a known shape.
That part is table stakes for this hackathon category — most stdlib
secret scanners stop here.

The part I actually care about is confidence.py: every finding gets
scored 0.00–1.00 by a hand-tuned naive-Bayes classifier evaluated in
log-odds space — the exact math behind a spam filter, done by hand
instead of scikit-learn.fit():

log_odds  = logit(prior_for_this_rule)
          + entropy_term
          + character_diversity_term
          + ordered_sequence_term
          + match_length_term
          + comment_marker_term
          + file_type_term
confidence = sigmoid(log_odds)

Seven signal families feed it — pattern specificity, Shannon entropy,
character-class diversity, monotonic-run detection (catches a
hand-written ABCDEF... charset that would otherwise score as
high-entropy and high-diversity), match length against known
fixed-width formats, inline # nosec/# noqa markers, and file-path
context. A hard override kills obvious placeholders like changeme or
<token> outright. Every weight is documented inline where it's added,
specifically so a judge — or me, six months from now — can read the
score instead of trusting it.

Underneath that, storage.py is an append-only, CRC32-checksummed,
log-structured key-value store with crash recovery, backing the shared
cache/history layer for every subcommand that needs to remember a
previous run.

Testing it on real repos found a real bug, not a hypothetical one

Talk is cheap for a secret scanner, so I shallow-cloned three public
repos I didn't pick for a favorable result — pallets/click,
psf/requests, expressjs/express, 596 files, ~10.4 MB — and read every
single finding by hand. None of the three has a real committed secret,
so the right answer for all three is "nothing," and the interesting
question is how much noise Forge generates getting there.

The first run of that validation found 327 findings in requests
alone. Not 2 — 327. 325 of them were base64 chunks inside one file:
ext/requests-logo.ai, a PDF-wrapped Adobe Illustrator asset. My binary
check only sniffed the first 4 KB of a file for a NUL byte, and .ai
files carry a long plain-text header (PDF structure plus XMP metadata)
that outruns 4 KB before the binary payload starts — so Forge read the
file as text and dutifully entropy-flagged every base64 run in it.

Fix: run the binary check over the full file content Forge already has
in memory, not just a 4 KB prefix, so a NUL byte anywhere marks the file
binary. After that, requests-logo.ai contributes zero findings, and
the three-repo total drops to 14 — down to genuine low/medium noise (a
hash in a doc, an ABCD…wxyz alphabet in a docstring, a dozen
ETag/hash literals in express's own test fixtures). Every one of those
12 express findings landed in confidence's medium band (0.41–0.49);
zero reached high-confidence — which is the actual claim --allowlist
and the pre-commit hook lean on: the ranking keeps ordinary noise out of
the tier that would actually block a commit.

repo	files	findings (after fix)
`click`	195	0
`requests`	159	2
`express`	242	12

I'm including the 327-finding number, not just the fixed 14, because a
write-up that only shows the clean result after the fact isn't proof of
anything — the bug is the evidence the validation actually happened.

Where the standard library actually made me suffer

Two places, honestly.

curses doesn't exist on Windows. Not "harder to use" — it's simply
not part of the CPython standard distribution there. forge dashboard
needed a live-updating terminal view, and the natural stdlib answer
(curses) evaporates the moment a judge runs this on Windows, which —
given this hackathon's toolchain — a lot of them will. The fallback is a
plain-text view that redraws on an interval using raw ANSI escape codes.
It works, but it's not the same widget, and I say so directly in the
README instead of pretending the two code paths are equivalent.

Determinism across platforms is not free. os.walk order is sorted
on NTFS and effectively hash-order on ext4. forge stats --json returned
its files_by_ext/bytes_by_ext/lines_by_ext maps straight from that
walk, so the same repo produced byte-different JSON depending on the
filesystem underneath it — invisible in the human-readable table (which
was already sorted by count for display), silently wrong in the machine
contract judges would actually diff. Fix was mechanical once I found it
— sort the maps at construction — but "the stdlib gives you an iteration
order, not a guarantee" is a lesson that doesn't show up until you
actually run the same code on two OSes.

The package I think I made unnecessary

detect-secrets (Yelp) — 1–2M downloads/month by PyPI's own tracker,
and the closest real category match to what forge scan does: regex
plus entropy heuristics, a baseline file, a CI/pre-commit gate. I
installed detect-secrets==1.5.0 into a throwaway venv and measured
instead of guessing:

	Forge	`detect-secrets` 1.5.0
Runtime deps	0	2 direct, 6 total installed
Installed footprint	0 bytes (whole 10-subcommand CLI is ~77 KB)	~4.3 MB
Ranks findings by confidence	Yes	No — flags only

That last row is the actual pitch, not the dependency count.
detect-secrets treats every regex hit as equally worth a human's time.
Forge ranks them, so a bare AWS key ID sitting in config/production.env
sorts to the top and a high-entropy string on a line ending in # noqa
inside tests/fixtures/ sorts to the bottom — the same signal every
commercial scanner (GitGuardian, GitHub secret scanning) sells as a
headline feature, done here in ~170 lines of math and re.

I'll say the honest gap too: detect-secrets has a real plugin
architecture and an interactive audit workflow for annotating false
positives over time that persists across a team. Forge's flat
--allowlist file is the lighter-weight version of that, not a clone.

The edge case that ate an afternoon

Reproducible builds and core.autocrlf do not get along.

Forge claims a byte-reproducible build — rebuild dist/forge.pyz from
source and it should hash identically every time, so a judge can verify
the exact artifact they're running matches the repo. On my machine, it
did. Then I checked out the repo fresh on a different Windows setup with
core.autocrlf=true (the Windows default) and the hash changed. Same
source, same commit, different SHA-256.

Two separate bugs stacked on top of each other:

scripts/build.py was reading source bytes verbatim into the zipapp. A Windows autocrlf clone silently converts every \n in the checkout to \r\n on the way out of git — so the "same" source file was actually different bytes depending on which machine cloned it, before my build script ever touched it.
scripts/bundle_single_file.py was writing its output with write_text(), which on Windows happily re-introduces platform line endings on the way out, even if the input was clean.

Neither bug is visible from reading either script in isolation — both
looked correct. It only showed up as "the hash doesn't match" after a
clean clone on a machine I hadn't built on before, which is exactly the
scenario a judge running the reproducibility check would hit.

The fix, once diagnosed: normalize all source to \n before archiving,
switch the bundler to write_bytes() with an LF-pinned bootstrap, and
add a .gitattributes pinning * text=auto eol=lf (with *.pyz
declared binary so git doesn't touch the archive itself). I also found
zipfile stamps each entry's create_system byte from the build OS
(0 on Windows, 3 elsewhere) — so even with identical zlib output, a
Windows-built and Linux-built .pyz still differed on that one metadata
byte. Pinned that too.

tests/test_build.py now has explicit regression tests —
test_pyz_hash_is_independent_of_source_line_endings,
test_single_file_hash_is_independent_of_source_line_endings,
test_committed_source_and_artifacts_are_lf_only — so this doesn't come
back quietly. And judge_mode.py re-derives both artifact hashes as one
of its 13 checks, so "reproducible" is something a judge runs in seconds
instead of taking on faith.

The lesson wasn't really about git or zipfile. It was that "reproducible
build" is a claim about bytes, and bytes have opinions about line
endings and OS metadata that never show up until someone else's
environment disagrees with yours.

What I'd do differently

Knowing what I know now, I'd build this exact reproducible-build check
first, not near the end. judge_mode.py's hash verification came
after the CRLF bug above had already shipped once — which meant the bug
reached a committed artifact before anything caught it. If the
byte-comparison check had existed from day one, that whole category of
bug gets caught in CI on the first Windows run, not discovered by
accident on a second machine days later. The fix itself — normalize line
endings before archiving — took twenty minutes once diagnosed.
Diagnosing it took the afternoon. Building the verification first would
have collapsed those into the same twenty minutes.

Where it landed

313 tests pass. dist/forge.pyz and dist/forge_single.py are
committed, byte-reproducible, and independently rebuildable from
src/forge/. deps-proof.txt statically checks every import against
the standard library, then re-checks it under an isolated interpreter
with no site-packages on sys.path at all. STDLIB.md documents 21
substitutions, each backed by an actual import in the codebase, not a
hypothetical.

python scripts/judge_mode.py

runs all of the above — plus the confidence-score reconciliation, the
risk-timeline formula, the SARIF 2.1.0 output, and a deliberate
storage-corruption test — as one command, in under 15 seconds, and exits
non-zero if any claim in this post doesn't hold up.

Forge is my submission to Hackathon Raptors'
Zero Dependency Hackathon — Track F (Open/Wildcard).

Closing thought

Anyone with an AI coding agent can produce a scanner that finds AWS keys
with a regex — that part isn't the hard 20%. The hard part was building
enough ways to catch myself being wrong: a judge_mode.py that
re-derives every claim in this post instead of asking a judge to trust
it, a real-world validation that kept the 327-finding run instead of
only publishing the fixed one, and a benchmark table honest enough to
put a 5–6× loss next to a 2× win. The scanner is the artifact. The
willingness to publish the run that didn't flatter it is the actual
submission.

Project

Built for Zero Dependency 2026, run by Hackathon Raptors
(@partnerships_raptors).

GitHub Repository — broforce6909-cmd/forge

ZeroDependencyHack #Python #OpenSource #DeveloperTools #CLI

PythonStandardLibrary #BuildInPublic