I Gave My Secret Scanner a Confidence Score — Then Found a Number I Didn't Want to Publish

Bro Force — Mon, 31 Aug 2026 19:12:40 +0000

TL;DR: I built forge for raptors.dev's Zero Dependency Hackathon
(Track F) — a single-binary secret scanner that ranks its own findings by
confidence instead of just flagging them, on top of a hand-rolled
log-structured store. Nothing outside Python's standard library. This is
the story of a 327-false-positive run I almost buried, a build hash that
changed on a machine I hadn't touched, and a benchmark number I genuinely
did not want to put in the README — and put in anyway.

For raptors.dev's Zero Dependency Hackathon (Track F, Open/Wildcard), I
built Forge: one forge binary that does secret scanning, file
search, dedup, repo stats, a terminal dashboard, run-to-run diffing, a
risk timeline, and per-finding confidence explanations. Underneath all
of it sits a hand-rolled log-structured key-value store. Nothing outside
Python's standard library.

5,502 lines in src/forge/. 313 tests. 0 runtime dependencies. ~77 KB
single-file build.

I'm not going to walk the whole feature list — the README does that.
This is the honest version: what I actually rebuilt, a real bug that
testing against live repos actually caught, where the standard library
fought back, which real package I think I made pointless, and the one
build bug that cost me an entire afternoon and taught me more than
anything else in the project.

The number I didn't want to publish

Every one of the numbers above makes Forge look good, so here's the one
that doesn't. I benchmarked forge scan against detect-secrets on two
targets — a small tree (42 files, 291 KB) and a large one (480 files,
5.1 MB) — 7 timed runs each, cold scans, no result cache, matching
detect-secrets' no-cache behavior:

	Forge	`detect-secrets` 1.5.0
Startup baseline (`--version`)	0.52s	1.14s
Small tree (42 files, 291 KB)	0.79s	1.61s
Large tree (480 files, 5.1 MB)	4.66s	1.81s
Scan-only throughput, large tree	~1.3 MB/s	~7 MB/s

Forge starts twice as fast and wins the small-tree case, because
detect-secrets pays import cost for pyyaml, requests, urllib3,
and its full plugin registry before it scans a single byte. On the large
tree, that startup advantage stops mattering and Forge's scan loop
runs 5–6× slower per byte. Twelve regexes plus entropy tokenization on
every line, no real thread parallelism because of the GIL, versus a
library that's had years to get its throughput right. I checked whether
the confidence scorer was the cause before writing this — it's under 4%
of scan time, so ranking isn't the tax. Raw regex throughput is.

I don't have a fix for this that doesn't compromise something else I
actually care about (zero deps, zero install, ranked output), so instead
of hiding it in a footnote I put it in the README next to the win. The
trade is real: Forge gives up large-tree throughput for zero
installation and confidence-ranked findings; detect-secrets gives up
both of those for raw speed. Different tools optimizing for different
things, and a judge should get to see the actual trade instead of a
cherry-picked number.

Architecture at a glance

One CLI entrypoint, ten subcommands, everything fanning down into a
single storage layer. This is the actual internal import graph, not an
idealized version of it — grouped into three layers so the shape reads
at a glance:

graph TD
    subgraph L1["CLI"]
        CLI["cli.py — 10 subcommands"]
    end

    subgraph L2["Feature modules"]
        Scanner["scanner.py<br/>12 regexes + entropy"]
        Dedup["dedup.py"]
        Stats["stats.py<br/>hand-parsed .git objects"]
        Dashboard["dashboard.py<br/>curses / ANSI fallback"]
        Watcher["watcher.py<br/>polling watch"]
        Baseline["baseline.py<br/>new-vs-existing gate"]
        Sarif["sarif.py<br/>SARIF 2.1.0"]
        History["history.py<br/>diff + risk timeline"]
    end

    subgraph L2b["Scoring"]
        Confidence["confidence.py<br/>naive-Bayes log-odds scorer"]
        Entropy["entropy.py"]
    end

    subgraph L3["Persistence"]
        Cache["cache.py"]
        Storage[("storage.py — LogStore<br/>CRC32 · crash recovery")]
    end

    CLI --> Scanner & Dedup & Stats & Dashboard & Watcher & Baseline & Sarif & History

    Scanner --> Cache
    Scanner --> Confidence
    Dashboard -.->|reuses| Scanner
    Watcher -.->|reuses| Scanner
    Confidence --> Entropy

    Cache --> Storage
    History --> Storage

    style Storage fill:#2d2d2d,stroke:#f5a623,stroke-width:2px,color:#fff
    style Confidence fill:#2d2d2d,stroke:#f5a623,stroke-width:2px,color:#fff

Solid arrows are "needs this to function"; dashed arrows labeled reuses
are "calls this subcommand's logic directly" (dashboard and watcher
both call into the scanner rather than re-implementing scan logic). Two
things worth noticing about the shape: everything that needs to
persist between runs — the cache and the risk/diff history — funnels
through one storage engine, so scan, dedup, stats, and diff all
get crash-safe persistence for free instead of each subcommand rolling
its own file format. And confidence scoring gets its own layer, not
tucked inside the scanner as an afterthought — ranking isn't
post-processing, it's a first-class step between "regex matched" and
"finding reported," which is why forge explain can reconcile its
per-term breakdown back to the exact score forge scan printed.

What I reimplemented

The obvious piece is the scanner itself: twelve regexes for AWS keys,
GitHub tokens, Stripe secret keys, GitLab PATs, JWTs, PEM blocks, plus a
Shannon-entropy fallback for anything that doesn't match a known shape.
That part is table stakes for this hackathon category — most stdlib
secret scanners stop here.

The part I actually care about is confidence.py: every finding gets
scored 0.00–1.00 by a hand-tuned naive-Bayes classifier evaluated in
log-odds space — the exact math behind a spam filter, done by hand
instead of scikit-learn.fit():

log_odds  = logit(prior_for_this_rule)
          + entropy_term
          + character_diversity_term
          + ordered_sequence_term
          + match_length_term
          + comment_marker_term
          + file_type_term
confidence = sigmoid(log_odds)

Seven signal families feed it — pattern specificity, Shannon entropy,
character-class diversity, monotonic-run detection (catches a
hand-written ABCDEF... charset that would otherwise score as
high-entropy and high-diversity), match length against known
fixed-width formats, inline # nosec/# noqa markers, and file-path
context. A hard override kills obvious placeholders like changeme or
<token> outright. Every weight is documented inline where it's added,
specifically so a judge — or me, six months from now — can read the
score instead of trusting it.

Underneath that, storage.py is an append-only, CRC32-checksummed,
log-structured key-value store with crash recovery, backing the shared
cache/history layer for every subcommand that needs to remember a
previous run.

Testing it on real repos found a real bug, not a hypothetical one

Talk is cheap for a secret scanner, so I shallow-cloned three public
repos I didn't pick for a favorable result — pallets/click,
psf/requests, expressjs/express, 596 files, ~10.4 MB — and read every
single finding by hand. None of the three has a real committed secret,
so the right answer for all three is "nothing," and the interesting
question is how much noise Forge generates getting there.

The first run of that validation found 327 findings in requests
alone. Not 2 — 327. 325 of them were base64 chunks inside one file:
ext/requests-logo.ai, a PDF-wrapped Adobe Illustrator asset. My binary
check only sniffed the first 4 KB of a file for a NUL byte, and .ai
files carry a long plain-text header (PDF structure plus XMP metadata)
that outruns 4 KB before the binary payload starts — so Forge read the
file as text and dutifully entropy-flagged every base64 run in it.

Fix: run the binary check over the full file content Forge already has
in memory, not just a 4 KB prefix, so a NUL byte anywhere marks the file
binary. After that, requests-logo.ai contributes zero findings, and
the three-repo total drops to 14 — down to genuine low/medium noise (a
hash in a doc, an ABCD…wxyz alphabet in a docstring, a dozen
ETag/hash literals in express's own test fixtures). Every one of those
12 express findings landed in confidence's medium band (0.41–0.49);
zero reached high-confidence — which is the actual claim --allowlist
and the pre-commit hook lean on: the ranking keeps ordinary noise out of
the tier that would actually block a commit.

repo	files	findings (after fix)
`click`	195	0
`requests`	159	2
`express`	242	12

I'm including the 327-finding number, not just the fixed 14, because a
write-up that only shows the clean result after the fact isn't proof of
anything — the bug is the evidence the validation actually happened.

Where the standard library actually made me suffer

Two places, honestly.

curses doesn't exist on Windows. Not "harder to use" — it's simply
not part of the CPython standard distribution there. forge dashboard
needed a live-updating terminal view, and the natural stdlib answer
(curses) evaporates the moment a judge runs this on Windows, which —
given this hackathon's toolchain — a lot of them will. The fallback is a
plain-text view that redraws on an interval using raw ANSI escape codes.
It works, but it's not the same widget, and I say so directly in the
README instead of pretending the two code paths are equivalent.

Determinism across platforms is not free. os.walk order is sorted
on NTFS and effectively hash-order on ext4. forge stats --json returned
its files_by_ext/bytes_by_ext/lines_by_ext maps straight from that
walk, so the same repo produced byte-different JSON depending on the
filesystem underneath it — invisible in the human-readable table (which
was already sorted by count for display), silently wrong in the machine
contract judges would actually diff. Fix was mechanical once I found it
— sort the maps at construction — but "the stdlib gives you an iteration
order, not a guarantee" is a lesson that doesn't show up until you
actually run the same code on two OSes.

The package I think I made unnecessary

detect-secrets (Yelp) — 1–2M downloads/month by PyPI's own tracker,
and the closest real category match to what forge scan does: regex
plus entropy heuristics, a baseline file, a CI/pre-commit gate. I
installed detect-secrets==1.5.0 into a throwaway venv and measured
instead of guessing:

	Forge	`detect-secrets` 1.5.0
Runtime deps	0	2 direct, 6 total installed
Installed footprint	0 bytes (whole 10-subcommand CLI is ~77 KB)	~4.3 MB
Ranks findings by confidence	Yes	No — flags only

That last row is the actual pitch, not the dependency count.
detect-secrets treats every regex hit as equally worth a human's time.
Forge ranks them, so a bare AWS key ID sitting in config/production.env
sorts to the top and a high-entropy string on a line ending in # noqa
inside tests/fixtures/ sorts to the bottom — the same signal every
commercial scanner (GitGuardian, GitHub secret scanning) sells as a
headline feature, done here in ~170 lines of math and re.

I'll say the honest gap too: detect-secrets has a real plugin
architecture and an interactive audit workflow for annotating false
positives over time that persists across a team. Forge's flat
--allowlist file is the lighter-weight version of that, not a clone.

The edge case that ate an afternoon

Reproducible builds and core.autocrlf do not get along.

Forge claims a byte-reproducible build — rebuild dist/forge.pyz from
source and it should hash identically every time, so a judge can verify
the exact artifact they're running matches the repo. On my machine, it
did. Then I checked out the repo fresh on a different Windows setup with
core.autocrlf=true (the Windows default) and the hash changed. Same
source, same commit, different SHA-256.

Two separate bugs stacked on top of each other:

scripts/build.py was reading source bytes verbatim into the zipapp. A Windows autocrlf clone silently converts every \n in the checkout to \r\n on the way out of git — so the "same" source file was actually different bytes depending on which machine cloned it, before my build script ever touched it.
scripts/bundle_single_file.py was writing its output with write_text(), which on Windows happily re-introduces platform line endings on the way out, even if the input was clean.

Neither bug is visible from reading either script in isolation — both
looked correct. It only showed up as "the hash doesn't match" after a
clean clone on a machine I hadn't built on before, which is exactly the
scenario a judge running the reproducibility check would hit.

The fix, once diagnosed: normalize all source to \n before archiving,
switch the bundler to write_bytes() with an LF-pinned bootstrap, and
add a .gitattributes pinning * text=auto eol=lf (with *.pyz
declared binary so git doesn't touch the archive itself). I also found
zipfile stamps each entry's create_system byte from the build OS
(0 on Windows, 3 elsewhere) — so even with identical zlib output, a
Windows-built and Linux-built .pyz still differed on that one metadata
byte. Pinned that too.

tests/test_build.py now has explicit regression tests —
test_pyz_hash_is_independent_of_source_line_endings,
test_single_file_hash_is_independent_of_source_line_endings,
test_committed_source_and_artifacts_are_lf_only — so this doesn't come
back quietly. And judge_mode.py re-derives both artifact hashes as one
of its 13 checks, so "reproducible" is something a judge runs in seconds
instead of taking on faith.

The lesson wasn't really about git or zipfile. It was that "reproducible
build" is a claim about bytes, and bytes have opinions about line
endings and OS metadata that never show up until someone else's
environment disagrees with yours.

What I'd do differently

Knowing what I know now, I'd build this exact reproducible-build check
first, not near the end. judge_mode.py's hash verification came
after the CRLF bug above had already shipped once — which meant the bug
reached a committed artifact before anything caught it. If the
byte-comparison check had existed from day one, that whole category of
bug gets caught in CI on the first Windows run, not discovered by
accident on a second machine days later. The fix itself — normalize line
endings before archiving — took twenty minutes once diagnosed.
Diagnosing it took the afternoon. Building the verification first would
have collapsed those into the same twenty minutes.

Where it landed

313 tests pass. dist/forge.pyz and dist/forge_single.py are
committed, byte-reproducible, and independently rebuildable from
src/forge/. deps-proof.txt statically checks every import against
the standard library, then re-checks it under an isolated interpreter
with no site-packages on sys.path at all. STDLIB.md documents 21
substitutions, each backed by an actual import in the codebase, not a
hypothetical.

python scripts/judge_mode.py

runs all of the above — plus the confidence-score reconciliation, the
risk-timeline formula, the SARIF 2.1.0 output, and a deliberate
storage-corruption test — as one command, in under 15 seconds, and exits
non-zero if any claim in this post doesn't hold up.

Forge is my submission to Hackathon Raptors'
Zero Dependency Hackathon — Track F (Open/Wildcard).

Closing thought

Anyone with an AI coding agent can produce a scanner that finds AWS keys
with a regex — that part isn't the hard 20%. The hard part was building
enough ways to catch myself being wrong: a judge_mode.py that
re-derives every claim in this post instead of asking a judge to trust
it, a real-world validation that kept the 327-finding run instead of
only publishing the fixed one, and a benchmark table honest enough to
put a 5–6× loss next to a 2× win. The scanner is the artifact. The
willingness to publish the run that didn't flatter it is the actual
submission.

Project

Built for Zero Dependency 2026, run by Hackathon Raptors
(@partnerships_raptors).

GitHub Repository — broforce6909-cmd/forge

ZeroDependencyHack #Python #OpenSource #DeveloperTools #CLI

PythonStandardLibrary #BuildInPublic

DEV Community: Bro Force