I built a secret detector. GitHub blocked my push for containing secrets.

Every rule in a secret scanner needs a fixture — a value that proves the pattern fires. Mine looked like this:

{ id: "stripe-secret", label: "Stripe secret key", severity: "critical",
  re: /\b(?:sk|rk)_live_[0-9a-zA-Z]{24,}\b/g,
  example: "sk_live_0123456789abcdefghijklmn" }

Fabricated, obviously. Sequential filler after the prefix. Not a key, never was.

GitHub push protection disagreed:

remote: — Stripe API Key —————————————————————————
remote:  locations:
remote:    - commit: 02345e9…
remote:      path: test/scan.test.js:37
remote: ! [remote rejected] main -> main (push declined due to repository rule violations)

Which is, when you think about it, the system working exactly as intended. My fixtures are realistic enough to trip a serious scanner. That is the point of them.

There's an "allow this secret" link in the rejection. I didn't click it. Here's what I did instead, and the two things I got wrong on the way.

Mistake 1: assuming push protection is the whole check

I fixed the four types it flagged — Stripe live, Stripe test, Slack, Mailgun — by assembling the values from fragments so no whole token-shaped literal exists in the source:

const ex = (...parts) => parts.join("");

example: ex("sk_", "live_0123456789abcdefghijklmn")

The runtime value is byte-identical. The literal in the file is not.

Push succeeded. Ten minutes later, an email: Google API Key and Telegram Bot Token detected.

Push protection and secret scanning are different systems with different coverage. The first blocks a subset of high-confidence provider types at push time. The second scans the repository afterwards and covers more. Clearing one tells you nothing about the other, and I had quietly assumed it did.

So I stopped playing whack-a-mole and applied the fragment treatment to every realistic fixture in the corpus at once.

Mistake 2: fixing it by hand and calling it done

A convention that lives only in someone's memory is a convention that dies at the next commit. So the build now fails on any credential-shaped literal in the source:

const CREDENTIAL_SHAPED = new RegExp([
  "ghp_[A-Za-z0-9]{36}", "AKIA[0-9A-Z]{16}", "AIza[0-9A-Za-z_-]{35}",
  "sk_live_[0-9a-zA-Z]{24}", "xox[baprs]-[0-9]{12}", "hf_[A-Za-z0-9]{34}",
  "\\d{9}:[A-Za-z0-9_-]{35}", "eyJ[A-Za-z0-9_-]{10,}\\.eyJ"
].join("|"));

I verified the guard by injecting a violation. The build failed with the exact file and line. A check you haven't seen fail is a check you don't have.

The actual hard part isn't detection

While I'm here — the reason I was building this at all is that most paste guards are unusable, and it isn't because they miss things. It's because they cry wolf.

Match [A-Za-z0-9]{32} and call it a secret, and you flag UUIDs, git SHAs, minified bundles and build numbers. After the third false alarm the user stops reading the warning, which is strictly worse than having no warning.

Three things fixed most of it:

Prefixes beat entropy. A rule keyed on a documented token format is near-zero false positive. Entropy alone is a coin flip on base64. I kept entropy, but only as a secondary gate on assignment-shaped lines.

Validate where an algorithm exists. Card numbers get Luhn. Brazilian CPF and CNPJ have check digits — verify them. Roughly 99% of random 11-digit strings fail CPF's check digits, so a naive \d{11} flags every order number in your logs and a validated one flags none.

Recognise placeholders by vocabulary, not by shape. This one took two attempts. My first fix rejected anything shaped like word-word-word, which killed your-api-key-here nicely and also killed correct-horse-battery-staple — a real password. Vocabulary works: random credentials do not contain the word "your".

And a small trap inside that: \b treats underscore as a word character, so \bchange_?me\b never matches CHANGE_ME_PLEASE. Normalising separators to spaces before matching fixed it — and then broke the multi-word patterns, which were still expecting [-_]. Two bugs, one line apart.

The suite that matters

The false-positive suite in this project is larger than the detection suite, deliberately. Detection tests prove the tool does something. False-positive tests prove it is worth leaving switched on.

FALSE POSITIVES — the cases that make people uninstall
  PASS  UUID v4
  PASS  git commit SHA
  PASS  semver + build number
  PASS  invalid card (Luhn)
  PASS  random 11 digits
  PASS  invalid CPF
  PASS  placeholder secret

PRECISION — placeholders die, real passphrases survive
  PASS  'your-api-key-here'
  PASS  'CHANGE_ME_PLEASE'
  PASS  real passphrase kept

---

The thing I was building is LeakGuard — it checks text for credentials and personal data before you paste it into an AI chat. 50 rules across 36 providers. The web version runs entirely in your browser: no upload, no account, no network requests at all, which you can confirm from an empty network tab. Source, MIT.

Comments

[Alex Shev]

That is the perfect failure mode for a secret detector project. The detector itself becomes part of the threat model, including fixtures, examples, screenshots, docs, and test data. Fake secrets need to be fake enough for scanners too.

[Henrique Yuri]

"Fake enough for scanners too" is a better way to put it than anything in the post, and you named a gap I had left open — so I went and closed it before replying.

My build check only covered code. Docs were exempt, which is exactly backwards: a README is where a credential-shaped string gets pasted without thinking, because it's "only documentation". It now covers README.md and the store listing, and I verified that by injecting a token into the README and watching the build fail on the right line. Docs were clean, as it happens — but they were clean by luck, not by construction, and those are different things.

Screenshots are the one you can't regex, and I don't have a clean answer. The best I have is designing the UI so the interesting part of any capture is already masked — findings show sk_l…klmn (32 chars) rather than the value — which means a screenshot of the results is safe even when a screenshot of the input box would not be.

The wider version of your point is the uncomfortable one: for this class of project the fixtures are load-bearing. They have to be realistic enough to prove the rule fires, which is the same property that makes them look like a leak. Assembling them from fragments at runtime is the only resolution I found that keeps both — the matcher sees the real shape, the file never contains it.

[Alex Shev]

That is a good catch. Docs are often the place where people paste the most dangerous examples because they feel outside the runtime. Extending the check to README and store text makes the detector part of the release surface, not just the code path.