DEV Community: Navesz

My code checker was wrong. How I turned its false positives into tests

Navesz — Wed, 23 Sep 2026 00:37:58 +0000

Disclosure: I'm the author of rebar, the open-source checker this post is about. It is alpha.

rebar fails a CI job when a repository breaks a rule. Examples: formatting nobody checks, a hook git will never run, a credential in a tracked file. A checker like that can fail in one way that matters more than any other: accusing a repository that is doing the right thing. After a couple of those, people stop reading the output. Soon after, they turn it off.

So I ran it against 28 public repositories and read what it said. Three of the accusations were wrong, and I could prove it. Each case below follows the same order: the accusation, the evidence it was wrong, the fix, and the test that keeps it fixed. The test has to work both ways. It must fail on the old code, and the real violation must still be caught.

1. "Git ignores this hook." Not with husky 9.

rebar told mastra (commit 7e21ff5):

hooks-executable ✗ no execute bit in the index, git ignores them on Linux: .husky/pre-commit
                   — fix with: git update-index --chmod=+x .husky/pre-commit

The file really is mode 100644, and on Linux git skips a hook that isn't executable. The accusation sounds right, and husky 8 users run into exactly that.

mastra uses husky 9, which works differently. husky 9 makes three changes:

it points core.hooksPath at .husky/_;
it writes its own .husky/_/pre-commit with mode 0o755;
that generated file calls a small script that runs yours with sh -e .husky/pre-commit.

Git never executes the committed file, so its mode doesn't matter.

I checked this by running git, not just by reading the husky source. I made a repository with .husky/pre-commit at 100644, cloned it, ran npm install, and ran git commit with GIT_TRACE=1, which prints the file git runs:

	husky 9.1.7	husky 8.0.3
file git ran	`.husky/_/pre-commit` (generated)	`.husky/pre-commit` (committed)
rebar before	✗	✗
rebar after	not applicable	✗

I ran it on Windows because Docker wouldn't start on my machine. Windows shows which file git runs. That Linux skips a non-executable hook comes from git's documentation, not from this run.

The fix exempts the hooks husky 9 manages. It doesn't switch the rule off.

The tests:

husky-v9 has two sides with the same bytes and the same 100644 mode. The only difference is the husky version the repository declares. The new code passes one side and fails the other. The old code fails both, which is what makes it a regression test.
husky-own-hook declares husky 9, but also has a hook that git runs directly (tooling/hooks/pre-push) at 100644. Declaring husky must not silence that one, and it doesn't.

2. "Rotate this credential." It's AWS's documentation example.

rebar told dotenv (commit f6390d1):

hardcoded-secret ✗ 10 credential(s) in tracked files … deleting the line is not enough,
                   what is in history has to be ROTATED

Six of the ten were not credentials:

AWS_ACCESS_KEY_ID=AKIAIOSF••••••••••••, twice. That's the example key pair from AWS's own IAM documentation.
A PRIVATE_KEY in the README whose body is Kh9NV..., twice. It was cut short on purpose.
PASSWORD: 'password', in two tests. The value is just the word.

A secret scanner that tells people to rotate AWS's example key loses its audience fast.

The fix releases those three shapes and nothing near them. The tests are built so the fix can't turn into a hole:

vendor-example: the failing side is the same file, and it even keeps AWS's example secret key. Only the access key ID changes, to one that isn't a published example, and it must still be flagged. A check for "contains EXAMPLE" would let it through.
elided-pem: the failing side has an ellipsis too, but between real 64-character key lines. That's what someone does when they paste a real key into docs and cut out the middle.
value-is-the-word: the same test fixture and the same key, with a value that isn't the word.

This fix nearly shipped wrong. Its first version removed 45 findings from mastra and added 6, and the total still went down. A count only shows the net change. So for this post I ran the scanner again with full lists on every repository, at the same commits, and compared finding by finding:

151 findings removed, 0 added;
I read each removed line in the original repository;
107 were the word itself, 28 were AWS's example pair, 9 were shortened keys, and 7 were placeholders like mypassword;
none was a real credential.

dotenv's other four findings are .env files used as test fixtures. rebar still flags them, on purpose, because it flags any committed .env whatever is in it. I haven't settled that design question, and I didn't touch it.

3. "Nothing fails when a file is out of format." The next step does.

rebar told e2b (commit ccaf9fc):

formatter ✗ script format only rewrites — nothing fails when a file is out of format

The format script is prettier --write, which only rewrites. But e2b's lint workflow runs pnpm run format and then fails the job if git status --porcelain isn't empty. Rewriting and then failing on any difference is a formatting check.

The fix recognizes that pattern, but only inside a single job. A git diff --exit-code in another job looks at a different checkout.

The tests:

write-then-verify must pass;
write-then-verify-other-job splits the same two commands across two jobs, and it must fail;
both of the sides that should pass fail on the old code.

I also ran rebar's GitHub Action on a runner with only this rule. prettier --write . exits 1 and prettier --check . exits 0.

Did the fixes generalize?

Seven of the 28 repositories are where I found these problems, and I wrote the fixes looking at them, so of course they pass there. The other 21 came from a search rule I wrote before measuring anything. Those 21 are the real test.

	7 chosen, before → after	21 searched, before → after
`hooks-executable`, repos flagged	1 → 0	4 → 1
`hardcoded-secret`, findings	1319 → 1202	664 → 630
`formatter`, repos flagged	4 → 2	14 → 14

husky: 4 → 1. The one left, agentscope, is still a false positive. Its husky 9 hook lives in examples/web_ui/, a package inside the repository, and rebar only looks for husky at the root. Still open.
Secrets: 34 fewer findings in the searched group, all placeholders, none added. I haven't classified the 1832 findings that remain, so I make no claim about them.
formatter: no. 14 → 14. I read all 16 repositories still flagged across both groups and followed everything their CI calls:
- 12 are correct: nothing fails on bad formatting.
- 4 are still wrong:
- serena runs poe lint, which runs ruff format --check;
- cognee runs ruff format --check;
- page-agent runs a scripts/ci.js that calls prettier --check;
- dotenv runs npm test, which runs standard. That last one is arguable.

Ten verdicts changed in total. Every one went from failing to passing or not applicable, and no repository was newly accused.

The checker's checker was wrong too

My first script for sorting those formatter results said "13 correct, 1 false positive". It looked for the check only on the workflow line itself, and didn't follow poe lint or node scripts/ci.js: the same blind spot as the rule it was auditing. The version before that read files through the API, swallowed an error, and reported "no check" for cognee. Both are fixed, and both now fail loudly. I mention them because they are the same bug as the rule: a checker that is wrong without saying so.

What this does not show

The searched repositories were picked partly by the mcp topic, and rebar has rules aimed at MCP server setups. This audit says nothing about those rules. I didn't classify their findings, and there's no comparison group of projects without MCP. So I can't say whether MCP projects are riskier, or whether those signatures catch a real attack. Answering that would need a study of its own.

Three fixed false positives aren't an accuracy rate either. They are three bugs, and each one now has a test that fails on the old code.

Run it on your project and send me a reproducible false positive

npx github:Navesz/rebar#v0.1.0 .
npx -p github:Navesz/rebar#v0.1.0 rebar-security .

Or in CI, as a GitHub Action, pinned to the release commit. A tag can be moved; a commit can't:

- uses: Navesz/rebar@9d39f1e3ed4cd3190b4635dccf22d7b815a74d5b # v0.1.0
  with:
    ruler: both   # check · security · both

If it flags something that is correct, open a false-positive issue with the rule id, the line it printed, and a public commit or a minimal repository. Every case above started as one wrong line of output, and each one is now a test.

Appendix: method and full tables

Versions. Before: rebar 9f78d9b. After: 58aff2c. The v0.1.0 release (9d39f1e) has the same rules: only the Action, its workflow and the changelog differ. The fixes landed in #41 (husky, formatter, ci-gates, env-example) and #40 (secrets).

Sample. 28 repositories, fixed before any result was seen and not extended afterwards.

7 chosen, because the fixes came from them: mastra, dotenv, stagehand, biome, e2b, serena, skyvern.
21 from a GitHub search:
- topic mcp, ai-agents or developer-tools;
- 300–40,000 stars, pushed after 2026-07-24;
- not archived, not a fork;
- TypeScript, JavaScript or Python;
- not a collection (awesome lists, skills, templates);
- at most 300 MB;
- top by stars.
mastra (1.8 GB) and skyvern (623 MB) break the size cap. I kept them because they're among the chosen seven.

Measurement. Each repository was cloned at depth 50 and measured with both rulers and both versions: 112 runs, 0 errors. The commit measured for each repository is in the raw data. The secrets comparison ran the scanner again with full lists at those same commits, because the ruler's summary cuts each list at about 12 items.

Other verdicts that changed (not among the three cases):

ci-gates on dotenv, because npm test chains npm run lint;
ci-gates on stagehand, because pnpm check runs turbo run fmt:check lint typecheck;
env-example on serena, because the variable it flagged is one the repository writes into a child process's environment and never reads.

Full table. Repositories flagged before → after, with findings in parentheses. Heuristic rules are reported but don't fail the job unless you pass --heuristics. Overall, rebar-check exits 1 on 28 of 28 repositories and rebar-security on 24 of 28, both before and after. Most of that comes from house-convention rules, such as editorconfig (20 of 21 searched) and ai-coauthorship (12 of 21). Those rules encode one project's conventions, and these projects never adopted them. That's not a defect on their side.

Ruler	Rule	Kind	7 chosen: repos (findings)	21 searched: repos (findings)
check	`ai-coauthorship`	deterministic	6 → 6 (48 → 48)	12 → 12 (740 → 740)
check	`ci`	deterministic	0 → 0	1 → 1 (1 → 1)
check	`ci-gates`	deterministic	2 → 0 (2 → 0)	4 → 4 (4 → 4)
check	`dependabot`	deterministic	4 → 4 (4 → 4)	5 → 5 (5 → 5)
check	`editorconfig`	deterministic	4 → 4 (4 → 4)	20 → 20 (20 → 20)
check	`env-example`	deterministic	6 → 5 (6 → 5)	17 → 17 (17 → 17)
check	`fake-ui`	deterministic	1 → 1 (1 → 1)	1 → 1 (1 → 1)
check	`formatter`	deterministic	4 → 2 (4 → 2)	14 → 14 (14 → 14)
check	`git-identity`	deterministic	1 → 1 (1 → 1)	9 → 9 (9 → 9)
check	`hooks-executable`	deterministic	1 → 0 (1 → 0)	4 → 1 (4 → 1)
check	`notice`	deterministic	3 → 3 (3 → 3)	7 → 7 (7 → 7)
check	`orphan-schema`	deterministic	0 → 0	1 → 1 (1 → 1)
check	`phone`	deterministic	0 → 0	2 → 2 (2 → 2)
check	`production-url`	heuristic	4 → 4 (174 → 174)	14 → 14 (664 → 664)
check	`raw-hex`	heuristic	4 → 4 (59 → 59)	8 → 8 (200 → 200)
check	`tests`	deterministic	0 → 0	2 → 2 (2 → 2)
check	`typecheck`	deterministic	3 → 3 (3 → 3)	9 → 9 (9 → 9)
security	`agent-bypass-invocation`	deterministic	0 → 0	2 → 2 (5 → 5)
security	`control-bytes`	deterministic	3 → 3 (84 → 84)	5 → 5 (5737 → 5737)
security	`disabled-defense`	deterministic	1 → 1 (1 → 1)	1 → 1 (2 → 2)
security	`env-committed`	deterministic	2 → 2 (7 → 7)	0 → 0
security	`hardcoded-secret`	deterministic	6 → 6 (1319 → 1202)	16 → 15 (664 → 630)
security	`hidden-markdown-directive`	heuristic	1 → 1 (1 → 1)	0 → 0
security	`hidden-unicode`	deterministic	4 → 4 (114 → 114)	10 → 10 (508 → 508)
security	`indirect-exec-change`	heuristic	2 → 2 (5 → 5)	1 → 1 (1 → 1)
security	`mcp-ansi-escape`	heuristic	1 → 1 (1 → 1)	2 → 2 (91 → 91)
security	`mcp-server-launch`	deterministic	2 → 2 (2 → 2)	5 → 5 (10 → 10)
security	`mixed-script-token`	heuristic	1 → 1 (2 → 2)	0 → 0
security	`password-without-kdf`	deterministic	1 → 1 (10 → 10)	4 → 4 (11 → 11)

Only the rows the three cases touch were classified by reading. Every other row is a count, not a verdict on whether rebar is right.

Raw data: the raw JSON from every run, the commit list, the scripts and the classification files are attached to the v0.1.0 release.

Rules in markdown vs rules in CI: what I measured in my own repositories

Navesz — Sun, 20 Sep 2026 23:32:07 +0000

Disclosure: I'm the author of rebar, the open-source tool this post is about. It is alpha, and I'm its only contributor so far.

I keep a written standard for the sites I build with AI coding agents. The complaint that started this project was mine: a lot of it got ignored. Values were hardcoded, parts of the stack were forgotten, the AI was listed as a co-author, the component library was skipped.

The obvious fix is to write the rules more clearly. Before doing that, I measured.

What I measured

On 2026-08-25 I went through 161 commits across six of my own repositories:

Measurement	Result
Repositories without CI	3 of 6
Repositories with broken lint at that moment	the same 3
Commits with an AI co-author trailer	41 of 161 (25.5%)
Repository with the most governance documents	35 lint errors

Six repositories, all mine, is a small sample. I'm not claiming a general result. The last row is what changed my approach. That repository had an AGENTS.md with a "Hard rules" section, plus SECURITY.md, GOVERNANCE.md, CONTRIBUTING.md, and a check script that chained format, lint, types, tests and build. Nothing ever ran that check.

Every rule was written down. None of them was enforced. The short version, as the README puts it: a rule written in markdown has close to zero compliance, and a rule in CI has 100%. The 100% is not a discovery. A required check that fails blocks the merge by construction. The useful part is the consequence: if a rule can move down a level, from prose into something that blocks, it should.

That needs one condition, and most of what I built came from learning it the hard way: the enforcement has to be more reliable than the rule it replaces.

Five times the enforcement was the problem

1. A score that counted nothing as a pass. The first version of the checker treated "this rule does not apply here" as a pass. An empty folder with an empty .git/ scored 8 of 14 and tied with the tool's own repository. Now every rule ends in one of three states: passed, failed, or not applicable, with the reason printed. Not applicable leaves the denominator.

rebar-check · prumo
  ✓ editorconfig       has .editorconfig
  ✗ dependabot         automated dependency updates  no dependabot, no renovate
  ✓ ci                 has CI
  ✓ ci-gates           CI reaches the verification the repository declares
  – typecheck          has a typecheck script  no TypeScript here
  11 of 13  ·  1 not applicable

(An excerpt of the sample in the README, measured on 2026-08-30 against another project of mine; only five of its rules are shown.)

2. Tests that could not tell passed from not applicable. Every rule has pass and fail cases. Each case builds a small repository in a temp directory with its own git init. At first the cases only read the exit code, but "passed" and "not applicable" both exit 0, so 13 of the 20 rules at the time could not be proved that way. When I applied 70 mutations to the checker, 30 survived with the suite fully green. The cases now declare the expected state and read it from --json.

3. A heuristic with zero true positives. A rule for literal colour values, measured on a real repository, gave 7 hits and zero true positives. Five of the seven were comments documenting the rule itself. So heuristics only warn. A heuristic that blocks teaches people to switch the whole output off.

4. A green gate over a broken product. During a rename, the generator started reading a workflow file from a folder where it had a different name. rebar new crashed with ENOENT, and npm run verify stayed green for six commits, because no step actually generated a project. There is now a step that does, and it is itself checked by mutation: putting the original defect back makes two of its five tests fail.

5. My own README. It got numbers wrong six times ("20 checks" when there were 19, "8 steps" when there were 12). The fix was not more care. Every number that belongs to the tree is now derived from the source and written between invisible markers, and a gate step fails if the document drifts.

The only layer an agent cannot edit

Hooks live in .git/hooks or a core.hooksPath setting. Workflows live in .github/workflows. Both are files, and an agent with write access can delete or bypass them. One attack I reproduced against my own gate: git update-index --skip-worktree on the config file, then rewrite it on disk. git status and git diff both came back empty, and the gate printed APPROVED. The gate now reads git ls-files -v, the one command that shows that flag.

The layer that holds is on the server. rebar's main has a GitHub ruleset with a required check and bypass_actors: [], so not even the owner goes around it. I tested it with a planted pull request that deleted .editorconfig. Both CI jobs failed and the merge was blocked. A direct push to main is refused too.

Agents read what reviewers don't see

The second ruler, rebar-security, looks at versioned files for known prompt-injection signatures. It checks invisible Unicode and terminal control bytes, agent settings that run commands or widen approval, and MCP server launches, each fingerprinted so an allowlisted command cannot quietly gain an environment variable. It also flags agent CLIs started with approval turned off, and AI steps in GitHub workflows that text from an outside account can reach. A finding names a position, never the payload, because the output is read by agents too.

A pass means "no known signature", never "free of injection". The signature list is public, and the attacker moves second.

Telling the agent before it writes

Projects generated by rebar new point their agent at an MCP server that serves rebar's rules, the gate steps, and the measured reason behind each rule. The rules artifact is generated from the rulers' source, and the gate regenerates it in memory and fails if the file on disk differs. Changing a rule and forgetting the MCP is not possible without a red build.

What it does not do

It does not check Broken Access Control, and it does not run your application.
Some rules reflect my own stack: a hardcoded Brazilian phone number check, shadcn's components.json, an allowlist for co-author trailers. On your repository they may be not applicable, or not useful.
The generator has one preset, a static Next.js site.
The MCP server needs a checkout and npm ci --prefix mcp. npx alone does not start it.
It is alpha. As of the project's state file it gates one repository for real: its own.

Try it, and tell me where it is wrong

It needs Node 22 or newer, has zero runtime dependencies, and never writes to the repository it audits:

npx github:Navesz/rebar .                       # format and gate rules
npx -p github:Navesz/rebar rebar-security .     # security and prompt-injection signatures

The most useful contribution right now is a false positive. Run it on a repository of yours. When a verdict is wrong, open an issue with the exact output and why it is wrong. One reported false positive is worth more than a new rule.