BrassCoders scanned 15 AI-generated Python files and emitted a finding in every one: 53 findings total, 16 of them critical or high. On triage, 9 of the 15 carried a real security or performance issue, and the rest were style notes or false positives. The split between those two numbers is the whole point of how BrassCoders is built.
The headline "every file had a finding" is true and nearly meaningless on its own. The number that matters is what survived a careful read.
How The Corpus Was Built
BrassCoders' corpus is 15 Python files generated by Claude Opus 4.8 from realistic coding prompts, each carrying a PROVENANCE docstring naming the prompt and model, written at normal effort with nothing planted or scrubbed. The prompts cover web handlers, data processing, subprocess calls, auth, and config.
The scoring was pre-registered before the scan: count the files for which brasscoders --offline scan emits at least one finding, then break the findings down by category, with no file added or removed after the first run. It's a single-model corpus of 15 files, which is a stated limit, not a hidden one. The point isn't a population-level rate; it's a reproducible look at what a deterministic scanner surfaces on un-planted AI code.
What BrassCoders Found
BrassCoders emitted 53 findings across the 15 files, 16 of them critical or high severity, with at least one finding in every file. The critical and high set clustered in the security categories.
The real, confirmed issues: SQL injection assembled with string formatting in user_lookup.py and bulk_insert.py; command injection through shell=True in run_command.py and os.system in thumbnail.py; hardcoded credentials in token_check.py and email_sender.py; an unsafe yaml.load in config_loader.py; a requests call with no timeout in api_client.py; Flask debug=True in user_lookup.py; and one genuine O(N^2) string-concatenation loop in csv_report.py. Nine of the fifteen files carried at least one of these.
After Triage: What Was Real
BrassCoders' value showed up at triage: of the 16 critical-and-high findings, the security structural ones held up, while several performance and style flags did not, leaving 9 of 15 files (60%) with a confirmed issue. A raw finding is a question, not a verdict.
This is the division of labor the tool is designed around. BrassCoders reports the pattern; a human or an LLM like Claude Code decides whether it's real. The confirmed-issue rate of 60% sits in the same range as external measurements of AI-code defects, which run from Veracode's 45% OWASP-vulnerability rate to CodeRabbit's 1.7x more issues per PR. Those are different methodologies on far larger samples, cited here for context, not as this corpus's result.
What The False Positives Teach
BrassCoders flagged four O(N^2) string concatenations, but only one was real: the other three were loops incrementing an integer counter, not building a string. A deterministic reporter that never inferred context produced exactly the noise the triage layer exists to remove.
The misfires are instructive. In api_client.py, log_parser.py, and retry.py, the flagged line was page += 1 or a counter increment, not string building, so the performance warning was wrong. In file_dedupe.py, MD5 was flagged as a weak security hash when it was being used for content deduplication, where it's fine. And in api_client.py, the scanner flagged a false string-concat while missing the real performance bug one line up: items.insert(0, item) in a loop, an O(N^2) list prepend. A scanner that guessed at intent would hide these tradeoffs; one that reports patterns lets the triage layer catch both the false alarm and the real miss.
Reproduce It
BrassCoders ships this corpus, its manifest, and the pre-registered scoring in the OSS repo, so the static findings reproduce exactly. Clone it, install the tool, and scan the files directory.
git clone https://github.com/CopperSunDev/brasscoders
pip install brasscoders
brasscoders --offline scan brasscoders/docs/benchmarks/ai-code-findings-corpus/files
The takeaway is the two-number story: a deterministic scan flags a lot, and the triage layer decides what's real. Run the scan on every commit; spend your reading on the findings that survive it.
Top comments (0)