SQL injection through a formatted string: BrassCoders catches it on every scan, byte-identical, at the exact line. Missing authentication on an /admin route: no static analyzer catches it without knowing which routes require authentication — a fact that lives in the spec, not the source file. The gap between those two cases is not a tool limitation. It's a category limit.
Static analysis matches patterns against source code structure. It can report only what the structure encodes. Every bug that depends on runtime state, caller intent, architectural policy, business rules, or deployment configuration sits outside that boundary — not because the tool is weak, but because those facts are not encoded in the source.
That distinction matters for teams shipping AI-generated code. AI assistants produce code that looks structurally sound while burying context-dependent bugs in the logic. A scanner catches the structural part. An AI assistant doing triage catches the rest. Neither layer replaces the other, and confusing their roles wastes both.
Quick Navigation
- What "Can't See" Means for a Pattern Scanner
- What BrassCoders Catches Deterministically
- What Requires Context the Scanner Can't Have
- The Bug-Category Visibility Table
- What False Positives Reveal About the Boundary
- How the Two Layers Compose
- What BrassCoders's Findings File Gives the AI Assistant
What "Can't See" Means for a Pattern Scanner
BrassCoders reports what has a structural marker in the source — patterns its 12 scanners can match deterministically on every run. What it cannot see is everything that depends on context the source code doesn't encode: intent, runtime state, caller provenance, business rules, production data distribution.
A structural marker is something an AST traversal finds without executing the code. The string f"SELECT * FROM users WHERE id={user_id}" is a structural marker for SQL injection: its AST contains a formatted string with a SQL keyword prefix and an interpolated variable, a pattern the rule matches deterministically on every run, regardless of where user_id came from or what database is running.
When there's no structural marker, the scanner has nothing to match against. A route handler that calls an admin-only query carries no "requires authentication" annotation in the AST. The requirement exists in a product document or in the developer's intent at the time of writing. Pattern rules evaluate syntax, not intent.
The implication is that the boundary isn't fuzzy. It's precise: a pattern with a fixed structural shape can be matched; a fact that requires executing the code, reading documentation, or knowing the deployment environment cannot. That boundary is the fundamental property that makes the scanner deterministic, not a shortcoming to engineer away. Give up the pattern-only constraint and you give up the guarantee of identical results on identical code.
What BrassCoders Catches Deterministically
BrassCoders's 12 scanners — Bandit, Pylint, Pyre/Pysa, Semgrep, ast-grep, detect-secrets, and six custom detectors — catch what has a fixed structural shape: SQL injection via string formatting (Bandit B608), shell injection (B602/B603), hardcoded credentials (detect-secrets), unsafe deserialization (B506), and phantom imports that don't exist on PyPI.
Each of these fires on every scan, byte-identical for the same commit. A formatted SQL string on line 42 today is the same finding on line 42 after twenty commits to unrelated files. That repeatability is what a CI gate requires: a check that fails consistently on the same code and passes consistently on clean code. A 2025 study by Gnieciak and Szandala, Large Language Models Versus Static Code Analysis Tools, measured this gap directly: deterministic analyzers pointed at the exact line while LLM reviewers mislocated findings due to tokenization artifacts. The LLMs led on recall; precision and exact localization lagged.
BrassCoders's JavaScript and TypeScript scanner (one of the six custom detectors) extends the same approach to six JS and TS file types via Babel AST parsing. The patterns it targets — eval() calls, dangerous innerHTML assignments, document.write usage, hardcoded API keys, and hardcoded passwords — are structural markers, the same category as the Python findings. Semgrep also runs against those file types; Bandit is Python-only and does not.
The structural coverage across categories is documented in BrassCoders's published benchmark: across 12 AI-generated bug categories, BrassCoders's scanners catch 11; Bandit running alone catches 6. For the corpus and scoring, see the full benchmark at /blog/ai-coder-bug-benchmark/. What makes these findings useful in CI is stability: the same code produces the same result on every run. That property is what Why AI Code Needs a Deterministic Gate, Not Just an LLM argues for with the Gnieciak benchmark as evidence.
What Requires Context the Scanner Can't Have
BrassCoders cannot determine whether a division operation will receive a zero denominator, whether a route requires authentication, or whether a query runs in O(N²) time against production-scale data — because these answers depend on runtime facts, architectural decisions, and domain knowledge that aren't encoded in the source file.
Four categories account for most of the scanner's blind spots.
Runtime behavior. A division that fails on a zero denominator only fails when that input arrives. The source shows the operation; it doesn't show the call path or what values flow to it across call boundaries. A ZeroDivisionError in a rarely-executed branch has no structural marker until it throws. The same applies to null-style errors in dynamically typed code: the source shows the attribute access, not the object's actual type at runtime. Logic Bugs No Scanner Can See works through concrete examples of context-dependent correctness bugs that require tracing call paths to find.
Authorization policy. Whether /api/admin/delete-user requires authentication is a policy decision. The source might show a decorator, or it might not — but knowing that it should have one requires reading the auth model and the route map together. BrassCoders flags an endpoint that lacks a recognizable auth decorator in a supported framework, but it cannot know the application's complete auth requirements from source alone. IDOR vulnerabilities, where a route accepts a user-controlled ID without checking ownership, are invisible at the structural level: get_record(user_id=request.args['id']) looks identical whether or not the result is ownership-checked. The Authorization Bug No Scanner Understands covers why these bugs fall outside static analysis.
Performance under production data. O(N²) behavior is visible only when data size context is known. String concatenation in a loop has a structural marker; a list prepend that degrades on a million-row dataset is structurally identical to one that runs fine on ten rows. The performance issue depends entirely on how large the input grows in production — a number the scanner doesn't have. The Slow Code No Scanner Can Flag makes the data-size argument with a concrete example.
Runtime configuration. Whether DEBUG=True reaches production, whether TLS is enforced in the deployment config, whether an environment variable points at the right secret store — these are facts determined at deploy time, not scan time. The source might contain DEBUG = os.getenv('DEBUG', 'false'), but what value the environment injects is invisible to the scanner. Security Misconfiguration Is a Runtime Fact covers the environment-dependent bug pattern and why it requires examining the deployment context alongside the source.
The Bug-Category Visibility Table
BrassCoders classifies every finding it surfaces as a structural pattern — not because it's uncertain, but because the scanner sees structure, not intent. The table below maps bug categories to scanner visibility.
| Bug Category | Scanner Visibility | What the AI Assistant Adds |
|---|---|---|
| SQL injection via f-string | ✅ Deterministic (Bandit B608) | Confirms real query vs test fixture |
| Hardcoded credentials | ✅ Deterministic (detect-secrets) | Confirms real credential vs FIXTURE placeholder |
| Command injection via shell=True | ✅ Deterministic (Bandit B602/B603) | Confirms user-controlled input reaches the call |
| Unsafe deserialization (yaml.load, pickle) | ✅ Deterministic (Bandit B506/B301) | Confirms external input reaches the deserializer |
| Phantom imports (PyPI cross-reference) | ✅ Deterministic (AI-pattern scanner) | Verifies whether package exists at scan time |
| Missing auth on a route | ❌ Needs route map + policy | ✅ AI reads the application structure |
| ZeroDivisionError in untested path | ❌ No structural marker | ✅ AI traces the call path for zero-denominator inputs |
| O(N²) loop under production data | Partial (string-concat pattern) | ✅ AI sees data size context |
| Race condition in asyncio | Partial (threading patterns) | ✅ AI traces shared state across await points |
| Business logic rule violation | ❌ No structural pattern | ✅ AI reads intent from the requirements context |
| Runtime config error (wrong env variable) | ❌ Runtime fact | ✅ AI knows the deployment environment |
The partial entries deserve attention. String concatenation in a loop gets a partial flag because the AST pattern (accumulator on the right side of += inside a loop body) is visible in the parse tree. Whether the accumulator holds a string depends on type inference the scanner doesn't run. BrassCoders flags the pattern; the AI assistant reads the variable declaration, the type annotation if present, and the surrounding call context to confirm.
Threading race conditions follow the same logic. threading.Thread without paired locks, and global mutable state in async functions, have structural markers BrassCoders matches. Asyncio races that leave no structural marker — two coroutines sharing state through an await point, with no visible locking primitive — require tracing the control flow across await boundaries. That's the AI's work.
Business logic violations fall entirely outside the green zone. Whether a coupon discount can be applied twice to the same order depends on a rule in a product document, not a pattern in the source. The source shows the discount being applied; it doesn't show whether the invariant is checked before or after. The AI assistant, given the requirements document alongside the code, can evaluate the invariant. The scanner cannot — because the invariant isn't in the file.
What False Positives Reveal About the Boundary
BrassCoders's false positives are structural — they occur when the scanner matches the pattern but the context makes it safe. A hardcoded credential in a test fixture with a FIXTURE-marked placeholder matches the detect-secrets pattern; the AI triage layer reads the fixture context and marks it a false positive.
This is the boundary made concrete. The scanner did its job: it found a pattern that matches the shape of a real credential. Deciding the value is a placeholder requires reading the variable name, the file path, the fixture directory context, and the FIXTURE annotation — information that isn't in the pattern rule. The AI reads it in a second. The division works.
It's worth pausing on the alternative: a scanner that suppresses findings when it infers fixture context. That scanner would stop reporting the credential in the test fixture. It would also stop reporting a real credential in a file a developer accidentally named test_helpers.py, or in a fixture file that was later refactored to hold production configuration. Inference-based suppression removes false positives and real bugs through the same mechanism — because the structural patterns are identical. The scanner can't know which is which without context.
The False Positive Is a Feature, Not a Bug works through BrassCoders's N=15 corpus scan in detail: 53 raw findings collapsing to 9 confirmed real security issues after one triage pass. It shows the false-positive cases — the counter-increment that looks like a string-concat loop, the MD5 call in a deduplication context — and explains why each context-based suppression would carry a corresponding risk. The ratio looks noisy; the triage session is fast. That's the design.
How the Two Layers Compose
BrassCoders runs first, narrows the scope to confirmed structural patterns, and emits a YAML file the AI assistant reads. The AI assistant verifies each finding in context — confirming whether the injection is real, whether the credential is a fixture, whether the route actually needs auth. Neither layer alone covers the full surface.
A raw AI review of an unscanned codebase starts from scratch: it reads the full source without any priority signal for what matters, and results vary run to run. A post-BrassCoders AI review starts from a prioritized work queue with exact line numbers and remediation notes. The AI spends its time on confirmation, not discovery — and answering a specific question about a specific line is faster and more reliable than open-ended review. Why Your AI Assistant Needs a Deterministic Pre-Pass describes this architecture and the practical speedup it produces.
A 2025 paper, ZeroFalse, measured this combination directly: feeding static-analyzer output to an LLM for adjudication produced F1 scores of 0.912 on the OWASP Java Benchmark and 0.955 on the OpenVuln dataset, with precision and recall above 90%. The pattern in the paper is the same BrassCoders uses: deterministic detection first, model judgment second.
Configuration bugs fit the composition exactly. BrassCoders catches debug=True in a Flask application as a structural flag — the literal Boolean in the source. Whether that value reaches production depends on how the environment variable loading chain works, which deploy scripts override it, and what the hosting environment sets. The AI assistant, reading the deployment scripts alongside the .brass/ai_instructions.yaml, can trace that chain. BrassCoders surfaces the lead; the AI confirms whether it's a real exposure. Security Misconfiguration Is a Runtime Fact shows how the combination handles this on a concrete deployment example.
What BrassCoders's Findings File Gives the AI Assistant
BrassCoders's .brass/ai_instructions.yaml gives an AI assistant a pre-digested work queue — severity, scanner rule, file path, line number, and remediation note for each finding — so it triages structured patterns rather than reading the full codebase from scratch.
The file has three sections. Findings sorted by severity lead the file, so the AI addresses critical and high-severity issues before low-severity style notes. Critical findings — SQL injection, command injection, hardcoded credentials, unsafe deserialization — appear first. The AI starts where the risk is highest. Scanner metadata follows: which of the 12 scanners ran, at what version, and how long the scan took. That section lets an AI assistant identify which rule engine fired and what its coverage scope is.
The third section, how_to_read_this_file, is a plain-language note explaining that BrassCoders is a pattern reporter, not a verdict machine. Findings are leads, not confirmed bugs. The AI's job is source verification on each flagged line — confirm it or dismiss it. That framing is why the triage session runs fast: the AI arrives knowing what to do with a finding, not wondering whether it can trust the verdict.
That contract produces a clean handoff. The scanner runs in seconds on a fresh commit. The YAML lands in the .brass/ directory. Claude Code or Cursor reads the file, opens the flagged lines, and runs the triage session against a bounded list of structural findings. The open-ended question ("what might be wrong?") becomes a specific list ("is this pattern a real bug in this context?"). The limit the scanner cannot cross is exactly where the AI assistant picks up — and the YAML format is what makes the crossing work at speed.
pip install brasscoders
brasscoders scan .
# .brass/ai_instructions.yaml is ready for your AI assistant
Top comments (0)