Vibe coding is fast. You describe what you want, the AI writes it, and it ships. The regrets show up later — in the O(N²) loop that's fine on 100 rows and unresponsive on 100,000, in the API key that ships in a string literal, or in the package that doesn't exist on PyPI.
The fix takes 30 seconds. But first, it helps to understand what you're actually dealing with.
What Vibe Coding Gets Right
Vibe coding — the practice of accepting AI-generated code with minimal line-by-line review, coined by Andrej Karpathy in early 2025 — genuinely speeds up development. For prototypes and early-stage products, shipping fast matters more than shipping perfect. BrassCoders is the safety-net scan that runs before the push, not a replacement for moving fast.
Karpathy described the pattern in a February 2025 post: you describe what you want; the AI writes it; you barely read it. The problem is that shipping fast and shipping without any safety net aren't the same thing.
An AI coding assistant generates code the way a confident but context-free collaborator would. It applies plausible patterns from training data, assembled quickly, without knowledge of your production data volumes or your secret management conventions. Speed is real. Verification is still your job.
What vibe coding does well: it eliminates the blank-page problem. You get something running in minutes instead of hours. The first 80% of any implementation arrives nearly free. That's a genuine productivity win for solo developers and small teams who can't afford to slow down.
What it doesn't do: read your deployment context. It doesn't know your database has 50M rows. It doesn't know your team commits .env files accidentally. It doesn't know the package it just confidently referenced was never published.
The Four Bug Classes That Slip Through
BrassCoders's benchmark against 12 AI-generated Python files (June 2026) found that AI coding assistants consistently produce four categories of bugs that reviewers typically miss: O(N²) performance anti-patterns in data processing, hardcoded credentials in example code that gets committed, imports of packages that don't exist on PyPI, and insecure subprocess and SQL patterns inherited from training data.
A frontier model reviewing its own generated code caught all of these when asked. It didn't warn while generating. The full benchmark is published at coppersun.dev/blog/ai-coder-bug-benchmark/.
Performance anti-patterns look fine until data volume hits. The canonical examples:
# O(N²) string concatenation — each += allocates a new string
csv_data = ""
for row in rows:
csv_data += format_row(row) # painful at 100K rows
# O(N²) list prepend — rewrites the entire list on every insert
results = []
for item in items:
results.insert(0, item) # use collections.deque or reverse after
# Triple-nested loop that wants a dict lookup
for user in users:
for order in orders:
for item in order_items:
if item.user_id == user.id: # O(N³); index by user_id instead
...
Hardcoded secrets appear when AI writes example code and the example travels into a commit:
# AI generates working example — developer copies it without noticing the key
client = openai.OpenAI(api_key="sk-proj-abc123realkey...")
db = psycopg2.connect("postgresql://admin:mypassword@prod-db/app")
Hallucinated imports compile fine and fail at runtime — or worse, become supply-chain attack vectors when a typosquatter registers the invented name. Lasso Security's 2024 research on AI-generated package names documented this as a measurable failure mode across major LLMs, not a rare edge case:
from fastapi_users_pydantic import UserManager # doesn't exist on PyPI
import langchain_memory_redis # hallucinated combination package
Insecure patterns are the ones that feel idiomatic because they're common in training data:
# SQL injection via f-string — classic, still generated constantly
query = f"SELECT * FROM users WHERE email = '{user_email}'"
cursor.execute(query)
# Shell injection waiting to happen
subprocess.run(f"convert {filename} output.png", shell=True)
# Deserializing untrusted data
model = pickle.loads(request.body)
Each category is a different kind of failure. All four are common outputs from AI coding assistants working normally.
The One Command That Runs in 30 Seconds
brasscoders --offline scan . runs 12 static-analysis scanners against a project directory, detects all four AI-coder bug categories above, and writes a ranked findings list to .brass/ai_instructions.yaml. The file is designed to paste directly into Claude Code or Cursor for triage. Zero network calls. One pip install brasscoders.
Six upstream tools run the core analysis — Bandit, Pylint, Pyre/Pysa, Semgrep, ast-grep, and Yelp’s detect-secrets. BrassCoders adds six custom detectors on top: the performance detector catches O(N²) anti-patterns; the AI-pattern detector surfaces hallucinated imports; and four more cover secret-format matching, PII, content moderation, and JavaScript/TypeScript.
BrassCoders's June 2026 benchmark planted 12 bugs across those categories and ran several scanners against the same files. Bandit caught 6 of 12 and zero of the four performance anti-patterns. Pylint caught 1 of 12. BrassCoders caught 11 of 12, including all four performance bugs. The benchmark is reproducible; the methodology is published.
Install and scan:
pip install brasscoders
brasscoders --offline scan .
The .brass/ai_instructions.yaml output is intentionally short — a ranked list of findings with severity, context, and a how-to-read note. BrassCoders reports the patterns. Your AI assistant triages them with full source context. Two tools, one job each.
Wiring It to Pre-Commit
BrassCoders works as a pre-commit hook — the check that runs before git commit succeeds. When configured this way, BrassCoders exits non-zero on CRITICAL findings and blocks the commit. Secrets, insecure patterns, and hallucinated imports don't enter the repo.
The .pre-commit-config.yaml entry takes five lines:
repos:
- repo: local
hooks:
- id: brasscoders
name: BrassCoders scan
entry: brasscoders --offline scan
language: system
pass_filenames: false
Add that to an existing pre-commit setup with pre-commit install and the hook runs on every commit. The first time you try to commit hardcoded credentials or a subprocess call with shell=True, the commit fails with a findings summary pointing at the exact line.
Pre-commit is a fast local gate that catches the obviously-wrong before it ever leaves your machine. The CI scan (see the GitHub Actions setup post) is the authoritative gate for teams; pre-commit stops you from wasting a CI run on a secret you forgot to remove. Both belong in the same workflow.
What BrassCoders Catches That a Model Review Misses
In BrassCoders's generation-mode benchmark probe, a frontier model given six realistic Python tasks generated clean code four of five times across the security and performance wedge categories — and warned proactively zero times during generation. The pattern is consistent with what published AI code review research shows: generation and review are different tasks, and a model optimizing for plausible code completion has no built-in reason to flag the patterns it's producing.
The specific gap: AI assistants generate O(N²) loops because they match pattern to context, not because they model your production dataset size. They generate shell=True subprocess calls because the pattern is common in training data, not because they've verified your input is sanitized. They generate hallucinated package names because the naming pattern is plausible, not because they've checked PyPI.
BrassCoders catches these deterministically — same rules on every run, same exit code on every violation, no variation based on model temperature or context window. The .brass/ai_instructions.yaml output then feeds back into your AI assistant for triage, which is where context-aware judgment belongs. The scanner reports the facts. The AI assistant, with access to your full codebase and your business context, decides what to fix.
Vibe coding is fine. Vibe coding with a 30-second safety net before every push is better.
Install BrassCoders with pip install brasscoders and run brasscoders --offline scan . against your next AI-generated diff. For the full benchmark methodology and planted-bug breakdown, see the AI Coder Bug Benchmark post.
Top comments (0)