The AI assistant doesn't know you're using HashiCorp Vault. It doesn't know your team agreed last quarter that all secrets go in AWS Secrets Manager. It doesn't know your .env convention, your CI/CD injection pattern, or your rotation policy. It knows one thing: you gave it a prompt, and the prompt has a field that expects a value.
So it fills in a value.
That's the policy gap. It's not a hallucination bug or a model alignment problem. It's structural. The model completes what's in front of it — and what's in front of it has no secrets-management context attached.
The Policy Gap
BrassCoders found hardcoded credentials in 2 of 15 files in a published corpus of naturally-generated AI Python code — files generated from realistic prompts, with no attempt to plant or avoid issues. The model filled in what looked right. Both files came from prompts that any developer might send to an AI assistant on a Tuesday afternoon.
That 2-in-15 rate isn't a catastrophe. It's a baseline. It tells you that on a typical vibe-coding session generating a handful of utility scripts, at least one of them is likely to contain a credential that shouldn't be there. Manual review would catch it. Vibe coding, by definition, skips the review step.
The organizational compounding factor: your AI assistant has no knowledge of your internal standards. Prompting it to "use environment variables" helps for that one file. It doesn't carry forward to the next file, or the next developer, or the next AI session. The policy gap opens again every time someone starts a new chat.
What the N=15 Corpus Found
BrassCoders scanned 15 AI-generated Python files — the full corpus is published at coppersun.dev/benchmarks and is reproducible from pinned source. Two files contained hardcoded credentials, both detectable by pattern matching alone.
token_check.py had SECRET_KEY = "s3cr3t-signing-key-change-me" — a literal HMAC signing key. The prompt asked for a function that signs and verifies a session token. The model generated working HMAC code. It also generated a signing key, because the code needs one to run.
email_sender.py had server.login("noreply@example.com", "hunter2-mailpassword") — a literal SMTP password embedded in the login call. The prompt asked for an email-sending function. SMTP login requires a password. The model provided one.
Both are textbook completions: the model sees a code pattern that requires a credential, and it provides a credential. Neither file was marked as example code or clearly labeled as a draft. Both would pass a quick visual scan if you're moving fast.
The detailed corpus walkthrough is in the companion post — if you want to see the exact scanner output for each file, that's the place to start.
The 20+ Formats BrassCoders Watches For
BrassCoders detects over 20 secret formats using Yelp's detect-secrets as the upstream library, plus 7 custom patterns added in BrassCoders's scanner layer. The combination covers the credential types that appear most often in AI-generated code.
Yelp's detect-secrets handles AWS access keys (the AKIA prefix), GitHub PATs (ghp_), Stripe live keys (sk_live_), OpenAI keys (sk-), Slack tokens, PEM-formatted private keys, JWTs, and high-entropy strings that don't match a known format. The custom layer adds patterns that detect-secrets doesn't cover by default.
What they don't cover: secrets your organization defines internally — custom API token formats, proprietary signing schemes, anything that doesn't follow a known structural pattern. That's a gap worth knowing. BrassCoders's YAML output names which scanner flagged each finding, so you can audit coverage and add .brassignore entries or upstream Semgrep rules for formats outside the default set.
How BrassCoders catches hardcoded secrets covers the detection mechanism in full — entropy scoring, format matching, and where the scanner draws the line on false positives.
Why a Placeholder Is as Dangerous as a Real Key
BrassCoders flags SECRET_KEY = "s3cr3t-signing-key-change-me" the same way it flags a real credential — because OWASP A02:2021 (Cryptographic Failures) makes no distinction. If your application uses that value in production for even a week while you "get around to replacing it," the security boundary is gone.
Git history doesn't forget. Once a credential string appears in a commit, it lives in the repository's history until someone runs a deliberate history rewrite. Teams that move fast often don't notice the commit until months later. By then the credential may be in backup snapshots, in CI logs, or in the IDE history of anyone who pulled the branch.
The organizational pressure makes it worse. A placeholder that looks like a real value doesn't trigger the same alarm as SECRET_KEY = "REPLACE_ME". The developer reading the PR sees something that looks like it was set intentionally. It passes.
This is why the broader secrets treatment covers both real and placeholder credentials — the rotation workflow is the same either way. Rotate it, replace it, and add the file to .brassignore with a note. Don't assume a placeholder is safe because it looks fake.
The Organizational Fix: Secrets Management Before First Commit
BrassCoders closes the policy gap at the tool layer because the prompt layer can't. Prompting your AI assistant to use environment variables is not a policy. It's advice that evaporates when the next developer opens a new chat session.
A concrete workflow that works:
Before generating code: decide which secrets manager your project uses — AWS Secrets Manager, HashiCorp Vault, a .env file loaded by python-dotenv, or your CI/CD platform's native injection. Document it in your README. One line is enough.
After every generation session: run brasscoders --offline scan before committing. The --offline flag makes zero outbound network calls — every scanner runs locally, nothing leaves the machine. The YAML output names the file, the line, and the pattern matched. Fix or .brassignore before the commit lands.
When a finding appears: rotate the credential. Treat it as compromised. If the string is a test fixture, switch to the FIXTURE-marked naming convention — AKIAFIXTURE000000000, ghp_FIXTUREbcdef... — which BrassCoders's scanner is trained to recognize as a test artifact, not a live key.
The two-minute scan before every commit is the review step that vibe coding removed. BrassCoders puts it back without requiring you to slow down the generation loop.
Install BrassCoders from PyPI:
pip install brasscoders
brasscoders --offline scan .
No account. No telemetry. Apache 2.0.
Top comments (0)