The 15 bugs AI coding assistants generate over and over (and a scanner that catches them)

#security #python #ai #webdev

AI coding assistants are fast. They're also surprisingly consistent at making the same class of structural mistakes.

After scanning hundreds of AI-generated files, I kept seeing the same patterns:

# Pattern 1: MISSING_WRITE
# AI generates a save function that never actually saves
def save_user(data):
    validate(data)
    return {"status": "saved"}   # no INSERT, no UPDATE, nothing

# Pattern 2: FAKE_ASYNC
# async keyword with no await anywhere
async def fetch_data(url):
    return requests.get(url)     # synchronous, blocks the event loop

# Pattern 3: STUB_SKELETON
# Placeholder that looks complete but does nothing
def analyze_sentiment(text):
    return {}                    # zero logic

These aren't random bugs. They're structural patterns that appear across languages and models — GPT-4, Claude, Gemini, Copilot. The AI writes code that looks correct at a glance but breaks at runtime.

The problem: existing scanners weren't designed for this. Bandit and Semgrep catch security vulnerabilities. They don't check whether your save_user() actually saves.

What I built

AINAScan — a deterministic AST scanner with:

15 vibe-coding patterns (the structural bugs above)
33 security patterns (SQL injection, SSRF, path traversal, command injection, XSS, etc.)
9 languages: Python, JS, TS, Go, Ruby, Java, PHP, Kotlin, C/C++
No LLM involved — same code always produces the same result

The 15 vibe-coding patterns

Pattern	What it catches
`MISSING_WRITE`	save/store function with no DB write
`FAKE_ASYNC`	async def with no await
`STUB_SKELETON`	function that just returns `{}` or `None`
`DEAD_CALL_RESULT`	calls 3 services, ignores all return values
`HARDCODED_TABLE`	40-key dict replacing what should be a DB query
`INPUT_OUTPUT_DISCONNECTED`	params never used in function body
`TRIVIAL_IF_CHAIN`	7+ elif branches with no DB lookup
`MOCK_PATTERN`	MagicMock in production code
`EMPTY_EXCEPT`	except: pass swallowing errors silently
`MISSING_ERROR_HANDLING`	external API calls with no try/catch
`TRIVIAL_ASSERT`	assert True in tests
`TODO_PLACEHOLDER`	TODO/FIXME left in production
`PARAM_SHADOW`	parameter shadowed by local variable
`SHORT_PASSTHROUGH`	wrapper that adds no value
`CONST_SQL_NO_PARAM`	SQL WHERE with hardcoded value

Try it (30 seconds, no signup)

curl -X POST https://pleasing-transformation-production-90c2.up.railway.app/v1/scan \
  -H "X-API-Key: vg_free_test" \
  -F "file=@your_agent.py"

Response looks like:

{
  "passed": false,
  "block_count": 2,
  "warn_count": 1,
  "issues": [
    {
      "kind": "MISSING_WRITE",
      "severity": "BLOCK",
      "line": 12,
      "detail": "function 'save_user' claims to save but contains no DB write call"
    }
  ]
}

GitHub Action — catch vibe-coding bugs in PRs

name: VibeGuard Security Scan
on: [pull_request]

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: Moonsehwan/aina-vibeguard-action@v1
        with:
          api-key: ${{ secrets.VIBEGUARD_KEY }}
          fail-on-block: 'true'

False positive design

One thing I spent a lot of time on: upstream sanitizer detection.

Before flagging a path traversal issue, the scanner checks 60 lines before the sink for guard patterns:

if '..' in path:
    return 400              # scanner sees this
filepath = open(path)       # and downgrades BLOCK to WARN

This cut the false positive rate on well-maintained open source repos (Django, FastAPI, celery) from ~40% down to near zero.

I also tested on 10 repos with 100k+ GitHub stars — 0 false positives on legitimate code.

DEV Community