DEV Community

sehwan Moon
sehwan Moon

Posted on

We Scanned the Vibe Coding Security Scanners. Here's What We Found — Including What We Missed.

#webdev

There are now more than ten security scanners specifically targeting vibe-coded apps. That happened in about three months.

We got curious. So we cloned three of them and ran AINAScan on the source code.

Here's what we found — including the two things AINAScan got wrong.

The setup

We scanned:

  • vibe-audit (ApacheWang) — CLI scanner for Cursor/Bolt/Lovable output
  • vibescan (Armur-Ai) — SAST+DAST pipeline, 15 languages, MCP support
  • VibeSecurity (abenstirling) — hackathon winner, Go scanner wrapped in FastAPI

Three real tools, all open source, all actively maintained. The intent wasn't to rank them — it was to see whether the same patterns that show up in vibe-coded apps also show up in tools built to catch them.

They do.

What AINAScan caught ✅

FAKE_ASYNC in the scanner itself

VibeSecurity/scanner.py, line 50:

async def scan(self) -> dict[str, Any]:
    """Run security scan using Go executable"""
    result = subprocess.run(          # no await
        [self.executable_path, self.url],
        capture_output=True,
        text=True,
        timeout=60,                   # blocks for up to 60 seconds
    )
    return json.loads(result.stdout)
Enter fullscreen mode Exit fullscreen mode

async def with subprocess.run() inside and no await. This blocks the entire FastAPI event loop for up to 60 seconds per scan request. Every other request queues behind it. The hackathon demo probably had one user at a time — this wouldn't surface until real traffic hit it.

This is exactly the pattern we see most often in AI-generated backends. The model learned to write async def but didn't learn why.

SENSITIVE_LOG_LEAK in the auth module

VibeSecurity/auth.py, line 26:

if "uid" not in decoded_token:
    print(f"UID missing from token: {json.dumps(decoded_token)}")
Enter fullscreen mode Exit fullscreen mode

decoded_token is the full Firebase JWT payload — uid, email, provider, any custom claims. All of it goes to stdout on every failed auth attempt. In production, that's your application logs, potentially forwarded to a log aggregation service, potentially accessible to multiple people.

It's a one-line fix. It also ships in the auth module of a security tool.

What AINAScan got wrong ⚠️❌

We're publishing this part because we think it's more useful than pretending the tool is perfect.

False positive: STUB_SKELETON on a Click group function

vibe-audit/cli.py:

@click.group()
@click.version_option(__version__, prog_name="vibe-audit")
def cli():
    """Security scanner for AI-generated code."""
    pass
Enter fullscreen mode Exit fullscreen mode

AINAScan flagged this as STUB_SKELETON — a vibe-coding pattern where a function has a meaningful name but no implementation.

It isn't. pass is the standard body for Click CLI group functions. The framework handles everything; the function itself needs nothing. AINAScan didn't know the difference.

We shipped a fix: @click.group() and @click.command() decorated functions are now exempt from STUB_SKELETON.

False negative: unauthenticated debug endpoints

VibeSecurity/routes/debug.py:

@router.get("/api/debug/all-scans")
async def all_scans_debug(request: Request):
    # returns all users' scan history + emails
    # no authentication dependency
    ...
Enter fullscreen mode Exit fullscreen mode

There are four endpoints under /api/debug/ — none of them require authentication. /api/debug/all-scans returns every user's scan history and email address from Firestore. Anyone who finds the URL can call it.

AINAScan missed this entirely. It caught the FAKE_ASYNC and the log leak, but it doesn't yet analyze whether sensitive route paths have auth dependencies wired up. That's an interoperational check — cross-referencing the route path against the function signature — and it wasn't implemented.

We shipped a fix: New UNAUTH_SENSITIVE_ROUTE rule now flags routes with paths containing debug, admin, repair, or internal that have no Depends(auth_func) parameter. Running the updated scanner on the same file returns four BLOCK findings, one per endpoint.

Results

Tool Real bugs caught FP FN
vibe-audit 0 BLOCK (some WARN) STUB_SKELETON on CLI —
vibescan test data file only (intentional) — —
VibeSecurity FAKE_ASYNC, SENSITIVE_LOG_LEAK — unauthenticated debug routes

vibescan's vulnerable.py had nine BLOCK findings — but that file lives in testdata/ and is intentionally full of vulnerabilities. Their actual source code was clean.

The pattern

All three tools were built quickly, by developers who clearly know what they're doing. And all three have at least one instance of the exact patterns they're designed to catch.

That's not a criticism. It's the point. Vibe-coding patterns aren't a sign of carelessness — they're what fast, AI-assisted development produces by default. The async declaration without semantics, the log statement that made sense in development, the debug route that never got locked down. These appear in mature codebases written by experienced engineers.

The right response isn't to move slower. It's to add a scan step before you ship.

Try it 

# scan before you deploy
curl -X POST https://pleasing-transformation-production-90c2.up.railway.app/v1/scan \
  -H "X-API-Key: vg_free_test" \
  -F "file=@routes/debug.py"
Enter fullscreen mode Exit fullscreen mode

Free key (vg_free_test) covers 50 files/day. Source: github.com/Moonsehwan/aina-scan

What's the most surprising bug you've found in a tool you were already using? Drop it in the comments.

Top comments (0)