Vibe Coding Without the Technical Debt: A Python Guardrail Pipeline That Catches What the AI Missed

Andrej Karpathy coined “vibe coding” in a post on X on February 2, 2025: you describe what you want, accept what the model writes, and “fully give in to the vibes.” By November, Collins had picked it as its Word of the Year. Adoption followed the hype: in the 2025 Stack Overflow Developer Survey, 84% of respondents said they use or plan to use AI tools, yet more of them distrust the accuracy of those tools (46%) than trust it (33%).

So is vibe coding bad? The research on AI generated code vulnerabilities gives a consistent answer: the code is not bad by default, but it is not safe by default either.

• In “Asleep at the Keyboard?” (IEEE S&P 2022), Pearce et al. generated 1,689 programs with GitHub Copilot across 89 security-relevant scenarios. About 40% were vulnerable.
• Veracode’s GenAI Code Security Report tested code from more than 100 LLMs in Java, JavaScript, Python and C#. AI-generated code introduced risky security flaws in 45% of tests, and larger, newer models did not do better.
• Perry et al. (“Do Users Write More Insecure Code with AI Assistants?”) found that participants with an AI assistant wrote significantly less secure code, and were more likely to believe it was secure.
• Spracklen et al. (USENIX Security 2025) analyzed 576,000 generated code samples from 16 models: on average at least 5.2% of the packages suggested by commercial models and 21.7% by open-source models did not exist.

The fix is not to stop using the assistant, but to make every AI-written change pass the checks you would demand from a new hire. In this post we take one small, realistic “vibe-coded” Flask module, run a guardrail pipeline over it, look at what each tool caught and what it missed, fix the code, and wire everything into pre-commit and GitHub Actions.

What we will do:

1. Look at the vibe-coded module and its two happy-path tests.
2. Run ruff, mypy (strict), bandit, detect-secrets, vulture and pip-audit on it.
3. Write the tests the assistant did not: API edge cases plus Hypothesis property-based tests, behind a 90% coverage gate.
4. Fix the module and get a fully green run.
5. Lock it in with pre-commit and a GitHub Actions workflow, and watch the hooks block a bad commit.

Versions (pinned in requirements-dev.txt): Python 3.13.18, Flask 3.1.3, ruff 0.17.3, mypy 2.4.1, bandit 1.9.4, detect-secrets 1.5.0, vulture 2.16, pip-audit 2.10.1, pytest 9.2.0, pytest-cov 7.1.0, Hypothesis 6.170.13, pre-commit 4.7.0. Every output block below is copied from a real run.

The specimen: a vibe-coded orders API

The prompt was the kind you type in a hurry: “Flask API for orders: list a customer’s orders with pagination, apply a percentage discount, notify a webhook.” I wrote the result by hand to reproduce the patterns the studies above keep finding, so every issue here is deliberate, but none of it is exotic. It runs and it demos well:

import json
import os
import sqlite3

import requests
from flask import Flask, jsonify, request

app = Flask(__name__)

DB_PATH = "shop.db"
PAYMENTS_API_KEY = "sk_live_51NzQ8vKp2ExampleKey9fT3"
WEBHOOK_SECRET = "whsec_3f9a1c7e2b"
WEBHOOK_URL = os.getenv("WEBHOOK_URL")


def get_db():
    conn = sqlite3.connect(DB_PATH)
    conn.row_factory = sqlite3.Row
    return conn


def apply_discount(total, percent):
    return round(total - total * percent / 100, 2)


def paginate(items, page, per_page):
    start = (page - 1) * per_page
    return items[start:start + per_page]


def legacy_total(items):
    total = 0
    for i in items:
        total += i["price"] * i["qty"]
    return total


@app.route("/orders")
def list_orders():
    customer = request.args.get("customer")
    page = request.args.get("page", 1)
    db = get_db()
    rows = db.execute(
        f"SELECT id, customer, total, status FROM orders WHERE customer = '{customer}'"
    ).fetchall()
    orders = [dict(r) for r in rows]
    return jsonify(paginate(orders, page, 20))


@app.route("/orders/<int:order_id>/discount", methods=["POST"])
def discount(order_id):
    data = request.get_json()
    percent = data["percent"]
    db = get_db()
    row = db.execute("SELECT total FROM orders WHERE id = ?", (order_id,)).fetchone()
    new_total = apply_discount(row["total"], percent)
    db.execute("UPDATE orders SET total = ? WHERE id = ?", (new_total, order_id))
    db.commit()
    notify(order_id, new_total)
    return jsonify({"id": order_id, "total": new_total})


def notify(order_id, total):
    try:
        requests.post(
            WEBHOOK_URL,
            json={"id": order_id, "total": total},
            headers={"X-Signature": WEBHOOK_SECRET},
            verify=False,
        )
    except:
        pass


if __name__ == "__main__":
    app.run(host="0.0.0.0", debug=True)

The assistant also wrote tests. They pass:

from shop.app import apply_discount, paginate


def test_apply_discount():
    assert apply_discount(100, 10) == 90


def test_paginate():
    assert paginate([1, 2, 3, 4, 5], 1, 2) == [1, 2]

Try to count the problems before the tools do. The matrix near the end lists sixteen, and no tool in this post flags three of them.

The guardrail pipeline at a glance

Each tool answers one narrow question. None of them replaces the others, and none of them replaces a reviewer.

Tool	Question it answers	Where it runs
ruff	Is this code idiomatic, without obvious bugs (unused imports, bare except)?	pre-commit, CI
mypy --strict	Do the types line up across function boundaries?	pre-commit, CI
bandit	Does the AST contain known insecure patterns (CWE-mapped)?	pre-commit, CI
detect-secrets	Does a staged file contain something that looks like a credential?	pre-commit, CI
vulture	Is there code nothing calls?	pre-commit, CI
pip-audit	Do pinned dependencies have published advisories?	pre-commit (on requirements change), CI
pytest + Hypothesis + coverage	Does the code do the right thing on inputs nobody thought of?	pre-push, CI

Pipeline diagram: commit-time hooks (ruff, bandit, detect-secrets, vulture, mypy, pip-audit on requirements change), pre-push tests with a 90% coverage gate, and the CI job that reruns everything with 1,000 Hypothesis examples

The project configuration that all tools read lives in one pyproject.toml:

[project]
name = "shop"
version = "0.1.0"
requires-python = ">=3.12"

[tool.ruff]
line-length = 100
target-version = "py312"
# vulture whitelists are bare expressions by design; force-exclude also applies to pre-commit
extend-exclude = ["vulture_whitelist.py"]
force-exclude = true

[tool.ruff.lint]
select = ["E", "F", "B", "BLE", "UP", "SIM"]

[tool.mypy]
strict = true
packages = ["shop"]

[tool.bandit]
exclude_dirs = ["tests", ".venv"]

[tool.pytest.ini_options]
addopts = "--cov=shop --cov-branch --cov-report=term-missing --cov-fail-under=90"

[tool.coverage.run]
omit = ["tests/*"]

Step 1. ruff: fast, but only the surface

$ ruff check --output-format concise .
shop/app.py:1:8: F401 [*] `json` imported but unused
shop/app.py:64:5: SIM105 Use `contextlib.suppress(BaseException)` instead of `try`-`except`-`pass`
shop/app.py:71:5: E722 Do not use bare `except`
Found 3 errors.
[*] 1 fixable with `--fix` (1 unsafe fix available with `--fix --unsafe-fixes`).
[exit 1]

ruff caught the unused json import and the bare except. That is useful hygiene, but notice SIM105: it proposes contextlib.suppress(BaseException). Accepting that “unsafe fix” (ruff labels it that way itself) keeps the real bug, a webhook failure that silently disappears, and just makes it shorter. The right fix is to catch requests.RequestException and log it.

ruff also ships rules derived from flake8-bandit under the S prefix (the f-string query would be S608), and you can enable them. I keep bandit separate here so you can see exactly what each tool finds; in a large repo, ruff --select S is the faster option.

Step 2. mypy –strict: noise first, then the real bug

On the unannotated vibe code, strict mode returns twelve errors, and every one of them says the same thing:

$ mypy
shop/app.py:16: error: Function is missing a return type annotation  [no-untyped-def]
shop/app.py:22: error: Function is missing a type annotation  [no-untyped-def]
shop/app.py:26: error: Function is missing a type annotation  [no-untyped-def]
shop/app.py:31: error: Function is missing a type annotation  [no-untyped-def]
shop/app.py:39: error: Function is missing a return type annotation  [no-untyped-def]
shop/app.py:42: error: Call to untyped function "get_db" in typed context  [no-untyped-call]
shop/app.py:47: error: Call to untyped function "paginate" in typed context  [no-untyped-call]
shop/app.py:51: error: Function is missing a type annotation  [no-untyped-def]
shop/app.py:54: error: Call to untyped function "get_db" in typed context  [no-untyped-call]
shop/app.py:56: error: Call to untyped function "apply_discount" in typed context  [no-untyped-call]
shop/app.py:59: error: Call to untyped function "notify" in typed context  [no-untyped-call]
shop/app.py:63: error: Function is missing a type annotation  [no-untyped-def]
shop/app.py:66: error: Argument 1 to "post" has incompatible type "str | None"; expected "str | bytes"  [arg-type]
Found 13 errors in 1 file (checked 2 source files)
[exit 1]

By default, mypy does not type-check the bodies of unannotated functions, and AI assistants often skip annotations in quick scripts. So mypy has nothing to check until you add signatures. After adding only the signatures (no logic changes), the noise disappears and a real bug appears:

$ mypy
shop/app.py:48: error: Argument 2 to "paginate" has incompatible type "str | int"; expected "int"  [arg-type]
shop/app.py:67: error: Argument 1 to "post" has incompatible type "str | None"; expected "str | bytes"  [arg-type]
Found 2 errors in 1 file (checked 2 source files)
[exit 1]

request.args.get("page", 1) returns the string "2" when the parameter is present and the integer 1 when it is not. The happy-path test never sends page, so the code passed. In production, GET /orders?customer=alice&page=2 crashes with TypeError and returns a 500.

The second error is a deployment bug: os.getenv("WEBHOOK_URL") returns None when the variable is not set, and the code passes it straight to requests.post. Because of the bare except, that failure would also be silent.

Step 3. bandit and detect-secrets: security patterns and leaked keys

bandit -q -c pyproject.toml -r shop -f custom \
  --msg-template "{relpath}:{line}: {test_id}[{severity}/{confidence}] {msg}"
shop/app.py:12: B105[LOW/MEDIUM] Possible hardcoded password: 'whsec_3f9a1c7e2b'
shop/app.py:44: B608[MEDIUM/MEDIUM] Possible SQL injection vector through string-based query construction.
shop/app.py:69: B501[HIGH/HIGH] Call to requests with verify=False disabling SSL certificate checks, security issue.
shop/app.py:65: B113[MEDIUM/LOW] Call to requests without timeout
shop/app.py:71: B110[LOW/HIGH] Try, Except, Pass detected.
shop/app.py:76: B201[HIGH/MEDIUM] A Flask app appears to be run with debug=True, which exposes the Werkzeug debugger and allows the execution of arbitrary code.
shop/app.py:76: B104[MEDIUM/MEDIUM] Possible binding to all interfaces.
[exit 1]

Seven findings, all real: the f-string SQL (B608, CWE-89), verify=False (B501), no timeout (B113), the swallowed exception (B110), debug=True that exposes the Werkzeug debugger (B201), and binding to all interfaces (B104).

Now look at what is missing. WEBHOOK_SECRET was flagged, but PAYMENTS_API_KEY = "sk_live_..." was not. B105 is name-based: it looks for variables whose names resemble password, pass, passwd, pwd, secret, token or secrete. “API key” is not on that list, so bandit let a live-looking payment key through.

This is why secure AI generated code needs a dedicated secret scanner. detect-secrets combines provider regexes, entropy checks and keyword rules:

$ detect-secrets-hook shop/app.py
ERROR: Potential secrets about to be committed to git repo!
Secret Type: Base64 High Entropy String
Location:    shop/app.py:11
Secret Type: Secret Keyword
Location:    shop/app.py:11
Secret Type: Stripe Access Key
Location:    shop/app.py:11
Secret Type: Secret Keyword
Location:    shop/app.py:12
[exit 1]
# (help text with links removed)

It flagged the Stripe-style key three ways (entropy, keyword, Stripe detector) and also caught the webhook secret.

Step 4. pip-audit: the dependency the model remembered

The assistant pinned requests==2.31.0, an older release of the kind models reproduce from their training data. pip-audit resolves requirements.txt and checks every package against an advisory database:

$ pip-audit -r requirements.txt --aliases
Found 6 known vulnerabilities in 1 package
Name     Version ID              Fix Versions Aliases
-------- ------- --------------- ------------ -----------------------------------
requests 2.31.0  PYSEC-2026-1873 2.32.0       CVE-2024-35195, GHSA-9wx4-h78v-vm56
requests 2.31.0  PYSEC-2026-1872 2.32.4       CVE-2024-47081, GHSA-9hjg-9r4m-mvj7
requests 2.31.0  PYSEC-2026-2275 2.33.0       CVE-2026-25645, GHSA-gc5v-m9x4-r6x2
requests 2.31.0  PYSEC-2026-1873 2.32.0       CVE-2024-35195, GHSA-9wx4-h78v-vm56
requests 2.31.0  PYSEC-2026-1872 2.32.4       CVE-2024-47081, GHSA-9hjg-9r4m-mvj7
requests 2.31.0  PYSEC-2026-2275 2.33.0       CVE-2026-25645, GHSA-gc5v-m9x4-r6x2
[exit 1]

Three advisories, each printed twice in this run (the JSON output shows the same duplication; the unique set is three). The first one is directly relevant: CVE-2024-35195 means that once a Session makes its first request to a host with verify=False, later requests to that host keep skipping certificate verification for the life of the pooled connection. Our module disables verification on purpose, which makes this worse, not better. The fix is requests==2.34.2 and removing verify=False.

If the advisory service is unreachable. pip-audit queries PyPI by default and also supports OSV (--vulnerability-service osv). In my sandbox, the PyPI service worked, but the OSV endpoint was blocked by the network proxy (api.osv.dev ... Tunnel connection failed: 403 Forbidden). If your build agents cannot reach either service, do not drop the check. Run pip-audit in a CI job that does have network access, or rely on GitHub’s Dependabot alerts on the repository, and keep --strict so a dependency that cannot be resolved fails the job instead of being skipped.

One more caution from the pip-audit README: pip-audit -r is functionally equivalent to pip install -r, and its isolation is not a security boundary. Do not run it on requirements files you do not trust.

Step 5. vulture: dead code and its false positives

$ vulture shop --min-confidence 60
shop/app.py:1: unused import 'json' (90% confidence)
shop/app.py:11: unused variable 'PAYMENTS_API_KEY' (60% confidence)
shop/app.py:18: unused attribute 'row_factory' (60% confidence)
shop/app.py:31: unused function 'legacy_total' (60% confidence)
shop/app.py:38: unused function 'list_orders' (60% confidence)
shop/app.py:50: unused function 'discount' (60% confidence)
[exit 3]

vulture found legacy_total, which nothing calls, and the unused PAYMENTS_API_KEY. It also reported list_orders and discount as unused, which is wrong: Flask calls them through @app.route. The vulture README says it directly: code that is only called implicitly may be reported as unused, and it suggests --ignore-decorators "@app.route" or a whitelist. With 60% confidence for functions, you cannot simply raise the threshold without also hiding legacy_total. In the fixed version I generated a whitelist with vulture shop --make-whitelist.

Step 6. Tests the assistant did not write: edge cases, properties and a coverage gate

The two vibe-coded tests pass, and the coverage gate (--cov-fail-under=90 with branch coverage) already says something is off:

$ pytest -q -p no:cacheprovider tests/test_app.py
..
ERROR: Coverage failure: total of 44 is less than fail-under=90
shop/app.py           50     27      4      1    44%   17-19, 32-35, 40-47, 52-60, 64-72, 76
FAIL Required test coverage of 90% not reached. Total coverage: 44.44%
2 passed in 0.20s
# (output filtered to the relevant lines)

Coverage is a weak signal for AI code: an assistant can produce 90% coverage with assertions that check nothing. The real value is in tests that state what must always be true. First, API tests that describe the contract the prompt never mentioned:

from decimal import Decimal

from flask.testing import FlaskClient


def test_customer_filter_is_not_injectable(client: FlaskClient) -> None:
    resp = client.get("/orders", query_string={"customer": "x' OR '1'='1"})
    assert resp.status_code == 200
    assert resp.get_json() == []


def test_page_query_param_is_parsed(client: FlaskClient) -> None:
    resp = client.get("/orders", query_string={"customer": "alice", "page": "1"})
    assert resp.status_code == 200
    assert [o["id"] for o in resp.get_json()] == [1]


def test_bad_page_is_a_client_error(client: FlaskClient) -> None:
    resp = client.get("/orders", query_string={"customer": "alice", "page": "abc"})
    assert resp.status_code == 400


def test_discount_applies(client: FlaskClient) -> None:
    resp = client.post("/orders/1/discount", json={"percent": 10})
    assert resp.status_code == 200
    assert Decimal(str(resp.get_json()["total"])) == Decimal("90.00")


def test_discount_over_100_percent_is_rejected(client: FlaskClient) -> None:
    resp = client.post("/orders/1/discount", json={"percent": 150})
    assert resp.status_code == 400


def test_non_numeric_percent_is_rejected(client: FlaskClient) -> None:
    resp = client.post("/orders/1/discount", json={"percent": "ten"})
    assert resp.status_code == 400


def test_missing_percent_is_rejected(client: FlaskClient) -> None:
    resp = client.post("/orders/1/discount", json={"pct": 10})
    assert resp.status_code == 400


def test_unknown_order_is_404(client: FlaskClient) -> None:
    resp = client.post("/orders/999/discount", json={"percent": 10})
    assert resp.status_code == 404

Second, property-based tests with Hypothesis. Instead of picking examples, we describe the input space and an invariant, and Hypothesis searches for a counterexample and then shrinks it to the smallest one it can find:

from decimal import Decimal

from hypothesis import given
from hypothesis import strategies as st

from shop.app import apply_discount, paginate

money = st.decimals(min_value=0, max_value=1_000_000, places=2)
percent = st.decimals(min_value=-1000, max_value=1000, places=2)


@given(total=money, pct=percent)
def test_discount_never_increases_total_or_goes_negative(total: Decimal, pct: Decimal) -> None:
    try:
        result = Decimal(str(apply_discount(total, pct)))
    except ValueError:
        return  # rejecting bad input is allowed; returning garbage is not
    assert 0 <= result <= total


@given(items=st.lists(st.integers(), max_size=60), per_page=st.integers(1, 25))
def test_pages_cover_every_item_exactly_once(items: list[int], per_page: int) -> None:
    pages = [paginate(items, p, per_page) for p in range(1, len(items) // per_page + 2)]
    assert [x for page in pages for x in page] == items


@given(items=st.lists(st.integers(), min_size=1), page=st.integers(max_value=0))
def test_page_below_one_is_rejected(items: list[int], page: int) -> None:
    try:
        result = paginate(items, page, 10)
    except ValueError:
        return
    raise AssertionError(f"page={page} returned {result!r} instead of raising")

Against the vibe-coded module:

$ pytest -q -p no:cacheprovider --tb=short
TypeError: unsupported operand type(s) for -: 'str' and 'int'
TypeError: unsupported operand type(s) for -: 'str' and 'int'
TypeError: can't multiply sequence by non-int of type 'float'
KeyError: 'percent'
TypeError: 'NoneType' object is not subscriptable
E   Failing test case: test_discount_never_increases_total_or_goes_negative(
E       total=Decimal('0.01'),
E       pct=Decimal('-50.00'),
E   AssertionError: page=0 returned [] instead of raising
E   Failing test case: test_page_below_one_is_rejected(
E       items=[0],
E       page=0,
FAIL Required test coverage of 90% not reached. Total coverage: 77.78%
FAILED tests/test_api.py::test_customer_filter_is_not_injectable - AssertionE...
FAILED tests/test_api.py::test_page_query_param_is_parsed - assert 500 == 200
FAILED tests/test_api.py::test_bad_page_is_a_client_error - assert 500 == 400
FAILED tests/test_api.py::test_discount_over_100_percent_is_rejected - assert...
FAILED tests/test_api.py::test_non_numeric_percent_is_rejected - assert 500 =...
FAILED tests/test_api.py::test_missing_percent_is_rejected - assert 500 == 400
FAILED tests/test_api.py::test_unknown_order_is_404 - assert 500 == 404
FAILED tests/test_properties.py::test_discount_never_increases_total_or_goes_negative
FAILED tests/test_properties.py::test_page_below_one_is_rejected - AssertionE...
9 failed, 4 passed, 4 warnings in 1.09s
# (output filtered to the relevant lines)

Nine failures, each a real defect:

• x' OR '1'='1 returned all three customers’ orders: SQL injection, confirmed end to end, not just flagged.
• page as a string crashes the endpoint, the same bug mypy found.
• A 150% discount returns 200 OK and a negative total. Hypothesis found the mirror case on its own: a total of 0.01 with a -50% “discount” becomes 0.02, so the discount increased the price.
• A non-numeric percent, a missing key and an unknown order id all return 500 instead of 400 or 404.
• paginate(items, 0, 10) quietly returns an empty list instead of rejecting the input.

The output also contained ResourceWarning: unclosed database: the vibe-coded get_db() opens a new SQLite connection on every request and never closes it.

Coverage before and after: 44% with the assistant’s two tests, 78% with the guardrail tests against the vibe-coded module, 96% after the fixes, against a 90% gate

Step 7. The fix, and a green run

The fixed module keeps the same features. What changed:

• Parameterized SQL (WHERE customer = ?); the Python sqlite3 docs say to always use placeholders instead of string formatting. Plus a length limit on customer.
• Input validation at the edge: parse_page() and parse_percent() turn bad input into 400 responses, and a missing order becomes 404.
• Money as Decimal with explicit rounding, and apply_discount() rejects percentages outside 0 to 100.
• No secrets in code: config comes from environment variables through an app factory.
• The webhook is signed with HMAC-SHA256 instead of sending the secret itself in a header, it has a 5-second timeout, certificate verification stays on, and failures are logged.
• Connections are closed in teardown_appcontext, and the app no longer starts with debug=True.
• Dead code removed, and requests upgraded to 2.34.2.

Full fixed module: after/shop/app.py

"""Orders API after the guardrail pass."""

import hashlib
import hmac
import json
import logging
import os
import sqlite3
from decimal import ROUND_HALF_UP, Decimal, InvalidOperation
from typing import Any

import requests
from flask import Flask, Response, abort, current_app, g, jsonify, request

log = logging.getLogger(__name__)

CENT = Decimal("0.01")
PER_PAGE = 20
MAX_CUSTOMER_LEN = 100


def apply_discount(total: Decimal, percent: Decimal) -> Decimal:
    if not total.is_finite() or total < 0:
        raise ValueError("total must be a non-negative amount")
    if not percent.is_finite() or not 0 <= percent <= 100:
        raise ValueError("percent must be between 0 and 100")
    return (total - total * percent / 100).quantize(CENT, rounding=ROUND_HALF_UP)


def paginate[T](items: list[T], page: int, per_page: int) -> list[T]:
    if page < 1 or per_page < 1:
        raise ValueError("page and per_page must be >= 1")
    start = (page - 1) * per_page
    return items[start : start + per_page]


def get_db() -> sqlite3.Connection:
    if "db" not in g:
        g.db = sqlite3.connect(current_app.config["DATABASE"])
        g.db.row_factory = sqlite3.Row
    conn: sqlite3.Connection = g.db
    return conn


def close_db(_exc: BaseException | None = None) -> None:
    conn = g.pop("db", None)
    if conn is not None:
        conn.close()


def parse_page(raw: str | None) -> int:
    if raw is None:
        return 1
    if not raw.isdigit() or int(raw) < 1:
        abort(400, description="page must be a positive integer")
    return int(raw)


def parse_percent(body: Any) -> Decimal:
    value = body.get("percent") if isinstance(body, dict) else None
    if isinstance(value, bool) or not isinstance(value, int | float | str):
        abort(400, description="percent must be a number")
    try:
        percent = Decimal(str(value))
    except InvalidOperation:
        abort(400, description="percent must be a number")
    if not percent.is_finite() or not 0 <= percent <= 100:
        abort(400, description="percent must be between 0 and 100")
    return percent


def list_orders() -> Response:
    customer = request.args.get("customer", "")
    if not customer or len(customer) > MAX_CUSTOMER_LEN:
        abort(400, description="customer is required (max 100 chars)")
    page = parse_page(request.args.get("page"))
    rows = (
        get_db()
        .execute(
            "SELECT id, customer, total, status FROM orders WHERE customer = ? ORDER BY id",
            (customer,),
        )
        .fetchall()
    )
    return jsonify(paginate([dict(r) for r in rows], page, PER_PAGE))


def discount(order_id: int) -> Response:
    percent = parse_percent(request.get_json(silent=True))
    db = get_db()
    row = db.execute("SELECT total FROM orders WHERE id = ?", (order_id,)).fetchone()
    if row is None:
        abort(404, description="order not found")
    new_total = apply_discount(Decimal(str(row["total"])), percent)
    db.execute("UPDATE orders SET total = ? WHERE id = ?", (str(new_total), order_id))
    db.commit()
    notify(order_id, new_total)
    return jsonify({"id": order_id, "total": str(new_total)})


def notify(order_id: int, total: Decimal) -> None:
    url = current_app.config["WEBHOOK_URL"]
    secret = current_app.config["WEBHOOK_SECRET"]
    if not url or not secret:
        return
    body = json.dumps({"id": order_id, "total": str(total)}).encode()
    signature = hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()
    try:
        requests.post(
            url,
            data=body,
            headers={"Content-Type": "application/json", "X-Signature-256": signature},
            timeout=5,
        )
    except requests.RequestException:
        log.warning("webhook delivery failed for order %s", order_id, exc_info=True)


def create_app(config: dict[str, Any] | None = None) -> Flask:
    app = Flask(__name__)
    app.config.update(
        DATABASE=os.environ.get("SHOP_DB", "shop.db"),
        WEBHOOK_URL=os.environ.get("SHOP_WEBHOOK_URL", ""),
        WEBHOOK_SECRET=os.environ.get("SHOP_WEBHOOK_SECRET", ""),
    )
    app.config.update(config or {})
    app.teardown_appcontext(close_db)
    app.add_url_rule("/orders", view_func=list_orders)
    app.add_url_rule("/orders/<int:order_id>/discount", view_func=discount, methods=["POST"])
    return app

A third test file covers the webhook: the signature is a valid HMAC, the secret never appears in headers, there is a timeout, and a network error is logged instead of raised.

Webhook tests: after/tests/test_webhook.py

import hashlib
import hmac
from decimal import Decimal
from typing import Any

import pytest
import requests

from shop import app as shop_app


def test_webhook_is_signed_and_has_timeout(monkeypatch: pytest.MonkeyPatch) -> None:
    sent: dict[str, Any] = {}

    def fake_post(url: str, **kwargs: Any) -> None:
        sent.update(url=url, **kwargs)

    monkeypatch.setattr(shop_app.requests, "post", fake_post)
    app = shop_app.create_app({"WEBHOOK_URL": "hooks.test/orders", "WEBHOOK_SECRET": "s3cret"})
    with app.app_context():
        shop_app.notify(7, Decimal("90.00"))
    expected = hmac.new(b"s3cret", sent["data"], hashlib.sha256).hexdigest()
    assert sent["headers"]["X-Signature-256"] == expected
    assert "s3cret" not in str(sent["headers"])
    assert sent["timeout"] == 5
    assert "verify" not in sent


def test_webhook_failure_is_logged_not_raised(
    monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
) -> None:
    def boom(url: str, **kwargs: Any) -> None:
        raise requests.ConnectionError("down")

    monkeypatch.setattr(shop_app.requests, "post", boom)
    app = shop_app.create_app({"WEBHOOK_URL": "hooks.test/orders", "WEBHOOK_SECRET": "s"})
    with app.app_context():
        shop_app.notify(7, Decimal("1.00"))
    assert "webhook delivery failed for order 7" in caplog.text


def test_webhook_skipped_without_config() -> None:
    app = shop_app.create_app({"WEBHOOK_URL": "", "WEBHOOK_SECRET": ""})
    with app.app_context():
        shop_app.notify(7, Decimal("1.00"))

The same test files, run against the fixed module:

$ pytest -q -p no:cacheprovider --tb=short
..............                                                           [100%]
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.18-final-0 _______________

Name               Stmts   Miss Branch BrPart  Cover   Missing
--------------------------------------------------------------
shop/__init__.py       0      0      0      0   100%
shop/app.py           89      2     24      3    96%   24, 38->41, 75
--------------------------------------------------------------
TOTAL                 89      2     24      3    96%
Required test coverage of 90% reached. Total coverage: 95.58%
14 passed in 0.72s
[exit 0]

And the static tools:

$ ruff check --output-format concise .
All checks passed!
[exit 0]
$ mypy
Success: no issues found in 2 source files
[exit 0]
$ bandit -q -c pyproject.toml -r shop -f custom --msg-template {relpath}:{line}: {test_id}[{severity}/{confidence}] {msg}
[exit 0]
$ detect-secrets-hook shop/app.py
[exit 0]
$ vulture shop vulture_whitelist.py --min-confidence 60
[exit 0]
$ pip-audit -r requirements.txt --aliases
No known vulnerabilities found
[exit 0]

Step 8. Lock it in: pre-commit and GitHub Actions

Guardrails only work if nobody has to remember to run them. Fast checks go into pre-commit; the test suite runs on git push; CI reruns everything because hooks can be skipped with --no-verify.

# Every hook runs the tool pinned in requirements-dev.txt, so commit-time
# checks and CI use exactly the same versions. Install: pre-commit install
default_install_hook_types: [pre-commit, pre-push]
repos:
  - repo: local
    hooks:
      - id: ruff-check
        name: ruff check
        entry: ruff check --force-exclude
        language: unsupported
        types: [python]
      - id: ruff-format
        name: ruff format
        entry: ruff format --check --force-exclude
        language: unsupported
        types: [python]
      - id: bandit
        name: bandit
        entry: bandit -c pyproject.toml -q
        language: unsupported
        types: [python]
        files: ^shop/
      - id: detect-secrets
        name: detect-secrets
        entry: detect-secrets-hook --baseline .secrets.baseline
        language: unsupported
        exclude: ^\.secrets\.baseline$
      - id: vulture
        name: vulture
        entry: vulture shop vulture_whitelist.py --min-confidence 60
        language: unsupported
        pass_filenames: false
        types: [python]
      - id: mypy
        name: mypy --strict
        entry: mypy
        language: unsupported
        pass_filenames: false
        types: [python]
      - id: pip-audit
        name: pip-audit
        entry: pip-audit -r requirements.txt
        language: unsupported
        pass_filenames: false
        files: ^requirements.*\.txt$
      - id: pytest
        name: pytest + hypothesis + coverage gate
        entry: pytest -q
        language: unsupported
        pass_filenames: false
        stages: [pre-push]

Three choices here are deliberate:

• Every hook is repo: local and runs the tool from the project venv. I started with the upstream hook repositories, then noticed that the mypy mirror’s newest tag was v2.4.0 while PyPI already had mypy 2.4.1. With local hooks, requirements-dev.txt is the single source of versions for laptops and CI, and mypy and pytest can see your installed dependencies.
• language: unsupported is the current name for what used to be language: system; the pre-commit docs say the old alias will be removed.
• default_install_hook_types makes a plain pre-commit install set up both the commit and the push hooks. --force-exclude keeps ruff from linting files you excluded, because pre-commit passes file names explicitly.

The CI workflow:

name: guardrails

on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read

jobs:
  checks:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-python@v6
        with:
          python-version: "3.13"
          cache: pip
          cache-dependency-path: requirements-dev.txt
      - run: pip install -r requirements-dev.txt

      - name: Lint and format
        run: |
          ruff check --output-format github .
          ruff format --check .
      - name: Types
        run: mypy
      - name: Security lint
        run: bandit -c pyproject.toml -r shop
      - name: Secrets
        run: detect-secrets-hook --baseline .secrets.baseline $(git ls-files)
      - name: Dead code
        run: vulture shop vulture_whitelist.py --min-confidence 60
      - name: Dependency audit
        run: pip-audit -r requirements.txt --strict
      - name: Tests, properties, coverage gate
        run: pytest -q --hypothesis-profile=ci

The CI job loads a Hypothesis profile with 1,000 examples per property instead of the default 100, registered in tests/conftest.py. I validated both configs and then ran them:

$ pre-commit validate-config .pre-commit-config.yaml
[exit 0]
$ actionlint .github/workflows/guardrails.yml
[exit 0]
$ check-jsonschema --builtin-schema vendor.github-workflows .github/workflows/guardrails.yml
ok -- validation done
[exit 0]
$ pre-commit install
pre-commit installed at .git/hooks/pre-commit
pre-commit installed at .git/hooks/pre-push
$ pre-commit run --all-files
ruff check...............................................................Passed
ruff format..............................................................Passed
bandit...................................................................Passed
detect-secrets...........................................................Passed
vulture..................................................................Passed
mypy --strict............................................................Passed
pip-audit................................................................Passed
[exit 0]
$ pre-commit run --all-files --hook-stage pre-push pytest
pytest + hypothesis + coverage gate......................................Passed
[exit 0]

actionlint and the GitHub workflow JSON schema both accept the workflow. To check the run: steps themselves, a small script (ci_local.py in the repo) executes each step of the workflow locally:

$ python ci_local.py
Lint and format                    exit=0  7 files already formatted
Types                              exit=0  Success: no issues found in 2 source files
Security lint                      exit=0  Files skipped (0):
Secrets                            exit=0  
Dead code                          exit=0  
Dependency audit                   exit=0  No known vulnerabilities found
Tests, properties, coverage gate   exit=0  14 passed in 5.50s
[exit 0]

Finally, the real test. Here is the kind of change an assistant might add in a follow-up prompt (“add a customer lookup helper”), committed on top of the clean repo:

STRIPE_KEY = "sk_live_51NzQ8vKp2ExampleKey9fT3"


def find_customer(name: str) -> list[sqlite3.Row]:
    return get_db().execute(f"SELECT * FROM orders WHERE customer = '{name}'").fetchall()
$ git commit -m "add customer lookup"
ruff check...............................................................Passed
ruff format..............................................................Passed
bandit...................................................................Failed
>> Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector through string-based query construction.
detect-secrets...........................................................Failed
Secret Type: Base64 High Entropy String
Secret Type: Stripe Access Key
vulture..................................................................Failed
shop/app.py:133: unused variable 'STRIPE_KEY' (60% confidence)
shop/app.py:136: unused function 'find_customer' (60% confidence)
mypy --strict............................................................Passed
pip-audit............................................(no files to check)Skipped
[exit 1]
# (output filtered to the relevant lines)

The commit was rejected by three different tools for three different reasons. Notice that ruff and mypy passed: the code is well-formatted and correctly typed, and still dangerous.

What the pipeline caught, and what it did not

Matrix of 16 issues in the vibe-coded module against seven tools: which tool caught each issue, and three issues that no tool flagged

Three problems in the original module were not flagged by any tool:

1. The webhook “signature” was the secret itself. headers={"X-Signature": WEBHOOK_SECRET} sends the shared secret to the receiver, and to any proxy that logs headers, on every request. bandit flagged the hardcoded string, not the protocol design.
2. There is no authentication on the discount endpoint. Anyone who can reach POST /orders//discount can set a price. No static tool knows that this route needs auth; neither did my fixed version, which is out of scope here and needs a human decision.
3. Money stored as REAL. I switched the arithmetic to Decimal, but the column is still a float. In production, store integer cents.

These are context problems: the model has no idea what your business rules are, and neither do linters. The best use of AI-generated code review tools is to automate the mechanical checks, so that human review time goes to the decisions only a person can make.

Gotchas I hit while building this

• with sqlite3.connect(...) does not close the connection. The context manager only commits or rolls back. My first test fixture leaked connections until I used contextlib.closing(), which is what the Python docs recommend.
• Flask returns 415, not 400, for a non-JSON body. My first “missing body” test expected 400 and failed against the vibe code for the wrong reason. Since Werkzeug 2.3, get_json() raises 415 Unsupported Media Type; use silent=True if you want to handle it yourself.
• ruff flagged the vulture whitelist (F821 Undefined name '_'), because a whitelist is just bare expressions. Excluding it takes extend-exclude plus force-exclude = true: pre-commit passes file names explicitly, and ruff lints explicit paths even when they are excluded unless force-exclude is on.
• Committed pycache made the pytest hook fail with “files were modified by this hook”. Add a .gitignore before the first pre-commit run --all-files.
• mypy strict on untyped code looks like pure noise. Add signatures first; the real errors only appear after that.

A checklist for vibe coding tools in a real repo

• Treat AI output like a pull request from a stranger: it goes through the same hooks and the same CI.
• Turn on mypy --strict and require annotations, so the type checker can see the code at all.
• Run a secret scanner in addition to bandit; name-based rules miss API keys.
• Audit dependencies in CI with --strict, and check that every suggested package actually exists before you install it.
• Write property-based tests for anything with arithmetic, pagination, parsing or money.
• Keep a coverage gate, but do not trust coverage alone: it measures what ran, not what was checked.
• Reserve human review for authentication, authorization, data model and protocol design.

Wrapping up

Is vibe coding bad? On this module, the AI-style code ran, demoed fine, and passed its own tests while hiding SQL injection, a leaked API key, a vulnerable dependency, a crash on a common query parameter and a discount that could raise prices. A pipeline that runs on every commit caught almost all of it. The parts it missed were design decisions, which is where your attention should go anyway.

What does your team run on AI-written code before it merges? I would like to hear which checks caught something real for you, and which ones turned out to be noise.