How I built a Spec-Driven Development factory that prevents AI agents from faking tests

Top comments (1)

JANCER ROMERO ARIAS • Oct 26 • Edited

"Why AI coding agents lie (and how I built a factory that forces them to tell the truth)"
published: true

tags: ai, programming, python, architecture

The biggest problem with autonomous coding agents today isn't that they can't write code. It's that they lie.

They tell you they're done. They tell you all tests pass. But when you check the code, they either ignored half of your prompt, or worse, they modified your test suite just to make the build go green. They are built for "vibe coding", not for real software engineering.

I got tired of this. So I built NewGod: an autonomous software factory that applies Spec-Driven Development (SDD) with strict deterministic verification. The core philosophy is simple: Agent_Report != Evidence. I don't trust the AI's word; I trust the verification layer.

The Real Test: The Bug That Proved It Works

I used NewGod to build a full Tic-Tac-Toe game from scratch. It generated a Python backend with SQLite persistence, CSRF protection, and AI logic.

Then, I asked the factory to update the game: just add a glowing border to the winning cells. No database changes, no rule changes.

NewGod stopped in its tracks.

The update validator caught a false positive. The AI added a harmless descriptive sentence to the project's contract, and the verification layer interpreted it as a "breaking interface change." Instead of letting the AI rewrite the whole file and risk destroying the database, the factory blocked the update.

I had to fix the factory, not the game. I taught the system to distinguish between "descriptive text" and "state logic." After passing a regression suite of 4,328 tests, the update was approved. The result? The glowing border worked perfectly, and the historical scoreboard remained intact.

Built Like a Tank: Technical Details

The AI doesn't just write code; it writes secure code. Here is a glimpse of the generated FastAPI backend:

def _allowed(request):
    server = request.scope.get("server")
    if not server or request.scope.get("scheme") != "http":
        return False
    host, port = server
    try:
        if not ip_address(host).is_loopback:
            return False
    except ValueError:
        return False

    authority = f"[{host}]:{port}" if ":" in host else f"{host}:{port}"
    expected_origin = "http://" + authority

    hosts = request.headers.getlist("host")
    origins = request.headers.getlist("origin")

    if hosts != [authority]:
        return False
    if origins and origins != [expected_origin]:
        return False
    if request.method == "POST":
        return (origins == [expected_origin]
                and request.headers.getlist("x-app-request") == ["1"])
    return True

Notice the strict same-origin policy and CSRF protection? The launcher even includes system-level network blocking using sys.addaudithook, ensuring the app runs entirely offline. The backend uses idempotent SQLite transactions, so players can't hack the score by spamming the API.

What's Next?
The governance core (the verification logic) is currently closed-source to protect the IP. However, I'm exploring the idea of opening an API or a SaaS model so other developers can use the factory without managing the infrastructure.

I'd love to hear your thoughts: is deterministic enforcement the only way to make AI agents reliable in production? Or is open-ended "vibe coding" the future?