Executable Contracts as Guardrails for AI-Generated Code

AYON ARYAN — Mon, 13 Jul 2026 05:23:37 +0000

Integrating Specmatic into my NL-to-SQL engine (DATABASE-MANAGER)

Submitted for the Specmatic Full Stack AI Engineering Intern challenge.

The problem nobody warns you about with AI coding agents

I build with AI coding agents every day — Claude Code, Cursor. They're incredible at velocity. But they have one quiet, expensive failure mode: they drift. You ask for a small change to an endpoint, and somewhere in the diff the agent renames a field, flips a status code, or drops a required parameter. The code still runs. The tests (if you have them) still pass. And then three layers downstream — in the frontend, in a consumer service — something breaks, and you spend an afternoon debugging an integration bug that was never a logic bug at all. It was a contract bug.

That's the exact problem Specmatic's challenge asked me to explore: can Spec-Driven Development and executable contracts improve AI-assisted software development? After integrating Specmatic into one of my projects, my answer is an emphatic yes — and the reason is that an executable contract is a guardrail for AI-generated code.

The project: DATABASE-MANAGER

DATABASE-MANAGER is a natural-language-to-SQL engine I built that works across 8 database engines (SQLite, MySQL, PostgreSQL, MSSQL, Oracle, MongoDB, Cassandra, Redis). It has a Flask /api blueprint consumed by a React frontend, and — importantly — a human-in-the-loop write-safety model: any write/schema command doesn't execute directly. The API returns { "needs_review": true, ... }, a human approves, and only then does /api/execute run it, with snapshot rollback via /api/undo.

So the project already had a philosophy: don't let the model run unchecked. Specmatic let me extend that exact philosophy from database writes to the API contract itself.

Step 1 — The contract becomes the source of truth

I wrote an OpenAPI contract (api_contract.yaml) describing the real API. The interesting part is the /api/command response, which is a union of three shapes — a READ result, a write staged for review, or an error:

/api/command:
  post:
    responses:
      "200":
        content:
          application/json:
            schema:
              oneOf:
                - $ref: "#/components/schemas/ReadResult"
                - $ref: "#/components/schemas/NeedsReview"   # { needs_review: true, sql, explanation, task }
                - $ref: "#/components/schemas/Error"

This contract is now the single thing both the backend and the frontend agree on.

Step 2 — Executable contract: testing the real API

With Specmatic, the contract isn't documentation that rots — it's executable. Specmatic auto-generates positive and negative requests from the spec and verifies the live API conforms:

docker run --rm --network host -v "$PWD:/specs" -w /specs \
  specmatic/specmatic:latest test --host localhost --port 5001

No hand-written test cases. The spec is the test suite.

Step 3 — Service virtualization: the frontend stops waiting

Specmatic can also be the API — a spec-conformant stub — so my React frontend can develop with no backend, no database, and no LLM keys running:

docker run --rm --network host -v "$PWD:/specs" -w /specs \
  specmatic/specmatic:latest stub --port 9000

Same contract, two uses: it tests the provider and mocks it for the consumer. That's contract-driven development.

Step 4 — The payoff: catching AI drift in seconds

Here's the demo that made it click for me. I asked an AI agent to "add a confidence score and rename needs_review to requiresReview." It did — silently changing the response shape. To a human reviewer skimming the diff, it looks harmless. To the React frontend, it's a broken integration.

Then I re-ran the contract test:

Specmatic caught it instantly and told me exactly what drifted. A bug that would have surfaced as a confusing frontend error became a precise, one-line test failure — before it ever left my machine.

What I learned

Building with LLMs taught me a mantra: it's 20% prompting and 80% guardrails. The interesting engineering is everything you put around the model to make it safe. Specmatic is that idea applied to AI-assisted development itself:

For humans: the contract is unambiguous truth; no more "wait, what does this endpoint return?"
For AI agents: the contract is a fence. The agent can move fast inside it; the moment it drifts outside, the executable contract fails loudly.

For a tool like DATABASE-MANAGER — where I already gate database writes behind human review — adding a contract guardrail at the API layer felt like the natural second half of the same idea. Two guardrails, one principle: don't let generated actions run unchecked.

That's how Spec-Driven Development improves AI-assisted software engineering. It turns the AI's biggest weakness — confident, silent drift — into the cheapest possible signal: a failing test.

Going deeper (round 2): examples, resiliency, and virtualizing the LLM

After the first pass I pushed three Specmatic techniques further:

1. Inline + external examples. Instead of letting Specmatic only check shapes, I added examples that pin values — so the contract test now verifies real cases: admin1→ADMIN, bad creds→Invalid credentials, and editor1→EDITOR (the last loaded as an external example file). One generic test became three value-checked ones, and the stub now returns lifelike data.

2. Schema-resiliency testing found a real bug. Turning on generative negative tests, Specmatic mutated the login request — empty body, username as null / number / boolean — and expected a 4xx. My API returned 200 for all of them: it had no input validation, silently treating a malformed body as a failed login. 14 negative tests failed. I added body validation (400 on a non-object body or non-string fields), documented the 400, and re-ran: 24/24 green. The resiliency suite turned an invisible robustness gap into a fixed, tested guarantee.

3. Virtualizing the LLM itself. Meridian's NL-to-SQL calls an LLM (Groq, which is OpenAI-compatible). Real tests would burn tokens on every CI run and tolerate non-determinism. So I wrote a contract for the LLM's /v1/chat/completions, ran specmatic stub, and pointed the app at it with a single env var.

How it's wired: core/llm_manager.py reads GROQ_API_URL (unset in prod → real Groq); in CI a step boots specmatic stub llm_contract.yaml, sets GROQ_API_URL to it, and runs the real NL-to-SQL path against the stub — so the AI path is offline, deterministic, and zero-token. It's a separate CI step, after the API contract tests, because it virtualizes an upstream dependency the app consumes, not the app's own API. The stub spec is a deliberately reduced version of the real provider API — every deviation (and why) is documented in LLM_CONTRACT_NOTES.md. The same tool that guards my API also virtualizes the dependency my API consumes — exactly the kind of guardrail an AI-native codebase needs.

All three run in CI on every push, and the contract test reports 100% coverage (generative negative tests cover the 400 that examples can't). Spec-Driven Development didn't just document my API — it found a bug, made my AI tests free, and became the project's source of truth.

Closing the loop: testing the LLM-calling endpoints (and the bugs that fell out)

The real test of "mock the LLM" is to run the contract suite against the endpoints that actually call the LLM (/api/command) with the provider stubbed — so the AI path is exercised, deterministically, for zero tokens. Those endpoints are behind a session cookie, which Specmatic's test mode can't drive, so I added a gated test-auth (a Bearer path enabled only when SPECMATIC_TEST is set, never in production) and pointed the app's LLM calls at the stub.

Running the full api_contract.yaml this way immediately earned its keep — it surfaced five real issues I'd never have found by reading the code:

/api/connections leaked an undocumented config object (DB path/host) the contract didn't model — caught as an "unknown property."
The /api/command error response was ambiguous in its oneOf (it carries sql/task alongside error, so it half-looked like a ReadResult). I gave errors their own CommandError schema.
/api/command returned HTTP 500 on a non-string command (None.strip() crashed) — a resiliency negative test caught it; I fixed it to a clean 400.
/api/execute mishandled malformed sql the same way — fixed to 400.
/api/execute had its body marked required, but the human-in-the-loop flow calls it with no body (run the last-reviewed SQL) — the contract was lying about the design; I corrected it.

Two of those (3, 4) were genuine server crashes on bad input. The whole point landed: mocking the LLM let the contract suite exercise the AI endpoints for free, and the resiliency tests turned "we'll find out in production" into "we found out in CI."

Making the pipeline legible: one job per concern, real HTML reports

The last round was about signal quality. Bundling "does it conform?" and "does it survive garbage input?" into a single green check hides which one broke when it does. So I split the CI by concern: for each app spec (contract_public.yaml, api_contract.yaml) there are now two independent jobs — a contract job (value-checked conformance against inline + external examples) and a resiliency job (generative negative/boundary tests). Two specs × two modes = four test jobs, plus a fifth that smoke-tests the LLM virtualization. A conformance regression and a robustness regression now light up as different red checks.

I also stopped hand-writing text summaries and let Specmatic emit the real thing: specmatic.yaml configures report.formatters: [text, html], so every job produces a proper HTML coverage report (build/reports/specmatic/test/html) — committed under reports/ and uploaded as a CI artifact. The numbers are honest per concern: contract_public contract-mode covers the 200s (50%) while its resiliency job reaches 100% by exercising the 400; api_contract sits at 43% → 64% because the authenticated-only paths and 401s aren't auto-driven by bearer test-mode — the report shows that gap rather than papering over it.

Finally, I broadened the external examples — the login contract now pins all three roles and the rejection path (admin1→ADMIN, editor1→EDITOR, viewer1→VIEWER, bad creds→Invalid credentials) as separate example files, so the contract job checks real values across the whole auth surface, not just one case.

The throughline across every round: the contract isn't documentation you write once — it's an executable artifact you keep sharpening, and each time you sharpen it, it hands you back a bug or a blind spot you didn't know you had.

Driving coverage to 100% — and what the gap was hiding

The per-concern reports were honest, and they exposed a real hole: my protected endpoints were only ~43–64% covered. Digging into Specmatic's coverage table, the pattern was clear — every endpoint's 200 was tested, but the 400s and especially the 401s were "not tested." The 401s were the interesting ones: my CI test-auth accepted any bearer token, and Specmatic always sends one in test mode, so the unauthenticated path was structurally unreachable. My own test shortcut was hiding a whole column of behavior.

The fix was to make auth example-driven instead of blanket: the test-auth gate now accepts exactly one token (the value of SPECMATIC_TEST). Then external examples do the rest — they carry the right token to exercise the authenticated 200/400 responses, and a wrong token (or none) to exercise the real 401. One authenticated run now covers both the authorized and the unauthorized response of every secured endpoint. With examples added for every 400 and 401, all four jobs — contract and resiliency, on both specs — report 100% coverage. The lesson that stuck: a convenient test backdoor can quietly suppress coverage; tightening it to be precise is what let the contract test the thing that actually matters (is auth enforced?).

Migrating the config to v3: wiring that matches the architecture

Finally I migrated specmatic.yaml from v2 to v3, which replaces v2's implicit provides/consumes with explicit service wiring. That turned out to describe this system almost perfectly: the systemUnderTest is the Meridian Data API (run type: test), and its one external dependency — the LLM provider — is declared as a service run in type: mock. The config now reads like the architecture diagram: test my API; virtualize the LLM it depends on. Reports moved under specmatic.governance.report (formats: [html, ctrf]). Six rounds in, the config isn't just settings — it's an honest, executable description of what this service is and what it leans on.

Actual coverage: making the app tell Specmatic what it really exposes

There's a difference between "every operation in the contract is tested" and "every operation is tested and actually exists in the running app." Specmatic can prove the second — but only if the app exposes its route table. Spring Boot apps get this free via /actuator/mappings; this is a Flask app, so I built the equivalent: a test-only /actuator/mappings endpoint that generates the Spring-Actuator JSON from Flask's own url_map. Point Specmatic at it (it auto-discovers /actuator), and the "cannot calculate actual coverage" warning disappears — the reports now show 100% Absolute Coverage, meaning each contracted endpoint is confirmed live in the app and exercised, not merely matched against a document. It also draws a clean line between the API the contract governs and the app's broader surface.

Dependency hygiene: install that just works, 3.11 through 3.14

A reviewer on Python 3.14 hit a wall: pymssql==2.3.4 has no wheel for 3.14 and fails to build. The fix taught a nice lesson about optional dependencies. Meridian speaks to eight database engines, but every driver is lazy-imported — the app and the entire test suite run on the bundled SQLite databases with none of them installed. So the heavy external drivers didn't belong in the core requirements.txt at all. I split them into requirements-optional.txt (loosely pinned so newer Pythons resolve a compatible wheel) and left the core install lean. Verified by importing the app with every driver blocked: 91 routes, zero errors. A dependency you only need for one optional path shouldn't be able to break everyone's install.

The last mile: getting the details exactly right

Three small things, each a good reminder that "works" and "configured correctly" aren't the same:

The actuator was read but not registered. My report kept showing the actuator as disabled even though actual coverage was computing. The tell was in the access log: Specmatic was hitting /actuator (my Spring-style root, which returns _links) but never following to /actuator/mappings. Pointing actuatorUrl directly at the mappings endpoint fixed it — the report now flips the actuator flag to enabled. A 200 was enough to silence the warning but not enough to actually ingest the routes.

Auth belongs in the config, not scattered across examples. Following Specmatic's security-schemes pattern, the bearer token is now declared once in specmatic.yaml under securitySchemes (overridable via SPECMATIC_BEARER_TOKEN), and the one-command test runner starts the app with the matching test-auth token. So both the authenticated paths and the 401s are exercised, and there's a single place that owns the credential — no more hunting through fixtures to see how auth works.

python -m pip, always. A reviewer on Python 3.14 hit an install-vs-run mismatch that traced back to bare pip resolving to a different interpreter than the app runs on. The dependency split already made 3.14 install cleanly (verified on 3.11–3.14); documenting python -m pip inside the venv closes the last gap. Small, but it's the difference between "it works on my machine" and "it works on yours."

Don't test through a backdoor

The sharpest feedback I got was also the most obvious in hindsight: your code shouldn't have a separate path for the test tool. I'd been authenticating Specmatic through a SPECMATIC_TEST-gated bypass — a branch that only existed for contract testing. That's a smell: you're no longer testing the thing you ship. So I removed it and made bearer-token auth a real, first-class feature of the API: alongside the web UI's session-cookie login, the API accepts a Bearer token (configured via API_BEARER_TOKEN, off by default) that any programmatic client can use — curl, a gateway, CI, or Specmatic. Specmatic just declares that token in securitySchemes and sends it like any other client. The auth path under test is now the same path real callers use, and the 401s are exercised by simply sending a wrong token. No backdoor, no drift between "tested" and "deployed."

Scope, documented. With the actuator now honestly reporting the app's 52 routes against a 6-endpoint contract, the report shows 46 "Missing in Spec." Rather than paper over that, I wrote CONTRACT_SCOPE.md: the six are the trust boundary external consumers depend on (contract-tested at 100%); the rest are the SPA's own feature endpoints, shipped in the same unit, with an explicit promotion path. A contract's job is to guard boundaries, and the gap is now a visible, justified backlog rather than a silent omission.

And it should just run. A reviewer hit port conflicts on the fixed test ports, and later found the setup easier to trust as plain, explicit commands than as a wrapper script. The README now documents the exact per-job commands directly, with every environment variable spelled out, so nothing about running the tests is hidden behind tooling. "Clone and run" shouldn't depend on nothing else living on port 5001, or on trusting a script you haven't read.

Code + integration: github.com/AYON-ARYAN/DATABASE-MANAGER (branches: main + react_build; see api_contract.yaml, SPECMATIC_INTEGRATION.md). Specademy course completed — certificate attached.

— Ayon Aryan

DEV Community: AYON ARYAN