Your OpenAPI Spec Is Decoration Until CI Enforces It

My rule: if an OpenAPI spec is not wired into CI as an executable contract, it is decoration. A spec that no test asserts against will drift within two sprints.

Here's how I take a YAML file sitting in a repo and turn it into a gate that blocks pull requests when the provider stops matching the promise it made to its consumers.

---

The gap unit and integration tests keep missing

Unit tests pass. Integration tests pass. Then a mobile client crashes in production because the status field your service returns as "PAID" became "paid" after someone inlined an enum refactor.

No test caught it because no test was looking at the wire-level shape. That's the gap contract tests fill:

• Field drift — a rename, a type change, a nullability flip
• Response shape regression — a consumer depended on items[] always being present; now it's omitted when empty
• Error code surface — a 409 became a 422 because a library upgrade changed default behavior
• Header contracts — idempotency keys, pagination cursors, content types

None of those show up in a green unit suite.

Consumer-driven vs provider-driven: pick based on who calls you

Two schools, and I've shipped both.

Provider-driven tools (Schemathesis, Dredd) take your OpenAPI spec and pound the running server with generated requests.

Consumer-driven tools (Pact) flip it: each consumer writes tests against a mock of the provider, publishes the interactions to a broker, and the provider verifies those interactions against its real server on every build.

My heuristic:

• You own all the consumers (one mobile app, one web app, one internal worker)? Go consumer-driven with Pact. You get proof of exactly the subset of the contract your consumers actually use — nearly always a small slice of the full surface.
• Public API with unknown consumers? Go provider-driven with Schemathesis. Anyone could be depending on any field, so you validate the full declared contract.
• Internal platform team? Usually both — Pact for known consumers, Schemathesis fuzzing for everything else.

Generate the tests from the spec — don't write them by hand

Hand-written contract tests rot. Generated ones stay honest because the spec is the source:

1. Schemathesis reads openapi.yaml and generates property-based cases: every endpoint, every status code branch, every schema boundary. One command, thousands of cases.
2. Prism stands up a mock server from the same spec so consumer tests run without a real backend — and it validates the consumer's requests against the spec too, so a malformed FE payload gets rejected at the mock.
3. Dredd pins the documented happy paths when the spec includes explicit examples: blocks.

Keep the generated suite in its own CI job named contract, separate from unit and integration, so a failure is immediately legible as "the wire shape moved."

Contract tests need a harness, or they get deleted

Contract tests are the fourth of five layers every backend test setup needs. If the layers underneath are missing, the contract job flakes — and a frustrated team deletes it.

Layer	Tooling	Effort	Build order
Fixtures + data factories	pytest fixtures, factory_boy	1–2 days	First
Mock servers	Prism, WireMock	2–3 days	Second
Contract test runner	Pact, Schemathesis	3–5 days	Third
Environment bootstrap	Docker Compose, CI scripts	1 sprint	Fourth

Build bottom-up: each layer depends on the one below, and each returns value before the next exists.

Flaky fixtures are the real enemy

Every contract-test flake I have ever debugged traced back to one of four things, and all four have boring fixes:

• Non-deterministic IDs. Seed the test DB with known UUIDs. Never assert equality against a generated id — assert it matches ^[0-9a-f-]{36}$ via a schema matcher.
• Time. Freeze the clock. Every created_at assertion should be a type check, not a value check.
• Ordering. Collections need a stable sort from the provider. If your endpoint returns items in whatever order the DB felt like today, contract tests will catch it — in the bad, intermittent way.
• External calls. Contract tests verify your contract, not Stripe's. If a contract test reaches the internet, it is misdesigned.

The CI gate that actually changes behavior

On every provider PR, the contract job:

1. Boots the service against a test database with deterministic seeds
2. Runs Schemathesis against the running server using the committed spec
3. Pulls the latest pacts for this provider from the Pact Broker and verifies them
4. Fails the build with a PR comment naming the consumer and the broken interaction

A developer renaming total_cents to amount sees — inside their PR — that the mobile team's checkout flow depends on the old name. They fix it before merge. They do not find out next Tuesday from a Slack message.

The loop working in practice

The web team wanted a loyalty_tier field on the order confirmation page. They added it to their Pact test, published the pact, and pushed their PR (blocked behind a feature flag).

Next day, a backend engineer opened an unrelated PR on the provider. The contract job pulled the broker's pacts and failed: the web consumer expected loyalty_tier, the provider didn't return it yet.

The two teams found the dependency at PR time — not in staging, not in production. That's the entire value proposition in one anecdote.

---

Start here

If you have an OpenAPI file and zero contract tests, the first afternoon looks like this:

1. pip install schemathesis
2. schemathesis run openapi.yaml --url http://localhost:8000
3. Triage what fails (something will fail)
4. Add it as a CI job once it's green

That single job will catch the next silent enum rename before your consumers do.

---

I write about spec-first delivery, API contracts, and release engineering at spec-coding.dev. The full version of this guide — including schema-diff review before release and CI ownership — is at spec-coding.dev/blog/contract-testing-plan-from-openapi-to-ci.