Updated November 2026: PactFlow pricing corrected (it now has a free Starter plan), and a new section on catching security regressions in the same gate. The opening example is now framed as what it always was: an illustration, not a specific incident.
Here's a change that passes every test: rename a response field from customer_id to customerId so it matches the rest of the codebase. Unit tests pass, because they test your code. Review approves it, because it's a tidy-up. And any client still reading customer_id breaks the moment you deploy.
Nothing in the pipeline was broken. What broke was the contract between the API and someone else — and in most CI setups, nothing is looking at that.
I've spent the past year building tooling around this problem. This is where I think each of the existing tools actually wins.
Disclosure before we start: I'm the author of SpecShield, one of the tools below. I'll be straight about where the free alternatives beat it, because for a lot of teams they will. If all you need is what oasdiff does, use oasdiff — it's excellent and it costs nothing.
"Contract testing" is four different jobs
Most of the confusion in this space comes from one phrase covering four unrelated problems. Once you separate them, tool choice gets obvious.
Job 1 — Did this change break anything at all?
Diff two OpenAPI specs, classify what changed, flag what's backwards-incompatible. Removed endpoints, narrowed types, newly-required request fields.
Job 2 — Is this spec any good?
Linting. Missing descriptions, inconsistent casing, no operationId, auth described in ways that will bite you later.
Job 3 — Will this break a specific consumer I know about?
Different question entirely. Removing an endpoint nobody calls is free. Removing one that three teams depend on is an outage. This needs knowledge of who consumes what.
Job 4 — Should this deploy be blocked?
The organisational one. Someone has to decide that a finding stops a merge, and someone has to be able to override it without turning the whole check off.
Most tools do one or two of these well. Nobody does all four, including me.
Job 1: breaking-change detection — oasdiff wins
oasdiff is open source, has several hundred breaking-change checks, ships a CLI and a GitHub Action, and has a hosted web diff where you paste two specs.
oasdiff breaking spec-v1.yaml spec-v2.yaml
It's the reference implementation. If your requirement is "tell me when a PR breaks the spec," install it this afternoon and skip the rest of this post.
I'll say the uncomfortable part plainly, since it's the honest framing: breaking-change detection is commoditised. It's table stakes. Nobody should pay for it as a standalone feature, mine included.
Job 2: linting — Spectral and vacuum
Spectral is the de facto standard, and vacuum is a much faster Go implementation that reads the same ruleset format.
extends: [[spectral:oas, recommended]]
rules:
operation-operationId: error
operation-description: warn
Both free. Both good. Stoplight, Apigee API hub and Azure API Center all bundle Spectral under the hood, so if you're on one of those you already have this.
The gap isn't the linter. It's what happens to the output. spectral lint prints a wall of warnings, someone puts it in CI, it fails on 200 findings the first day, and by week three it's || true. It's one of the most common ways a lint step dies.
Job 3: consumer-aware verification — PactFlow, and this is where money lives
Pact and PactFlow solve the real version. Consumers publish what they actually use, providers publish what they offer, and a broker tells you whether a given pair is compatible.
can-i-deploy is the piece worth stealing conceptually:
pact-broker can-i-deploy --pacticipant payments-api \
--version abc123 --to-environment production
That's not "did the spec change." It's "will this specific build break a consumer that is live right now."
PactFlow is the mature product here. There's a free Starter plan; the Team plan is around $115–127/month depending on billing. If you want full consumer-driven contract testing with the Pact DSL and runtime verification, they go deeper than anything spec-only can — including mine. Spec-to-spec comparison can't tell you a consumer reads a field it never declared. Runtime pact verification can.
Worth knowing: Optic was archived on 12 January 2026 (Optic Labs is now part of Atlassian). If you're running it, it's unmaintained.
Job 4: the gate — the part I think is underserved
Every tool above produces findings. Very few help you turn findings into a decision that survives contact with a deadline.
This is the gap I built SpecShield for, so weigh what follows accordingly.
A score, not a wall of warnings
Governance findings get weighted by severity into a 0–100 score and an A–F grade:
specshield govern openapi.yaml --min-score 80
Any error-severity finding caps the grade at C, on the theory that an API with an unresolved error is not an "A" API no matter how many nice descriptions it has. A threshold is something a team can agree on. "Zero warnings" is something a team agrees on once and then abandons.
Three rule packs ship built in:
| Pack | Rules | For |
|---|---|---|
recommended |
20 | Design consistency — operationId, descriptions, kebab-case paths, camelCase properties, array and string bounds |
owasp |
4 | HTTPS-only servers, security field defined, no credentials in query strings, no HTTP Basic |
minimal |
3 | The highest-signal rules only, for adding a gate to a legacy spec without drowning |
Waivers, so the gate survives
This is the feature I'd argue matters most, and it's the least exciting to demo.
When a gate blocks something at 6pm on a Friday, the team has two options: fix it, or disable the check. Everyone disables the check. Then it never comes back on.
So findings can be waived at the org level, with the waiver recorded and revocable:
specshield govern openapi.yaml --org acme --min-score 80
The waiver applies to the verdict, the rule stays on for everyone else, and there's a record of who accepted what. The gate stays credible because there's a pressure valve that isn't "delete the CI step."
Rulesets that aren't a lock-in trap
Custom rulesets are Spectral-format YAML. There's a portability validator that actively rejects engine-specific constructs — vacuum-only extends, vendor rule-id prefixes, engine-specific functions.
That's deliberate. A ruleset you write should run somewhere else if you leave. Governance rules encode institutional knowledge, and holding that hostage is a bad way to keep customers.
Output CI already understands
specshield govern openapi.yaml --output report.sarif
SARIF, JUnit, HTML and JSON. SARIF matters most — GitHub ingests it natively, so findings show up as annotations on the diff instead of buried in log output nobody opens.
The deploy gate
specshield bdct publish-provider --spec openapi.yaml --env production
specshield bdct can-i-deploy --provider payments-api --env production
Consumers register the contract they depend on. Before a deploy, you ask whether it's safe. Same shape as PactFlow's, built on OpenAPI rather than the Pact DSL — lighter to adopt, less powerful at runtime. That's a genuine trade, not a free win.
Where it runs
The same engine is exposed through a CLI, a GitHub Action, a GitHub App that posts a check run and a sticky PR comment, an IntelliJ plugin, and an MCP server so coding agents can run the gate before opening a PR.
One thing worth stating clearly: it's fully deterministic. No LLM anywhere in the analysis. The same spec always produces the same score, and every deducted point traces to a named rule. In a market currently bolting language models onto linters, "explainable and reproducible" felt like the more useful property for something that blocks deploys.
New since this post: security regressions in the same gate
A contract can break safely — every client keeps working — and still get worse. An endpoint loses its security requirement. A bearer token becomes an API key in the query string. A response starts returning client_secret. None of that is a breaking change, so a breaking-change diff stays green.
So the gate now has a security mode:
specshield gate openapi.yaml --security
It blocks a PR when authentication is removed or weakened, a required scope is dropped, a new endpoint ships without auth while the rest of the API has it, or credentials and regulated data appear in a response. Lower-risk changes (a new public endpoint marked security: [], an email field on an authenticated endpoint) are reported for review instead of blocking. Waivers work the same way as everything else in the gate: owner, reason, expiry.
And for the API as it is today, rather than the diff:
specshield audit openapi.yaml --logs /var/log/nginx/access.log*.gz --sarif audit.sarif
That checks the spec against the OWASP API Security Top 10 and compares real traffic with it: endpoints answering that the spec doesn't document (shadow APIs), old /v1 routes still serving after /v2 shipped, and files like /.env returning 200. It sends no requests — it reads the spec and the logs you already have.
To be clear about the limits: this reads what the spec declares and what the logs recorded. It isn't a DAST scanner and doesn't test what the running service actually enforces. Tools like ZAP or StackHawk do that, and they're complementary.
The comparison, honestly
| Breaking changes | Linting | Consumer-aware | Deploy gate | Waivers | Cost | |
|---|---|---|---|---|---|---|
| oasdiff | ✅ Best in class | ❌ | ❌ | Partial | ❌ | Free |
| Spectral / vacuum | ❌ | ✅ Standard | ❌ | ❌ | ❌ | Free |
| PactFlow | ✅ | Partial | ✅ Deepest | ✅ | Partial | Free Starter; Team ~$115/mo+ |
| Bump.sh | ✅ | Partial | ❌ | ❌ | ❌ | ~$50/mo+ |
| Optic | ✅ | ✅ | ❌ | ❌ | ❌ | Archived |
| SpecShield | ✅ | ✅ | ✅ | ✅ | ✅ | Free tier |
Reading that table fairly:
- oasdiff beats me at raw breaking-change detection. More checks, longer track record, larger community.
- PactFlow beats me at consumer-driven testing. Runtime verification catches things spec comparison structurally cannot.
- Spectral has a far bigger ruleset ecosystem than my three packs.
- Apigee and Azure API Center aren't competitors at all — they're runtime API management. Different layer, and complementary.
Where I think the combination earns its place: one gate that does breaking changes and governance, with a threshold and a waiver workflow, wired into the PR rather than a separate dashboard.
If you don't need the gate, the free tools are genuinely enough. I'd rather say that than have you find out in week two.
What I'd actually recommend
Just starting? oasdiff in CI. Today. Takes twenty minutes and catches the majority of what hurts.
Spec quality drifting across teams? Add Spectral or vacuum. Start with three rules, not thirty.
Real consumers, real runtime coupling, budget? PactFlow. It's the mature answer.
Want one gate, a score you can threshold, and a waiver trail? That's the gap I built for — specshield.io, free tier. CLI is npm i -g specshield (npm); add --security to the gate if you also want auth regressions blocked; there's also an IntelliJ plugin if you'd rather see it in the editor.
Still on Optic? It's been archived since January. Please migrate to something.
The lesson of the customerId rename isn't that you need more tests. You probably have plenty. It's that nothing in the pipeline knows a contract exists, or who is on the other end of it.
Whichever tool you pick, pick one that does.
What are you using for this? Genuinely curious whether people are gating on contracts yet or still finding out in production.
Top comments (0)