DEV Community

Deepak Satyam
Deepak Satyam

Posted on Edited on

OpenAPI contract testing in 2026: oasdiff vs Spectral vs PactFlow (and what I built)

Updated November 2026: PactFlow pricing corrected (it now has a free Starter plan), and a new section on catching security regressions in the same gate. The opening example is now framed as what it always was: an illustration, not a specific incident.

Here's a change that passes every test: rename a response field from customer_id to customerId so it matches the rest of the codebase. Unit tests pass, because they test your code. Review approves it, because it's a tidy-up. And any client still reading customer_id breaks the moment you deploy.

Nothing in the pipeline was broken. What broke was the contract between the API and someone else — and in most CI setups, nothing is looking at that.

I've spent the past year building tooling around this problem. This is where I think each of the existing tools actually wins.

Disclosure before we start: I'm the author of SpecShield, one of the tools below. I'll be straight about where the free alternatives beat it, because for a lot of teams they will. If all you need is what oasdiff does, use oasdiff — it's excellent and it costs nothing.


"Contract testing" is four different jobs

Most of the confusion in this space comes from one phrase covering four unrelated problems. Once you separate them, tool choice gets obvious.

Job 1 — Did this change break anything at all?
Diff two OpenAPI specs, classify what changed, flag what's backwards-incompatible. Removed endpoints, narrowed types, newly-required request fields.

Job 2 — Is this spec any good?
Linting. Missing descriptions, inconsistent casing, no operationId, auth described in ways that will bite you later.

Job 3 — Will this break a specific consumer I know about?
Different question entirely. Removing an endpoint nobody calls is free. Removing one that three teams depend on is an outage. This needs knowledge of who consumes what.

Job 4 — Should this deploy be blocked?
The organisational one. Someone has to decide that a finding stops a merge, and someone has to be able to override it without turning the whole check off.

Most tools do one or two of these well. Nobody does all four, including me.


Job 1: breaking-change detection — oasdiff wins

oasdiff is open source, has several hundred breaking-change checks, ships a CLI and a GitHub Action, and has a hosted web diff where you paste two specs.

oasdiff breaking spec-v1.yaml spec-v2.yaml
Enter fullscreen mode Exit fullscreen mode

It's the reference implementation. If your requirement is "tell me when a PR breaks the spec," install it this afternoon and skip the rest of this post.

I'll say the uncomfortable part plainly, since it's the honest framing: breaking-change detection is commoditised. It's table stakes. Nobody should pay for it as a standalone feature, mine included.


Job 2: linting — Spectral and vacuum

Spectral is the de facto standard, and vacuum is a much faster Go implementation that reads the same ruleset format.

extends: [[spectral:oas, recommended]]
rules:
  operation-operationId: error
  operation-description: warn
Enter fullscreen mode Exit fullscreen mode

Both free. Both good. Stoplight, Apigee API hub and Azure API Center all bundle Spectral under the hood, so if you're on one of those you already have this.

The gap isn't the linter. It's what happens to the output. spectral lint prints a wall of warnings, someone puts it in CI, it fails on 200 findings the first day, and by week three it's || true. It's one of the most common ways a lint step dies.


Job 3: consumer-aware verification — PactFlow, and this is where money lives

Pact and PactFlow solve the real version. Consumers publish what they actually use, providers publish what they offer, and a broker tells you whether a given pair is compatible.

can-i-deploy is the piece worth stealing conceptually:

pact-broker can-i-deploy --pacticipant payments-api \
  --version abc123 --to-environment production
Enter fullscreen mode Exit fullscreen mode

That's not "did the spec change." It's "will this specific build break a consumer that is live right now."

PactFlow is the mature product here. There's a free Starter plan; the Team plan is around $115–127/month depending on billing. If you want full consumer-driven contract testing with the Pact DSL and runtime verification, they go deeper than anything spec-only can — including mine. Spec-to-spec comparison can't tell you a consumer reads a field it never declared. Runtime pact verification can.

Worth knowing: Optic was archived on 12 January 2026 (Optic Labs is now part of Atlassian). If you're running it, it's unmaintained.


Job 4: the gate — the part I think is underserved

Every tool above produces findings. Very few help you turn findings into a decision that survives contact with a deadline.

This is the gap I built SpecShield for, so weigh what follows accordingly.

A score, not a wall of warnings

Governance findings get weighted by severity into a 0–100 score and an A–F grade:

specshield govern openapi.yaml --min-score 80
Enter fullscreen mode Exit fullscreen mode

Any error-severity finding caps the grade at C, on the theory that an API with an unresolved error is not an "A" API no matter how many nice descriptions it has. A threshold is something a team can agree on. "Zero warnings" is something a team agrees on once and then abandons.

Three rule packs ship built in:

Pack Rules For
recommended 20 Design consistency — operationId, descriptions, kebab-case paths, camelCase properties, array and string bounds
owasp 4 HTTPS-only servers, security field defined, no credentials in query strings, no HTTP Basic
minimal 3 The highest-signal rules only, for adding a gate to a legacy spec without drowning

Waivers, so the gate survives

This is the feature I'd argue matters most, and it's the least exciting to demo.

When a gate blocks something at 6pm on a Friday, the team has two options: fix it, or disable the check. Everyone disables the check. Then it never comes back on.

So findings can be waived at the org level, with the waiver recorded and revocable:

specshield govern openapi.yaml --org acme --min-score 80
Enter fullscreen mode Exit fullscreen mode

The waiver applies to the verdict, the rule stays on for everyone else, and there's a record of who accepted what. The gate stays credible because there's a pressure valve that isn't "delete the CI step."

Rulesets that aren't a lock-in trap

Custom rulesets are Spectral-format YAML. There's a portability validator that actively rejects engine-specific constructs — vacuum-only extends, vendor rule-id prefixes, engine-specific functions.

That's deliberate. A ruleset you write should run somewhere else if you leave. Governance rules encode institutional knowledge, and holding that hostage is a bad way to keep customers.

Output CI already understands

specshield govern openapi.yaml --output report.sarif
Enter fullscreen mode Exit fullscreen mode

SARIF, JUnit, HTML and JSON. SARIF matters most — GitHub ingests it natively, so findings show up as annotations on the diff instead of buried in log output nobody opens.

The deploy gate

specshield bdct publish-provider --spec openapi.yaml --env production
specshield bdct can-i-deploy --provider payments-api --env production
Enter fullscreen mode Exit fullscreen mode

Consumers register the contract they depend on. Before a deploy, you ask whether it's safe. Same shape as PactFlow's, built on OpenAPI rather than the Pact DSL — lighter to adopt, less powerful at runtime. That's a genuine trade, not a free win.

Where it runs

The same engine is exposed through a CLI, a GitHub Action, a GitHub App that posts a check run and a sticky PR comment, an IntelliJ plugin, and an MCP server so coding agents can run the gate before opening a PR.

One thing worth stating clearly: it's fully deterministic. No LLM anywhere in the analysis. The same spec always produces the same score, and every deducted point traces to a named rule. In a market currently bolting language models onto linters, "explainable and reproducible" felt like the more useful property for something that blocks deploys.


New since this post: security regressions in the same gate

A contract can break safely — every client keeps working — and still get worse. An endpoint loses its security requirement. A bearer token becomes an API key in the query string. A response starts returning client_secret. None of that is a breaking change, so a breaking-change diff stays green.

So the gate now has a security mode:

specshield gate openapi.yaml --security
Enter fullscreen mode Exit fullscreen mode

It blocks a PR when authentication is removed or weakened, a required scope is dropped, a new endpoint ships without auth while the rest of the API has it, or credentials and regulated data appear in a response. Lower-risk changes (a new public endpoint marked security: [], an email field on an authenticated endpoint) are reported for review instead of blocking. Waivers work the same way as everything else in the gate: owner, reason, expiry.

And for the API as it is today, rather than the diff:

specshield audit openapi.yaml --logs /var/log/nginx/access.log*.gz --sarif audit.sarif
Enter fullscreen mode Exit fullscreen mode

That checks the spec against the OWASP API Security Top 10 and compares real traffic with it: endpoints answering that the spec doesn't document (shadow APIs), old /v1 routes still serving after /v2 shipped, and files like /.env returning 200. It sends no requests — it reads the spec and the logs you already have.

To be clear about the limits: this reads what the spec declares and what the logs recorded. It isn't a DAST scanner and doesn't test what the running service actually enforces. Tools like ZAP or StackHawk do that, and they're complementary.


The comparison, honestly

Breaking changes Linting Consumer-aware Deploy gate Waivers Cost
oasdiff ✅ Best in class ❌ ❌ Partial ❌ Free
Spectral / vacuum ❌ ✅ Standard ❌ ❌ ❌ Free
PactFlow ✅ Partial ✅ Deepest ✅ Partial Free Starter; Team ~$115/mo+
Bump.sh ✅ Partial ❌ ❌ ❌ ~$50/mo+
Optic ✅ ✅ ❌ ❌ ❌ Archived
SpecShield ✅ ✅ ✅ ✅ ✅ Free tier

Reading that table fairly:

  • oasdiff beats me at raw breaking-change detection. More checks, longer track record, larger community.
  • PactFlow beats me at consumer-driven testing. Runtime verification catches things spec comparison structurally cannot.
  • Spectral has a far bigger ruleset ecosystem than my three packs.
  • Apigee and Azure API Center aren't competitors at all — they're runtime API management. Different layer, and complementary.

Where I think the combination earns its place: one gate that does breaking changes and governance, with a threshold and a waiver workflow, wired into the PR rather than a separate dashboard.

If you don't need the gate, the free tools are genuinely enough. I'd rather say that than have you find out in week two.


What I'd actually recommend

Just starting? oasdiff in CI. Today. Takes twenty minutes and catches the majority of what hurts.

Spec quality drifting across teams? Add Spectral or vacuum. Start with three rules, not thirty.

Real consumers, real runtime coupling, budget? PactFlow. It's the mature answer.

Want one gate, a score you can threshold, and a waiver trail? That's the gap I built for — specshield.io, free tier. CLI is npm i -g specshield (npm); add --security to the gate if you also want auth regressions blocked; there's also an IntelliJ plugin if you'd rather see it in the editor.

Still on Optic? It's been archived since January. Please migrate to something.


The lesson of the customerId rename isn't that you need more tests. You probably have plenty. It's that nothing in the pipeline knows a contract exists, or who is on the other end of it.

Whichever tool you pick, pick one that does.

What are you using for this? Genuinely curious whether people are gating on contracts yet or still finding out in production.

Top comments (0)