"Should we use Pact, or just test against the OpenAPI spec?" comes up in almost every team that starts on contract testing. The two approaches get described as rivals, but they answer different questions, and each one is blind to something the other sees.
This post compares them through concrete failures: what each approach catches, what it misses, and what it costs to run. Where I quote limitations, they come from the tools' own documentation.
(Disclosure: I build SpecShield, which does schema-based and bi-directional contract testing. I've tried to keep this useful whichever way you go — and for some teams the right answer is Pact.)
The two approaches in one minute
Consumer-driven contract testing (Pact). Each consumer writes tests against a Pact mock. Those tests record exactly what the consumer sends and which response fields it reads. That recording — the pact — is published to a broker and then replayed against the real provider in the provider's CI. The contract is "what consumers actually use," and it's verified against running code.
Schema-based contract testing. The OpenAPI spec is the contract. You check that changes to the spec don't break it (a spec diff), and separately that the implementation matches the spec (conformance testing). The contract is "what the provider says it offers."
There's also a middle path, bi-directional contract testing, which I'll cover after the comparison.
Six failures, and which approach catches them
1. The provider removes a response field that a consumer reads.
- Pact: caught. The pact says the consumer reads that field, and provider verification fails.
- Schema diff: caught. Removing a response field is a breaking change.
Both work. This is the case every demo shows.
2. The provider removes a response field that nobody reads.
- Pact: passes, correctly. No pact mentions the field.
- Schema diff: fails. It can't know nobody reads it, so every removal looks breaking.
This is Pact's real advantage, and it's bigger than it sounds: a schema-only gate that cries wolf on every removal is how teams end up turning the gate off. (Schema-based tools can close part of this gap with traffic data — more on that below.)
3. The spec says a field exists, but the implementation stopped returning it.
- Pact: caught. Verification runs against the real provider.
- Schema diff: missed. The spec didn't change, so there's nothing to diff.
- Conformance testing (running requests against the provider and checking responses against the spec) is what catches this on the schema side. Without it, schema-based contract testing is only as good as the spec's accuracy.
4. The consumer is a third party you've never met.
- Pact: can't help. Pact's own docs list public APIs under what it is not good for: consumers can't be identified, and provider states would have to be set up through the API under test.
- Schema-based: works. The spec is the promise to everyone, known or not.
5. The response has the right shape but the wrong data (the order total ignores the discount).
- Neither catches it. Pact checks the request and response format, and its docs say functional testing of the provider belongs in the provider's own tests. PactFlow's own comparison table rates both consumer-driven and bi-directional testing poorly on "testing data semantics."
6. An edge-case input makes the provider return a 500.
- Pact: only if a consumer test happens to send that input.
- Schema-based: only with a test generator. Tools like Schemathesis generate inputs from the spec, including edge cases and invalid ones, and report server errors and responses that don't match the schema.
Summary
| Failure | Pact | Spec diff | Spec diff + conformance |
|---|---|---|---|
| 1. Removed field a consumer reads | ✅ | ✅ | ✅ |
| 2. Removed field nobody reads | ✅ passes | ❌ false alarm | ❌ false alarm |
| 3. Implementation drifts from spec | ✅ | ❌ | ✅ |
| 4. Unknown / third-party consumers | ❌ | ✅ | ✅ |
| 5. Wrong data, right shape | ❌ | ❌ | ❌ |
| 6. 500 on edge-case input | rarely | ❌ | ✅ with a generator |
What each one costs
Pact asks a lot of both sides. Every consumer team writes Pact tests in its own language. Every provider team sets up provider states (the data each interaction needs) and runs verification in CI. You need a broker: the open-source Pact Broker you host, or PactFlow. And Pact's docs are explicit that it fits best where you control both sides, with a small number of consumers per provider — and that it's a poor fit when the other team won't also use Pact.
When that holds — internal microservices, teams that will all adopt it — Pact gives the strongest guarantees of anything here.
Schema-based testing asks for a spec you trust. If the spec is generated from code or kept honest by conformance tests, it's cheap: one file, no consumer-side work, and tools that run in a single CI step. If the spec is hand-written and nobody checks it against the implementation, failure 3 will eventually hurt you.
The middle path: bi-directional contract testing
PactFlow introduced bi-directional contract testing (BDCT) for teams where full Pact is a poor fit:
- The provider publishes its OpenAPI spec, plus evidence that the implementation matches it: results from the provider's own verification tool (Postman, Dredd, Schemathesis, SmartBear's Drift…), uploaded with
--verification-results. - Consumers publish what they use, either as a pact generated from their existing mocks or as an OpenAPI subset.
- The broker checks statically that every consumer expectation is satisfied by the provider spec.
The consumer contract is never replayed against the provider's code, which is what makes it cheaper. PactFlow is candid about the trade-off. Its comparison table rates BDCT weaker on guarantees than consumer-driven testing (+ vs +++) and better on team decoupling (+++ vs +), and its summary is "weaker guarantees than CDC but decouples teams and can be used with third parties/unknown consumers." The docs also say BDCT isn't meant to replace Pact, but to be an alternative where Pact isn't suitable.
In terms of the six failures above, BDCT behaves like a schema diff that knows which fields each consumer uses. It gets failure 2 right, and failure 3 is only as good as the provider's conformance evidence.
The tools, briefly
| Tool | Approach | License / cost | Notes |
|---|---|---|---|
| Pact + Pact Broker | Consumer-driven | Open source (self-hosted broker) | Libraries for most languages; spec v4 |
| PactFlow | Consumer-driven + bi-directional | Free Starter; Team ~$115–127/mo | SmartBear; hosted broker |
| oasdiff | Spec diff | Apache-2.0 | Deepest breaking-change catalog |
| Schemathesis | Conformance + generated tests | MIT | Property-based (Hypothesis); OpenAPI, GraphQL, gRPC |
| Specmatic | Spec as executable contract + stubs | MIT core | OpenAPI, AsyncAPI, GraphQL, gRPC, WSDL |
| Prism | Mock server + validation proxy | Apache-2.0 | Validate live traffic against the spec |
| Dredd | Conformance | MIT | Archived — don't start new projects on it |
| SpecShield (mine) | Spec diff + bi-directional | Free tier; Team $89/mo | Consumer contracts from OpenAPI, Pact JSON or a HAR capture |
Closing the "nobody reads it" gap without Pact
Failure 2 is the main reason teams pick Pact over a spec diff. There's another way to learn which fields and endpoints are actually used: your access logs already record who calls what.
That's the approach I took in SpecShield: import the logs you already have, and a breaking change on an endpoint no client has called in the window stops failing the build, while the rest show which clients they'd break. It's weaker than a pact in one important way — logs show which endpoints and query parameters are used, not which response fields a client reads — so it narrows false alarms rather than removing them. If you need field-level certainty, that's what Pact is for.
For consumers who won't write Pact tests, SpecShield can also turn a HAR capture from tests they already run (Playwright, Cypress, a browser) into a consumer contract, which is then checked bi-directionally. Same trade-off as any BDCT: lighter to adopt, weaker than replaying against the real provider.
How I'd choose
- Internal microservices, a handful of consumers per provider, and every team willing to write consumer tests: Pact. Strongest guarantees, and the cost is justified.
- Public or partner API, or consumers you can't make write tests: schema-based. A spec diff in CI, plus conformance testing (Schemathesis or similar) so the spec stays true.
- Somewhere in between, or Pact adoption stalled after one team: bi-directional. PactFlow is the mature option; SpecShield is a lighter, OpenAPI-native one.
- In every case: add a conformance check. A lot of "contract" failures are really failure 3: a spec everyone trusts that no longer matches the code.
Which failure has actually bitten your team? I'd like to know whether failure 2 (false alarms) or failure 3 (spec drift) is the bigger problem in practice.
Top comments (0)