I'm working on Impact Gate, a warn-only PR check that tells you which consumers an API change can affect. Before claiming anything about it, I wanted a public set of cases anyone can run. So I wrote one: github.com/impact-gate/impact-gate-benchmarks.
What it is: 13 before/after OpenAPI specs on one small Orders API, each with a label (breaking or not) and two made-up consumers, one of which is affected or neither is. 10 breaking changes, 3 controls (a new optional field, a new endpoint, a renamed path parameter).
What I ran: oasdiff v1.33.0 with --fail-on ERR. It caught 8, missed 2, raised no false alarms. The misses: removing an optional response field, and adding an API key requirement to an operation. oasdiff reports both as info, which is a defensible choice for a spec diff. But a consumer that reads that field, or calls without a key, breaks all the same. That gap between spec-level breaking and consumer-level breaking is what I want to measure.
Limits, plainly: the cases are synthetic, I wrote the labels myself, and 13 cases proves nothing about real-world accuracy. Next: real-project cases and automated consumer scoring. If you disagree with a label, open an issue, I'd like to be wrong in public. To score your own tool: TOOL_CMD="your-tool" bash scripts/run.sh.
Top comments (1)
Does Impact Gate learn who the consumers are from real traffic, or from something they declare in the repo? From my experience building Voiden (open source, I work on it), the request files people keep next to their code are a pretty honest list of what they actually call. github.com/VoidenHQ/voiden