I built a deterministic OpenAPI diff tool because I was tired of LLM-written release notes that hallucinated breaking changes. Feed it a base spec and a new spec, get back a structured changelog. No model in the loop, no vibes — just a tree walk and a diff.
The first version was naive. I compared paths, operations, schemas. If a property was added to a response schema, I flagged it as non-breaking. If a required request field appeared, I flagged it as breaking. Clean, defensible, wrong.
Then a user wired it into their CI and their release pipeline started crying wolf. Turns out "added a required field with a default value" is breaking in spec-land but non-breaking in reality-land — most HTTP clients tolerate defaults fine. And "renamed an enum value" is technically the same type but every downstream consumer that string-matches is now on fire.
So I had to encode judgment. Not AI judgment — deterministic judgment. The tool now classifies changes into tiers: definitely breaking (removed path, changed type, new required field without default), probably breaking (renamed enum value, tightened validation pattern, removed response property), additive (new optional field, new path, new operation). Each tier carries a human-readable explanation of why it's classified that way, so a human can override without reading the diff.
The lesson: a spec diff isn't a git diff. Git diffs bytes. OpenAPI diffs contracts, and contracts have semantics that live in the minds of the humans who wrote the clients. My tool can't read minds. But it can be honest about its uncertainty — and that turns out to be the feature people actually want. "Here's what I'm sure about, here's what I'm guessing about, here's the command to tell me I'm wrong."
If you're versioning an API and writing changelogs by hand, the deterministic part is freeing. You stop arguing about whether something is breaking and start arguing about whether your classification rules match your actual consumers. That's a much better argument.
I packaged it as https://x402.freeq.one/tools/changelog_openapi.html — give it two spec URLs and see what it thinks changed.
Top comments (2)
The three-tier classification is the smart part of this design. "Probably breaking" admits the honest middle ground most compatibility checkers refuse to model, which is exactly where renamed enum values and tightened patterns live. One follow-up: do the human-readable explanations mention the consumer behavior that could break, or just the spec-side reason for the classification?
The tier model's real test surface isn't the spec semantics, it's the classification rules themselves, and those will be edited every time a consumer complaint comes in. Pin them with a golden corpus: a fixed set of spec pairs whose tier assignments are checked into the repo. Every rule edit then shows up as a deliberate diff in expected output - "we decided renamed enums are probably-breaking, not definitely" - instead of silent drift where a tightened pattern quietly slides from probably-breaking to additive and nobody notices until a consumer does. The second fixture I'd want is for rename detection specifically, because rename versus delete-plus-add is a pairing heuristic, and heuristics have failure modes: rename one enum value AND add a genuinely new one in the same diff. Does the tool pair the rename correctly, or does it report a removal and two additions? That case decides whether "probably breaking" stays calibrated or fires on every enum touch, which is how a CI gate goes back to crying wolf.