I built a changelog generator that diffs two OpenAPI specs and tells you exactly what broke between versions. The tempting shortcut was to hand the diff to an LLM and let it write human-readable release notes. I tried it. It was charming — and it hallucinated a breaking change that didn't exist, because the LLM guessed at semantics from a field name. In a changelog, a hallucinated breaking change is worse than no changelog: someone's deploy automation trusts it.
So I stripped the LLM out of the critical path. The generator now computes a deterministic structural diff — paths, methods, schemas, required fields — and only then formats the result. No model in the loop means no invented changes, and the output is reproducible: same two specs in, same changelog out, every time. I kept the machine-readable version alongside the human one, because agents consuming release notes want JSON, not prose.
The lesson I keep re-learning as I ship tools in this space: cleverness is a liability when the output feeds decisions. If your pipeline produces something that triggers deploys, alerts, or billing, put boring deterministic machinery in the path and save the generative model for the layer where a mistake is cosmetic. I ended up packaging it as https://x402.freeq.one/tools/changelog_openapi.html, and the same pattern now runs through my git-commit changelog generator too.
Top comments (0)