DEV Community

Discussion on: Your API Tests Are Lying to You, The Schema Drift Problem Nobody Talks About

Collapse
 
rockgecko_dev profile image
Rockgecko

Why do you manually write the openAPI spec? If there's drift, how do you consume new endpoints in the client?

I just have the BE generate it automatically so it's never out of date.
My FE deployment pipelines fetch the latest spec, then generate the client code and compile. If there's a build error, output a diff of the schema changes so it's immediately clear what field or endpoint changed (eg int to string would fail, int to long would likely pass fine).
This works for native apps, Flutter, typescript Web projects etc.

You can do the same for third party services, whether they publish openAPI or an sdk.

Collapse
 
tanvi_mittal_9996387d280 profile image
tanvi Mittal AI and QA Leaders

Great setup, auto-generated specs with client codegen and build-time validation is genuinely one of the strongest workflows out there. If you have that running end-to-end, you're ahead of 90% of teams I've worked with.
That said, here's where the problem still lives even with that pipeline:

  1. Not everyone controls the backend. If you're consuming Stripe, Twilio, a partner API, or any third-party service that doesn't publish an OpenAPI spec (and many don't or publish one that lags behind reality), you have no spec to generate from. You're relying on their changelog, which... good luck.
  2. Auto-generated specs reflect code, not intent. If a developer adds null=True to a Django model field or changes a serializer, the auto-generated spec will update but nobody decided to make that a public contract change. The build might pass (nullable string is still a string) while every consumer that doesn't handle nulls breaks silently at runtime. The spec changed, the codegen passed, and you still have a production incident.
  3. Build errors only catch type incompatibilities, not structural drift. A new field appearing? Build passes. A field going from required to optional? Depends on the generator. An array that used to always have items now sometimes returns empty? Build passes. These are exactly the drifts that cause runtime bugs and they're invisible to compile-time checks.
  4. The "generate client + compile" approach assumes typed languages. Works beautifully for TypeScript, Flutter, Swift. Doesn't help Python, Ruby, or any dynamically typed consumer which is still a huge chunk of the ecosystem, especially on the QA/automation side. Your workflow is the right answer for the teams that can implement it. The problem I'm describing hits hardest at the teams that can't which in my experience is the majority. Different layers of the same problem. Would be curious though in your pipeline, when the build does fail on a schema diff, how do you handle the decision of "is this an intentional API change or an accidental one"? That triage step is where I see even good pipelines struggle