DEV Community: Rohit Mittal

Catch LLM Schema Drift Before It Breaks Production

Rohit Mittal — Mon, 22 Jun 2026 10:08:53 +0000

Your AI Returns a 200 OK. That Doesn't Mean It's Right.

A problem I kept hitting while building developer tools, and what I learned trying to solve it.

A few months ago I started noticing something strange in a feature I'd built. The flow was simple: send some text to an LLM, ask for a structured JSON response, use that response in the app. It had been working fine for weeks. Then, with no code changes on my end, a field that used to always be a number started showing up as a string. Nothing crashed. No error in the logs. The API call returned 200 OK every single time. The response just quietly stopped looking the way my code expected it to.

It took me longer than I'd like to admit to figure out what was happening, mostly because I was looking in the wrong place. I kept checking my own code, assuming I'd introduced a bug. I hadn't. The model's output had simply drifted.

Why this is different from a normal API breaking

If you've worked with REST APIs for any length of time, you have a mental model for how things break. A provider deprecates a field, they put it in a changelog, you get a few months' notice, you migrate. Annoying, but predictable. There's a contract, and when the contract changes, there's usually a paper trail.

LLMs don't work like that, and I don't think enough people building with them have fully internalized why.

When you call an AI API and ask for JSON, you're not getting a structured response in the way a traditional API returns one. You're getting text that the model generated, which happens to look like JSON because you asked it to. There's no schema enforcement on the provider's end unless you've gone out of your way to add it yourself. The model is doing its best to satisfy your prompt, and "its best" can shift for reasons you have zero visibility into: a model version update, a subtle change in how it interprets ambiguous input, even just natural variance in generation.

I'd ask the same prompt with the same model and get a slightly different shape back depending on the day. A field would get renamed. An array would occasionally come back as a single object when there was only one item, instead of a list with one item. A model would decide to wrap its JSON in a markdown code fence one time and not the next. None of this is the model being "wrong," exactly. It's just not playing the same enforcement role as a versioned API contract does.

The part that actually worried me

Here's the thing that changed how seriously I took this. The scariest version of this problem isn't a prompt change you made yourself, because at least you'd remember making it and could investigate. The scariest version is when nothing on your end changed at all, and the output still drifted, because the model provider updated something behind the scenes.

You ship a feature. It works. Weeks go by. You're not actively looking at it because it's "done." And then one day a field your downstream code depends on just isn't there in the format you expect, and you find out from a user, or from an error several layers removed from the actual cause, by which point you're debugging backwards trying to figure out when this started and why.

For a feature you control end to end, that's a bad afternoon. For something with real consequences attached, the cost of that gap between "the output changed" and "I noticed the output changed" is the whole problem.

What I ended up building

The fix, once I framed it correctly, was a familiar pattern borrowed from a different problem. If you've ever set up monitoring for a third-party API you depend on, you already know the shape of this solution: define what a healthy response looks like, check periodically that reality still matches the definition, get told immediately when it doesn't.

So that's what I built. You define a JSON Schema describing the shape your AI response should always have, you give it the actual prompt and provider you use in production, and it runs that prompt on a schedule, checking the real output against your schema. The moment something drifts, you get an alert, not a support ticket from a confused user three weeks later.

I also added a version of this that runs synchronously inside a CI/CD pipeline, because there's a second failure mode worth catching separately: a developer (often me) tweaking a prompt to "sound more natural" and accidentally breaking the structure the application depends on. Scheduled monitoring catches drift you didn't cause. A CI check catches drift you're about to cause, before it ships.

What I'd tell someone building with LLM APIs today

If you're returning structured output from an AI model and feeding it directly into application logic, the single most useful thing you can do is stop assuming a 200 response means a correct response. Those are two different claims, and the gap between them is exactly where this kind of bug lives.

You don't need anything elaborate to start. Even just logging the shape of what comes back and eyeballing it occasionally is better than nothing. But if the feature matters enough that a silent structural change would actually hurt — a customer-facing flow, anything billing-adjacent, anything where wrong data propagates somewhere consequential — it's worth treating the AI's output with the same seriousness you'd treat any other external dependency you don't fully control.

I ended up building this into Fixzi, the developer tools product I work on, because it was a problem I genuinely needed solved and couldn't find a simple version of anywhere else. If you're hitting the same thing, I'd be curious to hear how you're currently catching it — or if you're not catching it at all yet, which I suspect is more common than people want to admit.

How to Detect API Breaking Changes Before They Break Your App

Rohit Mittal — Mon, 01 Jun 2026 12:22:34 +0000

What Are API Breaking Changes?

API breaking changes happen when an API response changes in a way that existing applications can no longer handle. These changes can silently break frontend apps, mobile apps, dashboards, and third-party integrations even if the API still returns a successful status code.

Common Types of API Breaking Changes

Missing fields in the response
Renamed keys (e.g. name → full_name)
Changed data types (number → string)
New nested structure
Unexpected error response format

Example of a Breaking Change

Before:

{
  "user": {
    "id": 101,
    "name": "Rohit",
    "plan": "pro"
  }
}

After:

{
  "user": {
    "id": "101",
    "full_name": "Rohit",
    "subscription": "pro"
  }
}

The API still works, but your app may break because keys and types changed.

Why Traditional Monitoring Fails

Uptime monitoring only checks if an API endpoint is reachable. It does not verify whether the response is correct. This means your API can return HTTP 200 and still break your application.

Step 1: Validate API Response
Always validate the API response structure before using it. Check for missing fields, invalid JSON, and incorrect types.

Try: API Response Validation Tool

Step 2: Compare Old vs New Response
Use a response diff tool to compare previous and current API responses. This helps identify structural changes instantly.

Try: Compare API Responses

Step 3: Use API Contract Monitoring
The best way to detect breaking changes automatically is to define an expected response and monitor it over time.

Save expected response
Check API regularly
Compare live vs expected
Alert on mismatch

Try: API Contract Monitoring Tool

Best Practices

Never assume API response is stable
Validate responses before using them
Use JSON diff for debugging
Use schema validation for strict contracts
Monitor APIs in production

Conclusion

API breaking changes are common and often silent. By combining validation, response diff, and contract monitoring, you can detect issues early and protect your application from unexpected failures.

Stop Writing JSON Schemas Manually — Generate Them Instantly from JSON

Rohit Mittal — Tue, 19 May 2026 06:05:26 +0000

Writing JSON schemas manually is one of those tasks developers hate but still have to do.

You copy an API response, start defining types, add required fields, handle nested objects… and before you know it, you’ve spent 20–30 minutes on something that should have taken 2.

What if you could generate a complete JSON schema instantly from your data?

The Problem with Manual JSON Schema Creation

Let’s say you have this API response:

{
  "id": 101,
  "email": "john@example.com",
  "is_active": true,
  "created_at": "2026-01-01T12:00:00Z"
}

Now you need to write a schema like:

{
  "type": "object",
  "properties": {
    "id": { "type": "number" },
    "email": { "type": "string" },
    "is_active": { "type": "boolean" },
    "created_at": { "type": "string", "format": "date-time" }
  },
  "required": ["id", "email", "is_active", "created_at"]
}

Not hard… but:
Time-consuming
Error-prone
Annoying for nested structures

💡 The Smarter Approach

Instead of writing schemas manually, you can:

👉 Paste your JSON
👉 Generate schema automatically
👉 Use it instantly for validation or documentation

🛠️ Try It Yourself

You can generate JSON schema instantly using this tool:

👉 https://fixzi.ai/json-schema-generator

Just paste your JSON and it will:

Detect data types automatically
Build a full schema structure
Handle nested objects and arrays

🔍 What Makes This Useful?

1. Instant type detection
Numbers, strings, booleans — all inferred automatically.

2. Works with nested JSON
Even deeply nested APIs can be converted in seconds.

3. Saves development time
What used to take 20 minutes now takes 5 seconds.

🔗 Bonus: Validate Your Data

Once you generate the schema, you can validate your JSON here:

👉 https://fixzi.ai/json-schema-validator

This helps ensure:

API responses match expected structure
No breaking changes go unnoticed
⚠️ Common Mistake Developers Make

Many developers:

Skip schema creation entirely
Or write incomplete schemas

This leads to:

Production bugs
Broken API integrations
Hard-to-debug issues

👉 Schema = contract
And contracts should not be guessed.

🚀 Pro Tip

If you're working with APIs regularly:

Generate schema from response
Save it
Use it for validation

This creates a simple API contract system without heavy tools.

🧩 Where This Fits in Your Workflow
Backend dev → validate responses
Frontend dev → ensure data structure
QA → test API consistency
Automation → enforce contracts
🎯 Final Thoughts

JSON schema is powerful — but writing it manually is not.

If you’re still doing it by hand, you’re wasting time.

👉 Generate it
👉 Validate it
👉 Move faster

If you’ve been doing this manually till now, try automating it once — you won’t go back.

5 Common JSON Errors That Break APIs (and How to Fix Them)

Rohit Mittal — Wed, 13 May 2026 10:00:41 +0000

5 Common JSON Errors That Break APIs (and How to Fix Them)

If you’ve worked with APIs for even a short time, you’ve probably run into this:

“Why is this JSON not working?!”

Everything looks fine… until your parser throws an error and your app breaks.

The truth is, most JSON issues are small but deadly — a missing comma, a trailing bracket, or an incorrect structure can completely break your API flow.

In this post, I’ll walk through the 5 most common JSON errors developers face, along with simple fixes and examples.

1. Missing Commas Between Fields

This is probably the most common issue.

❌ Invalid JSON:

{
  "name": "John"
  "age": 30
}

✅ Fixed JSON:

{
  "name": "John",
  "age": 30
}

Why it happens:
When manually editing JSON, it’s easy to forget commas between key-value pairs.

2. Trailing Commas

Some languages allow trailing commas — JSON does NOT.

❌ Invalid JSON:

{
  "name": "John",
  "age": 30,
}

✅ Fixed JSON:

{
  "name": "John",
  "age": 30
}

Tip:
If you're copying JSON from JavaScript objects, watch out for this.

3. Unquoted Keys or Strings

JSON requires double quotes for keys and string values.

❌ Invalid JSON:

{
  name: "John",
  age: 30
}

✅ Fixed JSON:

{
  "name": "John",
  "age": 30
}

4. Incorrect Data Types

Sometimes values are accidentally wrapped as strings instead of numbers or booleans.

❌ Problematic JSON:

{
  "isActive": "true",
  "count": "10"
}

✅ Better JSON:

{
  "isActive": true,
  "count": 10
}

Why it matters:
Your backend or frontend logic may behave incorrectly if types are wrong.

5. Unclosed Brackets or Braces

This is another classic.

❌ Invalid JSON:

{
  "user": {
    "name": "John"

✅ Fixed JSON:

{
  "user": {
    "name": "John"
  }
}

How to Fix JSON Faster

Manually debugging JSON is frustrating — especially with large API responses.

A better approach is to use a tool that can:

Validate JSON instantly
Highlight errors
Format and beautify the structure
Fix common issues automatically

I’ve been using a simple tool called Fixzi for this:

👉 https://fixzi.ai/json-validator

It helps quickly:

Validate JSON
Fix invalid structures
Format and clean large payloads
Compare JSON responses (useful for APIs)

Final Thoughts

JSON errors are small, but they can cause big production issues — especially when working with APIs, webhooks, or third-party integrations.

If you’re debugging JSON frequently:

Always validate before using it
Use proper formatting tools
Avoid manual edits when possible

If you’ve faced other tricky JSON issues, I’d love to hear them 👇