How I Built SchemaWatch: A CLI Tool That Catches Breaking API Changes Before Production

This is a submission for the Gemma 4 Challenge: Write About Gemma 4

The Problem I Kept Running Into

It was a regular sprint. Backend team updated the API. Frontend team wasn't notified. The field user.id changed from integer to string. Nobody caught it in code review.

Production broke at 3AM.

Sound familiar?

I kept running into this exact problem — not because anyone was careless, but because there was no simple, automated way to catch breaking API changes before they shipped.

Existing tools like oasdiff are powerful, but they felt heavy for what I needed. I just wanted one command that would tell me: "Hey, this change will break your clients."

So I built SchemaWatch.

---

What SchemaWatch Does

SchemaWatch compares two OpenAPI schema files and detects breaking changes — classified by severity.

pip install schemawatch
python -m schemawatch.cli old.yaml new.yaml

That's it. No config files. No setup. Just two YAML files and one command.

The output looks like this:

====================================
🚨 SchemaWatch Report
====================================

Breaking changes detected: 5

── CRITICAL (2)
🔴 Endpoint removed: /orders
🔴 Method removed: GET /users

── WARNING (3)
🟡 Response field removed: User.email
🟡 Field type changed: User.id integer -> string
🟡 Field became required: User.id

------------------------------------
Summary:
- Total changes: 5
- Critical: 2
- Warning:  3
- Info:     0
------------------------------------

---

Why Severity Levels Matter

One thing I noticed with other tools: they treat all changes equally. But a removed description field is not the same as a removed endpoint.

SchemaWatch classifies changes into three levels:

Severity	Examples
🔴 Critical	Endpoint removed, HTTP method removed, schema removed
🟡 Warning	Field removed, type changed, field became required
🔵 Info	Enum changed, array item type changed

In a CI/CD pipeline, this means you can decide: "Fail the build only on Critical changes, warn on everything else."

---

CI/CD Integration

SchemaWatch is designed to live inside your pipeline:

# .github/workflows/schemawatch.yml
- name: Check for breaking API changes
  run: python -m schemawatch.cli openapi_old.yaml openapi.yaml

• Exit code 1 → breaking changes detected ❌ — build fails
• Exit code 0 → no breaking changes ✅ — build passes

---

How I Built It

The core logic is straightforward:

1. Parser — load and validate both YAML files
2. Diff Engine — compare paths, methods, schemas, fields recursively
3. Severity Classifier — tag each change as critical/warning/info
4. CLI — format and display output (text, JSON, or Markdown)

The trickiest part was recursive nested object comparison. OpenAPI schemas can have deeply nested properties, and a type change three levels deep is just as breaking as one at the top level.

def compare_properties(schema_name, old_props, new_props, path=""):
    for field in old_fields & new_fields:
        # Recursive check for nested objects
        if old_field.get("type") == "object" and new_field.get("type") == "object":
            changes.extend(
                compare_properties(schema_name, old_nested, new_nested, path=f"{field}.")
            )

---

🤖 Taking It Further: AI-Powered Explanations with Gemma 4

Detecting changes is useful. But understanding them is even better.

I integrated Gemma 4 with SchemaWatch so it can automatically explain
why each change is dangerous and how to fix it.

Here's the pipeline:

1. SchemaWatch detects the breaking changes
2. The changes are sent to Gemma 4 (via Google AI Studio API)
3. Gemma 4 explains each one in plain English

from google import genai
from dotenv import load_dotenv
import os

load_dotenv()
client = genai.Client(api_key=os.getenv("GEMINI_API_KEY"))

response = client.models.generate_content(
    model="gemma-4-26b-a4b-it",
    contents=prompt
)

The output from Gemma 4:

[CRITICAL] Endpoint removed: /orders
Any client attempting to access this URL will receive a 404 Not Found.
Fix: Use API versioning — move to /v2/orders and provide a deprecation period.

Full code: gemma_explainer.py

What's Next

SchemaWatch is actively developed. On the roadmap:

• [ ] Request body change detection
• [ ] Response status code comparison
• [ ] Custom GitHub Action (Marketplace)
• [ ] .schemawatch-ignore file for intentional breaking changes

---

Try It

pip install schemawatch
python -m schemawatch.cli old.yaml new.yaml

GitHub: https://github.com/CemCelik79/schemawatch

If you've ever been burned by an undocumented API change, give it a try. And if you have feedback — especially on what breaking change patterns SchemaWatch doesn't catch yet — I'd love to hear it in the comments.

---

Built with Python, colorama, and a lot of 3AM production incidents as motivation.

Comments

[Gilder Miller]

This addresses a real pain point. The 3AM production break scenario is more common than most teams admit, and existing tools often require significant setup for what should be a simple diff operation.
The recursive nested object comparison is the right approach. OpenAPI schemas get deep quickly, and a type change three levels down breaks clients just as hard as a top-level field change. The severity classification also maps cleanly to CI/CD decisions.

One technical question on the Gemma 4 integration. How do you handle API latency or failures in a CI/CD pipeline context? If the explanation step hangs or times out, does the build fail, or do you fall back to raw output? A blocking external API call in the critical path could introduce its own failure mode, especially at scale.

[CemCelik]

Great question! Currently the Gemma 4 explanation step is optional and separate
from the main SchemaWatch CLI. The core tool (schemawatch old.yaml new.yaml)
works independently with no external API calls — so CI/CD pipelines are never
blocked by AI latency.

The gemma_explainer.py is an experimental add-on. For production use in CI,
I'd recommend running SchemaWatch first (exit code 0/1), then optionally
running the explainer as a non-blocking step with a timeout wrapper.

Good point on the failure mode — I'll add a timeout and fallback to raw output
in the next version. Thanks for the feedback!

[Gilder Miller]

Thanks for your time and nice article!