DEV Community

Taylor Wang
Taylor Wang

Posted on

48 Hours of API Contract Drift: Field Notes from a Free Model on a Free Server

#ai #api #devops #python

Most API breakages don't announce themselves in advance; they arrive as a 500 in a client you haven't touched in weeks. After a Monday morning debugging session that lasted until lunch, I wondered whether a free model on a free server could catch those silent contract changes before my colleagues did.

The Setup

The idea was simple: a cron job on a free server fetches a representative API response every 15 minutes, validates it against a saved JSON Schema, and when something doesn't match, it sends the diff to a free model for a plain-English summary. That summary gets posted to a Slack webhook. No persistent database, no orchestration platform—just a Python script, a systemd timer, and a few environment variables.

For the model calls, I used the free model access that comes with MonkeyCode's free tier. Disclosure: This article was prepared as part of MonkeyCode's product outreach. That doesn't change the experiment, but you should know where the free tokens came from.

Here's the skeleton of the detector (only the interesting parts):

import json
import requests
import jsonschema
from datetime import datetime, timezone

# In practice, replace this with a call to MonkeyCode's free model endpoint.
def summarize_change(actual, expected):
    prompt = f"""The API response no longer matches our schema.
Expected keys: {sorted(expected.keys())}
Actual keys: {sorted(actual.keys())}
Explain the change in one sentence, and rate the impact as low/medium/high."""
    return call_free_model(prompt)
Enter fullscreen mode Exit fullscreen mode

The important part wasn't the code itself; it was the loop that turned a 200 response into a signal. My first version only compared top-level keys, and that missed almost everything.

What Broke Within the First Six Hours

The free server did what free servers do: it slept. My initial cron expression ran every 15 minutes, but the process was killed after two hours of inactivity. When the next run started, it had to re-fetch the schema from a URL that required authentication—and the token had expired.

I also discovered that the model, when given a raw diff, would sometimes invent missing fields that hadn't actually disappeared. One morning it flagged widget_count as missing when the real change was that the response now returns a string instead of an integer. The model saw a type mismatch and hallucinated a higher impact than the facts deserved.

The Fix That Mattered Most

Instead of feeding raw JSON snippets, I started passing a normalized diff—one line per field, with only three states: added, removed, or type-changed. That cut the false alarms dramatically. The model's job became classification, not discovery.

def diff_schema(expected, actual):
    lines = []
    for key in sorted(expected.keys() | actual.keys()):
        if key not in actual:
            lines.append(f"- {key}: missing")
        elif key not in expected:
            lines.append(f"+ {key}: new")
        elif expected[key] != actual[key]:
            lines.append(f"~ {key}: type changed {expected[key]} -> {actual[key]}")
    return "\n".join(lines)
Enter fullscreen mode Exit fullscreen mode

That tiny normalization was the difference between "everything is on fire" and "one endpoint changed its date format."

The Two Interesting Findings

Twice in 48 hours the detector fired with a real signal. The first was a deprecated author object that became a flat list of strings. The model summarized it as "the author field is now exposed as an array, not a nested object," which was exactly what a developer needed to see.

The second was more subtle: the API started returning timestamps without timezone offsets. No keys changed, but the values were no longer ISO 8601 compliant. My schema validation didn't catch it because I hadn't declared format: date-time in the schema. The model caught it because I had pasted an example timestamp into the prompt, and it flagged the inconsistency.

So yes: a free model caught a bug that my deterministic validation missed. But it also produced two false positives that I had to inspect by hand. Would I trust this as the only warning system? Not yet.

What I Would Repeat (and What I Wouldn't)

I'd absolutely run this kind of drift detector again for a public API with a slow release cycle. The cost is near zero, and the signal-to-noise ratio becomes manageable once you normalize diffs and give the model a strict output format.

I wouldn't use it as the only gate for a production migration, and I wouldn't rely on the model's severity ratings. Treat the output as a triage suggestion, not a verdict.

Who Should Skip This

If your API uses public OpenAPI documents and versioned clients, your ecosystem already gives you clearer signals than any free-model summary. This approach makes sense for internal microservices where the consumer is you and your five teammates, and where the change happens without announcement.

Limitations

  • Free tier servers sleep; build in a retry or use a lightweight process manager.
  • The model's free endpoint can return 429 under burst load, so wrap your calls in a backoff loop.
  • This is not a substitute for contract testing; it's an additional tripwire.

If you've tried something similar, I'd be curious to hear what worked—drop a comment below.

Top comments (0)