The Model Did Not Change: The Test Was Comparing Strings, Not Behavior

A red build that feels random is worse than a consistent failure. Recently, a generated integration test passed locally for days and then failed in CI with a tiny diff. The only visible difference was the order of two JSON keys. My first assumption was that the model had started returning a different shape. It had not. The real problem was a test that treated a deterministic API contract as a fixed string, and a server routine that built a dictionary from a set.

I am reconstructing this as a debugging exercise rather than a production war story, so the service names are placeholders. The lesson is the reusable part.

Start from the diff, not from blame

When a test passes locally and fails remotely, the environment is the first suspect. I checked the Python version, the dependency lock file, and the request fixture. They matched. The diff showed the response body had node before status in CI, while my local mock had the opposite order. Because the test used assert response.text == expected_text, any harmless reordering became a failure.

# The brittle assertion generated by the model
def test_payload():
    expected = '{"status":"ok","node":"api-1","retries":0}'
    assert response.text == expected

This assertion is not wrong because an AI wrote it. It is wrong because it freezes a serialization detail. JSON object key order is not part of the API contract unless the contract explicitly says it is.

Find the nondeterminism before touching the model

Before asking whether the model changed, I ran the server routine in a loop. It exposed the bug without an API call.

for i in $(seq 1 20); do
  python -c 'import json; print(json.dumps({k: 0 for k in {"status","node","retries"}}))'
done | sort -u

The output varied between runs:

{"node": 0, "retries": 0, "status": 0}
{"retries": 0, "node": 0, "status": 0}
{"status": 0, "node": 0, "retries": 0}

The cause was hidden in the implementation:

def build_payload(fields):
    body = {}
    for name in set(fields):  # unordered source
        body[name] = registry[name]()
    return body

In Python, iterating over a set is not guaranteed to preserve insertion order. The set() call removed the order from fields, so the dictionary was built in unpredictable order across processes. CI creates a fresh process, so the key order changed there.

The fix is to preserve the declared order:

def build_payload(fields):
    body = {}
    for name in fields:  # preserve the caller's order
        body[name] = registry[name]()
    return body

But fixing the implementation is only half the work. The test should not have depended on a detail that could change again.

Test behavior, not formatting

A more durable test checks the structure and values, not the raw text.

def sort_keys(value):
    if isinstance(value, dict):
        return {key: sort_keys(value[key]) for key in sorted(value)}
    if isinstance(value, list):
        return [sort_keys(item) for item in value]
    return value

def test_payload_contract(response):
    body = response.json()
    body.pop("timestamp", None)  # intentionally nondeterministic field
    assert sort_keys(body) == {
        "node": "api-1",
        "retries": 0,
        "status": "ok",
    }

sort_keys makes nested key order irrelevant. Removing timestamp acknowledges the field that changes on every request. If the API grows a new field, this test stays focused on the parts that matter, while a stronger OpenAPI or JSON Schema check can cover the full contract.

Where a free model fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode's free model access and free server option to draft the normalizer and generate a throwaway test runner. I did not ask the model to verify my production code. I ran the generated script against the fixed routine and watched the output. The free access removed the pressure to eyeball a diff instead of building a small experiment.

That is a narrow use case. The free access is useful for short, non-confidential debugging loops, not for latency-sensitive traffic, long-running work, or anything that sends secrets. The model can still suggest a snapshotted assertion, so the real correctness check has to come from measuring nondeterminism rather than asking for confidence.

Who should not copy this workflow

Teams with strict data boundaries should keep reproduction data synthetic. Teams that need guaranteed model availability, version pinning, or response-time SLAs should run the same experiment on their own infrastructure. Anyone tempted to treat a free model as a contract checker should use a local schema test instead.

Reusable debugging checklist

• Reproduce the failure multiple times before changing code.
• Compare the whole object, not just the string diff.
• Suspect nondeterminism: sets, timestamps, locale, timezone, and request IDs.
• Distinguish an API contract from a serialization detail.
• Normalize only the fields that can legally vary.
• Fix both the brittle implementation and the brittle test.
• Keep a minimal reproduction in the repository.

The model was never the source of this failure. The test had frozen a nondeterministic implementation detail. If I had forced the model to emit a specific key order, I would have hidden the real bug and made the suite even more fragile. The useful skill was noticing that a string diff is not a contract test.