Find Contract Drift by Running Two Free Models, Not by Reading One Diff

A payment webhook arrived at 3 a.m. with a status the codebase had never seen. The enum had grown by one value somewhere between the AI-generated implementation and the database migration.

Both files looked reasonable in review. Both were wrong together.

The model did not invent the value. It picked a name the contract never specified, and the contract was silent enough to allow it. This is the failure mode that reading diffs cannot catch. The diff shows what the model wrote, not what the contract left open. A single generated implementation is a single point of failure. The review process becomes the only place where the contract's ambiguity can surface. Most review processes are not equipped for that job.

Differential testing closes the gap. Generate two independent implementations of the same contract. Deploy them to isolated environments. Send both the same request sequence. Where the responses diverge, the contract is ambiguous. Where they agree, the contract is probably precise enough to ship. The divergence report becomes the review artifact. A human reads contract gaps instead of a wall of code.

This is where free resources stop being a perk and start being a methodology. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's open-source project provides free models with a ten-million-token allocation. It also offers a free server option. Both map directly onto differential testing. The free models pay for the two candidate implementations. The free server gives each candidate an isolated URL. The probes run there without touching anything a customer depends on.

The harness itself is small. It reads a list of request cases. Each case goes to both deployments. The responses are normalized. The first differing field is reported.

# differential.py — find contract drift between two implementations
import json
import sys
import urllib.request
import urllib.error

def call(base, path, payload):
    req = urllib.request.Request(
        base + path,
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    try:
        with urllib.request.urlopen(req, timeout=10) as resp:
            return resp.status, json.load(resp)
    except urllib.error.HTTPError as e:
        return e.code, json.load(e)

def normalize(status, body):
    body.pop("request_id", None)
    body.pop("processed_at", None)
    return status, json.dumps(body, sort_keys=True)

cases = json.load(open(sys.argv[1]))
base_a, base_b = sys.argv[2], sys.argv[3]

for case in cases:
    status_a, body_a = call(base_a, case["path"], case["payload"])
    status_b, body_b = call(base_b, case["path"], case["payload"])
    if normalize(status_a, body_a) != normalize(status_b, body_b):
        print(f"DRIFT: {case['name']}")
        print(f"  A: {status_a} {body_a}")
        print(f"  B: {status_b} {body_b}")

The case file is a plain JSON array. Each entry names the scenario, the path, and the payload that exercises one contract rule.

[
  {
    "name": "finalize unknown job",
    "path": "/jobs/finalize",
    "payload": {"job_id": "missing-42", "action": "finalize"}
  },
  {
    "name": "finalize twice",
    "path": "/jobs/finalize",
    "payload": {"job_id": "job-7", "action": "finalize"}
  },
  {
    "name": "cancel after finalize",
    "path": "/jobs/cancel",
    "payload": {"job_id": "job-7", "action": "cancel"}
  }
]

Run the harness with the two deployment URLs. The URLs come from the two free server deployments.

python differential.py cases.json https://impl-a.free-server.example https://impl-b.free-server.example

The output is the review. A clean run means both models read the contract the same way. A drift line means the contract failed to constrain the implementation. That line becomes a new contract test before either implementation ships.

The workflow has four steps. Write the smallest possible contract. Endpoints, status codes, and the meaning of each field.

Generate two implementations with the free model allocation. Use only the contract as the prompt. Deploy both to the free server and run the harness.

Convert every drift into a contract test. Regenerate until the drift disappears.

The interesting part is what the drift reveals. In the webhook incident, the contract said a job could be "pending", "running", or "done". It never said what happened after a cancel raced a finalize.

One model invented "cancelled". The other returned "done" with a cancellation timestamp. Both were internally consistent. Both violated the unstated expectation that a finalized job stays finalized.

A decision table helps decide when differential testing earns its cost.

Situation	Run differential?	Reason
New endpoint with a written contract	Yes	Ambiguity is cheapest to fix before consumers exist
Refactor of an existing endpoint	No	The old implementation is the reference; use golden tests instead
Error-path heavy API	Yes	Models guess error semantics most freely
Internal helper with no external consumers	No	Drift has no blast radius
Compliance or regulated data	No	Free shared servers are for failure rehearsal, not regulated workloads

The limitations are real. Two models trained on overlapping data can share the same blind spot. Agreement is evidence, not proof. The free token allocation is a budget. The harness should run only contract-rule cases, not every possible input. The free server is a disposable environment. Nobody should put customer data on it. When the contract itself is wrong, differential testing exposes the disagreement. It cannot decide which side is right. That judgment stays with a human who knows the business.

Some teams should skip this approach. Teams with a stable contract and no error-path complexity will find the harness mostly quiet. Golden tests against the existing implementation are cheaper. Teams under compliance constraints should not route regulated data through a shared free environment. Teams that cannot convert a drift into a contract test will accumulate divergence reports. Those reports never close.

The broader lesson is that the model is a dependency. Dependencies need verification beyond reading. Differential testing turns the free model allocation into a probe. The probe finds the places where the contract is too weak to constrain an implementation. The free server gives that probe a place to run without risking production. Run two models against each other before you run one model against your users. The drift report will tell you where the contract needs to be stricter.