I Kept the Free Model on the Drafting Table. The Runtime Held the Gavel.

Every 48-hour experiment I have run with a free AI stack has ended the same way: the model proposes, and something concrete vetoes. The scheduler rejected my assumptions, the semantic cache returned a near-miss instead of a hit, and the rate-limit storm came from inside my own retry loop. So when I wanted to test contract drift without burning a week on green-field code, I decided the pattern was the point. I would let a free model draft test skeletons, but I would force the runtime to be the only judge that matters.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The workflow below was designed to run on MonkeyCode's free model access and free server option, though the same pattern works with any zero-cost endpoint and a Python-capable server. You deserve to know my bias before we go further.

Why contract tests are the right job for a free model

A contract test checks that an API still speaks its documented language: the right status code, the right shape, the right required fields. That is a messy task for an LLM if you ask it to judge business logic, but a surprisingly good task if you ask it to propose inputs. The model does not need to know why a user signs up; it only needs to read an OpenAPI spec and produce a few representative request bodies that a real HTTP client can throw at the server.

The key is to stop asking for a verdict. When I ran PR reviews with a free model, the golden set saved me from its false positives. When I graded failing tests, the labels were the flaky part. In both cases the model was doing judgement work, and judgement needs context that a 48-hour window cannot reliably provide. So for contract tests, I flipped the division of labor: the model drafts, pytest cross-examines.

The workflow: draft, run, discard

The experiment is a simple loop that looks like this:

1. Load every operation from your OpenAPI document.
2. For each operation, call the free model with a narrowly scoped prompt: "Given this path, method, and schema, return one valid request body and the expected status code."
3. Generate a pytest function from that response, but wrap all values in explicit type coercion.
4. Run the whole suite against a test instance. Do not let the model run, only the generated tests.
5. Collect every failure and classify it manually: wrong expectation from the model, or a real contract regression in your code.

The grammar of the prompt matters more than the model. You are not asking for prose or a dry run; you are asking for two JSON fields: args and expected_status. Keep the output contract tiny, and the model's hallucinations become easier to spot.

A condensed, unexecuted sketch

This is an illustrative skeleton, not a production script. I have not executed this exact code in a public repo, mainly because your HTTP client and model endpoint will differ. Fill in the missing pieces, keep the coercion layer, and do not trust the model's output blindly.

# contract_drafter.py — condensed sketch, not executed
import json


def load_operations(spec_path: str) -> list:
    with open(spec_path) as f:
        spec = json.load(f)
    operations = []
    for path, methods in spec["paths"].items():
        for method in ("get", "post", "put", "patch", "delete"):
            if method in methods:
                operations.append((method.upper(), path, methods[method]))
    return operations


def draft_from_model(operation: tuple) -> dict:
    """Call the free model here. Return dict with args and expected_status."""
    # Replace this stub with your MonkeyCode client call.
    return {
        "args": {"name": "sample-user"},
        "expected_status": 201,
    }


def build_test_source(operation: tuple, draft: dict) -> str:
    method, path, _ = operation
    safe_path = path.strip("/").replace("/", "_").replace("-", "_")
    body = json.dumps(draft["args"])
    status = draft["expected_status"]
    return f"""
def test_{safe_path}_{method.lower()}():
    import json as _json
    response = client.request("{method}", "{path}", json={body})
    assert response.status_code == {status}
"""

# In your runner: for each op, call draft_from_model, compile the generated
# source with exec(), and let pytest collect it. Always wrap the response in
# real type checks before putting it into the assertion.

The coercion layer is not optional. My earlier field notes kept showing the same failure mode: the model would return "status": "201" as a string, or "items": 3 for a list field. The runtime never complained, because Python does not care until the assertion blows up. If you cast every field against a schema from the spec, you convert model noise into test failures, which is exactly what you want.

What broke in the broader 48-hour arc

Even with this design, several things broke in the surrounding infrastructure, and they should change how you schedule the experiment.

• The free server fell asleep before the first request. Cold starts made the initial test hang long enough that I suspected the endpoint was dead. The fix is a trivial warm-up call before the suite runs.
• The model produced the same happy path over and over. When prompted for a valid request, it returned the example from the spec rather than a boundary case. Add a second prompt that explicitly asks for "one invalid but well-formed request" and assert a 4xx status.
• The generated tests passed, but real callers would not. That sounds backwards, but it happens because the model copied the spec's example verbatim. The suite caught nothing, and my manual audit caught the blind spot. Coverage tools are not optional.
• The retry storm returned once I added a real model call. The free tier rate limit hit at the worst moment, and my client automatically retried ten times. I finally learned to treat every retry as a state change, not a prayer.

A decision table for the drafting table

Use this table to decide where the free model earns its keep in your own contract-test loop:

Task	Let the model do it	Write it yourself
Generate request bodies from a schema	✅	❌
Pick expected status codes	⚠️ Only with explicit schema rules	✅
Write the assertion expression	❌	✅
Decide whether a failure is a bug or a bad draft	❌	✅
Create smoke tests for every endpoint	✅	❌

Notice the pattern: the model is strongest at generating volume, weakest at making judgments. The moment you give it a binary decision with downstream consequences, you are back to the golden-set problem from my earlier experiments.

Who should not use this approach

If your API is still changing every hour, do not bother. Contract tests become noise when the contract itself is the moving target. If you have no test instance or no way to roll back schema changes, the runtime cannot be the gavel because the courtroom is not stable. And if you cannot spare twenty minutes to manually review every generated test, the model will quietly codify its own mistakes faster than your CI can report them.

The approach also fails for APIs with deeply contextual validation rules, like multi-step wizards or retry idempotency semantics. A free model might draft the shape of those tests, but you will need a human to fill the body.

What I would repeat

I would repeat the one rule that made every previous experiment useful: never let the model judge its own output. The scheduler judged nothing, the cache judged by similarity, and the runtime judged by loading code. In this design, the runtime still has the last word, and the model simply supplies the questions. That is the only division of labor that survived contact with a free server and a free model.

If you are planning a similar experiment, start with a messy spec and a timer. Your first generated suite will be wrong, and that is fine. Let the wrongness show up in the test report, not in your head.