Stop Mocking the Model. Run CI Against the Free Tier Instead.

Last month, a colleague's PR merged cleanly because every test mocked the model as returning perfect JSON. Then the real model wrapped its answer in markdown code fences on the first production request, and the parser died instantly. The post-mortem read like a confession: we had tested everything except the one thing that could actually fail. That is when I stopped believing that mocking a model is a testing strategy, and the free tier became the only honest test double available.

Here is the position I want to argue: free AI compute is not a production discount, and it is not a demo budget either. It is a CI test runner that happens to speak HTTP, and the teams that understand this are the ones whose AI features survive contact with reality. The teams that deploy free tiers to serve real traffic are building on a foundation that can vanish mid-request. The difference is not model quality; it is the honesty of the environment you test against.

MonkeyCode's open-source gateway and free server option fit this pattern because they give you a real endpoint with a real token allowance. Disclosure: This article was prepared as part of MonkeyCode's product outreach. As of this writing, the free tier includes a ten-million-token allowance and a deployable server, which is enough for thousands of CI runs but nowhere near enough for production traffic. That asymmetry is not a limitation; it is a specification for where the resource belongs.

The artifact I want to share is a test suite that treats the model as an external service with a contract, not as a function you can mock away. The first test verifies that the response schema matches what the frontend expects, the second enforces a latency budget, and the third deliberately exhausts the quota to confirm the error path is structured and parseable. None of these tests are possible with a mock, because a mock cannot produce a 429, a slow response, or a malformed JSON body.

# tests/test_ai_contract.py
import os
import time
import httpx
import pytest

GATEWAY_URL = os.environ.get("GATEWAY_URL", "http://localhost:8000")

@pytest.fixture(scope="session")
def client():
    with httpx.Client(base_url=GATEWAY_URL, timeout=20.0) as c:
        yield c

def test_response_schema_matches_frontend(client):
    payload = {
        "messages": [{"role": "user", "content": "Summarize this PR in one sentence."}],
        "max_tokens": 128,
    }
    r = client.post("/chat", json=payload)
    assert r.status_code == 200, r.text
    body = r.json()
    assert "choices" in body, f"missing choices key: {body.keys()}"
    content = body["choices"][0]["message"]["content"]
    assert isinstance(content, str) and len(content) > 0

def test_latency_stays_under_budget(client):
    start = time.monotonic()
    r = client.post("/chat", json={
        "messages": [{"role": "user", "content": "Reply with OK."}],
        "max_tokens": 16,
    })
    elapsed = time.monotonic() - start
    assert r.status_code == 200
    assert elapsed < 10.0, f"latency breach: {elapsed:.1f}s"

def test_quota_exhaustion_returns_structured_error(client):
    r = client.post("/chat", json={
        "messages": [{"role": "user", "content": "hi"}],
        "max_tokens": 10_000_000,
    })
    assert r.status_code in (429, 503)
    assert "retry" in r.text.lower() or "exhaust" in r.text.lower()

The CI workflow is where this pattern earns its keep, because it runs on every pull request without anyone remembering to do anything. You start the gateway as a service container, point the test suite at it, and let the free allowance absorb the cost of catching regressions early. The first time I wired this up, the suite caught a breaking prompt-template change within three minutes of the commit, which is faster than any code review I have ever attended.

# .github/workflows/ai-contract.yml
name: ai-contract
on:
  pull_request:
    paths: ["app/**", "prompts/**"]
jobs:
  contract:
    runs-on: ubuntu-latest
    services:
      gateway:
        image: ghcr.io/your-org/ai-gateway:latest
        ports: ["8000:8000"]
        env:
          PRIMARY_URL: ${{ secrets.PRIMARY_URL }}
          PRIMARY_KEY: ${{ secrets.PRIMARY_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements-dev.txt
      - run: pytest tests/ -v
        env:
          GATEWAY_URL: http://localhost:8000

Before you copy this pattern, your gateway needs to meet three contract requirements, and the workflow above assumes you have containerized it and pushed it to a registry your runner can pull from. It must translate provider errors into structured HTTP responses, it must expose a token counter that resets on a fixed window, and it must fail with a 503 when every provider is exhausted. If your gateway returns raw provider exceptions, the test suite will still pass on the happy path, and you will have learned nothing about your real failure modes.

Now the honest limitations, because this pattern is not for everyone. If your feature requires deterministic outputs, like a grading rubric or a financial summary, a live model in CI will flake, and you need a recorded-replay approach instead. If your organization blocks outbound traffic from CI runners, the free server becomes a manual step, and manual steps get skipped, and skipped tests are worse than no tests because they create false confidence.

The teams that should not use this pattern are the ones that need a guarantee, because a free tier provides none. You cannot promise a stakeholder that the model will respond within two seconds when the provider can throttle you at any moment. You also cannot promise that the server will be warm when the pipeline starts, so your tests must treat cold starts as a normal state. What you can promise is that your code handles every one of those failures gracefully, and that promise is worth more than any benchmark score.

Here is the checklist I run before wiring any free model tier into a pipeline. First, verify the gateway translates provider errors into structured responses with status codes your frontend understands. Second, confirm the test suite fails loudly when the model is down, so a green build actually means something. Third, set a per-run token budget so one pathological test cannot drain the monthly allowance, and fourth, document who owns the gateway configuration, because a shared endpoint with no owner is an incident waiting for a victim.

So the next time someone argues that free AI compute is too unreliable for real work, agree with them, and then ask why they are trying to serve production traffic from a test environment. The free tier is not a server that happens to be cheap; it is a test runner that happens to be free. The teams that understand that distinction are the ones whose AI features survive contact with the real world. If you want to see the failure modes yourself, point this test suite at MonkeyCode's free server and let the 429s teach you what your mocks have been hiding all along.