A Fixture Is a Contract: Reviewing Agent PRs That Call a Free Server Instead

Replacing a checked-in fixture with a live model call is a dependency change. It is not a better test merely because the endpoint is free. Trust the diff where the input, the response shape, and the failure path stay pinned in git. Revert it where the default job grows a network requirement, a moving model, or a credential.

Agent pull requests often call that swap realism. The summary is not the change. The file list is: a JSON fixture disappears, a client appears, and a URL becomes the oracle.

Classify the call before you comment on style

Treat the patch as four edits, even if the agent shipped them as one feature.

1. A client, including base URL, headers, timeout, and buffered versus streamed body.
2. A credential: environment read, default argument, or a literal in an example file.
3. A test that changed class, from a pure function over bytes to an integration test with a remote side effect.
4. A fallback policy: offline mock, opt-in live call, or live call on every run.

If a new call site does not fit one row, the pull request is incomplete. Copy this table into the review comment and fill it from the diff, not from the generated summary.

Call site	Default CI	Data allowed	Bucket
Fixture, no socket	On	Synthetic only	Keep, if the schema is asserted
Opt-in sandbox, marker-gated	Off	One redacted sample	Prove with a recorded response
Live call on the unit path	On	Whatever the test builds	Send back
Hosted URL as the application default	On	Real user data	Send back the default

A free server does not move a row from send-back to keep. Cost is not a contract.

Rows that look closed and are not

A call looks closed when the URL comes from an environment variable. It is not closed if the default value is a hosted origin. Missing configuration should fail the process, not dial out.

A marker is also not a control unless the default CI job excludes it. Check the workflow file, not the docstring. The next scan is a proposed command for the branch under review. It was not executed for this article.

git diff -U0 origin/main -- '*.py' '*.yml' '*.toml' '*.env*'

Search that diff for URL literals, timeout=None, and key assignments. The search is a pointer. The table is the decision.

A reduced diff, and the lines that decide it

The next snippet illustrates the patch shape. It is not a capture from a real repository. The host is reserved and intentionally unusable.

import os
import httpx

def complete(prompt: str) -> str:
    url = os.getenv('MODEL_URL', 'https://example.invalid/free')
    resp = httpx.post(url, json={'prompt': prompt}, timeout=None)
    resp.raise_for_status()
    return resp.json()['text']

Four facts fall out of those lines.

• The default URL changes behavior for every caller, including tests that used to pass a dict.
• timeout=None lets a stalled peer stall the worker. Set a bound from latency you have measured. A number copied from an article, including this one, is not a measurement.
• The return type collapses to a string. Callers that used to branch on status now infer outcomes from prose.
• The sent payload is not recorded. The diff cannot show whether logs or customer text are included.

A shape that survives review separates transport from policy. This version is a proposal, not a benchmarked client, and it was not executed here.

class Transport:
    def post(self, path: str, *, json: dict, timeout: float) -> dict:
        raise NotImplementedError

def complete(prompt: str, transport: Transport) -> dict:
    body = transport.post('/complete', json={'prompt': prompt}, timeout=5.0)
    required = {'id', 'text', 'finish'}
    if set(body) != required:
        raise ValueError('schema mismatch')
    return body

Tests supply a fake transport. Only a sandbox script binds a real URL. The keys are a local contract you impose on the saved response. They are not a description of any vendor payload. The 5.0 argument exists so the timeout cannot be omitted in this sketch. Replace it with a bound from your own run.

What the default job may keep

Keep evidence that can be re-run without a network.

• The request is built from a fixture whose keys and size are known.
• Non-success HTTP statuses fail the test instead of becoming empty stand-ins.
• The oracle is a schema plus a cassette, not a substring of generated prose.
• Any network test carries an explicit marker, and the default job excludes it.
• The written summary matches the file list. A claim that the suite stayed hermetic is false if conftest.py session-scopes a live client.

A green run on the agent machine against a hosted sandbox is a sample. It is not a pin. Do not treat that sample as coverage for the default job.

What to send back

Send these back in the same review cycle. A follow-up issue is how a default path ships.

• A hosted URL, free or paid, used when configuration is absent.
• Any key, token, or cookie in the diff, including files named example.
• Fixture deletion without a replacement artifact in the same commit.
• Assertions on free-form model wording.
• A health or readiness handler that performs generation. Process liveness and model availability are different questions.
• A streaming parser with no test for a truncated body. Half an object is not a successful completion.

None of those lines become acceptable because the invoice is zero. Price can explain why someone tried the call. It cannot explain why the suite should depend on it.

What a closed socket must still prove

Prove the boundary with a check that a later generation pass cannot quietly delete. The harness is a proposal. It was not executed for this article. Point it at the repository you are reviewing.

import re
from pathlib import Path

LIVE_CALL = re.compile(r'(httpx\.|requests\.|aiohttp\.|websocket)')
MARKER = 'pytest.mark.network'

def test_default_suite_does_not_call_out(root: Path = Path('tests')):
    offenders = []
    for path in root.rglob('*.py'):
        text = path.read_text(encoding='utf-8')
        if LIVE_CALL.search(text) and MARKER not in text:
            offenders.append(path.as_posix())
    assert offenders == [], offenders

Run the focused file before you debate naming. This command is also unexecuted here.

pytest -q tests/test_offline_boundary.py

A failure means send the call site back. Do not ask for a weaker pattern until the call is gone or excluded from the default job.

Then mint one offline oracle, outside that job.

1. Write a single synthetic input to testdata/sandbox/case-01.json. No customer text, no internal hostnames, no keys.
2. Send that case once to a scratch server, with a timeout you selected. Save the raw response beside the input.
3. Commit the cassette. Do not commit a switch that re-enables the live call in CI.
4. Load the cassette in a local test. Assert status, the key set, and the absence of fields you did not request.

Cassette checks that stay stable

Prefer fields you control over fields the model phrases.

• Status code from the transport, not from a sentence in the body.
• Exact key set. Extra keys fail the test.
• Declared types for the keys you required.
• A maximum byte length, so a sudden verbose shift fails closed.

The live call exists to mint the file. The file exists so the next review can rerun without a network.

A scratch model and a scratch server, then disconnect

Drafting the client and minting one cassette still requires a model and a place to run a single prompt. A pure mock cannot show a real error envelope. That is the only gap this section fills.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode, an open-source project, fits here as the scratch environment and nowhere else in the merge. Free model access can draft the client and the failing boundary test. The free server option can replay the sanitized case so the cassette has a real envelope to freeze. After that file is saved, disconnect. The branch you ask others to review should pass with the network disabled.

Both availability claims are product facts supplied for this draft. They are not a token quota, a machine size, a latency target, an uptime figure, or a promise that the offer stays fixed. Read the current project documentation before you plan around them. If those docs and this article disagree, follow the docs.

A hosted scratch server is still another party. Do not upload production logs, customer content, or unpublished secrets in the name of realism. If the agent pointed the application default at that free server, split the change. Keep the cassette and the offline test. Remove the default URL.

Who should skip the recording step

This filter does not score model quality. It does not replace a contract test negotiated with a vendor.

Skip the live recording when:

• The build must stay air-gapped, and an opt-in marker would be too easy to flip on the default branch.
• The only useful payloads contain regulated or customer data.
• You have no schema. A reasonable paragraph is not an oracle.
• The branch is a demo with no production caller. Review cost will exceed the diff.
• You need capacity or duration numbers. This article does not establish any.

Cassettes rot when a provider changes shape. Refresh them in a reviewed commit after you have read that notice. Do not let a scheduled job overwrite fixtures from a live model. That job is the same dependency again, only slower.

Close on the bucket, not on the price

The review comment should name the bucket. State that the default job dials out, that the URL default should be removed, and that the cassette can stay. Calling the endpoint acceptable because it is free is not a review.

Classify each new call, delete unmarked network use, record one redacted case, and merge only what still passes with the socket closed. For the draft and that single recording run, check MonkeyCode's current documentation for free model access and the free server option, then leave both out of the default configuration.