Fail the Job If the Agent Invents a Field

Stop treating a green unit test as proof the agent spoke to your API correctly. If the model invents a query field, a required header, or a verb your service never shipped, the merge path should fail before any live call leaves CI.

That is the whole method. Schema-check the tool JSON. Keep live inference off the merge job. Replay recorded fixtures so a flaky upstream cannot buy you a green check.

You already know the failure mode. An agent “helps” by filling in limit, pretty, or async=true because those names showed up in training data. Your stub server shrugs. Production 400s after merge. The CI log still says success because nobody asserted the call shape.

What you are actually gating

You are not grading prose. You are grading a contract.

A tool call is an HTTP intent with a name, a method, a path, and a JSON body. If that payload does not validate against the schema you committed, the agent did not finish the work. Fail the job. Do not retry the model hoping the next sample is nicer.

Live model access belongs in a labeled soak job, not in pull_request. Free endpoints are still endpoints. They rate-limit. They change. They are the wrong merge dependency.

Decision table: merge vs soak vs skip

Signal in the PR	Merge job	Soak job (opt-in label)
Agent only edits application code	Replay fixtures, schema-check tool JSON	Off
Agent adds or renames a tool	Replay fixtures and fail if schema file was not updated in the same diff	Off until a human adds soak-tools
Fixture hash mismatch	Fail. Do not silently rerecord	Allowed to rerecord behind the label
HTTP 429 / timeout from a hosted model	Must not run	Retry with a hard cap, then skip (non-blocking)
Schema itself changed	Contract tests must change in the same commit	Optional live probe of the new fields

Print this table in the PR template if your reviewers skim. One row per signal. No extra narrative.

1. Commit the contract next to the tool

Put the schema where the agent can see it and where CI can hash it. Do not hide it in a wiki.

{
  "$id": "https://example.local/schemas/search-tickets.json",
  "title": "search_tickets",
  "type": "object",
  "additionalProperties": false,
  "required": ["query", "status"],
  "properties": {
    "query": { "type": "string", "minLength": 1, "maxLength": 200 },
    "status": { "enum": ["open", "pending", "closed"] },
    "page": { "type": "integer", "minimum": 1, "maximum": 50 }
  }
}

additionalProperties: false is the whole point. Invented fields must be a CI failure, not a warning the log swallows.

Pin a tiny allowlist of tool names in the same directory.

tools/
  search_tickets.schema.json
  create_ticket.schema.json
  ALLOWLIST.txt

ALLOWLIST.txt is one name per line. If the agent emits a tool that is not on that list, fail. New tools need a schema file and a human-edited allowlist in the same commit.

2. Record the tool JSON, not the essay

You need a fixture the merge job can replay without a network. Capture only the structured call.

# tests/support/record_tools.py
# Proposal: drop this next to your agent runner. Label it experimental until you wire it to your real tracer.
from __future__ import annotations

import json
from pathlib import Path
from typing import Any

FIXTURE_DIR = Path("tests/fixtures/tool_calls")


def dump_tool_call(run_id: str, tool_name: str, arguments: dict[str, Any]) -> Path:
    FIXTURE_DIR.mkdir(parents=True, exist_ok=True)
    path = FIXTURE_DIR / f"{run_id}__{tool_name}.json"
    payload = {"name": tool_name, "arguments": arguments}
    path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8")
    return path

Do not record chain-of-thought. Do not record API keys. If your tracer includes headers, strip Authorization before write.

Rerecord on purpose:

python -m tests.support.record_tools --run-id pr-842 --from-session ./tmp/agent-session.jsonl
git add tests/fixtures/tool_calls

If the agent changed behavior and nobody updated fixtures, merge CI must go red. That is the receipt.

3. Schema-check in pytest, on every PR

# tests/test_tool_contracts.py
from __future__ import annotations

import json
from pathlib import Path

import jsonschema
import pytest

SCHEMA_DIR = Path("tools")
FIXTURE_DIR = Path("tests/fixtures/tool_calls")
ALLOWLIST = {
    line.strip()
    for line in Path("tools/ALLOWLIST.txt").read_text(encoding="utf-8").splitlines()
    if line.strip() and not line.startswith("#")
}


def load_schema(name: str) -> dict:
    path = SCHEMA_DIR / f"{name}.schema.json"
    if not path.exists():
        pytest.fail(f"tool {name!r} has no schema at {path}")
    return json.loads(path.read_text(encoding="utf-8"))


@pytest.mark.parametrize("fixture_path", sorted(FIXTURE_DIR.glob("*.json")))
def test_recorded_tool_matches_schema(fixture_path: Path) -> None:
    payload = json.loads(fixture_path.read_text(encoding="utf-8"))
    name = payload["name"]
    if name not in ALLOWLIST:
        pytest.fail(f"{fixture_path.name}: tool {name!r} is not in ALLOWLIST.txt")
    jsonschema.validate(instance=payload["arguments"], schema=load_schema(name))

Run it locally the same way CI will.

python -m pip install jsonschema pytest
python -m pytest tests/test_tool_contracts.py -q

One assertion per fixture. No sleeps. No retries. If this file needs the network, you built the wrong test.

4. Keep the live model off the merge path

Merge CI should be boring. Checkout. Install. Replay fixtures. Validate schemas. Upload the junit file. Stop.

# .github/workflows/tool-contracts.yml
name: tool-contracts
on:
  pull_request:
  push:
    branches: [main]

jobs:
  replay:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install jsonschema pytest
      - run: pytest tests/test_tool_contracts.py --junitxml=junit-tools.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: tool-contract-junit
          path: junit-tools.xml

That job never calls a model. If an agent rewrites this workflow to inject a live key, you have a different problem: protect the workflow file with CODEOWNERS and a required review. Do not solve that inside the pytest file.

5. Optional soak: live calls behind a label

You still want a place to see what a hosted model does with the same prompt. Make it opt-in, non-blocking, and budgeted.

This is the only place a free model path and a free server option belong. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you already use MonkeyCode, its free model access and free server option can host that soak runner so the merge job stays fixture-only. They are extra capacity, not a required check name.

# .github/workflows/tool-soak.yml
name: tool-soak
on:
  pull_request:
    types: [labeled, synchronize, opened]

jobs:
  soak:
    if: contains(github.event.pull_request.labels.*.name, 'soak-tools')
    runs-on: ubuntu-latest
    timeout-minutes: 15
    continue-on-error: true
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Probe live tool calls with a hard retry cap
        env:
          SOAK_BASE_URL: ${{ secrets.SOAK_BASE_URL }}
        run: python tests/support/soak_probe.py --max-attempts 3 --timeout-sec 20

The probe must refuse to invent retries. Three attempts. Then skip.

# tests/support/soak_probe.py
# Proposal: wire SOAK_BASE_URL to whatever host you actually control.
from __future__ import annotations

import os
import sys
import time
import urllib.error
import urllib.request


def main(max_attempts: int, timeout_sec: int) -> int:
    url = os.environ.get("SOAK_BASE_URL")
    if not url:
        print("SOAK_BASE_URL unset; skipping soak")
        return 0
    last_err = None
    for attempt in range(1, max_attempts + 1):
        try:
            with urllib.request.urlopen(url, timeout=timeout_sec) as resp:
                print(f"soak ok status={resp.status} attempt={attempt}")
                return 0
        except (urllib.error.URLError, TimeoutError) as err:
            last_err = err
            print(f"soak attempt {attempt} failed: {err}")
            time.sleep(min(2 ** attempt, 8))
    print(f"soak skipped after {max_attempts} attempts: {last_err}")
    return 0  # non-blocking


if __name__ == "__main__":
    raise SystemExit(main(max_attempts=3, timeout_sec=20))

Notice the exit code. Soak never blocks merge. If the free server is busy, you learn that in the log, not by wedging the queue.

6. Fail closed on three cheap signals

Add a short guard so the agent cannot “fix” CI by deleting tests.

# scripts/assert_contract_surface.sh
set -euo pipefail
test -f tools/ALLOWLIST.txt
test -f tests/test_tool_contracts.py
count=$(find tests/fixtures/tool_calls -name '*.json' | wc -l)
if [ "$count" -lt 1 ]; then
  echo "no tool-call fixtures; refusing to pass"
  exit 1
fi

Call it from the merge workflow before pytest. Empty fixture directories are not a pass. They are an untested agent.

You can also hash the schema directory and echo it into the job summary. When the hash changes without a matching fixture update, reviewers see it without reading YAML.

schema_hash=$(find tools -type f -print0 | sort -z | xargs -0 sha256sum | sha256sum)
echo "schema_hash=$schema_hash" >> "$GITHUB_STEP_SUMMARY"

What this does not prove

A valid tool payload is not a valid product decision. The agent can still search the wrong tenant, page past your max, or call a legal tool in an illegal order. Schema checks do not replace authz tests, idempotency tests, or a human reading the diff.

This approach also assumes you own the schema. If the upstream API is a moving SaaS spec you do not vendor, freeze a copy. Do not fetch OpenAPI from the internet during pull_request. That is another flaky dependency wearing a documentation hat.

Hosted inference — free or not — can still return a well-typed wrong answer. Soak is for drift detection. It is not a substitute for the replay job.

Who should not use this

Skip this workflow if your “agent” never emits tool JSON. Skip it if every call is already a hand-written client with compile-time types and no LLM in the loop. Skip it if policy forbids sending prompts or customer text to a hosted model, free server included.

Do not use soak-as-merge if you cannot tolerate a 429. Do not store raw session traces that contain secrets. Do not let continue-on-error: true leak onto the replay job.

A 15-minute checkout you can run today

1. Add tools/search_tickets.schema.json and tools/ALLOWLIST.txt.
2. Drop one recorded fixture under tests/fixtures/tool_calls.
3. Copy tests/test_tool_contracts.py and run pytest until it fails on an invented field.
4. Wire .github/workflows/tool-contracts.yml as a required check.
5. Only then add tool-soak.yml behind the soak-tools label.

Break the fixture on purpose. Add "pretty": true to the arguments. Watch pytest fail. That red log is the feature.

If you want a soak host that is not your laptop, MonkeyCode’s free model access and free server option are enough to run the labeled job. Keep them off the required check list. The merge path should still be fixtures, schemas, and a boring green.