Postmortem: A Tool Result Schema Defaulted Missing Failures to Zero

A merge gate treated a missing failed key as zero failures. The coding agent returned a partial tool object after a client timeout. Pytest never emitted junit XML for that SHA. The patch still reached main.

This article reconstructs that failure class in detail. It is a labeled worked example, not a named outage. The durable fix is fail-closed parsing plus a second pytest run on a disk the agent cannot seed.

What broke

The job used an LLM agent to invoke pytest through a tool interface. The tool result was a small JSON object. The merge gate decoded three fields and then stopped.

• passed
• failed
• duration_ms

A JSON Schema document marked those fields optional. It also set "default": 0 on failed. A missing key therefore became a green count. Absence was stored as evidence.

Timeline (reconstructed)

The times below are relative offsets. They describe one realistic sequence. They are not production timestamps.

1. T+0s — The agent received a patch and a test command.
2. T+12s — The tool started pytest -q --tb=no in the job root.
3. T+90s — The HTTP client hit its timeout. Pytest was still collecting.
4. T+91s — The tool returned {"passed": 0, "duration_ms": 90000}.
5. T+91s — No failed key. No junit.xml. No returncode field.
6. T+92s — The schema filler wrote failed: 0 into the object.
7. T+93s — The gate logged TOOL_OK failed=0 and merged.

Nothing in that path read pytest's process status. Nothing hashed an artifact on disk.

Contributing factors

Several small choices stacked. None looked reckless in isolation.

Schema defaults on gate fields

JSON Schema defaults help document APIs. They are hostile to merge gates. A missing integer is not a measured zero. Defaults convert silence into a pass.

No required exit code

The tool object omitted returncode. Timeouts often surface as 124. SIGTERM can surface as 143. The gate never saw those numbers.

The object replaced the process

Pytest was still collecting tests. There was no session footer. The agent still produced JSON. The gate trusted the object over the subprocess.

Shared .pytest_cache on the volume

A prior agent attempt on the same volume had a lastfailed file. Later parsers could read that cache. This incident did not need that extra leak. The default already sufficed. Cache reuse remains a second path to the same class of error.

No artifact digest

The job did not require a SHA-256 of junit.xml. Without a file, there was nothing to hash. The gate did not demand the file. Green became a property of JSON, not of pytest.

How partial tool JSON is born

Agent tool calling maps a model request onto a function. The function returns a structured object. HTTP layers then serialize that object.

Timeouts cut the function short. Retry wrappers may return the last partial dict. JSON encoders omit keys with None values. Schema fillers then insert defaults. Each step looks locally correct. The chain is fail-open.

A gate that reads payload.get("failed", 0) repeats the same bug in Python. Default arguments and schema defaults are the same hazard. Both treat silence as zero.

Artifact: fail-closed tool result contract

The contract below is executable. It rejects missing fields. It rejects schema defaults. Copy it into the repo that owns the merge gate. Label: this is a proposal, not a published standard.

Schema with required fields and no defaults

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://example.invalid/agent-tool-result.schema.json",
  "title": "AgentPytestToolResult",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "ok",
    "returncode",
    "failed",
    "passed",
    "skipped",
    "junit_sha256",
    "junit_path",
    "cmd"
  ],
  "properties": {
    "ok": { "type": "boolean" },
    "returncode": { "type": "integer", "minimum": 0, "maximum": 255 },
    "failed": { "type": "integer", "minimum": 0 },
    "passed": { "type": "integer", "minimum": 0 },
    "skipped": { "type": "integer", "minimum": 0 },
    "junit_sha256": { "type": "string", "pattern": "^[a-f0-9]{64}$" },
    "junit_path": { "type": "string", "minLength": 1 },
    "cmd": {
      "type": "array",
      "minItems": 1,
      "items": { "type": "string" }
    }
  }
}

Draft 2020-12 does not insert defaults during validation. Older drafts might. Pin the validator. Assert that a payload without failed is rejected before enabling the gate.

Validator that fails closed

# validate_tool_result.py
# Proposal: run this before any merge gate reads failed counts.
from __future__ import annotations

import hashlib
import json
import sys
import xml.etree.ElementTree as ET
from pathlib import Path

from jsonschema import Draft202012Validator

SCHEMA = json.loads(Path("agent-tool-result.schema.json").read_text())


def sha256_file(path: Path) -> str:
    data = path.read_bytes()
    if not data:
        raise ValueError("junit artifact is empty")
    return hashlib.sha256(data).hexdigest()


def junit_failure_count(path: Path) -> int:
    root = ET.parse(path).getroot()
    nodes = [root] if root.tag.endswith("testsuite") else []
    nodes.extend(root.iter())
    total = 0
    seen = False
    for node in nodes:
        if not node.tag.endswith("testsuite"):
            continue
        seen = True
        total += int(node.attrib.get("failures", "0"))
        total += int(node.attrib.get("errors", "0"))
    if not seen:
        raise ValueError("junit xml has no testsuite node")
    return total


def main(argv: list[str]) -> int:
    if len(argv) != 3:
        print("usage: validate_tool_result.py RESULT.json JUNIT.xml", file=sys.stderr)
        return 2
    result_path = Path(argv[1])
    junit_path = Path(argv[2])
    try:
        payload = json.loads(result_path.read_text())
    except json.JSONDecodeError as exc:
        print(f"reject: invalid json ({exc})", file=sys.stderr)
        return 1
    validator = Draft202012Validator(SCHEMA)
    errors = sorted(validator.iter_errors(payload), key=lambda e: list(e.path))
    if errors:
        for err in errors:
            loc = ".".join(str(p) for p in err.path) or "$"
            print(f"reject: {loc}: {err.message}", file=sys.stderr)
        return 1
    if payload["ok"] is not True or payload["returncode"] != 0:
        print("reject: pytest did not pass", file=sys.stderr)
        return 1
    if payload["failed"] != 0:
        print("reject: failed count is not zero", file=sys.stderr)
        return 1
    if not junit_path.is_file():
        print("reject: junit path missing on disk", file=sys.stderr)
        return 1
    digest = sha256_file(junit_path)
    if digest != payload["junit_sha256"]:
        print("reject: junit digest mismatch", file=sys.stderr)
        return 1
    if payload["junit_path"] != str(junit_path):
        print("reject: junit path mismatch", file=sys.stderr)
        return 1
    xml_failed = junit_failure_count(junit_path)
    if xml_failed != payload["failed"]:
        print("reject: json failed != junit failures+errors", file=sys.stderr)
        return 1
    print("accept: tool result matches junit artifact")
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

The XML walk counts failures and errors. Skips do not fail the job. An empty file fails the job. A missing testsuite node fails the job.

Commands the job should run

# Label: pin pytest in the lockfile before copying this.
rm -rf .pytest_cache
pytest -q --junitxml=junit.xml --cache-clear
python validate_tool_result.py tool_result.json junit.xml

--cache-clear is not optional in agent jobs. Sibling attempts share volumes more often than humans expect. rm -rf .pytest_cache is the belt. --cache-clear is the suspenders.

Decision table

Use this table when reviewing a tool payload. Any reject row must fail the merge. Do not add a fourth column for "warn".

Condition	Gate action	Reason
failed absent	reject	Absence is not zero
returncode absent	reject	Timeouts hide here
ok true and returncode != 0	reject	Fields disagree
failed == 0 and no junit file	reject	No evidence
junit digest mismatch	reject	Stale or swapped artifact
junit failures+errors != json failed	reject	Parser drift
failed > 0	reject	Suite is red
All required fields present, returncode 0, digest matches, failed 0	accept	Evidence exists

Durable fix

The schema change is necessary. It is not sufficient. The gate still needs a file the agent cannot forge cheaply.

1. Make failed, returncode, and junit_sha256 required.
2. Remove every default from the gate schema.
3. Delete .pytest_cache at job start.
4. Parse junit XML. Do not parse model prose.
5. Rerun pytest in a workspace the agent cannot write before the run.

Step 5 needs a second machine or a second container. Local reruns keep the same cache. They also keep the same sitecustomize.py and the same env.

Rollout order

Do not flip the gate to reject on day one without a shadow log. Run the validator in warn mode for a few jobs. Compare reject reasons against later production failures. Then enforce.

Shadow mode must not write defaults back into the payload. Log the raw bytes. Log the validator exit code. Keep those logs next to the junit artifact.

Where a clean remote pytest run fits

Some teams lack spare CI minutes for a second pytest. Isolation still matters. A second host that never mounts the agent workdir removes cache leaks.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode currently offers free model access and a free server option. Those two facts are the only product claims in this article. No model names, quotas, or hardware details are stated here.

The useful part is isolation, not branding. The merge job should ship validate_tool_result.py, the schema, and pytest --junitxml=junit.xml --cache-clear. The remote job should clone the candidate SHA. It should not mount the agent's workdir. It should not copy .pytest_cache. Free model access may draft the schema text. Humans still commit the required-field list. The gate still executes the validator, not the model.

Readers who already have isolated CI runners do not need another host. Readers who want a scratch box for that isolated pytest run can use the free server option as one such box.

What this does not fix

• Tests that assert the wrong behavior still look green.
• Mocks that swallow HTTP errors still look green.
• Import hooks in the job root can still rewrite modules.
• A validator that catches all exceptions can still fail open.

Those are separate incident classes. This post covers only schema defaults and missing junit evidence. It does not recertify autouse fixtures, path allowlists, or git apply error pages.

Who should not use this approach

• Repos without pytest or junit output.
• Jobs that cannot write a readable artifact path.
• Teams that parse only natural language summaries from an agent.
• Public forks that cannot keep a second runner limited to pytest.

If the suite cannot emit junit XML, fix that first. Do not add a JSON schema on top of logs. If operators cannot pin jsonschema and pytest, the validator itself becomes drift.

Limits of the worked example

No production timestamps are claimed. No customer names are claimed. The Python snippet is a proposal. Operators must pin jsonschema, pytest, and Python themselves. Draft 2020-12 required-field semantics differ from older drafts. Test the validator against a payload that omits failed before enabling the gate.

Tool calling remains useful for agents. The API shape is not the hazard. The hazard is treating a partial object as a test report. Fail closed. Hash the artifact. Run pytest twice if the first run shared a disk with the agent.