Postmortem: When Truncated Test Logs Look Like a Pass

You stare at a green check on a Friday night pull request and feel the kind of relief that makes people merge too fast. The coding agent has already posted a summary that says every integration test passed after the refactor of the retry budget. You skim three lines of the comment, squash the branch, and walk away from the laptop before the on-call rotation even starts. Ninety minutes later the pager fires because checkout is throwing 500s on a path the tests were supposed to cover.

The failure is not mysterious once you open the raw log instead of the agent's confident paragraph. A single assertion in checkout_service_test never ran to completion, and the process died while flushing JUnit XML. The agent never saw the stack trace because the tool result was clipped at a character budget you never configured on purpose. It then did what many agents do under uncertainty: it summarized the visible fragment as success and invited you to merge.

Customers were not the first to notice; your canary instance was, and that is the only reason this stayed a postmortem instead of a headline. Inventory locks then stacked up behind a nil pointer that the truncated suite never actually reached. You roll back in twelve minutes, which feels fast, and still remains slow compared with never merging. The question you should ask next is not which model is smarter, but which evidence the agent was allowed to treat as complete.

Call this a reconstructed timeline rather than a war story from a specific company, because the pattern shows up in more than one codebase. At 23:08 the agent starts the suite through a shell tool and streams stdout into its context window. At 23:11 the runner prints a long fixture dump, then the tool host cuts the result and appends a quiet truncation marker. At 23:12 the model posts a pull request comment citing all tests passed, because the last visible lines look like setup noise.

You merge at 23:18 without opening the CI artifact, trusting the agent like a colleague who was already in the room. Deploy starts at 23:31 on the canary pool, and error rates bend upward before the dashboard even finishes loading. The rollback completes at 23:43, which means the window of customer pain is short and still entirely avoidable. The durable lesson arrives later, when you diff the CI log against the tool payload the agent actually received.

Several ordinary decisions combine here, the way small leaks combine until a boat is just a suggestion of wood. The test command dumps fixtures at debug level, so a single failing case produces more text than the tool channel will keep. The agent prompt tells the model to be concise and to avoid pasting raw logs, which trains it to guess from fragments. Nobody defined a completion contract, so a missing JUnit footer is treated as an aesthetic problem rather than a failed observation.

There is also a human factor that looks like laziness until you name it as an interface bug. You asked whether tests passed, which is a boolean question that invites a coin flip when the evidence is only partial. A better question is whether the agent observed a complete, signed test report from a runner it controls. Until that report exists, the only honest status is unknown, and unknown must block merge the same way a red X does.

Think of the agent as a night-shift operator watching a security camera that randomly freezes on a calm hallway. You would not certify the building from a frozen hallway frame, and you should not certify a suite from frozen tokens. The durable fix is not a sterner system prompt, and it is not a larger model with more patience. It is a small program that withholds a pass verdict unless the report is complete, plus an agent rule that will not summarize missing evidence.

Local agent runs are useful because they are fast, and that speed is exactly why they lie so cleanly. A CI system usually archives the full log, the XML report, and the exit code as three separate objects you can reconcile. An agent tool call often collapses those objects into one string, then cuts the string to protect the context window. If you do not rebuild the three objects before the model speaks, you have rebuilt a slot machine and called it quality engineering.

Treat the following helper as a proposed workflow, not a production service, and place it in front of the agent's test tool. The script runs your suite, writes the full log to disk, and emits a tiny verdict document the model is allowed to read. It fails closed when the log looks clipped, when the XML is not well formed, or when the runner never printed a footer. Proposed code like this is the contract; the agent is only a narrator sitting on top of it.

#!/usr/bin/env python3
"""Fail closed when test evidence is truncated or incomplete.

Proposed helper for agent-run suites. Not a measured benchmark.
"""
from __future__ import annotations

import argparse
import hashlib
import json
import os
import subprocess
import sys
import xml.etree.ElementTree as ET
from pathlib import Path

TRUNCATION_MARKERS = (
    "[truncated]",
    "output truncated",
    "result truncated",
    "... (truncated)",
    "max output length",
)
FOOTERS = (
    "==== ",
    "failed",
    "passed",
    "error",
    "seconds",
)


def sha256_file(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(65536), b""):
            digest.update(chunk)
    return digest.hexdigest()


def looks_truncated(text: str) -> bool:
    lowered = text.lower()
    if any(marker in lowered for marker in TRUNCATION_MARKERS):
        return True
    if not text.endswith(("\n", "\r\n")):
        return True
    tail = lowered[-400:]
    return not any(token in tail for token in FOOTERS)


def junit_ok(path: Path) -> tuple[bool, str, int, int]:
    if not path.exists() or path.stat().st_size == 0:
        return False, "missing_junit", 0, 0
    try:
        root = ET.parse(path).getroot()
    except ET.ParseError:
        return False, "malformed_junit", 0, 0
    suite = root if root.tag.endswith("testsuite") else root.find(".//{*}testsuite")
    if suite is None:
        return False, "no_testsuite", 0, 0
    tests = int(suite.attrib.get("tests", "0"))
    failures = int(suite.attrib.get("failures", "0")) + int(suite.attrib.get("errors", "0"))
    if tests < 1:
        return False, "empty_suite", tests, failures
    return True, "ok", tests, failures


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--cmd", required=True)
    parser.add_argument("--junit", default="report.xml")
    parser.add_argument("--log", default="suite.log")
    parser.add_argument("--out", default="verdict.json")
    parser.add_argument("--cwd", default=".")
    args = parser.parse_args()

    log_path = Path(args.log)
    junit_path = Path(args.junit)
    completed = subprocess.run(
        args.cmd,
        cwd=args.cwd,
        shell=True,
        text=True,
        capture_output=True,
    )
    combined = completed.stdout + "\n" + completed.stderr
    log_path.write_text(combined)

    truncated = looks_truncated(combined)
    xml_ok, xml_reason, tests, failures = junit_ok(junit_path)

    if truncated:
        status = "incomplete"
    elif not xml_ok:
        status = "incomplete"
    elif completed.returncode != 0 or failures > 0:
        status = "failed"
    else:
        status = "passed"

    verdict = {
        "status": status,
        "runner_exit": completed.returncode,
        "truncated_log": truncated,
        "junit": xml_reason,
        "tests": tests,
        "failures": failures,
        "log_sha256": sha256_file(log_path),
        "junit_sha256": sha256_file(junit_path) if junit_path.exists() else None,
        "bytes": log_path.stat().st_size,
    }
    Path(args.out).write_text(json.dumps(verdict, indent=2) + "\n")
    print(json.dumps(verdict))
    return 0 if status == "passed" else 2


if __name__ == "__main__":
    sys.exit(main())

Run it locally before you let any agent interpret pytest, because the verdict file is the only document the model is allowed to read. Keep the raw log and the XML beside that file so a human can still open the camera when the night operator is unsure. The commands below are a rehearsal, not evidence from a scored bake-off, and you should swap in your own suite command.

chmod +x complete_test_verdict.py
python complete_test_verdict.py \
  --cmd "pytest -q tests/checkout_service_test.py --junitxml=report.xml" \
  --junit report.xml \
  --log suite.log \
  --out verdict.json
echo $?
cat verdict.json
sha256sum suite.log report.xml verdict.json

If status is incomplete, the agent must stop and ask you to raise the log budget or to run the suite on CI instead. Do not paste suite.log back into the model as a recovery trick, because that is how the original truncation bug returns through a side door. Give the agent the JSON, the hashes, and a short rule that treats unknown as blocked. A proposed instruction block looks like the following, and you should paste it beside the tool definition rather than into a motivational preamble.

You may run tests only through complete_test_verdict.py.
Read verdict.json. Do not read suite.log unless status is failed and a human asks.
If status is incomplete, say the observation is incomplete and refuse to approve.
If status is failed, cite tests and failures from the JSON only.
If status is passed, cite the log_sha256 and junit_sha256 before you say green.
Never infer pass from stdout fragments, progress bars, or missing stack traces.

You can keep the agent in the loop without giving it the raw firehose, and that is the point of the verdict file. The model should read status, counts, and a hash, then decide whether to request a targeted rerun or to stop. It should not receive ten thousand lines of fixture dumps and then apologize for missing the assertion. That division of labor looks boring, which is a compliment, because boring contracts survive Friday night better than clever summaries.

A truncated XML file is the other half of the same camera freeze, and you should look at it once with your own eyes. Complete reports close the suite tag and carry a tests count that is not zero. Clipped reports often end in the middle of a failure message, which is enough for a parser to throw and enough for a tired agent to ignore.

<!-- complete enough to parse: the footer exists, tests is not zero -->
<testsuite name="checkout" tests="4" failures="1" errors="0" time="3.1">
  <testcase classname="checkout" name="test_lock_release" time="0.4">
    <failure message="AssertionError: expected 0 open locks">nil pointer</failure>
  </testcase>
</testsuite>

<!-- incomplete: tool output ended in the middle of the failure element -->
<testsuite name="checkout" tests="4" failures="1"
  <testcase classname="checkout" name="test_lock_release"
    <failure message="AssertionError: expected 0 open lo

When you want to rehearse that loop without spending a paid coding endpoint, a free model path is enough for the language half of the exercise. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding project with free model access and a free server option, which matters here as a bounded place to practice the verdict wrapper. Point the agent at the helper, give it only verdict.json, and watch whether it still invents a green summary from an incomplete status.

If you want to rehearse the contract on MonkeyCode's free models and free server, treat that environment as a sandbox rather than as CI. The server is useful when you need a shared shell for the helper, and it is the wrong place to store production secrets or release attestations. Use it to see whether the model obeys the incomplete branch, then bring the same helper back to your own runner. The product is a rehearsal room in this story, not the judge of whether checkout should ship.

This approach will not catch a test that never existed, and it will not catch a mock that lies with a well-formed XML file. It also will not help if your runner streams forever without a footer, because unknown must still mean do not merge. Teams that need cryptographic attestations, regulated release evidence, or hardware-in-the-loop traces should keep humans and real CI in that loop. You should not treat this wrapper as permission to skip CI, nor treat a free remote server as the system of record.

Skip the approach entirely if your agent cannot call a local program, or if you cannot store the raw report beside the verdict. Skip it if the suite is nondeterministic and a hash would only decorate flaky noise. Skip it if you need the model to read every assertion message in order to propose a patch, because then you need CI artifacts with full retention, not a clipped tool string. In those cases the honest move is to stop the agent before the boolean question is even asked.

Friday night merges will still happen, because software still ships when people are tired and the check is green. The difference is whether green means a complete observation or a camera that froze on a quiet hallway. Make the agent earn the boolean, and the next truncated log becomes a blocked merge instead of a pager.