Postmortem: Truncated Tool JSON Applied Half a Schema Change

A truncated tool payload is still a write. This reconstructed lab incident applied six of eleven statements. The job record still showed a success status. The durable fix is a trailer, a checksum, and a two-phase apply.

Scope and disclosure

This write-up reconstructs a local fixture, not a production outage. Logs, patches, and counts come from that fixture. No customer workload is claimed in this write-up.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. Those two availability claims are the only product facts used here. Model names, quotas, and hardware are omitted on purpose.

The fixture used a shared apply worker for agent jobs. The worker streamed tool JSON from one session. The session targeted a small SQL schema change only.

What broke

The agent emitted an apply_schema tool call as JSON. The byte stream ended inside the seventh statement. The worker parsed the prefix it already held.

It executed every complete statement in that prefix. It then marked the job complete without a trailer. Review later opened the database and found a hole.

Six ALTER statements were present on disk. Five planned statements never reached the engine. The migration table still listed the job as applied.

Timeline from the lab fixture

All times below use the fixture clock only.

1. T+0s. The worker accepts job job_7f3a with a clean tree.
2. T+2s. The agent plans eleven statements and a trailer hash.
3. T+9s. The stream starts as one JSON object over stdout.
4. T+14s. The connection closes after byte 1882 of 3410.
5. T+14s. The streaming decoder returns a partial object without error.
6. T+15s. The apply loop runs six complete SQL statements in order.
7. T+16s. The seventh statement is a fragment, so the loop stops.
8. T+16s. The worker writes status=success because error is empty.
9. T+40s. The review bot reads job status and skips row counts.
10. T+6m. A later job assumes users.email_verified exists, then fails.

The failure was delayed by several minutes. The first job looked healthy in the queue UI. The second job discovered the missing column during apply.

Contributing factors

Three independent mistakes stacked in one path. Any one of them could hide for weeks. Together they turned a closed socket into a schema change.

Best-effort JSON parsing

The worker used a streaming decoder on tool output. That decoder yielded a dict when the socket closed. It did not require a complete end-of-object marker.

Incomplete input became a Python object anyway. Missing keys were treated as optional by the apply loop. Truncation looked like a short but valid plan.

Success meant no exception

The apply loop treated a stop as a finish. A SQL fragment raised no engine error. Autocommit had already sealed statement one through six.

Empty error became status=success in the job row. No statement counter was stored beside that status. Review therefore had nothing to diff against the plan.

The agent treated silence as completion

The agent never received a retry prompt after close. It stored the partial tool result as memory. Later steps treated that memory as ground truth.

This is a common agent failure mode under load. A closed stream looks like a finished thought. The worker must not agree with that assumption.

Artifact: reject incomplete apply payloads

The fix is mechanical and testable in one file. An apply payload must carry a length, a checksum, and a trailer. Missing any field is a hard fail, not a retry hint.

The worker stages SQL only after verify. It commits only inside one transaction. A truncated body never reaches BEGIN.

Payload contract

{
  "tool": "apply_schema",
  "job_id": "job_7f3a",
  "stmt_count": 11,
  "sha256": "hex digest of joined statements",
  "statements": ["...", "..."],
  "trailer": "END_APPLY"
}

Rules the worker enforces on every job:

• stmt_count must equal len(statements) exactly.
• sha256 covers the joined statements plus one newline.
• trailer must be the exact token END_APPLY.
• Parse must use a complete decoder, never a streaming guess.
• SQL runs inside one transaction with autocommit off.

Validator and two-phase apply

# apply_gate.py
from __future__ import annotations

import hashlib
import json
import sqlite3
from dataclasses import dataclass
from typing import Any

TRAILER = "END_APPLY"
REQUIRED = ("tool", "job_id", "stmt_count", "sha256", "statements", "trailer")


class IncompleteApply(Exception):
    """Raised when the tool payload is truncated or unsigned."""


@dataclass(frozen=True)
class ApplyPayload:
    job_id: str
    statements: list[str]
    sha256: str


def parse_complete(raw: bytes) -> ApplyPayload:
    stripped = raw.rstrip()
    if not stripped.endswith(b"}"):
        raise IncompleteApply("stream closed before JSON object ended")
    try:
        data: dict[str, Any] = json.loads(stripped.decode("utf-8"))
    except json.JSONDecodeError as exc:
        raise IncompleteApply(f"json decode failed: {exc}") from exc
    missing = [k for k in REQUIRED if k not in data]
    if missing:
        raise IncompleteApply(f"missing keys: {missing}")
    if data["trailer"] != TRAILER:
        raise IncompleteApply("trailer mismatch")
    stmts = data["statements"]
    if not isinstance(stmts, list) or data["stmt_count"] != len(stmts):
        raise IncompleteApply("stmt_count does not match statements")
    joined = "\n".join(stmts).encode("utf-8")
    digest = hashlib.sha256(joined).hexdigest()
    if digest != data["sha256"]:
        raise IncompleteApply("checksum mismatch")
    if data["tool"] != "apply_schema":
        raise IncompleteApply("unexpected tool")
    return ApplyPayload(data["job_id"], stmts, digest)


def apply_in_transaction(conn: sqlite3.Connection, payload: ApplyPayload) -> None:
    conn.execute("BEGIN")
    try:
        for stmt in payload.statements:
            conn.execute(stmt)
        conn.execute(
            "INSERT INTO schema_jobs (job_id, stmt_count, sha256, status) "
            "VALUES (?, ?, ?, ?)",
            (payload.job_id, len(payload.statements), payload.sha256, "applied"),
        )
        conn.commit()
    except Exception:
        conn.rollback()
        raise

Repro test

# test_apply_gate.py
import hashlib
import json
import sqlite3

import pytest

from apply_gate import IncompleteApply, apply_in_transaction, parse_complete

STMTS = [
    "CREATE TABLE IF NOT EXISTS schema_jobs ("
    "job_id TEXT PRIMARY KEY, stmt_count INT, sha256 TEXT, status TEXT)",
    "CREATE TABLE IF NOT EXISTS users (id INTEGER PRIMARY KEY, email TEXT)",
    "ALTER TABLE users ADD COLUMN email_verified INTEGER DEFAULT 0",
]


def _payload(stmts):
    joined = "\n".join(stmts).encode("utf-8")
    body = {
        "tool": "apply_schema",
        "job_id": "job_7f3a",
        "stmt_count": len(stmts),
        "sha256": hashlib.sha256(joined).hexdigest(),
        "statements": stmts,
        "trailer": "END_APPLY",
    }
    return json.dumps(body).encode("utf-8")


def test_truncated_json_is_rejected():
    raw = _payload(STMTS)[:-40]
    with pytest.raises(IncompleteApply):
        parse_complete(raw)


def test_count_mismatch_is_rejected():
    data = json.loads(_payload(STMTS))
    data["stmt_count"] = 99
    with pytest.raises(IncompleteApply):
        parse_complete(json.dumps(data).encode())


def test_full_payload_commits_all_or_nothing():
    conn = sqlite3.connect(":memory:")
    payload = parse_complete(_payload(STMTS))
    apply_in_transaction(conn, payload)
    cols = [r[1] for r in conn.execute("PRAGMA table_info(users)")]
    assert "email_verified" in cols
    row = conn.execute(
        "SELECT status FROM schema_jobs WHERE job_id='job_7f3a'"
    ).fetchone()
    assert row[0] == "applied"

Run the tests with a quiet pytest invocation:

python -m pytest test_apply_gate.py -q

A second check slices a real payload on disk. It proves the worker fails closed on short reads.

python -c "from apply_gate import parse_complete, IncompleteApply; \
from test_apply_gate import STMTS, _payload; \
raw=_payload(STMTS); open('/tmp/apply.json','wb').write(raw); print(len(raw))"
head -c 120 /tmp/apply.json > /tmp/apply.cut.json
python -c "from apply_gate import parse_complete, IncompleteApply; \
import pathlib; raw=pathlib.Path('/tmp/apply.cut.json').read_bytes();\
\ntry:\n    parse_complete(raw)\n    raise SystemExit('expected IncompleteApply')\nexcept IncompleteApply as exc:\n    print('nack', exc)"

The truncated case must fail closed every time. A green bar on this file is the gate. Do not ship the worker until both rejection tests pass.

Decision table

Use this table before an agent apply touches schema or config. Idle timeout is not success in this table. Empty error is not success either.

Signal	Treat as	Action
JSON decode error	Truncation	Nack job, do not write
Missing trailer	Truncation	Nack job, do not write
stmt_count mismatch	Logic error	Nack job, do not write
Checksum mismatch	Corruption or mix-up	Nack job, quarantine bytes
SQL error mid-loop	Partial apply risk	Rollback, mark failed
Stream idle timeout	Unknown completeness	Nack, retry with a new job id
All fields valid	Complete payload	Begin transaction, then commit

Only a verified trailer is success. Store truncated as a first-class job state. Do not overload failed for this case.

How the free-server path fits

A shared free server is a useful place to practice this gate. Jobs share bandwidth and session lifetime on that path. Streams can end for reasons the agent does not see.

Free model access is enough to generate tool JSON. It is not enough to trust that JSON on disk. The worker in this fixture sat in front of the model.

The model never talked to SQLite directly in the fixture. That split is the entire control point. Generation can be cheap on a free path. Apply must stay strict and local.

Readers who already isolate checkouts still need this check. Isolation does not detect a half object in memory. A fair queue does not detect a half object either. Completeness is a separate control from scheduling.

Durable fix in the worker

The production-shaped patch is small and local. Buffer the full tool body before any parse. Do not parse incrementally while tokens arrive.

Require trailer, stmt_count, and sha256 together. Disable autocommit for the whole job. Write status=applied inside that same transaction.

On any IncompleteApply, persist status=truncated and the byte length. Issue a new job_id on retry after truncation. Never resume a partial body under the old id.

Teach the agent one durable line of policy. A closed stream is not success. A missing trailer is not a hint to continue.

# worker_loop.py
import sys
from apply_gate import IncompleteApply, apply_in_transaction, parse_complete


def handle(raw: bytes, conn) -> str:
    try:
        payload = parse_complete(raw)
    except IncompleteApply as exc:
        return f"truncated:{exc}"
    apply_in_transaction(conn, payload)
    return "applied"


if __name__ == "__main__":
    raw = sys.stdin.buffer.read()
    print(handle(raw, None))

Replace conn with a real connection in the service. Keep the stdin path for tests and for head -c drills. Log the raw byte length beside every nack.

Limitations

This gate does not prove the SQL is correct. It only proves the payload is complete. A complete wrong migration still commits in full.

Checksums do not replace human review of statements. They replace silence from a closed socket. Review still reads the statements before merge.

SQLite in the tests is not Postgres or MySQL. Statement packing and transactional DDL differ by engine. Port the transaction model before using this on another engine.

The fixture does not measure model quality at all. It measures worker hygiene around tool JSON. Do not treat a passing pytest file as a model eval.

Who should not use this approach

• Do not use it as the only migration tool in regulated systems.
• Do not run it without backups and a migrate-down path.
• Do not let the agent retry the same job_id after truncation.
• Do not parse tool JSON with a regex or a best-effort decoder.
• Teams without a review step should not auto-apply schema from any model.

Shared free paths make truncation more likely, not less. That is a reason for the gate. It is not a reason to skip backups.

What the incident actually taught

Agents fill gaps when output stops early. A closed socket looks like a finished thought. The worker must refuse that story without a trailer.

Status fields need the same proof as data rows. success without a trailer is a false record. Store truncated beside the observed byte length.

The second job failed for a missing column. The first job was the real incident. Delayed detection is part of the bug, not a sequel.

Close

The reconstructed fixture is small on purpose. Copy the tests and break the payload on purpose. Confirm the worker nacks, then wire the same checks in front of any apply path.

That includes a path backed by free model access on a free server. Readers who already gate generated patches can add this trailer check beside those gates.