Postmortem: Truncated Stream Applied a Partial File, Then the Loop Deleted the Rest

A truncated model stream is not a paused write. Partial bytes on disk count as a failed write. Retry without a length and digest check stores those bytes. The next agent loop then deletes remaining valid code.

The writeup reconstructs one concrete failure class in full. The text does not describe a named production outage. The lab below is a reproducible local fixture. Operators can run every listed step locally.

The core failure

A file apply path treated socket EOF as success. The stream stopped after forty percent of the payload. Disk held a prefix of a Python module. The next test command then raised SyntaxError. The loop treated that error as a bad implementation. The loop then emitted a delete-and-rewrite plan. The follow-up rewrite stream never actually started. The delete tool still ran to completion.

The durable fix is narrow and mechanical. Every write tool must send content-length and sha256. The worker writes a temp file, fsyncs, and renames. Any length or digest mismatch refuses the apply.

Scope and product bounds

This article does not name a hosted model. The article does not cite any token quotas. It does not claim hardware, uptime, or speed. Free model access and a free server option are lab conditions only.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option. Operators can replay this fixture on that disposable server. Cheap retries make truncated streams much more common. Those retries still do not change apply semantics.

Timeline (lab clock)

The timestamps below belong only to the fixture. They are not drawn from customer telemetry.

1. T+0s — Operator opens a scratch git repo.
2. T+8s — Agent starts a full rewrite of src/billing/ledger.py.
3. T+19s — Stream delivers 1,204 of 3,010 promised bytes.
4. T+19s — Socket closes. Worker logs stream_complete=true.
5. T+20s — Worker writes the prefix in place and fsyncs.
6. T+27s — Pytest imports the module and raises SyntaxError.
7. T+31s — Loop prompt includes traceback plus partial source.
8. T+40s — Model output orders a delete, then a recreate.
9. T+41s — Delete tool removes ledger.py. Recreate never starts.
10. T+55s — git status shows a deleted tracked file.

The failure window is under one minute. The delete leaves lasting damage in git history. History now records a delete without a replacement blob.

Contributing factors

Three independent mistakes had to line up here. Any one gate would have stopped the loss.

EOF mapped to success

The worker called read() until the socket closed. The worker ignored length trailers and missing hashes. A dropped stream then matched a finished stream. Shared free servers often preempt long sessions. That mapping turns preemption into data loss.

In-place writes

Bytes landed on the real path immediately. No adjacent temp file existed at all. No atomic rename occurred after the write. A crash left a truncated canonical file. Git already observed a dirty working path. Later recovery had no clean original on disk.

Loop repair without a completeness bit

The agent loop only saw a syntax error. The loop never received an apply.incomplete flag. The model optimized for a green import. Deleting the file removed the traceback entirely. That deletion looked like real forward progress. The recreate step was a second stream. Second streams can drop in the same way.

Detection signals

Log these fields on every write tool result. Missing fields mean a refused apply, not a default.

• promised_length — integer byte count after decode
• bytes_written — integer count actually fsynced
• sha256_promised — digest from the tool payload
• sha256_disk — digest of the temp file
• apply.incomplete — boolean, true on any mismatch
• stream_end_reason — trailer, socket_close, timeout, cancel

A useful alert is tiny. Fire when stream_end_reason is not trailer. Fire again when bytes_written != promised_length. Do not wait for pytest to notice.

Sample worker log line:

apply path=src/billing/ledger.py promised=3010 written=1204
end=socket_close incomplete=true digest_ok=false

That line is the incident start. The delete is only the second act.

Lab reconstruction

The fixture is local Python. It does not call a hosted API. This is a simulator. It is not a production agent.

Create truncate_lab.py:

from __future__ import annotations

import hashlib
import json
import os
import unittest
from pathlib import Path

PROMISED = b"""def post_invoice(ledger, invoice):
    if invoice.total < 0:
        raise ValueError("negative invoice")
    ledger.append(invoice)
    return invoice.id
"""

DIGEST = hashlib.sha256(PROMISED).hexdigest()
LENGTH = len(PROMISED)


def write_inplace(path: Path, data: bytes) -> None:
    path.write_bytes(data)


def write_atomic(path: Path, data: bytes, length: int, digest: str) -> None:
    if len(data) != length:
        raise ValueError("length_mismatch")
    got = hashlib.sha256(data).hexdigest()
    if got != digest:
        raise ValueError("digest_mismatch")
    tmp = path.with_suffix(path.suffix + ".tmp")
    with tmp.open("wb") as fh:
        fh.write(data)
        fh.flush()
        os.fsync(fh.fileno())
    os.replace(tmp, path)


def simulate_truncation(path: Path) -> dict:
    cut = PROMISED[: int(LENGTH * 0.4)]
    write_inplace(path, cut)
    on_disk = path.read_bytes()
    return {
        "bytes_on_disk": len(on_disk),
        "promised": LENGTH,
        "complete": len(on_disk) == LENGTH,
        "digest_ok": hashlib.sha256(on_disk).hexdigest() == DIGEST,
    }


def simulate_loop_delete(path: Path, report: dict) -> dict:
    if not report["complete"]:
        # Unsafe loop: treat syntax failure as "remove the file".
        path.unlink(missing_ok=True)
        return {"deleted": True, "recreated": False}
    return {"deleted": False, "recreated": False}


class ApplyTests(unittest.TestCase):
    def setUp(self) -> None:
        self.partial = Path("ledger_partial.py")
        self.ok = Path("ledger_ok.py")

    def tearDown(self) -> None:
        self.partial.unlink(missing_ok=True)
        self.ok.unlink(missing_ok=True)
        Path("ledger_partial.py.tmp").unlink(missing_ok=True)
        Path("ledger_ok.py.tmp").unlink(missing_ok=True)

    def test_reject_truncation(self) -> None:
        data = PROMISED[:40]
        with self.assertRaises(ValueError):
            write_atomic(self.partial, data, LENGTH, DIGEST)
        self.assertFalse(self.partial.exists())

    def test_accept_full_payload(self) -> None:
        write_atomic(self.ok, PROMISED, LENGTH, DIGEST)
        self.assertEqual(self.ok.read_bytes(), PROMISED)


if __name__ == "__main__":
    target = Path("ledger.py")
    report = simulate_truncation(target)
    after = simulate_loop_delete(target, report)
    print(json.dumps({"report": report, "after": after}, indent=2))

Run the simulator with a single command.

python3 truncate_lab.py

Expected JSON includes "complete": false. It also includes "deleted": true. The file ledger.py is gone. That is the incident in miniature.

Prove the unsafe path with git as well.

mkdir -p /tmp/billing-lab && cd /tmp/billing-lab
git init
cp /path/to/truncate_lab.py .
python3 truncate_lab.py
git add ledger.py 2>/dev/null || true
git status --short

Expected status shows a missing or deleted ledger.py. Tests cannot reconstruct the lost tail. Only git history can, and only if a commit existed.

Completeness gate

Replace in-place writes. Pass length and digest from the tool result. Fail closed on mismatch. Keep the canonical path untouched until rename.

Tool results need an explicit schema. Label: this is a contract example, not a vendor payload.

{
  "tool": "write_file",
  "path": "src/billing/ledger.py",
  "content_length": 3010,
  "sha256": "6f0c0e2c0a1b2d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e",
  "encoding": "utf-8"
}

Hash decoded file bytes. Do not hash compressed frames. Do not hash base64 wrappers. The worker must refuse a result with no content_length.

Run the unit tests next.

python3 -m unittest truncate_lab.py

The first test is the gate. The second test is the happy path. Both must remain in CI.

Decision table

Drive the worker from this table. Do not add silent defaults.

Stream signal	Disk action	Loop hint
Length match and digest match	Atomic rename	apply.ok=true
Length short	Delete temp only	apply.incomplete=true
Digest mismatch	Delete temp only	apply.corrupt=true
Socket error mid-write	Delete temp only	apply.incomplete=true
Tool result lacks length	Refuse apply	apply.refused=true

Never map socket_close to apply.ok. Closed sockets are common on free servers. Closed sockets are not completion.

Durable control changes

Code alone will rot. Add process controls.

1. Tool schema requires content_length and sha256.
2. Apply worker rejects missing fields. No defaults.
3. Canonical files change only through os.replace.
4. Loop prompts must include the apply hint bit.
5. Delete tools cannot target a path with apply.incomplete.
6. Git pre-commit refuses commits while .apply/incomplete exists.

Install a tiny pre-commit hook. Large hooks get skipped.

#!/usr/bin/env bash
set -euo pipefail
if [[ -f .apply/incomplete ]]; then
  echo "incomplete apply; refuse commit"
  exit 1
fi

Record the apply id in .apply/incomplete. Clear that file only after rename. Also verify the digest after rename.

A second hook can block deletes during incomplete apply.

#!/usr/bin/env bash
set -euo pipefail
# labeled example: refuse delete of paths listed in incomplete apply
while read -r status path; do
  if [[ "${status}" == D* && -f .apply/incomplete ]]; then
    echo "refuse delete during incomplete apply: ${path}"
    exit 1
  fi
done < <(git diff --cached --name-status)

Label: this hook is an example. Teams must adapt path filters.

Why the loop made it worse

Agent loops optimize the last error string. A SyntaxError points at a file. Removing the file removes the error. The model is not malicious here. The prompt lacked a completeness flag. Without that flag, amputation looks like cleanup.

A safe prompt suffix is small.

apply.incomplete=true
forbidden_tools=delete_path,git_rm
required_next=retry_apply_or_abort

The worker injects those lines. The model does not get a vote. If the model still emits delete, the tool layer refuses it.

Keep delete tools on an allowlist. A rewrite is two operations. The second operation needs its own complete payload. Do not chain them under one success bit.

Limits of this fix

The gate does not detect a complete but wrong program. A full hash can still encode a logic bug. Tests remain mandatory after a green apply.

The gate does not provide multi-tenant isolation. A shared free server still needs separate OS users. It also needs separate worktrees. Hashing a file does not separate tenants.

The gate does not replace backups. A completed delete is still a delete. Recover from git. Do not recover from optimism.

The lab uses local bytes. Real streams add chunking and retries. Hash the decoded payload. Do not hash transfer frames.

This write protocol does not fix invented tool paths. It does not fix recycled session slots. Keep those gates separate. Do not merge them into one boolean.

Who should not use this approach

Do not use a free server as a production control plane. The option is for labs and spikes.

Do not drop the length field to move faster. That was contributing factor one.

Do not allow delete after a syntax error without the apply bit. That was the data-loss step.

Teams without version control should not run destructive agent loops. There is no rollback path.

Do not run the simulator against a real working tree. Copy the fixture first.

What to keep

Treat every streamed write as a transaction. Complete, hashed, then renamed. Otherwise it did not happen. The next loop step needs that boolean. Without it, the model tidies a wound by amputation.