The Agent Crash-Looped on a Truncated Line: A Ledger Debugging Retrospective

The workflow had been running for six hours when the first alert fired. The apply stage was exiting with a JSONDecodeError, the supervisor was restarting it, and every restart died on the same line of the same file. The queue showed zero unacked messages, which meant the work was already marked done. I had a crash loop that would not die, and a set of decisions that were recorded but never applied.

My first instinct was to blame the model, and that instinct was wrong. This is the story of how the real root cause turned out to be a boundary condition in the pipeline, and why a disposable server and a free token allowance turned a panicked debugging session into a reproducible experiment.

The symptom

Here is the failure signature in its simplest form. The apply stage read the ledger line by line, parsed each line as JSON, and applied the decision to the database. One line was truncated mid-object, json.loads raised, and the process exited with code 1. The supervisor saw a non-zero exit and restarted the stage, which re-read the ledger from the top and hit the same line again.

writer -> ledger.jsonl: append raw model output (truncated)
writer -> queue: ack(item)
apply -> ledger.jsonl: read line 1..N
apply -> json.loads: line N
json.loads -> apply: JSONDecodeError
apply -> supervisor: exit 1
supervisor -> apply: restart
apply -> ledger.jsonl: read line 1..N   # same line, same crash

That loop is deterministic, fast, and completely independent of the model. The tempting diagnosis was model degradation, so I replayed the prompt manually and got a perfect JSON object. The model was fine in isolation. The problem was not what the model returned; it was what the pipeline did with the return value.

Step 1: Reproduce the loop before you theorize

Step one was reproduction, and I did it on MonkeyCode's free server tier because a supervisor restart loop burns CPU continuously. Disclosure: This article was prepared as part of MonkeyCode's product outreach. A disposable VM kept that noise off the shared production box, and the repro was two docker-compose services: a writer that appends model output to ledger.jsonl, and an apply stage that parses each line.

git clone <your-repro-repo> ledger-repro
cd ledger-repro
docker compose up --build apply

Within seconds the apply stage was crash-looping with the exact error from production. That is the first debugging lesson: reproduce the loop before you theorize about it.

Step 2: Bisect the ledger by length, not by content

Step two was inspecting the ledger, which was a JSON Lines file holding one raw model output per line.

tail -n 5 ledger.jsonl
wc -c ledger.jsonl
awk '{ print NR, length($0) }' ledger.jsonl | tail -n 10

The last line was 214 bytes and the median line was about 1,400 bytes. A truncated line stands out immediately once you look at lengths instead of content. The request had carried a token budget, the output had exceeded it, and the writer had appended the cut-off string without any validation.

Step 3: Characterize the truncation

Step three was characterizing the truncation with repeated calls to the same prompt. MonkeyCode is open source, so I could read the endpoint configuration instead of guessing, and its free model access, which offered ten million tokens as of this writing, made thirty test calls cheap. The pattern was consistent: the output stops at the budget boundary, and that boundary never aligns with JSON structure. That is not a model bug; it is an unhandled boundary condition.

Root cause

The root cause, stated as a chain: the writer trusted the model output and appended it raw; the apply stage trusted every line and parsed without a guard; the supervisor trusted the exit code and restarted blindly; and the queue had already acked the item, so the decision was lost, not delayed. Every layer assumed the layer below it was correct. That assumption is the actual bug.

Fix 1: Envelope every record before writing

Treat the ledger as a protocol, not a dump. A record is not the raw output; it is a struct with an item id, a timestamp, the output, and a checksum. The writer validates that the output is complete JSON before appending, and quarantines anything that fails.

def envelope(item_id: str, output: str) -> dict | None:
    try:
        json.loads(output)
    except json.JSONDecodeError:
        quarantine(item_id, output)      # never write poison
        return None
    return {
        "item_id": item_id,
        "ts": time.time(),
        "output": output,
        "sha256": hashlib.sha256(output.encode()).hexdigest(),
    }

Fix 2: Make the read path fail soft

The apply stage skips a bad line, moves it to quarantine.jsonl, and continues; it only exits non-zero if the quarantine write itself fails.

def recover_ledger(path: Path) -> Iterator[dict]:
    for line in path.open():
        try:
            yield json.loads(line)
        except json.JSONDecodeError:
            with quarantine_path.open("a") as qh:
                qh.write(line)          # fail soft, never silent

Fix 3: Alert on quarantine

Skipping is only safe if someone counts the skips. A counter on quarantine.jsonl turned a silent gap into a visible metric, and the on-call rotation stopped guessing whether decisions were being applied.

Validate with failure injection

The fix is only credible if it holds under injected failures, so I wrote a small property test that truncates a valid output at every possible byte. It asserts two properties: the write path never appends poison, and the read path never crash-loops.

def test_truncation_never_poisons_the_ledger():
    valid = json.dumps({"label": "escalate", "reason": "x" * 500})
    for cut in range(1, len(valid)):
        record = envelope("item-1", valid[:cut])
        assert record is None                  # rejected at the boundary
        assert quarantine_count() == cut        # counted, not dropped

def test_bad_line_never_crash_loops():
    ledger = Path("fixtures/ledger.jsonl")
    ledger.write_text('{"item_id": "ok"}\n{"truncated"\n')
    decisions = list(recover_ledger(ledger))
    assert len(decisions) == 1                  # good line survived
    assert quarantine_path.exists()             # bad line was quarantined

Run this under a supervisor that restarts on non-zero exit and you have a testable acceptance rule: the apply stage must exit zero for every truncated input. That is the property I care about, not a benchmark score.

Tradeoffs

Every fix has a price, and the tradeoffs are worth making explicit.

Decision	Option A	Option B	What you pay
Ledger format	Raw lines	Enveloped records	~100 bytes per record for validation and checksums
Write path	Trust model output	Validate before append	A few ms of latency; poison never enters state
Read path	Crash on bad line	Quarantine and continue	Gaps are possible, so alerting becomes mandatory
Ack timing	Ack after write	Ack after apply	Later ack risks duplicates; earlier ack risks loss

Who should not use this

If your agent is a stateless request-response loop with no ledger, none of this applies. If you need strict exactly-once apply, quarantine is not enough; you need a transactional outbox or an idempotency key on the apply side. And if you cannot tolerate any gap in applied decisions, fail-soft recovery will scare you more than a crash loop, because a crash loop is loud and a quarantine gap is quiet.

The counterexample I keep asking myself

Suppose the writer appends a valid envelope and crashes before the queue ack; the item is re-delivered, the writer appends a second envelope with the same item_id, and the apply stage sees two decisions for one item. Does your recovery deduplicate by item_id, or does it apply both? If you cannot answer that, the ledger is still a protocol you do not fully control.

The whole repro, including the injected-truncation tests, fits comfortably inside a free server and a ten-million-token allowance. The MonkeyCode open-source project is a reasonable place to start if you want to run it yourself.