Postmortem: The Agent Loop That Died Quietly While the Eval Said Green

A green eval is not a green light for the pipeline. This article walks through a synthetic incident reconstructed in a local harness. No production outage is claimed. The goal is to expose a common failure mode and a durable fix.

At 03:47 UTC, batch job 8447 stopped producing results. The review harness reported 41/41 checks passed. No alert fired. Six hours passed before a human noticed.

This postmortem examines that silent failure. It covers the timeline, the contributing factors, and the durable fix. The pattern applies to any AI agent pipeline, including free-tier setups.

MonkeyCode provides free model access and a free server option for experiments. Limits and terms change. Check the current docs. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The same reliability rules apply whether the worker runs on a free server or a paid cluster.

Timeline

All timestamps are UTC in the reproduction scenario.

Time	Event
02:10	Batch job 8447 starts.
02:10–03:45	Worker processes 212 tasks.
03:45	Queue reaches capacity. Worker requests more tasks.
03:46	Worker receives an empty page. Marks itself idle.
03:47	A socket timeout kills the worker's heartbeat.
03:47	Queue has 1,890 tasks. Nobody is consuming them.
08:12	A developer checks the dashboard.
08:14	Operator restarts the worker.
08:15	Worker resumes processing from checkpoint.

The failure produced no exception. The log contained one line: heartbeat_interval exceeded. The monitoring dashboard showed the worker as "idle." Idle looked healthy.

What Actually Broke

Three components failed together.

1. The model review harness measured outputs only.
2. The free server option has limited resources, but nobody checked the worker's liveness.
3. The worker treated "idle" as a valid end state.

The review harness compared generated patches against a contract. Every patch was valid. The harness had nothing to say about the queue, the disk, or the heartbeat.

Contributing Factors

The harness measured the model, not the system

The harness scored model outputs. It did not score the delivery path. A model can generate a perfect patch and the pipeline can still lose it.

The worker had no liveness contract

The handler loop had no heartbeat. When the socket closed, the process stayed alive. It just stopped pulling work.

Retries were not idempotent

The job state lived only in memory. A restart lost the cursor. The operator had to guess which tasks were already complete.

The Durable Fix

The fix has three layers.

1. Infrastructure contract test

Before every batch, run a test that verifies the environment.

def infrastructure_check(settings):
    checks = {
        "queue_connect": ping_queue(settings.queues.url),
        "disk_headroom": disk_free_pct(settings.worker.path) > 20,
        "model_endpoint": model_ready(settings.model.url),
        "clock_delta": abs(clock_skew()) < 5,
    }
    failed = [k for k, ok in checks.items() if not ok]
    if failed:
        raise SystemExit(f"infra check failed: {failed}")

This test runs in the orchestrator. It does not run inside the model's prompt. It checks the actual environment the worker will use.

2. Heartbeat dead-man switch

The worker writes a heartbeat file every 15 seconds. A supervisor watches it.

import time, pathlib

HEARTBEAT = pathlib.Path("/var/run/agent/heartbeat")

def beat():
    HEARTBEAT.write_text(str(time.time()))

def heartbeat_fresh(timeout=45):
    return time.time() - float(HEARTBEAT.read_text()) < timeout

If the heartbeat goes stale, the supervisor kills the process. Then the orchestrator starts a fresh worker. Idle is no longer a valid silent state.

3. Idempotent checkpointing

The worker persists task state to SQLite before applying a side effect.

CREATE TABLE jobs (
    task_id   TEXT PRIMARY KEY,
    status    TEXT NOT NULL,
    updated_at TEXT NOT NULL
);

SELECT task_id FROM jobs WHERE status = 'queued' LIMIT 1;

A task is safe to process only after a transaction marks it in_progress. On restart, the worker scans for tasks in that state. It can safely re-run them because the application is idempotent.

def reserve_next(conn):
    conn.execute("BEGIN")
    row = conn.execute(
        "SELECT task_id FROM jobs "
        "WHERE status = 'queued' LIMIT 1"
    ).fetchone()
    if row:
        conn.execute(
            "UPDATE jobs SET status = 'in_progress', "
            "updated_at = datetime('now') "
            "WHERE task_id = ?", (row[0],))
    conn.commit()
    return row

Validation

The reproduction harness verified the fix.

• Infrastructure test passed before the worker started.
• The heartbeat monitor killed a simulated dead worker within one timeout window.
• Checkpoint resume restored the full task queue.
• Two forced restarts lost zero tasks.

The fix converted a six-hour outage into a 36-second restart.

Limitations

This approach does not fix everything.

• It does not detect degraded model quality.
• It does not replace integration tests run by a human.
• It assumes the side effect is idempotent. Non-idempotent operations need external idempotency keys.
• The free server option is suitable for experiments and low-stakes workloads. Critical production traffic needs a stronger resource contract.

Who Should Not Use This Pattern

Teams that cannot tolerate duplicate side effects should not adopt this pattern as-is. Payment operations and message sends need idempotency keys before retry. Teams that have no monitoring system should still implement the heartbeat. It is the cheapest alarm you can build.

Takeaway

The review harness answered the wrong question. It asked: "Does the model output meet the contract?" The real question was: "Will the output reach the user?" Those are different claims.

Run an infrastructure contract test before every batch. Give each worker a heartbeat. Persist the state. A green eval then means only one thing: the model was correct. It says nothing about the delivery path.

Measure the whole loop, not just the model. That is the only way an exhaustive review cannot hide an empty queue.