Code Agent Anatomy (14): Harness Design Part 4 — Fault Recovery

Errors Are the Norm, Not the Exception

For a long-running agent, errors are inevitable: the model occasionally returns empty content, context window exceeded, network jitter, a process crashes mid-tool-execution...

The question isn't "how to avoid errors" — it's what the agent should do after an error occurs. This article dissects MyCodeAgent's fault recovery system: runtime tiered retries, the persistent fact log, and state recovery after a crash.

---

Conclusions First

The fault recovery system has two dimensions:

Dimension	Mechanism	Problem Solved
Runtime self-healing	Error classification + tiered retry	Don't give up immediately on model errors; try automatic recovery first
Crash recovery	Transcript fact log + ResumeLoader	Resume from the interruption point after a process crash, no need to start over

The two dimensions work together: runtime self-healing handles predictable errors, while Transcript ensures that even if self-healing fails and the process crashes, no completed work is lost.

---

1. Error Classification: Understand the Error Type First

# runtime/model_errors.py
class ModelErrorKind(str, Enum):
    EMPTY_RESPONSE    = "empty_response"    # Model returned empty content — no text and no tool_call
    PROMPT_TOO_LONG   = "prompt_too_long"   # Context limit exceeded, model refused to process
    MAX_OUTPUT        = "max_output"        # Model output truncated (finish_reason="length")
    API_ERROR         = "api_error"         # RuntimeError, usually an API layer issue
    UNKNOWN_MODEL_ERROR = "unknown_model_error"

classify_model_error() accepts an exception object or response metadata and outputs a classification result with a recoverable flag. Classification is the prerequisite for retry decisions — different errors have completely different recovery strategies, and blanket retries only waste quota.

def _looks_like_prompt_too_long(message: str) -> bool:
    # Uses keyword matching rather than specific exception types
    # Reason: different providers throw different exception classes,
    # but error messages usually contain these phrases
    patterns = ("prompt too long", "context length", "context window",
                "too many tokens", "maximum context length", "request too large")
    return any(pattern in message for pattern in patterns)

String matching rather than exception types — this is an intentional design choice. Different LLM providers throw different exception classes, but the wording in error messages is highly consistent, making keyword matching more universal.

---

2. Runtime Self-Healing: Three Error Types, Three Recovery Paths

Errors occur inside the inner while True loop; successful recovery triggers continue to retry the current step without consuming the outer step quota.

Path 1: PROMPT_TOO_LONG → Compact and Retry

Model call throws exception → classify → PROMPT_TOO_LONG
  → reactive_compact() (LLM summarizes old history, produces a checkpoint)
  → build_model_view() (read-time projection, gets the compacted message list)
  → inner continue, retry invoke_raw()
  → compaction fails → MODEL_RECOVERY_FAILED → terminate

The retry limit is 1 (hardcoded in _get_model_recovery_limit). Compaction itself can also fail (LLM call timeout, insufficient turns) — if so, it goes directly to the termination path without looping forever.

Path 2: EMPTY_RESPONSE → Inject Hint and Retry

The model returned empty content (response_text is empty and no tool_calls) — usually because the model "hesitated" — unsure whether to continue with a tool or give a final answer.

if classification.kind is ModelErrorKind.EMPTY_RESPONSE and retry_count < retry_limit:
    hint = "Last response had empty content and no tool_calls. Please reply with a final answer in content, or use a tool call."
    # Append the hint to the end of this round's messages — NOT written to history_manager
    # Only affects this retry's model view, does not pollute the permanent history
    messages = base_messages + [{"role": "user", "content": hint}]
    continue  # inner retry

Notice the hint message is not written to history_manager — it's only temporarily appended to the message list for this send. This keeps history clean; the model won't see this "debug hint" on the next step.

The retry limit defaults to 1 (configurable via empty_response_retry_limit).

Path 3: MAX_OUTPUT → No Recovery Currently

finish_reason="length" means the model output was forcibly truncated. In theory this could be recovered with a "continuation" strategy, but the current recovery limit is 0:

if kind is ModelErrorKind.MAX_OUTPUT:
    return int(getattr(host, "max_output_recovery_limit", 0) or 0)
    # Returns 0 → retry_count >= retry_limit immediately → terminate directly

This effectively means no retry — it goes to MODEL_ERROR termination. This is an MVP placeholder: the structure is reserved, but the strategy hasn't been implemented yet.

---

3. Transcript: Append-Only Fact Log

Runtime self-healing handles "predictable errors." But what if the process crashes outright? In that case, a persistent fact log is needed — one that can reconstruct "where things were before the crash."

# runtime/transcript.py
class TranscriptStore:
    """One JSONL file per session, one event per line, append-only."""

    def append_event(self, event: TranscriptEvent) -> TranscriptEvent:
        with self._lock:
            self._repair_trailing_record()  # First repair any potentially incomplete last line
            with self.path.open("a", encoding="utf-8") as handle:
                handle.write(line)
                handle.write("\n")
                handle.flush()  # Flush immediately on every write to reduce data loss window

Five event types cover all key facts in the loop:

Event Type	Records	Recovery Use
MESSAGE	Message role + content + metadata	Reconstruct HistoryManager
STATE_TRANSITION	Transition reason + details	Reconstruct last LoopState
TOOL_LIFECYCLE	Four-phase tool status	Identify which tools didn't complete
CHECKPOINT	Compaction summary + split point	Reconstruct CompactStore
TERMINAL	Termination reason	Determine if completed normally

_repair_trailing_record() handles a special case: the process crashes between write and \n, leaving a partial JSON on the last line. Before each write, it checks and truncates this line, ensuring every line in the file is valid JSON:

def _repair_trailing_record(self) -> None:
    if data.endswith(b"\n"):
        return  # Normal ending, no repair needed
    # No newline at end: try parsing the last line
    try:
        json.loads(tail.decode("utf-8"))
        # Parse succeeded: just missing the newline, add it
        with self.path.open("ab") as handle:
            handle.write(b"\n")
    except (UnicodeDecodeError, json.JSONDecodeError):
        # Parse failed: this line is a partial write, truncate it
        handle.truncate(tail_start)

---

4. ResumeLoader: Rebuilding State from the Event Stream

After a crash and restart, ResumeLoader "replays" the event stream from the JSONL file to reconstruct ResumeState:

# Iterate over all events, handling each by type
for event in events:
    if MESSAGE:    → history_messages.append(...)
    if CHECKPOINT: → checkpoint = payload (passed to CompactStore later)
    if TERMINAL:   → terminal = payload (determine if it completed normally)
    if TOOL_LIFECYCLE: → tool_events[(run_id, tool_call_id)] accumulates phase states

Tool state handling is the most complex part, because a tool call has four phases and a crash can occur between any of them:

requested → started → completed / failed

ResumeLoader makes four judgments for each tool call's set of states:

State Set	Meaning	Recovery Handling
Contains completed	Successfully completed	completed_tool_results, don't replay
Contains failed	Already failed	failed_tool_results, don't replay
Only requested, no started	Requested but not yet started	pending_tool_calls, can be re-triggered
Has started, but no completed/failed	Started but result unknown	uncertain_actions

---

5. UncertainAction: Explicit Modeling of Uncertainty

started but no result is the most tricky situation — the tool executed, but we don't know if it succeeded.

UNSAFE_UNCERTAIN_REPLAY_TOOLS = {"Edit", "Bash", "Task"}

uncertain_actions.append(
    UncertainAction(
        tool_name=tool_name,
        tool_call_id=tool_call_id,
        step=step,
        replay_allowed=tool_name not in UNSAFE_UNCERTAIN_REPLAY_TOOLS,
    )
)

replay_allowed is classified by idempotency:

• Read/Grep/Glob: idempotent, replay has no side effects, replay_allowed=True
• Edit/Bash/Task: have side effects, unknown if the previous run succeeded — blindly replaying might duplicate file modifications or command executions, replay_allowed=False

Uncertainty is not silently handled — it's explicitly exposed to the user: when recovering via CLI, the uncertain actions list is printed, letting the user decide whether to continue. This is "transparent fault recovery" — the framework doesn't pretend to know what happened; instead, it honestly reports.

---

Design Highlights

1. Classification Before Retry

All recovery paths start with classification; different errors take different strategies. This avoids the most common fault-tolerance anti-pattern: blindly retrying all errors, wasting retry budget on both transient and unrecoverable errors alike.

2. Inner while Loop Isolates Retries

Retries happen via continue in the inner loop; the outer step counter step stays unchanged. The max_steps=50 quota is entirely spent on productive ReAct iterations, not consumed by error recovery.

3. Transcript Append-Only, Never Modified

Consistent with the HistoryManager design principle: write only, never modify historical events. This guarantees the integrity of the event stream — the state reconstructed after a crash and restart is identical to the state before the crash, with no risk of "introducing new problems during repair."

4. Uncertainty Explicitly Modeled

Uncertain actions are not an implementation detail — they're a design concept. It acknowledges that "there are situations the framework cannot automatically recover from." Rather than silently skipping or pretending to recover, it honestly informs the user and lets humans make the judgment.

---

Summary

Design Choice	Approach	Engineering Value
Error classification	Keyword matching + recoverable flag	Cross-provider compatible, classification before retry
Retry isolation	Inner while + continue	Retries don't consume step quota
Persistence	Append-only JSONL + flush	Small crash window, file always reconstructable
Last-line repair	_repair_trailing_record	Handles edge case of crash between write/flush
Uncertainty	UncertainAction + replay_allowed	Side-effect tools not blindly replayed, transparently informs user

---

About the Source Code for This Series

All analysis in this series is based on the open-source project MyCodeAgent.

The source code has been annotated at key locations following the order in which topics are covered in this series — you can read the articles alongside the code, or clone the repo and run, modify, and extend it to build your own agent.

git clone https://github.com/chendongqi/MyCodeAgent
cd MyCodeAgent
cp .env.example .env   # Fill in your LLM API key
uv sync
uv run python main.py

---

Visit PrimeSkills — a curated AI Agent and skills marketplace where every piece of content is validated against real enterprise workflows. No hype, only things that actually work.

For more practical insights and interesting products, visit my homepage