DEV Community

Cover image for Code Agent Anatomy (15): Harness Design Part 5 — Observability
WonderLab
WonderLab

Posted on

Code Agent Anatomy (15): Harness Design Part 5 — Observability

#ai #agents #harness #tracing

The Last Piece of the Puzzle

The previous four articles dissected the agent's control flow, context engineering, tool pipeline, and fault recovery. All of these mechanisms share a common problem: when something goes wrong, how do you know what happened?

An agent running in the background — which tool the model chose, how many times history was compacted, which permission rule blocked a call — these all happen inside the code but are invisible by default. This article dissects MyCodeAgent's observability system, examining how it turns internal behavior into queryable records.

Conclusions First

The observability system consists of three layers:

Layer Component Responsibility
Emission layer RuntimeRunner._emit() Unified event entry point — all observable facts in the loop originate here
Routing layer CompositeRuntimeEventSink Delivers one event to multiple sinks simultaneously, mutually isolated
Persistence layer TraceRuntimeEventSink + TranscriptRuntimeEventSink Write to JSONL diagnostic logs and append-only fact transcript respectively

One event is emitted, two paths are taken simultaneously — diagnostics and recovery each get what they need.

1. Unified Emission Point: All Events Originate from One Place

Every "thing worth recording" in the loop is emitted through the same method:

# runtime/loop.py  RuntimeRunner
def _emit(self, event_type: str, payload: dict[str, Any], *, step: int) -> None:
    self._emit_runtime_event(
        run_id=self._get_transcript_run_id(),
        step=step,
        event_type=event_type,
        payload=payload,
    )
Enter fullscreen mode Exit fullscreen mode

_emit accepts three elements: event type, payload, step number — constructing a RuntimeEvent object and handing it to the sink router.

Call sites are scattered throughout the loop, but there's only one entry point. This means: adding a new sink (for example, sending events to Prometheus) only requires modifying the CompositeRuntimeEventSink constructor — the loop itself doesn't need to change.

Emission points cover all key moments in the loop:

message          → message written to history (user/assistant/tool roles)
state_transition → state transitions (USER_INPUT / TOOLS_EXECUTED / MODEL_RECOVERY_RETRY ...)
tool_lifecycle   → tool four phases (requested / started / completed / failed)
checkpoint       → context compaction checkpoint
terminal         → loop termination (completed / max_steps / token_budget ...)
prompt_assembly  → fingerprints of each prompt layer per step
tool_schema      → hash of tool schema (detects tool list changes)
model_output     → raw model response (including token usage)
Enter fullscreen mode Exit fullscreen mode

2. Dual Sink: Diagnostics and Persistence Each Serve Their Own Purpose 

# runtime/events.py
def create_runtime_event_sink(trace_logger, recorder) -> CompositeRuntimeEventSink:
    return CompositeRuntimeEventSink(
        (TraceRuntimeEventSink(trace_logger), TranscriptRuntimeEventSink(recorder))
    )
Enter fullscreen mode Exit fullscreen mode

CompositeRuntimeEventSink delivers the same event to two sinks:

def emit(self, event: RuntimeEvent) -> None:
    for sink in self.sinks:
        try:
            sink.emit(event)
        except Exception as error:
            # Sink failure only logs a warning, does not affect loop state
            logger.warning("Runtime event sink failed for %s: %s", event.type, error)
Enter fullscreen mode Exit fullscreen mode

Isolation is key: a Transcript sink write failure doesn't affect the Trace sink, nor does it affect the loop continuing to run. Failures in observability infrastructure must not stop the agent.

Trace Sink: Diagnostic Logs

TraceRuntimeEventSink translates events into diagnostic format and forwards them to TraceLogger, writing to memory/traces/trace-{session}.jsonl. This is the log for development and debugging:

  • Each line is a JSON object with timestamp, session_id, step, event name, and payload
  • tool_lifecycle events are additionally split into tool_call (when called) and tool_result (when completed), allowing HTML reports to display them separately
  • After the file is written, an HTML report can be generated for direct human reading

Transcript Sink: Recoverable Facts

TranscriptRuntimeEventSink only handles five types of "recoverable fact" events (message/state_transition/tool_lifecycle/checkpoint/terminal), writing to memory/transcripts/transcript-{session}.jsonl. This is the log for crash recovery (see article 14) — it doesn't record debug details, only the minimal set of facts needed to reconstruct state.

The same event is emitted, two paths each take what they need: Trace takes all details, Transcript takes only the facts needed for state recovery.

3. TraceLogger: JSONL Streaming Write 

# extensions/tracing/logger.py
def log_event(self, event: str, payload: dict[str, Any], step: int = 0) -> None:
    event_obj = {
        "ts": _utc_now().isoformat(),
        "session_id": self.session_id,
        "step": step,
        "event": event,
        "payload": self._sanitizer.sanitize(payload),  # sanitize first
    }
    self._current_run_events.append(event_obj)  # keep in memory for this run
    self._write_line(event_obj)                  # write to disk immediately
    self._update_stats(event, payload, step)     # update statistics

def _write_line(self, event_obj: dict[str, Any]) -> None:
    with self._lock:                             # write lock to prevent concurrent corruption
        if self._file_handle:
            self._file_handle.write(json.dumps(event_obj, ensure_ascii=False) + "\n")
            self._file_handle.flush()            # flush immediately on every write
Enter fullscreen mode Exit fullscreen mode

Every log_event flushes immediately — same strategy as Transcript, aimed at shrinking the crash window: if the process crashes between two events, already-flushed events won't be lost.

_current_run_events keeps all events from the current run in memory, consumed when generating the HTML report (at finalize()).

4. Prompt Fingerprint: Detecting Prompt Drift

At the start of each step, trace_model_request_state() computes and emits fingerprints for each layer of the prompt:

# runtime/events.py  trace_model_request_state()
current = {
    "constitution": prompt_assembly.constitution_fingerprint,
    "tool_contracts": prompt_assembly.tool_contracts_fingerprint,
    "project_rules": prompt_assembly.project_rules_fingerprint,
    "runtime_signals": prompt_assembly.runtime_signals_fingerprint,
}
emit("prompt_assembly", {
    ...
    "changed_layers": [key for key, value in current.items()
                       if previous.get(key) not in (None, value)],
})
Enter fullscreen mode Exit fullscreen mode

changed_layers lists the layers that changed in this step compared to the previous step. For example, if Skills were updated, the tool_contracts layer fingerprint changes, and the trace will show "changed_layers": ["tool_contracts"].

This solves a debugging pain point: model behavior suddenly changes at a certain step, but it's unclear why. By comparing fingerprints, you can precisely identify which prompt layer drifted.

The same treatment applies to tool schemas:

fingerprint = hashlib.sha256(
    json.dumps(tools_schema, ...).encode()
).hexdigest()
emit("tool_schema", {"fingerprint": fingerprint, "changed": previous != fingerprint})
Enter fullscreen mode Exit fullscreen mode

5. TraceSanitizer: Sanitization Makes Logs Shareable

Trace logs may contain sensitive data. TraceSanitizer performs a scan before each event is written to disk:

# extensions/tracing/sanitizer.py
SENSITIVE_KEYS = {
    "api_key", "token", "password", "authorization", "session_id", ...
}
PATTERNS = [
    (re.compile(r"sk-[a-zA-Z0-9]{20,}"), "sk-***"),      # OpenAI key
    (re.compile(r"Bearer\s+[a-zA-Z0-9._+/=-]{20,}"), "Bearer ***"),
]

def _sanitize_dict(self, data: Dict[str, Any]) -> Dict[str, Any]:
    for key, value in data.items():
        if key.lower() in self.SENSITIVE_KEYS:
            result[key] = "***"     # key hit: replace value directly
            continue
        if "path" in key.lower() and isinstance(value, str):
            result[key] = self._sanitize_string(value)  # sanitize usernames in paths
            continue
        result[key] = self.sanitize(value)              # recursively handle nested structures
Enter fullscreen mode Exit fullscreen mode

Two types of coverage:

  • Key blacklist: any key matching api_key, token, etc. gets its value replaced with *** regardless of the value
  • Value regex: patterns matching sk-..., Bearer ..., etc. cover cases where the key name is innocuous but the value is sensitive

Usernames in paths (/home/alice/, /Users/alice/) are also replaced with ***, preventing developers from accidentally leaking their local username.

TRACE_SANITIZE=false disables this; it's enabled by default.

Design Highlights

1. Event-Driven, Not Log Instrumentation

Observable points in the loop emit semantic events (state_transition, tool_lifecycle) rather than print or logger.info. Events have structured payloads that downstream can process with code; logs can only be read by human eyes.

2. Sink Failure Doesn't Affect the Loop

CompositeRuntimeEventSink calls each sink inside try/except; failures only log a warning. Observability infrastructure is a "side channel" for the agent, not the main channel. Issues with the main channel are what matter — side channel failures shouldn't bring down the main channel.

3. Two JSONL Files, Two Perspectives

The Trace JSONL is the diagnostic perspective: what happened, at which step, when — aimed at developer debugging. The Transcript JSONL is the recovery perspective: which facts need to be retained to reconstruct state — aimed at crash recovery. The same event stream, two purposes, written to disk separately.

4. Fingerprint Rather Than Diff

Prompt change detection uses fingerprints (sha256) rather than storing full text for diffing: saves storage, comparison is O(1), and in the trace you just look at changed_layers to know which layer changed — no need to open two files and compare manually.

Summary

Design Choice Approach Engineering Value
Emission point Unified _emit(), not scattered Adding a sink doesn't require modifying the loop
Routing CompositeRuntimeEventSink Multiple sinks mutually isolated; single failure doesn't affect others
Diagnostic log TraceLogger JSONL + flush Streaming write, small crash window, can generate HTML report
Recovery log TranscriptRuntimeEventSink Writes only minimal fact set, not mixed with debug details
Drift detection Prompt fingerprint O(1) comparison, precisely locates which prompt layer changed
Sanitization TraceSanitizer key blacklist + value regex Logs are shareable without fear of leaking API keys

This completes all of Part 4 Harness Engineering. A recap of these five articles:

  • 11: Control flow — single loop, immutable state machine, completion gate, exhaustive termination paths
  • 12: Context engineering — History and ModelView separation, read-time projection, dual-trigger compaction
  • 13: Tool pipeline — concurrent grouping, four-checkpoint execution, two-layer byte budget
  • 14: Fault recovery — classified retry, Transcript fact log, UncertainAction explicit modeling
  • 15: Observability — event-driven, dual-sink routing, fingerprint drift detection

These five mechanisms together form the "skeleton" of an agent framework — determining how long it can run, how much it can self-heal after errors, how much it can recover after a crash, and how deep you can investigate when something goes wrong.

About the Source Code for This Series

All analysis in this series is based on the open-source project MyCodeAgent.

The source code has been annotated at key locations following the order in which topics are covered in this series — you can read the articles alongside the code, or clone the repo and run, modify, and extend it to build your own agent.

git clone https://github.com/chendongqi/MyCodeAgent
cd MyCodeAgent
cp .env.example .env   # Fill in your LLM API key
uv sync
uv run python main.py
Enter fullscreen mode Exit fullscreen mode

Visit PrimeSkills — a curated AI Agent and skills marketplace where every piece of content is validated against real enterprise workflows. No hype, only things that actually work.

For more practical insights and interesting products, visit my homepage

Top comments (0)