How I Built Observability for My Autonomous Coding Agent: 5 Lessons

TL;DR

I run an autonomous coding agent that works unattended for hours at a time, and for months I had almost no visibility into what it was actually doing between "started" and "done." I built a thin structured-logging layer on top of its tool calls, and it turned debugging from archaeology into just... reading. Here's what I logged, what I skipped, and the mistake that cost me the most time.

The Problem

My agent runs long sessions without me watching. It reads files, edits code, runs commands, and eventually reports back with a summary like "Refactored the auth module, all tests pass."

The problem: that summary is generated by the same agent that might be wrong. If it silently skipped a step, misread a file, or "fixed" the wrong function, the final report still reads as confident and clean. I had no independent record of what actually happened during the run — only the agent's own retrospective narration of itself.

This bit me hard once. The agent reported a successful refactor. Tests were green. Except it turned out that a step earlier in the session had silently failed to apply an edit, so the "passing tests" were testing the old code, unchanged. Nothing crashed. Nothing looked wrong. It just quietly did less than it claimed.

That's when I realized: a silent partial failure is worse than a loud one. A crash tells you where to look. A silently skipped step tells you nothing, and the final report actively lies to you by omission.

I needed a way to see the actual sequence of actions the agent took — independent of whatever story it told me afterward.

How I Solved It

I added a logging layer that sits between the agent and its tools, not inside the agent's own reasoning. That distinction matters: if the log is something the agent writes about itself, it can be wrong in the same way its summary can be wrong. If the log is emitted by the harness every time a tool actually executes, it's a fact, not a narration.

What I log

For every tool call, I capture:

• Timestamp
• Tool name and a redacted summary of its arguments (no secrets, no full file contents — just enough to identify what happened)
• Result status: success, error, or "no-op" (this last one turned out to be the most valuable category)
• Duration

import json
import time

def log_tool_call(tool_name, args_summary, status, duration_ms):
    entry = {
        "ts": time.time(),
        "tool": tool_name,
        "args": args_summary,
        "status": status,       # "success" | "error" | "noop"
        "duration_ms": duration_ms,
    }
    with open("agent_trace.jsonl", "a") as f:
        f.write(json.dumps(entry) + "\n")

Each run appends to a .jsonl file — one JSON object per line. That format matters more than it sounds: it's append-only, crash-safe (a truncated last line doesn't corrupt earlier ones), and trivially greppable with plain jq or Python, no database required.

The "noop" status was the actual fix

The bug that started all this — the silently skipped edit — showed up the moment I added a noop status. An edit tool that runs but changes nothing (because the target text wasn't found, or the file was already in the desired state) is not the same as a successful edit. Before I distinguished the two, both just showed up as "success" in my head. After I split them out, the failed run from before would have shown up immediately as:

{"tool": "edit_file", "args": "auth.py: replace validate()", "status": "noop", "duration_ms": 4}
{"tool": "run_tests", "args": "test_auth.py", "status": "success", "duration_ms": 812}

A noop immediately followed by a success on the thing it was supposed to fix is a huge red flag once you can see it. It was invisible when all I had was the agent's own final summary.

Visualizing a session

Once I had structured logs, tracing a single session as a sequence became easy:

sequenceDiagram
    participant Agent
    participant Harness
    participant FileSystem
    participant TestRunner

    Agent->>Harness: edit_file(auth.py)
    Harness->>FileSystem: apply patch
    FileSystem-->>Harness: no match found (noop)
    Harness-->>Agent: noop
    Agent->>Harness: run_tests(test_auth.py)
    Harness->>TestRunner: execute
    TestRunner-->>Harness: pass
    Harness-->>Agent: success

Seeing this rendered out made the gap obvious in about five seconds — something I'd missed across three separate debugging sessions reading the agent's prose summary instead.

Sampling, not everything

I don't log full file contents or full command output — that would balloon the trace file and reintroduce the "wall of text nobody reads" problem I was trying to escape. Instead I log a summary string (first ~100 chars of a diff, the command name and exit code, not stdout). If I need the full detail for a specific step, I re-run just that step manually. The trace's job is to tell me where to look, not to be a full replay.

Querying the trace instead of reading it

The other thing that changed once logs were structured: I stopped reading them top to bottom and started querying them. A one-liner like this answers "did anything go quiet on me today?" in about a second:

cat agent_trace.jsonl | jq -r 'select(.status == "noop") | "\(.tool) \(.args)"'

Once I started running that after every session, a pattern jumped out that I'd completely missed before: roughly 1 in 12 sessions had at least one noop on a write tool somewhere in the middle. Most of the time it was harmless — the file was already in the desired state. But about a quarter of those noop events were masking a real problem, the same class of bug that started this whole effort. Before I had the query, I had no way to even estimate that ratio; I was relying on noticing symptoms days later.

I also track a rolling count of tool calls per session and flag anything wildly outside the normal range. A session that usually makes 20–40 tool calls but suddenly makes 3 and stops is just as suspicious as one that runs 200 — both usually mean something upstream broke the loop, not that the task was unusually easy or hard.

Keeping the overhead honest

A fair objection: doesn't all this logging add overhead and complexity to the very system you're trying to keep simple? In practice, the logging layer is under 40 lines of code and the write itself takes single-digit milliseconds — it's one open().write() call per tool invocation, no network round-trip, no external service. The cost is trivial compared to the minutes (sometimes hours) I used to spend reconstructing a session from memory and scrollback after something went subtly wrong.

Lessons Learned

1. Log at the boundary, not inside the reasoning. A log the agent writes about itself inherits the agent's own blind spots. A log the harness emits every time a tool actually fires is ground truth, independent of what the agent believes happened.

2. "No-op" deserves its own status. Success/fail is not enough. A tool that ran without error but changed nothing is a distinct, important case — it's usually where the real bugs hide, and lumping it in with "success" is how I missed mine for weeks.

3. Append-only beats clever. I originally reached for a small SQLite log and it was immediately more fragile — a crash mid-write could corrupt state, and inspecting it needed a separate tool. Plain JSON Lines files needed nothing but cat and jq, survived crashes fine, and I could tail them live during a run.

4. Summarize, don't dump. Full stdout/diffs in every log line made the trace unreadable and expensive to store. Log enough to know where to look, and re-run the specific step for full detail when you actually need it.

5. Build the visualization after the data model, not before. I was tempted to build a fancy dashboard on day one. What actually mattered was getting the log schema right first — status categories, what counts as one "event." Once that was solid, even a five-line sequence diagram from the raw log was more useful than the dashboard I'd imagined.

What's Next

I'm working on turning the noop detector into an active check: instead of me eyeballing the trace after the fact, the harness flags a noop on any edit/write tool as a mid-run warning, so a silent no-op gets surfaced before the agent moves on and builds a false narrative on top of it.

I'd also like to add duration-based anomaly flags — if a step that normally takes 2 seconds suddenly takes 40, that's worth a second look even if it technically "succeeded." And I want to keep a small rolling baseline (average tool-call count, average duration per tool) per task type, so "anomalous" is measured against that task's own history instead of a single global threshold, which right now produces more false positives than I'd like on genuinely large tasks.

None of this needs to be fancy. The lesson underneath all of it is that observability doesn't require a dashboard or a vendor — it requires a boring, structured, append-only fact log that's independent of the thing you're trying to observe.

Wrap-up

If you're running any kind of autonomous agent unattended, don't trust its own summary as your only record of what happened — log the tool calls independently, give "did nothing" its own status, and keep the format boring enough that you can grep it at 2 AM.

If this resonated, follow me here on Dev.to — I write about the practical, occasionally painful lessons from building and running autonomous coding agents. Curious what your own logging setup looks like — drop it in the comments.