Welcome back to the Harness Engineering series — a 10-part journey from raw language model to production-ready agentic system. Made by builders. For builders.
In Part 7, I closed on a line worth expanding: "I built an agent" vs "I built an agent I can actually operate." The difference between those two sentences is the sixth and final component of the harness. It's called Observability, and without it, everything else you've read in this series is a bet you can't check.
Every previous component in this series does something the agent needs to work. Observability does something the engineer needs — to see what happened, to know when things are going wrong, and to have any hope of making the harness better over time.
What's ahead:
- Part 1: The Raw Model Problem
- Part 2: Defining the Harness — The Six Components
- Part 3: The Control Loop
- Part 4: The Tool Layer
- Part 5: Context Engineering
- Part 6: The Filesystem & Environment
- Part 7: The Memory Layer
- Observability ← You are here
- Part 9: The Harness Architecture
- Part 10: Decomposing Claude Code
By the end of this article, you'll know what Observability actually is, why non-deterministic multi-step systems can't be operated without it, and the three properties — full-fidelity logs, session-level traces, and fixed evals — that separate a real observability setup from an aspirational one.
Let's get started.
📚 Want to go deeper than the articles?
While you follow along with this series, I've put together two hands-on resources that go further than any single article can:
- Build a Harness from Scratch — Udemy Course — A self-paced course where I walk you through building a production-grade agentic harness from the ground up, in code.
- Harness Engineering for AI Agents — Live Maven Workshop — A live, cohort-based workshop for builders who want direct feedback, Q&A, and to work through the material with peers.
Both are optional — the series stands on its own. But if you want the full studio-quality version, that's where it lives.
What Observability Is
Observability is the instrumentation that exposes what the agent is doing. Concretely, it includes:
- Logs of each model call — what went in, what came out, how long it took, how many tokens it cost
- Traces of each tool execution — which tool ran, with what arguments, with what result, in what order
- Latency and token metrics — the operational health of the agent, tracked over time
- Evals — a fixed set of tasks you can run the agent against to score its behavior and catch regressions
Together, this instrumentation lets you answer basic operational questions about your agent: What happened on this run? Why did it fail? Is it getting better or worse over time? Without instrumentation, none of those questions have an answer.
Why Observability Exists
Agents are hard to operate. Three properties make them so:
- They're non-deterministic — the same input can produce different outputs on different runs
- They're multi-step — a single task can involve dozens of model calls and tool invocations
- They run for minutes or hours — long enough that you often can't watch every step live
Now stack those three properties on top of each other. When something goes wrong — and something always goes wrong — you need to be able to answer:
What did the agent actually do, in what order, with what inputs, and why?
You can't answer any of that without instrumentation. If your agent burned $12 in tokens producing a plausible-looking but wrong result, and you have no logs, then you have no way to know whether the failure was a bad tool call, a bad retrieval, a bad prompt, or the model just getting it wrong. Every failure becomes a mystery you can't investigate.
There's a second reason observability matters, and it's forward-looking rather than reactive: you can't improve a harness you can't measure. If you change the system prompt, or reshape the tools, or swap in a new model, and you have no way to compare "before" and "after" on a real set of tasks — then you're not iterating on the harness. You're rearranging deck chairs and hoping.
Observability is what turns agentic work from a black-box guessing game into an engineering discipline.
What a Good Observability Design Looks Like
Three properties separate a real observability setup from an aspirational one.
Log Every Call — Model and Tool
Both kinds of calls need to be logged in full, but the data you capture is slightly different for each.
For every model call, log the full context sent to the model, the full response returned, the latency of the call, and the token counts (in and out, and by whatever model tier the call hit). Model calls are the most expensive thing the agent does and the source of most surprising behavior — you'll want everything.
For every tool call, log the tool name, the arguments passed in, the result returned, and the duration of the execution. Tool calls are where the model's decisions meet the environment, so their logs are where you'll find the story of what actually happened in the world during a run.
The word "log" here is doing a lot of work. It doesn't just mean "print to stdout." It means: persisted, structured, queryable. If you can't go back a week later, filter by task ID, and pull up the exact prompt-response pair that caused a failure — you don't have logs. You have console output.
Traces That Tie a Session Together
Logs alone tell you what happened at each step. Traces tell you the shape of the run.
A trace stitches together every event in a single agent session — every model call, every tool invocation, every retry, every error — in a single timeline. With a good trace, you can replay an agent run end-to-end and see exactly what the agent tried, in what order, and how each step led to the next.
This is the difference between "I have 200 log lines" and "I have a story of one agent run." Debugging a multi-step failure without traces is like trying to diagnose a car accident from only the odometer readings of every car involved. You have data. You don't have narrative.
Evals That Let You Measure Regressions
The first two properties are about understanding runs that already happened. Evals are the proactive side — a fixed set of tasks you run the agent against to measure its behavior over time.
Evals don't need to be complicated. Ten to fifty representative tasks with known-good answers is often plenty. What matters is that they're fixed — the same tasks, run the same way, so that the numbers are comparable across changes.
Then, whenever you change something in the harness — a new system prompt, a reshaped tool, a different model, a memory-layer tweak — you re-run the eval set. If the score drops, you introduced a regression. If the score climbs, the change was probably good.
Without evals, you cannot tell whether the harness is getting better or worse. You can only tell whether the last thing you tested by hand worked or not. And "worked by hand" is not a reliable signal for a non-deterministic system.
Example: Claude Code
Watch Claude Code work, and you'll see observability baked into the surface itself.
Every tool call Claude Code makes is visible in the UI, live, as it happens. You can see exactly what file it read, what it edited, what shell command it ran, in what order. When it makes a decision that surprises you, you can scroll back and see the sequence of events that led there.
That's not just a UX choice. It's an observability decision that makes the agent debuggable. The visible tool log is essentially a trace, presented to the user in real time.
Compare that to a black-box agent that just hands you a result at the end. Something magical happened in the middle, and either it worked or it didn't, and you have no idea what actually went on. When it fails, you can't investigate. When it succeeds, you can't verify.
You can't trust what you can't see. That principle applies to human coworkers, and it applies double to AI agents. Observability is what makes trust possible — not in the sense of hoping the agent is right, but in the sense of being able to check.
Where This Leaves Us
We now have all six components. The Loop drives the cycle. The Tools give the model reach. The Context decides what the model sees on each call. The Environment provides where side effects land. The Memory Layer lets the agent carry state forward. And Observability makes the whole thing debuggable, measurable, and improvable.
Six components. Three verbs — action, persistence, measurable behavior — from the formal definition all the way back in Part 2. Every piece we've walked through serves at least one of those verbs, and together they add up to what an agentic harness actually is.
But naming the pieces is only half the job. So how do these six components fit together in practice? What does the architecture of a real harness look like when you draw it as a whole? That's what Part 9 is about: The Harness Architecture. We zoom out from individual components to see how they compose, where the seams are, and how a well-designed harness holds together as a system.
Remember that this article is part of a longer 10-part series that walks you through every component of an agentic harness.
Here's the roadmap:
- Part 1: The Raw Model Problem
- Part 2: Defining the Harness — The Six Components
- Part 3: The Control Loop
- Part 4: The Tool Layer
- Part 5: Context Engineering
- Part 6: The Filesystem & Environment
- Part 7: The Memory Layer
- Observability ← You just finished this one.
- Part 9: The Harness Architecture ← Move to this one.
- Part 10: Decomposing Claude Code
See you in the next one.
Happy coding :)
Top comments (1)
Fixed evals are where I see people quietly lose the plot. Logs tell you why this one run failed, but the frozen task set tells you whether your latest prompt/tool tweak merely moved the bug somewhere else. I usually want both before I call an agent harness operable.