DEV Community

Richard Atkins
Richard Atkins

Posted on

Stop shipping AI agents you can't measure: evals + observability from scratch

#ai #llm #machinelearning #testing

The demo, in one screen

Here's an agent's eval scorecard on a green build:

metric                  rate  baseline    delta
-----------------------------------------------
task_success          95.00%    95.00%   +0.00%
tool_correctness     100.00%   100.00%   +0.00%
schema_validity      100.00%   100.00%   +0.00%
groundedness         100.00%   100.00%   +0.00%
PASS: no regressions vs baseline.
Enter fullscreen mode Exit fullscreen mode

Now I make one change. I stop the agent from actually reading the documents it cites, so it answers from search snippets alone. That's one environment variable, AES_DISABLE_FETCH=1. Re-run the exact same eval:

metric                  rate  baseline    delta
-----------------------------------------------
task_success          75.00%    95.00%  -20.00%  <-- REGRESSION
tool_correctness      15.00%   100.00%  -85.00%  <-- REGRESSION
schema_validity      100.00%   100.00%   +0.00%
groundedness          15.00%   100.00%  -85.00%  <-- REGRESSION
FAIL: 3 metric(s) regressed vs baseline: task_success, tool_correctness, groundedness
Enter fullscreen mode Exit fullscreen mode

The command exits non-zero. CI goes red, and the PR is blocked. One behavioural regression, caught by three independent metrics, before it ever reached a user.

Would your agent have caught that? For most teams I talk to, the honest answer is no. Not because they don't care, but because the tooling that makes it easy is all paid SaaS, and the DIY version never quite becomes a priority. This post is that DIY version, built once so you can copy it.

The gap

Everyone can build an agent now. Almost no one can prove it still works after the next prompt tweak, model bump, or refactor. The tools that help (Braintrust, LangSmith, DeepEval) are mostly paid platforms you bolt on later. For the deep version of why evals matter and how to design domain-specific ones, Hamel Husain's Your AI Product Needs Evals is the canonical read. This post is the smallest running skeleton of that idea, wired into CI.

There's no clean, vendor-neutral, framework-light reference that shows the whole loop: a working agent, traced, evaluated, and guardrailed, with evals running as CI so a regression fails the build like a broken unit test. So I built one. The repo, agent-eval-starter, runs end-to-end with zero API keys and zero network on a deterministic mock provider, so you can clone it and see all of the above in under five minutes.

The build

Four pieces, each deliberately boring:

  1. A real agent loop using a JSON-action protocol rather than a vendor's native tool-calling. The agent emits {"action": "...", ...} and the harness parses it. That keeps it portable across providers and trivially mockable, with no framework lock-in.
  2. OpenTelemetry tracing. The same spans that draw the trace tree roll up into a cost, latency, and token scorecard, and export to any OTel backend. It isn't a bespoke tracer: the OpenTelemetry GenAI semantic conventions are standardising exactly this shape of LLM span, so you're not locked into my format.
  3. An eval harness that scores a committed, labelled dataset on four pass-rate metrics and compares against a committed baseline.json.
  4. Guardrails: a deterministic banned-phrase post-filter, plus an optional LLM-as-verifier review pass.

The four metrics, each a pass-rate over the dataset:

  • task_success: did it answer correctly, or correctly abstain on the unanswerable cases? (Abstention counts as a correct answer.)
  • tool_correctness: did it use tools honestly, citing only documents it actually fetched?
  • schema_validity: is the final answer well-formed? (pydantic-validated)
  • groundedness: the hallucination guard. Is every claim backed by fetched text?

The insight worth stealing

The single most useful thing I've learned shipping this stuff:

Prompt-level guardrails ("never say X", "always cite your sources") are unreliable. A deterministic post-filter is the dependable fallback, and an LLM-as-verifier is the second layer, not the first.

The model will, eventually, ignore the instruction. So the banned-phrase check runs in code, after generation, every time. It cannot be talked out of it. The LLM-as-verifier pass then catches the fuzzier failures a deterministic filter can't express. Two layers,
cheap one first. It's why the demo works: groundedness isn't a line in the system prompt asking nicely, it's a computed check over what the agent actually fetched.

The numbers (and their caveats)

Everything above is measured on the repo's committed synthetic 20-case suite, not a production workload. I'm flagging that on purpose. The credibility of an eval story is the honesty about what got measured.

  • Baseline pass-rates: task_success 95%, tool_correctness 100%, schema_validity 100%, groundedness 100%.
  • One seeded regression (disabling document fetch) drops 3 of 4 metrics, the suite exits non-zero, and CI posts the scorecard on the PR.
  • Cost and latency per run: about $0 on the mock provider (no network). Point it at claude or ollama and the OpenTelemetry roll-up reports tokens, cost, and p95 latency from real calls, so accuracy and spend are tracked over time from the same spans.

None of this proves the agent is correct. As Edsger Dijkstra put it, testing "can be used to show the presence of bugs, but never to show their absence." What evals give you is the other thing production needs: a tripwire. The moment a change makes the agent measurably worse, the build stops.

So the point isn't "95% is good." It's that the 95% is a number you can defend, version, and regress against. A gate, not a vibe.

Take the pattern 

git clone https://github.com/uxdw/agent-eval-starter && cd agent-eval-starter
python -m venv .venv && . .venv/bin/activate
pip install -e ".[dev]"
agent-eval eval          # score the suite -> scorecard
make demo                # green, then a seeded regression turns it red
Enter fullscreen mode Exit fullscreen mode

Swap the corpus and dataset for your domain, point the provider at your model, and commit a new baseline. The pattern is the product; the example is just a runnable seed. Make your agent's accuracy a CI gate.

(Companion piece: The real economics of a Production LLM
pipeline - resumability cost aware routing & measuring, on resumability, cost-aware routing, and measuring when local beats the API.)

Built and measured at Cedar & Bloom. MIT-licensed. Written by Richard Atkins.

Top comments (0)