DEV Community

Cover image for Measuring agents like systems: 6 metrics that tell you if your harness helps or hurts
Phạm Cảnh
Phạm Cảnh

Posted on Edited on

Measuring agents like systems: 6 metrics that tell you if your harness helps or hurts

On a task that fits in one read, my harness used 5.3× the tokens and 2× the wall time — for the same correctness.

On a 12-file task, a one-line change to the pruning rule cut 69% of input tokens and 60% of wall time.

Same harness. Same model. Same benchmark. If you only measure one of those results, you will draw the opposite conclusion about whether the thing works.

This post is not "use my harness." It is the scorecard I use to decide whether any agent setup is helping — and the six ways I got the measurement wrong at first.

Disclosure: I maintain opencode-harness-kit, the harness used in these numbers. The benchmark tables and the arm where it loses are in the repo, so you can check the numbers instead of trusting the summary.

An agent is three layers, and each layer has somewhere to measure

TL;DR

  • Model ≠ agent ≠ harness. The model sets the ceiling. The harness decides how much of that ceiling you actually use. The loop decides whether there is evidence.
  • "Effective" has to be answered with six metrics: correctness, cost, latency, tool efficiency, context hygiene, process safety.
  • Over 6 short tasks (4 lookups + 2 edits), 120 runs, the harness tied on correctness (54/60 vs 55/60) but cost 5.3× the uncached input and 2× the wall time.
  • Over 60 real sessions, the pruner cut 80.5% of context; per-turn history dropped 64%.
  • On a 12-file task, a pruner fix cut 69% of input and 60% of wall time versus the old pruner — and the harness still cost more than no harness.
  • Bad measurement is worse than no measurement: one grader scored the harness at 52/60; re-graded correctly, it won 34/40 on the four lookup tasks.

The point is not that harnesses are good or bad. It is that "effective" is a claim you have to pay for with a baseline, a counterfactual, and an external grader.


1. The wrong question: "Is my agent smart?"

That question is about the model. The model is part of the story, but it is rarely the deciding part of daily work.

The useful question is:

For this kind of work, is the agent more correct, cheaper, faster, or safer than what I accept today?

That forces three things: a baseline before you change anything, a counterfactual (the same work without the agent or harness), and a grader that is not the agent itself.

Without those three, "effective" is just a feeling you got after a nice demo.


2. Three layers, one equation

A real agent is not a model. It is three layers stacked:

Layer What it does What it decides
MODEL Reasons, produces actions, calls tools The capability ceiling
HARNESS Context, tool schema, permissions, pruning, routing, retention, verification How much of that ceiling you use
LOOP Plan → act → observe → verify → record Whether there is evidence, a stop condition, and a human in the loop

The harness is the model's operating system: it allocates context, limits permissions, and decides what is remembered and what is cut. The loop is the control loop: it decides when an action counts as done.

A better model does not rescue a bad harness. A better harness does not turn a weak model into a strong one. And a loop without verification is just a sequence of actions nobody checked.

The practical equation:

effectiveness = correct work / (tokens × price + time × waiting + human intervention × price)
Enter fullscreen mode Exit fullscreen mode

The numerator must be graded externally. The denominator must come from real logs, not estimates.


3. Effectiveness is a curve

Here is the real result from opencode-harness-kit, a harness for OpenCode. I ran the same set of 6 tasks, same model, two arms (harness vs. no harness). Ten runs per arm: 120 agent runs total.

Short tasks: the harness is a cost

Metric (median) With harness Without harness Read
Correctness 54/60 55/60 Tie within noise
Tool calls 12.5 8 +56%
Read calls 3 2
Steps 9 6
Uncached input 56,205 10,540 5.3×
Final context 39,928 12,718
Wall time 43 s 21 s 2×

On the free route, three runs per arm showed the same shape: 15/17 vs 16/16 correct, but 6.4× the context and 2.9× the time.

Why? The harness has fixed overhead:

  • The skill list is 25,151 tokens on every request — 70% of the harness arm's fixed cost.
  • Lane prompts, GLOBAL.md, and the tool schema make up the rest.
  • A short task is never long enough for that overhead to be repaid by context savings.

Put another way: a one-file task is not where you install a harness. It is like installing a full operating system to open a text file.

Long tasks: the harness starts to have a job

Once the task touches 12 files and needs about 50 tool calls, the picture changes. Same subject, same model:

Metric (median) With harness Without harness
Correctness 2/2 2/2
Tool calls 47 46
Read calls 12 18
Uncached input 78,280 37,171
Final context 68,794 42,732
Wall time 121.4 s 104.7 s

To be blunt: the harness still costs more here — about 2.1× the input and 1.2× the time. What it buys is fewer reads (12 vs 18) and more process, not a smaller bill.

But there is a more useful measurement: fixing the prune policy.

The old rule kept the 6 most recent tool outputs and replaced the rest with a stub. On a long session the agent read a file; six tool calls later the original output was stubbed, and the agent read the file again — the output was cut, but the work was not.

The new rule keeps the latest copy of each distinct output and only stubs repeated ones. Measured on the same task:

Within the harness arm Old pruner New pruner Change
Tool calls 67.5 51 −24%
Read calls 22.5 18.5 −18%
Uncached input 268,834 84,325 −69%
Final context 48,637 75,705 +56%
Wall time 307.5 s 123.5 s −60%

Final context is larger because the new rule keeps the latest copy of each distinct output; total input still fell because the agent stopped re-reading what it already had.

This is the most important lesson: a rule that sounds sensible can still make the agent worse. "Keep the last 6 outputs" is a sensible rule. It is only wrong when the work is longer than that window. Only measurement shows you that.

And across 60 real sessions, the pruner cut 80.5% of context — from 15.1M characters to 2.95M, with 6,608 parts stubbed. Per-turn history dropped from 763 to 272 tokens (−64%). The price of the harness is +469 tokens per request.

The whole story in one sentence:

A harness does not make an agent cheaper on every task. It keeps long sessions from growing without bound — at a fixed price.

Effectiveness is a curve


4. Six metrics you can actually measure

This is the scorecard I use. It is small enough to run every week and specific enough that you cannot quietly fool yourself.

# Metric Definition How to get it Trap
1 Correctness pass@k on graded tasks Compare files after edits, compare file:line citations The agent grading itself
2 Cost uncached input + output + cache read Provider or stream log Counting only tokens.input — cache read can be 20.6× input
3 Latency median wall time, p90 The harness clock Using the mean; one 262.7 s outlier ruins the table
4 Tool efficiency tool calls, reads/search, repeat-read rate Tool log More calls do not mean better
5 Context hygiene history/turn, prune %, final context Session log A smaller context with more re-reads is worse
6 Process safety out-of-scope writes, verify, exit code, human intervention Diff, run.json, exit code "Fix it" can override the reviewer lane — measured

Add one confidence factor: timeouts, excluded runs, and variance between runs. A clean number from 3 runs is a hypothesis, not a result.

The six-metric scorecard


5. Measure in five steps

Step 1 — Baseline before you install

Without a baseline, everything after it is a feeling. Record sessions, context per request, tool calls, time, and cost.

Step 2 — Pin the subject

Same commit, same copy. If the subject changes between arms, you are measuring the subject, not the harness.

Step 3 — Same model, same suite

This is the mistake I made: the "no harness" arm used the default model and the "with harness" arm used a different one. That result says nothing about the harness.

Step 4 — Grade externally

Grade the file after the edit, or the citation the agent produced. Do not ask the agent, "did you do it right?"

Step 5 — Median, n, and a gate

  • Median, not mean. Agents have huge outliers.
  • 10 runs per arm is the minimum for signal; 2–3 runs only tell you to measure again.
  • A gate should take its threshold from observed data, but a human decides the threshold. A gate that lowers its own threshold on failure is a gate that cannot catch a regression.

The real commands in the repo (OpenCode + SQLite; swap in your own logs):

npm run baseline          # measure a real install before changing anything
npm run measure           # before/after, same window length
npm run measure:prune     # replay the pruner over real sessions
npm run attribute         # where context goes: fixed overhead vs history
node eval/ab.mjs --target <scratch>/self --suite eval/tasks-self.json --reps 10
node eval/compare.mjs --target <dir> --suite <suite> --arm 'name=command "$PROMPT"'
npm run thresholds        # propose thresholds from data, never auto-apply
Enter fullscreen mode Exit fullscreen mode

The key distinction: measure observes an install; measure:prune is attributable because it replays the exact pruner over the exact messages; ab is a controlled comparison. Three measurements answer three different questions. Do not mix them.


6. Six ways to measure wrong

1. Mean instead of median

One run at 262.7 s next to runs at 42.4 s and 7.3 s turns a 42 s median into a 104 s mean — a number that represents no actual run. Agents have outliers; the median is your friend.

2. Comparing two different work windows

"Before install" and "after install" are not the same amount of work. One measurement showed context up 16%, but the later window had twice the reads and twice the messages. That number described the work, not the harness.

3. A broken grader

This is the most memorable one. One measurement said the harness lost 52/60 vs 59/59. It sounded clear. But the grader could not parse the citation format the agent wrote:

**File:** `src/profile.js`
**Line:** 105
Enter fullscreen mode Exit fullscreen mode

The old pattern stopped at the newline, so the file and line number were never joined, and correct answers were marked wrong. Re-grading the same runs and counting only the four lookup tasks (40 runs per arm), with the fixed grader: 34/40 vs 30/40 — the harness won.

A harness that measures itself with a broken ruler is worse than no measurement. Check the grader before you trust it.

4. Counting only tokens.input

In a 153-session baseline: input 66.07M, output 3.00M, reasoning 1.45M — but cache read 1,358.49M, 20.6× the input. If you only count "fresh input," you are measuring the smallest part of the bill. The same work was recorded at $0.40; the estimate on a paid route was $53.36.

5. Generalizing from short tasks

Six short tasks cannot measure the value of a pruner, because the session is never long enough for pruning to have a job. The honest conclusion has a condition: "on short tasks the harness is a cost; on long sessions it is insurance."

6. Letting the gate lower its own threshold

A threshold is a human decision based on an observed distribution. If it drops every time the check fails, the day the harness breaks is the day the threshold falls below it — and the gate goes green.


7. When to use a harness, and when not to

Use it when Do not use it when
The task is long, touches many files, needs many tool calls You are looking up one file or asking a short question
A session can exceed 10–15 turns You need the lowest possible latency
You need audit, verification, evidence, exit codes There is no verification — the harness only adds cost
The model route changes and you need failover One stable model for one repeated job
You need control over permissions, retention, data A minimal setup is enough
You are seeing re-reads, context bloat, rising cost Context is always small and there are no large tool outputs

If your job is one file, a harness is a cost. If your job is 12 files and 50 tool calls, a harness is insurance.


8. Ten questions before you say "the agent is effective"

  1. Did you have a baseline before you installed anything?
  2. Were both arms the same model, the same suite, the same commit?
  3. Was correctness graded externally?
  4. Did you count cache reads and output, or only tokens.input?
  5. Are you using the median or the mean?
  6. Do you have at least 10 runs per arm?
  7. Are timed-out runs excluded, or counted as "fast but wrong"?
  8. Did you measure repeat reads and out-of-scope writes?
  9. Did you measure the harness's own fixed overhead?
  10. If the numbers are bad, will you report them?

If the answer to #10 is "no," the other nine do not mean much.


9. The conclusion

After all the measurements, the conclusion is not "this harness is good" or "this harness is bad." It is:

  • On short tasks, the harness is a cost: 5.3–6.4× the tokens, 2–2.9× the time, correctness tied.
  • On long sessions, the harness is insurance: 80.5% of context cut, per-turn history down 64%, fewer re-reads, and a process with evidence.
  • On long tasks, the pruner fix cut 69% of input and 60% of wall time — but the harness can still cost more than a minimal arm. The value is in not losing control, not in a smaller bill.

Effectiveness is a trade with numbers attached. If you cannot state the numbers, it is a belief.

And if you are building an agent: build the ruler before you build the shield. A harness that cannot measure itself will struggle to prove it is helping anyone.


Sources and how to reproduce

The numbers are medians over recorded runs; they depend on the model, the route, and the time of day. Re-run them on your own logs before you use any number here as a decision.

Top comments (0)