DEV Community

Leandro Boeing Vieira
Leandro Boeing Vieira

Posted on

Repository Harness, Part 2: 30 Runs Later, It Wasn’t Cheaper. It Was More Predictable.

#ai #programming #productivity

This is the follow-up to Coding Agents Evolved. Our Repositories Didn’t..

My first Repository Harness benchmark produced an uncomfortable result.

The Harness completed the task faster than a monolithic AGENTS.md, but it used more fresh input tokens.

That could have meant the idea was wrong.

It could also have meant that my first implementation of progressive disclosure was not strict enough.

So I changed the routing model, rebuilt the benchmark baseline, and ran the experiment again.

This time, I ran 30 measured executions.

The result was not a dramatic reduction in token usage.

It was something more useful:

The Repository Harness made the agent’s behavior substantially more predictable.

The idea

Most repositories expose their knowledge through a mix of:

  • README files;
  • architecture documents;
  • contribution guides;
  • scripts;
  • tribal knowledge;
  • one increasingly large AGENTS.md.

A monolithic instruction file works, but every task receives the complete repository handbook, even when most of it is irrelevant.

The Repository Harness replaces that model with a small entry point and routed context:

AGENTS.md
    ↓
.harness/
├── harness.yaml
├── project-specific guidance
├── engineering guidance
└── asset guidance
Enter fullscreen mode Exit fullscreen mode

The root file explains how to use the Harness.

The manifest classifies the task and selects the repository guidance required for that task.

The original hypothesis was:

Selective context routing should preserve implementation quality while reducing unnecessary context.

The first benchmark did not prove that.

What changed in version 2

The first Harness used broad loading rules such as:

load_when:
  - implementation
  - feature
  - validation
Enter fullscreen mode Exit fullscreen mode

Those labels were too general.

Almost every coding task is an implementation. Many tasks are features. Every completed task needs validation.

The agent therefore had a reasonable incentive to load most of the documentation.

I had modularized the repository knowledge.

I had not created strict routing.

The second version added:

  • repository-defined concerns;
  • directory scopes;
  • required and optional context;
  • escalation rules;
  • context budgets;
  • route-specific validation.

Before explaining how that routing behaved, it helps to understand the repository used in the experiment.

The benchmark project

The benchmark used a real game project built with Godot and C#.

The task was a gameplay-input change involving camera controls.

The agent needed to:

  • move gamepad zoom from the triggers to the vertical right-stick axis;
  • preserve horizontal right-stick rotation;
  • preserve mouse-wheel zoom;
  • add horizontal mouse movement rotation;
  • update relevant prompts and tests;
  • avoid breaking gameplay and menus.

This repository was useful for the experiment because it contained different kinds of knowledge:

Godot and gameplay conventions
general engineering rules
asset rules
validation commands
Enter fullscreen mode Exit fullscreen mode

A camera-input task should need the Godot-specific guidance.

It should not need asset guidance.

It should only need broader engineering guidance if the implementation crosses an architectural boundary.

That made the task a practical test of selective routing.

The two variants

I created two isolated versions of the same repository.

Monolithic variant 

AGENTS.md
Enter fullscreen mode Exit fullscreen mode

All repository guidance lived in one concise root document.

Harness variant 

AGENTS.md
.harness/
Enter fullscreen mode Exit fullscreen mode

The root file acted as an entry point.

The Harness contained:

.harness/
├── harness.yaml
├── GODOT.md
├── ENGINEERING.md
└── ASSETS.md
Enter fullscreen mode Exit fullscreen mode

For this camera-input task, the expected route was:

AGENTS.md
+ .harness/harness.yaml
+ .harness/GODOT.md
Enter fullscreen mode Exit fullscreen mode

Across all 15 measured Harness runs, the agent inspected the manifest and the Godot-specific guidance without loading the unrelated engineering or asset documents.

That confirmed that the routing mechanism itself was working.

The remaining question was whether it improved the complete execution.

Benchmark design

Both variants used the same:

  • coding agent;
  • model;
  • source-code baseline;
  • implementation prompt;
  • test suite;
  • acceptance criteria.

I ran:

3 warm-ups per variant
15 measured runs per variant
30 measured runs total
Enter fullscreen mode Exit fullscreen mode

The order alternated between variants.

The prompt was identical in every run.

Each repository started from a clean, frozen commit.

Every measured execution satisfied the benchmark acceptance criteria.

AGENTS:  15/15 accepted
HARNESS: 15/15 accepted
Enter fullscreen mode Exit fullscreen mode

The Harness did not reduce reliability.

The average results were almost tied

Metric AGENTS Harness Harness difference
Duration 188.36 s 185.61 s -1.46%
Total input tokens 803,295 829,006 +3.20%
Uncached input tokens 63,147 61,859 -2.04%
Output tokens 5,947 5,630 -5.32%
Reasoning tokens 987 952 -3.63%
Commands 14.40 14.53 +0.93%
Failed commands 6.07 7.00 +15.38%
Changed files 6.20 6.00 -3.23%

The Harness was slightly faster.

It used slightly fewer uncached input tokens and produced less output.

It also used more total input tokens and executed more failed commands.

None of those mean differences was large enough to justify a strong claim that the Harness was reliably faster or cheaper.

The honest conclusion is:

Average efficiency was approximately tied.

That was not the most important result.

The routed context was only modestly smaller

The concise monolithic AGENTS.md was roughly 14 KB.

The Harness route loaded about:

small AGENTS.md       ~0.8 KB
harness.yaml          ~3.3 KB
GODOT.md              ~7.3 KB
--------------------------------
routed context       ~11.5 KB
Enter fullscreen mode Exit fullscreen mode

The Harness successfully avoided unrelated documents, but the direct reduction was only a few kilobytes.

That is small compared with a complete agent execution containing source files, search results, command output, tests, diffs, and repeated tool context.

The experiment averaged roughly 62,000 uncached input tokens per run.

Saving a few kilobytes of instructions was unlikely to transform the total.

This changed the main question.

The Harness may not primarily optimize the size of the initial prompt.

It may optimize how consistently the agent navigates the rest of the task.

The strongest result was lower variance

Metric AGENTS standard deviation Harness standard deviation Reduction
Duration 46.67 s 25.10 s 46.2%
Total input tokens 239,991 108,745 54.7%
Uncached input tokens 19,211 9,855 48.7%
Output tokens 1,430 772 46.0%
Commands 3.27 1.13 65.6%

In this sample, the Harness produced:

  • 46% less variation in duration;
  • 55% less variation in total input;
  • 49% less variation in uncached input;
  • 46% less variation in output;
  • 66% less variation in command count.

I did not begin this project with variance as the primary hypothesis.

But predictability may matter more than a small improvement in the mean.

Lower variance improves:

  • cost forecasting;
  • CI capacity planning;
  • timeout configuration;
  • rate-limit management;
  • anomaly detection;
  • user expectations.

The Harness did not make every run dramatically cheaper.

It made the execution path more repeatable.

What selective routing appears to change

A monolithic instruction file gives the agent every repository rule at once.

The agent must decide which ones matter while already performing the task.

A routed Harness makes the selection explicit:

task classification
        ↓
repository route
        ↓
required context
        ↓
optional escalation
        ↓
selected validation
Enter fullscreen mode Exit fullscreen mode

This does more than organize Markdown files.

It constrains the agent’s decision space.

That provides a plausible explanation for the lower variance, although this benchmark cannot prove causation.

The strongest conclusion supported by the data is:

Repository-defined routing preserved implementation success and made coding-agent behavior more consistent.

Did the Harness win?

Not in the simple way I originally imagined.

The experiment does not prove that the Harness always uses fewer tokens or is reliably faster.

It does show that:

  • selective routing worked;
  • unrelated documents were not loaded;
  • implementation reliability was preserved;
  • average efficiency remained close to the monolithic version;
  • execution variability was substantially lower.

So the result is not:

The Harness is cheaper.
Enter fullscreen mode Exit fullscreen mode

It is:

The Harness is more predictable.
Enter fullscreen mode Exit fullscreen mode

That may be the more valuable engineering property.

The next experiment should be full-stack

The current benchmark tested one task inside one Godot repository.

The next step should test whether the routing model generalizes to a repository with clearly different technical domains.

A backend-and-frontend project is a stronger test because it contains knowledge that should not be loaded for every task:

backend
database
API contracts
frontend
accessibility
design system
integration tests
deployment
Enter fullscreen mode Exit fullscreen mode

The next benchmark should include:

  1. a backend-only task;
  2. a frontend-only task;
  3. a cross-stack task.

A useful Harness should remain narrow for isolated work and expand only when the task crosses a boundary.

A practical setup would be a TypeScript monorepo:

apps/
├── api/
└── web/

packages/
├── contracts/
└── testing/
Enter fullscreen mode Exit fullscreen mode

Validation could be fully automated:

lint
type-check
unit tests
integration tests
production build
headless end-to-end tests
Enter fullscreen mode Exit fullscreen mode

The next benchmark should measure not only tokens and duration, but also:

  • documents loaded;
  • unnecessary documents loaded;
  • route escalations;
  • changed-file scope;
  • p95 duration;
  • variance.

The central question for Part 3 is:

Can repository-defined routing stay narrow for backend-only and frontend-only work, then expand correctly for a cross-stack task?

Why memory should come later

Persistent memory is promising.

A Harness could eventually learn architectural decisions, recurring failures, conventions, validation paths, and known issues.

But adding memory now would mix two different hypotheses:

  1. Does static routing generalize?
  2. Does accumulated experience improve future tasks?

Memory also makes later runs depend on earlier ones, which complicates a controlled comparison.

The cleaner sequence is:

Part 2
Selective routing and predictability

Part 3
Full-stack generalization

Part 4
Lifecycle hooks and persistent memory
Enter fullscreen mode Exit fullscreen mode

If the Harness generalizes in Part 3, memory becomes the next logical layer.

The project is not a failure

The original promise was token reduction.

The second benchmark did not produce a large or statistically conclusive reduction in average token use.

Instead, it revealed a different benefit:

Repository-defined context routing can make coding-agent behavior more predictable without reducing implementation success.

Reliable agent workflows need more than correct final code.

They need bounded behavior:

  • bounded context;
  • bounded validation;
  • bounded tool use;
  • explainable escalation;
  • repeatable execution paths.

That is what the Repository Harness is becoming.

The first article proposed the idea.

The first benchmark exposed a routing flaw.

The second benchmark showed that stricter routing worked and made the agent more consistent.

Part 3 will test whether that result generalizes beyond Godot.

The question is no longer only:

Can a Harness reduce tokens?

It is:

Can a repository define a predictable, auditable, and progressively disclosed operating environment for any coding agent?

The project is available at:

Repository Harness Specification

https://github.com/Repository-Harness-Specification

Top comments (1)

Collapse
 
alexshev profile image
Alex Shev •

Predictability is a better win than cheapness for agent harnesses. If 30 runs make the variance visible, you can start budgeting review time, failure recovery, and model routing instead of pretending every run is the same.