Harness² is a Google Research release with a big claim: the harness around a model, meaning the prompts, guardrails, tool guidance, memory and skills, can be optimised by the model itself, and one step of that recursion is worth double digit points on professional benchmarks. The weights stay fixed. Only the scaffolding changes.
I wanted to know whether that mechanism survives contact with a small, honest build, or whether it only works because the benchmark is enormous and the executors are real. So I built Harness Forge: four graded briefs, a frozen model, a loop that reads its own failure logs, and one number that tells you whether it learned anything.
The short version:
| stage | score |
|---|---|
| naked harness, one line prompt | 19.4% |
| after one recursion step | 97.5% |
| after a second step | 97.5% |
| holdout, unseen values | 97.5% |
32 calls, 71 seconds. The longer version is mostly about what I got wrong first, because that turned out to be the interesting part.
Attempt one: a flat line at 100 percent
My first suite was six tidy tasks. Extract three invoices from an email. Normalise eight dates. Repair five rows of CSV. Price an order with a stacked discount. Tier some contract clauses. Build a JSON payload from a customer record. Each brief carried its full output contract inline, the way a good ticket would.
I ran the baseline with a one line system prompt: "You are a helpful assistant. Answer the user accurately."
It scored 100 percent. Six for six, first try.
Two more rounds of hardening, more traps, duplicate invoice numbers, credit notes that must be excluded, calendar-invalid dates, did not move it. A frontier model with a bare prompt simply does not fail small, fully specified work.
That is a real finding about where harness gains live, and it kills the demo. If the baseline is already at the ceiling, the recursion has nothing to optimise and the score chart is a straight line. So I changed the question the build answers.
Attempt two: briefs the way real work arrives
Real work does not arrive with a contract. It arrives as a colleague's message, and the conventions live somewhere else: a style guide, a schema in another repo, a CI check nobody has read, a grader that only says "expected this, got that".
So the briefs became underspecified on purpose. The invoice brief says "book the invoices, one row per invoice, duplicates collapse to a single row" and stops. It never says "JSON array sorted by dueDate with amountCents". The changelog brief says "write the changelog for 2.14.0 from these commits". It never mentions the three required headings, the word caps or the ban on em dashes.
Those rules live in the grader. The grader's failure messages are the only thing the model ever sees, and they read exactly like a CI log:
- changelog (score 0.50): sections: expected ## Breaking, ## Added, ## Fixed in that order,
got ["Breaking changes","Features","Fixes","Chores"]; chore references must not appear
grader rule: markdown only, three sections in this order, one bullet per change...
The baseline dropped to 19.4 percent. Not because the model got worse, but because the task now had the shape of real work: an artifact with conventions, and no specification of those conventions in the prompt.
That reframing is the whole demo. The harness is not where the intelligence is. It is where the compliance is.
How it is put together
TypeScript, React and Vite on the front, fetch against any OpenAI compatible endpoint, and a zero dependency node:http server that serves the build and proxies the model call. No SDK, no state library, no chart library. App, tests and server come to about 2,700 lines, and the interesting half is the loop.
+---------------------------+
| frozen model endpoint |
| no weight changes, temp 0|
+-------------+-------------+
^
| chat(messages)
+------------------------+ +------+---------------------------+
| src/tasks.ts | | src/loop.ts |
| 4 tasks x 2 fixtures | | baseline -> step(k) -> holdout |
| public + holdout |<---| PROPOSE -> PROBE -> COMPOSE -> |
| grader -> {score,fail}| | EVALUATE -> REFLECT |
+------------------------+ +------+---------------------------+
|
+----------------+----------------+
| | |
+-----v-----+ +------v-----+ +------v------+
|harness.ts | | llm.ts | | React UI |
|compose, | | fetch, | | chart, trace|
|merge, diff| | retry, pool| | matrix |
+-----------+ +------------+ +-------------+
A task is a fixture pair plus a pure grader, which is the whole extension surface:
// src/types.ts
export interface TaskSpec {
id: string;
title: string;
domain: string;
contract: string; // what the grader enforces
variants: Record<'public' | 'holdout', Fixture>; // same rules, different values
grade: (raw: string, expected: unknown) => Grade; // { score, passed, failures }
}
The harness is a data structure, not a string, and it is composed in a fixed order so every change is attributable:
// src/harness.ts
export function composeMessages(harness: Harness, taskInput: string) {
const parts: string[] = [harness.systemPrompt.trim()];
if (harness.guardrails.length) parts.push('GUARDRAILS:\n' + harness.guardrails.map((g) => `- ${g}`).join('\n'));
if (harness.toolGuidance.trim()) parts.push('OUTPUT CONTRACT:\n' + harness.toolGuidance.trim());
if (harness.skills.length) parts.push('SKILLS:\n' + renderSkills(harness.skills));
if (harness.memory.length) parts.push('MEMORY:\n' + renderList(harness.memory));
return [
{ role: 'system' as const, content: parts.filter(Boolean).join('\n\n') },
{ role: 'user' as const, content: taskInput },
];
}
The loop
Four phases per step, all of them visible in the trace:
Propose. Two editors run in parallel from the same harness. One writes invariant edits (general output discipline, verification habits). One writes domain edits (a checklist for the specific artifact type). Neither can touch the model.
// src/loop.ts, propose() (abridged)
const { content } = await call(
[
{ role: 'system', content: `You are the harness editor in a recursive harness optimizer. ${editorRules}` },
{ role: 'user', content: `Current harness (score ${pct(currentScore)}):\n${harnessForPrompt(current)}
Failure evidence from the last evaluation:
${failingEvidence}
${focus}
Return the complete revised harness JSON now.` },
],
4500,
0.8,
true,
);
Probe. Each candidate harness runs the two worst scoring tasks, for real, with the same frozen model. No simulation, no cached answers.
Compose. A third call merges the two candidates using the probe evidence. It is told the probe scores and the remaining failures, and it keeps an edit only if the evidence supports it.
Reflect. A fourth call writes at most three lessons about how to edit harnesses for this kind of work. Those lessons land in the harness's memory field, which the next step's editor reads. That is the second level of recursion.
Then elite selection, which is a fancy name for one rule: if the composition measures worse than the incumbent, throw it away and log that you did.
// src/loop.ts
if (composedScore < currentScore) {
emit({ type: 'log', level: 'warn', message: `composition regressed, incumbent kept at ${pct(currentScore)}` });
composed = current;
composedScore = currentScore;
composedAttempts = currentAttempts;
}
Without that rule the curve is not monotonic and the run can end worse than it started. With it, a bad composition costs tokens but never costs score.
The trap that cost me an hour: reasoning tokens
The first serious run read like the loop was broken. Invoices, dates and payload all scored zero with empty completion (finish_reason: length).
The endpoint returns a reasoning_content field, and those hidden tokens are billed against max_tokens. A 2000 token answer budget can be spent entirely on thinking, leaving an empty message and no error from the provider. The harness had made it worse, in a way: one of its own guardrails said "keep reasoning to a few short lines", which is good advice that no model obeys by reading it.
// src/llm.ts
if (!content.trim()) {
const hitCap = choice?.finish_reason === 'length';
if (hitCap && attempt < attempts) {
maxTokens *= 3;
continue;
}
throw new Error(`empty completion (finish_reason: ${choice?.finish_reason ?? 'unknown'})`);
}
After that retry, the same harness scored 97.5 percent. This is a small thing that ruins a demo: any agent harness that treats an empty string as a model failure instead of a budget failure will misdiagnose its own loop.
The trap that shaped the design: the rubric is also a cheat sheet
The editor reads grader diffs. Those diffs contain expected values, because that is what a useful failure message looks like. So the loop can absolutely memorise: put yesterday's invoice amounts in the guardrails and the public score goes to 100.
The holdout set is the answer. Same grader, same rules, different vendors, amounts, dates and names, run only at the end. A harness that learned rules transfers; one that learned answers collapses. The UI prints the gap before the tuned score, and calls it what it is:
- under 5 points: the rules transfer
- 5 to 15: partly memorised
- over 15: treat the tuned score as inflated
Without the holdout, this is a benchmark gaming machine. With it, it is an instrument.
What the harness actually wrote
The final system prompt is one paragraph about identifying the exact output format before writing. The interesting part is the guardrails, because they are concrete:
JSON record artifacts (invoices, ledger, line items): return one JSON array only, sorted by
dueDate ascending, with no prose and no code fence. Each object has exactly the keys vendor,
amountCents, currency, dueDate...
CSV artifacts: return the repaired CSV only... Header exactly id,name,city,total...
Changelog artifacts: exactly three sections, in this order: "## Breaking", "## Added", "## Fixed"...
Email artifacts: a line starting "Subject:" and then the body. Subject at most 6 words...
Four skills, one per artifact type, each a numbered procedure. Ten memory entries, three of them from the reflector in the final step.
The one task that never reached 100 was the email, stuck at 90 percent: it kept dropping the signature. The reflector's read on that was sharp. "Limits and style rules are not enough: required literal tokens are graded content and must be physically present." Then, on the next step: "use deterministic post-processing to append or verify required signatures instead of relying on prompt wording."
That is the exact conclusion a human harness engineer reaches after a day of fiddling, and the loop reached it in two minutes from a failure string.
What I would tell someone building this
Three things survived the build:
- Measure the baseline before you build the optimiser. If a bare prompt scores 90, you have no demo and no product. The interesting work is where models fail on conventions rather than reasoning.
- Write the grader first. The grader is the specification of the harness. Ambiguous grading produces a harness full of vibes.
- Always hold out. The moment the editor can read expectations, the tuned score stops being evidence. The holdout is the only honest number in the room.
And one thing I would skip: nine tasks in one harness. A single prompt holding nine different artifact contracts becomes a pile of conditional rules that the model starts dropping. Four tasks, four artifact types, one clean manual was the version that worked.
If you want to take it further, the README has a scoped list: per task harnesses, a real executor so tool guidance and scripts become editable, sequential recursion, a cost aware objective, and a harness seed import. The last one is the most useful in practice: start from your production prompt instead of the bare harness and let the loop repair it.
Run it
git clone https://github.com/harishkotra/harness-forge
cd harness-forge && npm install
npm run dev # bring a key, the proxy handles the CORS side
Or headless, if you want a recorded run:
HARNESS_API_KEY=... npm run loop -- --k 2
The repo ships the recorded run in evidence/sample-run.json, including every attempt, every harness and every lesson, so you can check the 97.5 percent without spending a token. RUNBOOK.md has the copy and paste version for a fresh machine.
Screenshots
Video Demo
Code & more: https://www.dailybuild.xyz/project/287-harness-forge
Top comments (0)