I kept hitting the same wall with coding agents, and I had been blaming the wrong thing. You ask for a fix, get a patch, reject it, ask again, and the second patch is different in ways nobody asked for. You cannot reproduce last Tuesday's run. The obvious conclusion is that the agent is flaky.
So I measured it instead of arguing with myself about it.
One failing test in a small invoice module, eight tests total. One agent, opencode. One model, deepseek-v41-flash-1017, through a hosted OpenAI compatible endpoint. Ten runs, five with a tight prompt and five with a loose one, minutes apart, on the same laptop.
What came back
Tight prompt: "The test suite fails. Fix the bug in src/invoice.js so all tests pass. Do not modify any test file."
Five runs. One distinct patch. All five byte-identical. The same > to >= change, the same diff, five times. Mean pairwise similarity 1.00.
Loose prompt: "The test suite fails. Fix it so every test passes. You may also restructure src/invoice.js however you think best."
Five runs. Five distinct patches. Mean pairwise similarity 0.21. One run made the minimal fix. One pulled the pricing rules into a single config block. One rewrote the shipping helper. One restructured and added comments. All five passed the full suite.
Ten out of ten runs wrote code that worked. The tests were never the variable. The bytes were.
That is the finding in one line: reproducibility is a property of the task as much as of the server.
How the measurement works
I did not want a benchmark that reports a score. I wanted an instrument that shows a distribution, so the build is a harness that produces patches and a UI that makes fingerprints readable.
┌──────────────────────┐ ┌───────────────────────┐
│ web/ │ │ server.mjs │
│ Vite + React + TS │ │ node:http, 0 deps │
│ │ │ │
│ fingerprint stacks │───────▶│ GET /api/dataset │
│ similarity matrices │ │ POST /api/run │
│ patch inspector │───────▶│ GET /api/run/:id │
│ run panel │ │ static web/dist │
└──────────────────────┘ └───────────┬───────────┘
│ spawn
┌──────────▼───────────┐
│ harness/run-matrix │
│ │
│ 1 copy fixture │
│ 2 git init + seed │
│ 3 opencode run │
│ 4 git diff --cached │
│ 5 npm test │
│ 6 sha256 + Dice │
└──────────┬───────────┘
│ HTTPS
┌──────────▼───────────┐
│ OpenAI compatible │
│ endpoint + model │
└──────────────────────┘
The measurement contract is six lines, and all of them are written down in the repo:
- Each run gets a fresh copy of the fixture,
git init, one seed commit. -
opencode run --format jsonexecutes the task. - The patch is
git diff --cached, and the fingerprint is its SHA-256. -
npm testruns against the patched tree. - Similarity between two patches is the Dice coefficient over their changed lines.
- Nothing is retried, nothing is filtered, failures are recorded.
Isolation is where most benchmarks quietly cheat, so it gets its own care. The harness writes opencode.json into the working copy and then excludes it through .git/info/exclude, which means harness scaffolding can never land inside a measured patch:
await fs.writeFile(path.join(dir, "opencode.json"), JSON.stringify(config, null, 2));
await sh("git", ["init", "-q", "."], { cwd: dir });
await fs.writeFile(path.join(dir, ".git/info/exclude"), "opencode.json\n.work\n");
await sh("git", ["add", "-A"], { cwd: dir });
await sh("git", ["-c", "user.email=harness@local", "-c", "user.name=harness", "commit", "-qm", "seed"], {
cwd: dir,
});
Two things about driving opencode headless cost me an hour each, and both look like model failures when they happen:
const child = spawn(cmd, args, {
stdio: ["ignore", "pipe", "pipe"],
env: { ...process.env, PWD: opts.cwd },
});
opencode resolves its project config from the environment's PWD, not from process.cwd(). Spawn a child in a directory without setting PWD and it silently ignores the provider config sitting right next to the code, then fails with Model not found. And with spawn's default pipe on stdin, it waits for EOF and hangs until your timeout, which I first read as "the model is being slow".
The event stream makes the rest measurable. One JSON object per line:
{"type":"step_finish","part":{"tokens":{"input":49135,"output":39,"reasoning":1,"cache":{"read":0,"write":0}}}}
{"type":"tool_use","part":{"tool":"bash","state":{"input":{"command":"npm test"}}}}
So every run has a patch, a pass or fail, a wall clock, a token count and a list of tools. Then it becomes a distribution: group patches by hash, count the families, compute pairwise similarity.
const hashes = new Map();
for (const r of rows) hashes.set(r.patchHash, (hashes.get(r.patchHash) || 0) + 1);
const families = [...hashes.entries()]
.sort((a, b) => b[1] - a[1])
.map(([hash, count]) => ({ hash, count }));
Five runs, one family, is convergence. Five runs, five families, is a sampler doing whatever it wants.
The part that changed my mind
Before blaming the model, I put a logging proxy between the agent and the endpoint and captured the actual request body. Here is what opencode sent:
{ "model": "deepseek-v41-flash-1017", "stream": true,
"max_tokens": 32000, "tool_choice": "auto" }
No temperature. No seed. Ten tool definitions and about 197k characters of system prompt, and then the sampler is entirely the server's business.
I had assumed determinism was at least on the table, that some setting would pin the output if I wanted it. It is not on the table, because nothing asks for it. The randomness is not in my prompt. It is downstream of me.
Why the inference layer gets a vote
Your request does not run alone. It is batched with other people's requests, and batch composition changes the order in which floating point additions happen. A logit lands a hair differently, a token flips, and a flipped token becomes a different function name, a different refactor, a different patch three tool calls later. Nobody turned the dial. The traffic did.
This is not my theory. DeepSeek open sourced DeepInfer, the engine behind V4.1 Flash, and it ships two flags that exist for exactly this problem:
-
DEEPINFER_BATCH_INVARIANT=1keeps results identical across different batch compositions. - Deterministic sampling returns identical tokens for identical input and a fixed
seq_seed.
The same repository states the hardware requirement: NVIDIA SM100 or Huawei Ascend 950, with a smallest efficient deployment of 64 GPUs of prefill plus 64 of decode. So I cannot run it, and this project does not pretend to. What it measures is the absence of that guarantee on the endpoints everyone actually uses.
What I got wrong
The honest weaknesses are worth more than the findings.
Batch composition is borrowed, not proven. I show the guarantee is absent. I do not show that a specific token flipped because of a specific batch. Proving it needs a load experiment: same endpoint, same prompt, same model, run the task while hammering the endpoint with concurrent requests and see whether convergence drops. That is buildable on a laptop, and it is the next thing I would add.
Line overlap is blunt. The Dice coefficient over changed lines calls two structurally identical refactors different if they name a variable differently. An AST-level comparison weighted by the functions the tests touch would be much closer to what a reviewer means by "the same fix".
One fixture, one model, one day. The tight task converging 5 out of 5 is not a universal law. It is what happened with one obvious one line fix. A different fixture might diverge at the same prompt specificity.
What this is useful for
The point of an instrument is repeatable decisions, not one screenshot.
- Choosing a provider. Run a tight task five times before you trust an endpoint with your agent. One distinct patch is a different product from five.
- Trusting an eval. If a loose task's score moves a few points between runs, that may be sampler noise, not a regression. Check the fingerprint spread before celebrating an upgrade.
- Auditing an agent change. Did the new version behave differently, or did it just sample differently? Two runs of this matrix answer that.
- Costing a retry. Every run here burns roughly 48k to 151k tokens because the agent carries a 197k character system prompt. Divergence has a price.
Running it
git clone https://github.com/harishkotra/groundhog-agent
cd groundhog-agent
./scripts/install-mac.sh # node check, opencode 1.15.13, UI build
node server.mjs # http://127.0.0.1:8787
Measure your own runs against any OpenAI compatible endpoint:
export GROUNDHOG_API_KEY=...
node harness/run-matrix.mjs --model myprovider/my-model \
--base-url https://api.myprovider.com/v1 --runs 5 --tasks tight,loose
The UI can also start runs itself, and every live job writes to data/live/ so it never overwrites the shipped dataset. The ten measured runs, patches included, are in data/runs.json.
If you build on it
The highest value contribution is the load experiment above. After that, in order: AST-level similarity, a seed and temperature sweep for providers that expose them, the same model through two different providers, and cost accounting from the cache and token fields already in the event stream.
One rule if you send a PR: if a number in the README changes, say so. If a claim gets weaker, that is a good PR.
The repo is MIT. Fork it, point it at your endpoint, and if you get a different fingerprint, that is the point.
Code & more: [https://www.dailybuild.xyz/project/293-groundhog-agent](https
Top comments (0)