A model-effort experiment becomes fiction the moment a spreadsheet contains invented latency or quality numbers. The useful artifact is the empty harness: fixed tasks, captured settings, an independent oracle, and a stop rule. Then a solo developer can run it with their own account and current prices instead of borrowing a vendor headline.
Anthropic announced Claude Opus 5 on July 24 and says it is available, supports effort settings, and improves coding and long-running agent work. That latest verified official signal includes Anthropic's benchmark and pricing comparisons; those are vendor claims, not results from this article. Recheck current pricing and availability before filling any cost column. Unverified secondary July 27 claims are rejected.
The smallest honest matrix
Pick two effort settings that your actual client exposes and three repository tasks with deterministic acceptance tests. Do not assume setting names or API parameters from this pseudocode; inspect current provider documentation first.
{
"model": "record-exact-model-id",
"effort": "record-supported-setting",
"task": "T2",
"start_sha": "abc123...",
"attempt": 1,
"wall_ms": null,
"input_units": null,
"output_units": null,
"tests_passed": null,
"human_minutes": null,
"notes": "unexecuted template"
}
| Task | Fixed input | Oracle | Failure fixture |
|---|---|---|---|
| T1 parser repair | one malformed CSV case | repository tests | quoted newline |
| T2 CLI option | written acceptance contract | snapshot plus exit code | unknown flag |
| T3 refactor | pinned code and behavior tests | no diff in outputs | timeout test |
Run every model/effort/task cell at least three times if your budget allows. Randomize cell order so later runs do not always receive a warm cache or a tired reviewer. Use fresh worktrees and the same start commit.
A plain command contract keeps the runner replaceable:
./run-one.sh \
--model "$MODEL" \
--effort "$EFFORT" \
--task fixtures/T2.md \
--repo worktrees/T2-a \
> "runs/${MODEL}-${EFFORT}-T2-a.json"
run-one.sh should return 0 only when it produced a complete evidence record, not when generated code merely exists. The oracle runs afterward under your control.
Score outcomes, not vibes
Use an ordinal result that can be audited:
0 = no usable patch
1 = patch builds but acceptance test fails
2 = acceptance tests pass with disallowed changes
3 = acceptance tests pass within file and dependency limits
Report the distribution, not just the best attempt. Keep latency, provider usage units, billed amount, and human cleanup time as separate columns. A price-per-token comparison cannot by itself answer whether a task was cheaper, and Anthropic's published comparison should not be substituted for your invoice.
Normal path: the runner records settings and revision, the model returns a patch, the independent tests execute, and a reviewer records cleanup without seeing which effort label produced it.
Failure path: a request times out, hits a usage limit, returns no patch, or uses an unsupported setting. Preserve that row with a typed reason such as CLIENT_REJECTED_SETTING or PROVIDER_TIMEOUT; never rerun silently until it looks successful.
Decide before execution
Predeclare a monetary cap, required valid cells, accepted score, and cleanup-time ceiling. Stop when the cap is reached or the oracle cannot distinguish outcomes; these are policy choices, not measured results.
Reproduction checklist
- Pin repository SHA, dependency lockfile, client version, and exact model ID.
- Confirm each effort value is currently supported rather than guessing its spelling.
- Save raw response metadata while excluding secrets and sensitive prompts.
- Run the same deterministic oracle after every attempt.
- Blind the patch reviewer to the effort setting when practical.
- Count retries, failures, and abandoned cells in the denominator.
- Publish no winner if task mix or sample size cannot support one.
Limits: a tiny matrix estimates only these fixtures, environment, date, and acceptance rules. It does not verify Anthropic's broader coding or long-running-agent claims. Provider behavior and pricing can change after collection, and three repeats cannot characterize a heavy-tailed distribution.
Where MonkeyCode fits in the evaluation
Only after the matrix is fixed would I consider MonkeyCode as another evaluation environment. Its currently verified scope is an open-source AGPL-3.0 AI development platform, an overseas online option, managed server-side cloud development environments, model/task/requirement management, and build/test/preview; it is free to start. I did not run this matrix there, so the honest next step is to inspect the option and apply the same stop rules through the official campaign link, not infer a result.
Disclosure: This article promotes MonkeyCode using an official campaign link. Iām a MonkeyCode user, not affiliated with the project, and I receive no commission from this link.
AI assistance disclosure: This article was drafted with AI assistance and reviewed against the cited primary sources.
Top comments (0)