We run substantive work as three, in fixed seats, every day. This is a plain account of what each seat actually finds, which one a given question should go to, and the bill at the end of a normal day.
It is a follow-up to who your model works with matters more than which model you picked. That piece argued the case. This one is the field notes.
Who is in the room
Grok 4.6, over OpenRouter, holds the argument. It gets the claim, the reasoning and the thing about to be done, and it gets asked one question: what input would defeat this. It sees the codebase only when a piece is pasted in deliberately.
Codex, running GPT-5.5 at medium effort, holds the apparatus. It gets the repository, the data and the scripts, and it gets asked whether the code supports the claim. It builds the fixture, runs it, pastes the output.
Claude orchestrates. It writes the thing, briefs the other two, grades what comes back, and commits. An orchestrator that also implements ends up grading its own homework, which is the reason for the split.
And then there is me, the human, who decides what is worth doing and rules on anything going out in public.
Codex arrived as an implementer: write the feature, Claude reviews, it ships. The seat has grown well past that. Most of what it does now is audit, and the two best things it found this week were in our own measuring instruments, with product code untouched.
One detail worth stating because it is easy to get wrong. Codex cannot commit. Its sandbox mounts the git directory read-only, which is a sensible safety property with an ugly consequence: it cannot sign its own work. Unnamed in the commit message, its findings enter the history as the orchestrator's. Attribution here is checkable, and the check is whether it is in the log.
That cuts more than one way. An early draft of this very article described three seats and named only two of them, leaving the orchestrator as an unlabelled "me". A piece about absorbed contributions, absorbing one.
What Grok catches
Grok finds holes in the reasoning. Consistently, and almost never anything else.
It kills explanations that do not fit your own data. Last week we measured that our document search was failing to find the right document, and wrote up why. The search returns fragments, documents get split into many fragments, so a document with more fragments gets more chances to be picked and a short precise one gets almost none. Tidy story. It went into the internal record as the mechanism.
Grok read it and pointed at our own results table. The short one-fragment document ranked 22nd. The long eleven-fragment document ranked 41st or missed entirely. The explanation predicted the opposite of what the data showed, and the counterexample was sitting in a table written an hour earlier and re-read several times.
It catches a special case wearing a general principle's clothes. We had a theory that putting a hard size limit on a store forces it to decide what matters. Grok's answer was that one narrow case had been written up and then promoted. It also named the counterexample from our own history: we already had a hard size limit on a disk, and what it produced was a disk-full error.
It finds the checks that measure the wrong thing. One of our guards was looking for cautious wording in a file instead of measuring the thing the wording was about. Grok spotted it, we wrote five examples designed to sneak past, and all five sailed through.
Where Grok is useless: anything requiring the repository. Your test fixture could be contaminated all week and it would have no way to know, having never seen it.
What Codex catches
Codex finds holes in the apparatus. The instrument, the fixture, the data path.
The clearest case is the most recent, so here it is with the protocol on show, step by step.
The claim we were about to act on. Fourteen test questions, each with a known correct document. The search found the correct document zero times in the top twenty. On that basis a piece of planned work was about to be abandoned and a finding published.
What was sent. The brief named the exact file, gave the commit, and listed five things to attack in order of how much it would cost us to be wrong. It also wrote down what had been withheld. The phrase "check my conclusion" appeared nowhere.
What came back first was boring. Codex could not reproduce the numbers. In its sandbox the run produced nothing, and an environment problem is precisely what that was. Easy to shrug at.
What came back underneath it was not. It had read the code and explained why its run produced nothing. The script shells out to a search tool and reads standard output without ever checking whether the tool ran. In Codex's sandbox the tool died on startup with a cache permission error and printed nothing.
| what happened | what the script recorded |
|---|---|
| search tool exits with an error, prints nothing | "this question found no documents" |
| same thing, fourteen times | a full results table, 0 of 14 at every depth |
| no search ever ran | a verdict saying the planned work was the wrong project |
| the script finishes | exit code 0, success |
Three ordinary lines produced that. The subprocess call read output and discarded the return code. The loop wrapped it in a catch-all that turned failure into an empty result. The reporting function printed the table regardless and returned success. Any reviewer would read past all three.
Our own run had genuinely worked, and after the fix every number came back identical. The result survived. What went with it was the right to believe it, because for an hour the strongest evidence for that conclusion was a table a completely dead process would have produced.
The other things Codex found this week
To show the class rather than one instance: two of our checks were reading the working copy of files instead of the version being committed; a length limit could never fire because of how its pattern was built; a constant pointed at a directory that does not exist; and a matcher accepted any file whose name ended the right way, anywhere in the tree.
Where Codex is useless: deciding whether the thing is worth building. It will happily verify the apparatus behind a claim nobody needed.
Where each one is better, in one table
| the question | send it to | why |
|---|---|---|
| Should this exist at all | Grok | no stake in the code already written |
| What input would defeat this | Grok | imagination beats access here |
| Is this explanation consistent with our own results | Grok | reads the table without knowing what we hoped |
| Does the code support this sentence | Codex | it can run the thing |
| Build a fixture and paste the output | Codex | same |
| Is this number reproducible | Codex | it re-derives before it answers |
| Is this worth the effort | neither | that one is the human's |
Each is poor at the other's job by a wide margin, and we have wasted time proving it in both directions.
What it costs
Here are today's two Grok reviews, billed:
| review | tokens in | tokens out | cost |
|---|---|---|---|
| a research brief on reranking | 829 | 10,675 | $0.066 |
| an adversarial review of a finding | 1,049 | 5,455 | $0.035 |
Ten cents for both. Grok 4.6 lists at $2 per million tokens in and $6 per million out, and over half those output tokens were the model thinking, before it wrote a word back.
The interesting column is the input. Eight hundred tokens. A thousand. That is the whole reason it is this cheap, and it is the same fact as the seat design: Grok is inexpensive precisely because it never gets the repository. A reviewer that has to read your codebase costs what reading your codebase costs. A reviewer that only has to attack your argument costs about what a paragraph costs.
So the two things that looked separate turn out to be one. Giving them deliberately different material is what makes a disagreement informative, and it is also what makes one of them nearly free.
What is genuinely expensive is unbilled. Briefing well takes real thought, because a lazy brief produces a confident review of a problem you do not have. Grading takes longer than reading. Codex, working over the real repository, is a different cost profile entirely.
When they agree, and when that means nothing
Two reviewers help you to the extent that they are looking at different things. Hand the same brief to two models and you have bought the same opinion twice in two voices, and the agreement will feel like confirmation while carrying almost no information.
When it is real, it is the strongest signal available. Both of them killed the size-limit theory separately, reaching for the same incident from our own history, and then split cleanly on the follow-up: Grok on the logic, Codex on two checkable facts, one of which was a number we had formally retracted three months earlier and had just re-shipped.
That agreement counted for something because we could say what each of them had been denied.
When they are wrong
A headline can be confidently false. We asked for a look at a change and the report led with a finding: a rule could never trigger, because it needed a counter to reach nine while another limit stopped everything at three. Specific, plausible, real numbers from the real file, and wrong. The limits are checked in a different order than the report assumed. Three other findings in the same report were real and got fixed, so it earned its keep, but taking the headline on trust would have broken something that worked.
A brief written badly produces a confident wrong answer. Once a review came back saying we had never measured a thing we had measured. The reviewer was careful. The measurement was simply absent from the brief. Deferring would have withdrawn a claim we could defend. So now what got left out is written down before the response is read.
A dispatcher reporting success means nothing. One night two of four review requests produced no output at all. The tool reported fine. There was a large log of work and no deliverable, because it had hit a usage limit partway through. A missing review and a review that found nothing look the same from here.
What we are not claiming
In the earlier piece we reached for the claim that two reviewers beat one better model. That one is unearned and we have said so.
The reviewers are frequently poor. Two of the stories above are them being wrong, and one is them producing nothing while reporting success.
And most changes should skip all of this. The briefing and the grading are hours, and those hours are real even when the bill is ten cents.
The narrow claim is this. When two reviewers are handed genuinely different material and arrive at the same objection, that has been worth more than anything either said alone. Handed the same material, their agreement has been worth nothing, and it felt exactly the same from the inside.
The two lines worth keeping
Two reviewers with the same input are one reviewer with a stutter.
Agreement is information exactly when the other party could have said something you never fed them.
Top comments (0)