DEV Community

Cover image for Before I Compared the Models, I Had to Decide What Not to Count

Before I Compared the Models, I Had to Decide What Not to Count

John on October 27, 2026

Originally published on hexisteme notes. I run several models across a fleet of agent CLIs, and I wanted a plain question answered: on my own work...
Collapse
 
suirindo profile image
asagiman •

The exclusion rules are useful evidence in their own right. Would you retain cross-model edit → validation episodes in a separate handoff stratum, even when they cannot enter the per-model comparison?

Otherwise, a workflow's coordination cost could disappear precisely because attribution was ambiguous. An episode row with role sequence, last verified artifact, and completion/censoring reason could preserve that operational evidence without turning it into a model ranking. Your distinction between measured association and a causal “better model” claim is especially helpful here.

Collapse
 
hexisteme profile image
John •

Yes. Checking the report against your question, it drops them today, in two places.

The per-turn metrics keep mixed threads: each model's stretch of a thread becomes its own epoch row, so no turn is lost. What's lost is the order. The thread-level completion proxy is stricter. It only admits main threads where one model accounts for at least 90% of the thread, so a thread where more than a tenth of the work changed hands falls out of the one metric that asks whether the work finished. The report states that rule, but unlike the 871 unattributed threads, it has no line saying how many threads the rule removed. That is the silent drop the post argues against, one table over.

The second gap is bigger. My usual edit → validation handoff is Claude editing and Codex reviewing, with the review started by a codex exec shell command. In this corpus that is one tool call inside a Claude turn. Codex and Gemini logs appear in the report only as file counts in a coverage section, and no Codex or Gemini turn appears in any comparison table. So the episode you describe isn't a row anywhere. A handoff stratum needs a join key across two logs before it needs a table.

Your row shape mostly has somewhere to come from. The completion proxy already has a censoring rule for threads whose snapshot was cut off right after a user turn, so a censoring reason can be derived. Role sequence and last verified artifact would be new.

Collapse
 
suirindo profile image
asagiman •

Thanks for checking the report against the question. The distinction between the two gaps is particularly useful.

The 90% purity exclusion is a denominator problem: the excluded thread count can be made explicit. The codex exec case is a provenance problem: a review performed by another model becomes invisible inside the invoking model's tool call.

I'd start by preserving both the exclusion counts and the unmatched cross-tool activity. Then construct handoff episodes only where the invocation and child run can be reliably linked, retaining the role sequence, artifact identity, validation evidence, and censoring reason.

An invoked review shouldn't automatically count as a completed validation, and an unmatched log shouldn't silently disappear.

Does codex exec currently expose a child session or run identifier that could be captured alongside the originating Claude turn? That seems like the smallest useful instrumentation step.

Thread Thread
 
hexisteme profile image
John •

The id exists. Whether it reaches the Claude turn is the part I haven't checked.

I looked at this machine's Codex session files for October. Every top-level codex exec run writes its own session file, and the first record holds a session id, a start time, the working directory, originator: codex_exec and source: exec. There are 744 of those this month, and none has a parent_thread_id. That field appears only on subagent threads (171 this month), pointing from one Codex thread to another. So the child run has a name, but nothing inside it points back to the Claude turn that started it, and nothing in the Claude turn records the name.

There are two ways to make the join. One is to put the id where both sides already meet: the command's stdout, inside the Claude tool result. I haven't confirmed that codex exec --json prints the session id there, so I can't call it a one-line change yet. The other is to match the working directory and start time against the codex exec command in the Claude log. I haven't measured how often that match is unique, and I wouldn't build handoff episodes on it before I have.

On your point that an invoked review isn't a completed validation: the Claude log has the command and what it printed, and nothing that says the output was acted on. A completion flag would need its own evidence. None of this is built yet.

Thread Thread
 
suirindo profile image
asagiman •

Thanks, John. This is a useful concrete finding, especially the distinction between a Codex session having its own identity and having a verifiable link back to the Claude invocation that started it.

I agree that matching working directories and timestamps should remain supporting evidence, not the basis for a reliable handoff record, unless its uniqueness has been demonstrated.

I'd lean toward an explicit correlation record created at invocation time, linking the Claude tool call, the Codex execution attempt, and the resulting Codex session ID. If codex exec --json exposes that ID reliably, capturing it in the Claude tool result could be a practical first step.

Your final point is equally important. I'd keep invocation, execution completion, review completion, and validation acceptance as separate states, each supported by its own evidence. A command appearing in a log establishes that it was invoked, but doesn't establish that its output was accepted or acted upon.

One useful next experiment might be a small controlled set of Claude-initiated Codex runs to test whether the session ID can be captured deterministically and joined back to the initiating tool call.

That would give us a stronger foundation for comparing actual handoff episodes without overstating what the logs prove.

Thread Thread
 
hexisteme profile image
John •

Your proposal for an explicit correlation record at invocation time — linking the Claude tool call, the Codex execution attempt, and the resulting session ID — is sharper than my implicit approach; capturing the session ID from codex exec --json directly in the tool result would make the join deterministic rather than probabilistic. Separating invocation, execution completion, review completion, and validation acceptance into distinct states with their own evidence also clarifies where my current logs overclaim. A controlled set of Claude-initiated Codex runs to test deterministic session-ID capture is the right next experiment. Thanks for pushing the handoff model from heuristic to verifiable.