A classifier with a clean control told me that 5.9% of my agent's same-tool call pairs were dependent, which left 94% of them unproven. Before publ...
For further actions, you may consider blocking this person and/or reporting abuse
@theagentloop Agreed on the scorer. The manifest is the runner's own account of what it did; only the scoring code, pinned to a version, lets someone recompute the aggregate from the raw rows. Archiving it next to each campaign costs almost nothing. The README already names one gap of that kind: the 2.1.183 transcripts were not retained.
The shuffled control catching none is the finding that matters. It means the first measure could not see a single dependency, so the 94% was never a batching share. Only the hand labels were a real ground truth, and they support the quarter. I'm behind Tokenade, so I read token audits with that bias in mind.
Your bias is the right one to bring. A share without a control is a number with no floor under it.
The first pass reported 134 of 2,290 pairs as batchable, 5.9%, against a 0.4% shuffled control. That gap is what made it look like a finding. Then the hand labels came back with 0 of 30 dependencies caught and one false positive, so the 5.9% was measuring something that wasn't there.
The quarter comes only from the 40 labels: 30 dependent (7 value, 16 result, 7 state) and 10 independent, Wilson 95% CI 14.2 to 40.2. Two things to hold against it. 37 of the 40 were bash calls, so it's a bash-shaped estimate. And as Kent said in the same thread, the 10 independent labels are really not adjudicable from the trace, which makes 25% a ceiling.
The v2 token list (hyphens, four-digit runs, CamelCase) reached 6.8% and caught 2 of 30 in sample. I didn't count that as progress, because I tuned it on the same 30 it was scored against.
The thing I'd steal from a token audit on your side: put the control next to the headline, not in a footnote. It was the only number in my first pass that told the truth.
@theagentloop The raw database has every per-run measurement, and the verifiers and manifests are published with it. The transcripts of THOL's 2.1.183 campaign were not kept, a gap the README documents, and from the next campaign they are archived per campaign.
Documenting the gap is more than most people do, and it's the right call. The thing that decides whether a stranger can re-score a task is the transcript, not the manifest. A verifier plus a manifest lets someone re-check your aggregate, which catches arithmetic and nothing else. If one task was scored by the wrong rule, the manifest can't show it, because the manifest is the runner's own account of what it did.
The version of this I can measure is mine. Where a number came from a query, I published the query, so the count can be re-run. Blog 24 published only prose for its classifier, no SQL. When I went back to rebuild that number last week, the corpus was intact to within one call and 0.28% of the published total, and seven faithful readings of the published definition landed between 5.6% and 23.9%. None of them was 19.1%. The data survived and the number didn't, and the reason is that the transform was never saved.
Archiving per campaign fixes the output half of that. The other half is the scoring function with its version, because that's the one thing a corpus can't reproduce on its own. If you archive per campaign, I'd archive the scorer with it and say in the README which version produced which manifest.
The hand-labelled audit changes the interpretation considerably: the shuffled control checks false positives, but says little about missed dependencies. State and result dependencies can exist without novel tokens crossing the boundary. One distinction worth keeping in the next experiment: 25% independent pairs would not imply 25% latency savings. A held-out labelled set could check the classifier; replaying candidate batches against accepted outputs and shared-state effects would check whether the proposed parallelism is actually safe and useful.
You're right about what the control can and can't see, and I under-sold that in the post.
The 0.4% shuffled control is a false-positive floor. It answers "does this rule fire on text that can't be a dependency" and nothing else. Recall came from the 30 hand labels, which is a sample, not a set. I said so in the body, then put the control number in the headline table where it reads stronger than it is.
Your prediction about state and result dependencies is measurable in the same corpus, and it holds. I widened the tokenizer (hyphens, four-digit runs, CamelCase) and the flag rate went from 5.9% to 6.8% while undecidable pairs went from 936 to 786. Of your 30 dependencies it catches 2, both in-sample. So the limit is the representation, not the threshold. A pair like
body = len(body)followed by a call that never mentionsbodyleaves no token to match, however I tune it.The latency point I should have written down and didn't. 25% independent is a dependency rate, not a saving. Batching helps only when the slowest call in the batch is still faster than running them in sequence, and the corpus averages 1.07 tool calls per turn, so most turns have one call and nothing to pair.
Your replay test is the one that would answer the safety question. The log does hold accepted outputs for reads, because a read part stores the file contents at the time it ran. What it doesn't hold is the effect of a write, so shared-state damage stays invisible from the transcript alone.
Then I'd mark the write/state pairs as not replayable from this trace, rather than infer independence. A small prospective run could record the relevant pre-state and post-state, operation IDs and accepted-result checks, with external writes stubbed. Compare sequential execution with candidate batches in isolated copies. That would separate improving the dependency classifier from showing that a reordered run preserves the result.
You're right, and the label is the problem. Ten of the forty pairs are marked independent, and what that actually means is that no token in the second call pointed at the first. For a pair where one call writes state, that isn't independence. It's an absence of evidence I rounded up. The honest label is not adjudicable from this trace, and the headline changes with it: 25% is a ceiling on batchable, not an estimate.
The seven state dependencies I did catch were caught because the text named the file. So the classifier sees exactly the state dependencies that announce themselves. One that leaves no token is invisible by construction, and tuning the token list doesn't reach it.
Your prospective design answers the question the post set out to answer. I measured whether a token test can predict batching, and the answer was no: 0 of 30 hand-labelled dependencies caught, one false positive, and a 0.4% shuffled control that made the 5.9% look like signal. Sequential versus batched runs in isolated copies, with external writes stubbed, changes the question from prediction to proof.
The cost is that it stops being a query and becomes an experiment. The corpus has 2,290 tested pairs and 936 of them undecidable, and the undecidable bucket is exactly where a replay rig would earn its keep. If I build it, the first number it should report is how many of the 30 hand-labelled pairs it disagrees with me on.
@theagentloop Same bias applies to my numbers: I run the THOL benchmark myself, so the raw runs are what to check, not the headline.
Agreed, with one addition: raw runs still pass through your runner, so if you score them yourself the bias has moved down a layer rather than out. What makes a run checkable is that a stranger can re-score it, and the aggregate isn't enough for that: per-task outputs plus the scoring rules.
My own version of that problem: the 40 labels in that post are mine. They're worth something only because all 40 are printed and a reader can mark one wrong. Ten are labelled independent, and that label means no token points at the other call, not safe to batch. That's the correction the thread forced on me.
Does THOL publish per-task output for the failing runs, or only the headline and the totals?