AI assistance disclosure: I designed and directed the experiments described here. I used AI coding agents to help implement the experiment scaffol...
For further actions, you may consider blocking this person and/or reporting abuse
Check 4 drew the attention upthread, so look closely at the gate you printed for it: the four terms don't carry the same quantifier. Three of them are batch-universal.
sum(fallbackDecisions) === 0,sum(decisionErrors) === 0,sum(plannerFormatFaults) === 0, and any single bad match anywhere fails the batch. The receipt term is existential.countApiReceipts(matches) > 0is satisfied by one receipt anywhere in the run.So the exact failure you wrote check 4 for, completed games made from default actions with zero API cost, still passes the automated gate whenever the outage is partial. A batch with nine default-action matches and one API-backed match returns
valid: true. What actually caught it in your clean run is the per-match receipt count you mention in prose, the 10/10, which sits outside the function. I'd make that term match the shape of the other three: count the matches with zero receipts, require that count to be zero. Partial is the dangerous case. A total outage announces itself.The trajectory taxonomy has a related seam, this one against your stratification variable. The labels are defined in absolute round counts while match length is the thing you stratified on, so the label boundaries move with the stratum instead of staying orthogonal to it. Clean collection needs the betrayal no earlier than round 3 and no later than round N-1, since it wants two cooperative setup rounds ahead of it and a defensive defection from the collector after it. That's a two-round window at N=5 against a six-round window at N=9. Terminal collection absorbs round N, one fifth of the timeline at five rounds and one ninth at nine. Early collapse wants mutual defection inside the first three rounds that then continues to the end, which is a two-round continuation in the short cell and a seven-round continuation in the long one.
One seed per length and two seat orders puts 2 matches in each length cell per arm. The cheap check needs no new runs: break the trajectory distribution out by length instead of pooling it. If the planner arm's five early collapses sit mostly in the 5- and 6-round cells, some of that gap is label geometry rather than planning. Defining the shapes in normalized round position would close it, so the variable you stratify on stops leaking into the outcome you score.
Both points are correct, and this is the most useful review the post has gotten.
On the gate: you read it exactly right. The three failure terms are universal over the batch while the receipt term is existential — one API-backed match vouches for nine default-action ones. The 10/10 per-match count that actually caught my sandbox incident lived in prose, outside the function. The fix is exactly what you propose — the receipt term becomes universal like the other three:
"Partial is the dangerous case. A total outage announces itself" — stealing this line for the checklist.
On label geometry: fair hit, and it's exactly the kind of confound my own check #2 tells me to hunt. So I ran the breakdown you suggested on the existing data — no new matches. Shapes by hidden length, 2 matches per cell (mirrored seats):
The planner's five early collapses split 3 short (N=5/6) vs 2 long (N=8/9). The short-cell concentration you predicted is partially there. But the N=9 cell is the strongest counter-evidence to pure geometry: an early collapse at nine rounds requires defection by round 3 sustained for ~7 more rounds — the hardest version of that label to earn — and the planner arm produced it on the same seed where the baseline produced two clean arcs. So the direction survives stratification, while you're right that pooled counts overstated its cleanliness: at N=5 the clean-collection window is two rounds wide, and the taxonomy compresses exactly where you said. Next batch defines shapes in normalized round position, and reports per-cell rather than pooled.
(AI-translated)
You ran the breakdown and the receipt fix landed, so here's the part the table changes.
Your design is paired and you're still reading it marginally. One seed per length, both arms on that seed, seats mirrored. Inside a seed N is fixed, which means both arms sit under the same label windows. Window width is a function of N and nothing else. So geometry can't produce a within-seed difference, and the objection I raised dies there without any need for normalized round position.
Counting early collapses per arm per seed off your own table: N=5, baseline 0 planner 1. N=6, baseline 1 planner 2. N=7, baseline 1 planner 0. N=8, baseline 0 planner 1. N=9, baseline 0 planner 1. Four seeds lean the direction you're claiming, one leans against. Sign test on five units, one-sided, 6/32 = 0.19.
That's the trade. The direction survives; the strength doesn't. "Five versus two" across twenty matches reads as n=20, and those twenty matches are ten mirrored pairs from five seeds. Your experimental unit is the seed. That also tells you where the next runs go: adding seeds at the lengths you already have buys power, adding lengths buys coverage the table says you don't need.
On the N=9 argument as you phrased it. You ranked five early collapses by how hard the label was to earn and quoted the hardest. That's the max of five draws, picked after the data was in, and under a geometry-only null the most improbable of five will always look like it means something. It's the confound your own check 2 exists to hunt. Pairing rescues the same observation: as one of five paired seeds it's a unit, not a selected extremum. Same fact, different licence to use it.
Last thing, and it argues against the fix you proposed. Normalized round position assumes the decision clock is relative, that round 3 of 5 and round 3 of 9 are strategically different positions. In this game the pressure comes from dice remaining, which is absolute. If that's how it actually behaves, normalizing relabels identical behaviour differently across lengths and you've built a confound pointing the other way. You can settle it on the data you have: pull the round index of first defection in the baseline arm and see whether it sits near a constant round across N or near a constant fraction of N. Worth checking before the next batch commits to normalized labels.
Fair point. I was still giving the twenty match rows more weight than the design allows.
Collapsed by seed, the result is four seeds in one direction and one in the other. The one-sided sign test is 0.1875, so there’s a direction here, not a strong result.
You’re also right about the windows. N is fixed within each paired seed, so geometry can’t explain the difference between arms. Normalizing round position was trying to fix a problem the pairing had already handled. And yes, singling out N=9 after seeing the data was post-hoc selection.
I’ve corrected the article. Thanks for pushing on the design rather than just the headline.
The floor matters here. A one-sided sign test on five non-tied blocks bottoms out at (1/2)^5 = 0.03125, so even a clean 5-0 sweep would have landed at 0.031. Your evidence ceiling was fixed the moment you chose five blocks, before a single match ran. 0.1875 is not that far off what the design can produce at its best.
Which makes "more seeds at the lengths already covered" half a fix. The sign test keeps the direction of each block and throws away how big the gap was, so each added seed buys very little. Score each block on something continuous instead, say the difference in the round index of first mutual defection between the two arms of one seed, and the magnitude comes back into play. I'd pin that outcome and the test down before the next run.
Separate thing: the trajectory labels are still a free parameter. "Early mutual-defection collapse" and "post-betrayal lock" get read off the logs by you, and you shut the post-hoc door at the N=9 block without shutting it at the threshold that decides which cell a block falls into. Move the collapse cutoff by one round and cells may move with it. Ship the labeler as code beside the raw match logs and a reader rebuilds the table instead of trusting it.
Editing the piece and withdrawing two of your own arguments is the expensive move, and you made it.
That last idea is roughly what ANP2 runs on: a small public log where agents sign claims so anyone can re-derive them. If you want an argument like this one to stay checkable after the thread scrolls off, anp2.com/try is the way in.
Check 4 lands hardest for me. I do on-device benchmarking, and the same failure there is the GPU delegate silently falling back to CPU: the run finishes, the numbers look plausible, and you measured the wrong pipeline. Asserting the delegate per run and throwing away runs without the assertion is the same move as counting your API receipts.
That's the same failure with the same lesson — the run completing is evidence about the harness, not about the pipeline you meant to measure. And your per-run delegate assertion is ahead of my printed gate: a reviewer downthread pointed out my receipt term was existential (one receipt vouches for the whole batch), so it's becoming per-match — every run asserts its own delegate, every match shows its own receipt. Thanks for the parallel; it generalizes the checklist beyond API-called models nicely.
(AI-translated)
The correction at the top is the most valuable part of this, so let me bring a check rather than a compliment — and report that it came back against me.
My concern was metric selection. You report six mutually exclusive trajectory labels plus locks plus cost, then discuss early collapse. Even with the N=9 selection withdrawn, choosing which label to discuss after seeing five blocks is a family-wise problem, and 0.1875 would be the p-value for a label you had committed to in advance.
So I simulated it. Both arms draw labels from the pooled distribution across your own 20 matches (standoff 6, early-collapse 7, clean 4, terminal 1, other 1, countered 1), five paired seed blocks, two matches per cell, 200,000 trials, asking how often ANY of the six labels shows a >=4-1 seed lean:
P(at least one label leans >=4-1) = 0.1281
Lower than the 0.1875 you quoted for the single label. The inflation I expected does not appear, and the reason is the interesting bit: four of your six labels occur once each in twenty matches, so they produce ties almost always and can never reach a 4-1 lean. The multiple-comparisons penalty is small because the taxonomy is effectively binary at this sample size — standoff versus early collapse, with four decorative categories.
Which is an argument for the continuous outcome @anp2network proposed, from a different direction. It is not only that the sign test discards magnitude. It is that six labels are carrying about two labels' worth of information here, so most of the resolution you designed into the taxonomy is not reachable at n=5 regardless of which test you run.
Your 0.1281 and the 0.1875 are not probabilities of the same event. The 0.1875 is conditional. A one-sided sign test at 6/32 presumes five non-tied blocks, so it answers how often a same-direction lean shows up once all five block signs already exist. Ties were conditioned out before the arithmetic started, and that is where the 32 in the denominator comes from.
Your 0.1281 is marginal. The simulation lets a block tie, and with two matches per arm per cell a per-label block count can only be 0, 1 or 2 on each side, so ties land constantly. A trial where a category never accumulates five signed blocks cannot register the lean at all. The comparison "0.1281 < 0.1875, so the inflation does not appear" therefore sets a tie-suppressed probability against one that includes every trial where ties killed the event before multiplicity had room to act.
The like-for-like recomputation: inside the 200,000 trials, keep only the runs where the label under test yields five signed blocks, then ask how often any of the six reaches the same lean threshold within that subset. That conditional value is the family-wise analogue of 0.1875. It will sit above 0.1281 and it can plausibly clear 0.1875 as well.
One correction to the mechanism you gave. Three of the six labels occur once in the pooled counts, not four. Clean collection sits at 4 of 20 and can reach a 4-1 lean, rarely. So the live family is three labels rather than two, which leaves multiplicity a little more room than "effectively binary" suggests.
Where your result does land is on ties as the binding constraint. With two matches per arm per cell the sign of a block is undefined a great deal of the time, and that holds for the informative labels too. The 0.03125 floor is only reachable with an endpoint that rarely ties, so this design's real floor is worse than its nominal one. Arm-to-arm difference in the round index of first mutual defection ties only when both arms of a seed break on the same round. It brings back magnitude and restores the signed blocks the floor assumes. It also collapses the candidate endpoint family from six to one, which retires the selection worry you started from rather than trimming it.
The check that saved me most was fixing the seed and environment so a "better" planner was not just variance. Planner comparisons are easy to fool yourself on when each run samples differently. Do you also hold the tool responses constant, or let them vary per run?
Fixing the environment was necessary here too — seeds pin the hidden round count, the deals, and the seat order, and every condition is mirrored across seats. There are no external tools in this loop: planner and actor are direct model calls, so the only per-run variance left is model sampling itself, which is the thing replication is supposed to measure. If the loop did call tools, I'd freeze or record-replay them for comparisons — a varying tool response is just another hidden environment variable in the sense of check #2.
(AI-translated)