Level 3 of Arc Ops can prove a person said yes to these exact bytes. It cannot tell you why the agent proposed them, or whether it would propose them again. L4 asks, and the answer inverts the level before it: closing a source of non-determinism can make the system less reproducible.
Ten named entropy sources across three layers - harness 6, transport 2, model 2 - behind one Entropy door, and the door is parsed rather than asserted.
imports of random/time/uuid/secrets/datetime: 0
calls to time.time / random.* / uuid4 / urandom: 0
set/dict iterations with no sorted() around them: 0
entropy draw sites: 12, every one NAMED
Now close them in the order a team actually does it, 960 runs a rung, read off pair agreement - the chance two runs of the same input agree, which does not depend on which recording you happened to keep.
| fix | sources closed | pair agreement | gain |
|---|---|---|---|
| nothing | 0 | 0.8299 | +0.0000 |
| freeze the clock | 1 | 0.8373 | +0.0075 |
| + seed the rng | 2 | 0.8376 | +0.0003 |
| + sorted() x3 | 5 | 0.8072 | −0.0304 |
| + derive the request id | 6 | 0.8072 | +0.0000 |
| + the transport | 8 | 0.8676 | +0.0604 |
The three sorted() calls that fixed L3 are net negative here. Per input they are worse on 11 of 24, better on 3, mean delta −0.0247, worst case −0.1802. Freezing the whole harness layer scores 0.8072 against 0.8299 for freezing nothing, and harness+model (0.8849) is worse than model alone (0.8959). The mechanism is not subtle once you see it: freezing a source does not remove a near-tie, and it can push a decision into one.
Two more that went the other way
The cascade does not exist. "One flipped step means a different run" is the sentence everybody writes about agent replay; across 165 diverged pairs the mean number of decisions changed is exactly 1.0000, pairs where more than one changed is 0, and of 624 downstream steps, 624 of 624 prompts changed while 0 decisions followed. The text moves everywhere and the behaviour moves nowhere. Fitted against the exponential model it implies, 5 rows sit below the prediction and 4 above.
And reproducibility is not a rate at all. It is a set. Of 192 measured steps, 176 agree with their baseline every single time and 16 move - not slightly unreliable but running 0.0333 to 0.9833 across 12 distinct rates. The 0.9771 a determinism dashboard prints is the mean of that column: it describes exactly 1 step, and it sits above 15 of the 16 that actually flip. A free one-pass detector that flags any decision whose margin is inside the 12-unit band finds all 16 with zero false negatives, at 2.50x over-flagging.
Repo: https://github.com/dev48v/arc-ops - MIT, dependencies = [], 659 pytest with 147 new here, commit 00e4f20, 197 in-page assertions and a page that fetches nothing off itself: https://dev48.infy.uk/arcops/level4-determinism.html
Next is L5, sandboxing. L4 can tell you whether the agent would decide the same way again, and nothing in it bounds what that decision can reach. Nine levels, five unlocked: https://dev48.infy.uk/arcops.php
Top comments (0)