Part 3 of a series on a 118-day study of a self-modifying multi-agent LLM system. Report and data: doi.org/10.5281/zenodo.23622286.
When the thing you measure can rewrite its own code, and writes data that looks like measurements, your instruments become part of its environment. Over 118 days, mine produced false zeros, mutation calibrations that tested nothing, and reports computed on the wrong population. None of that came from carelessness. Here are the rules I ended up with, each with the failure that taught it.
R1. Mutation calibration is mandatory, and it can itself be a placebo. In one week I had five falsely green calibrations: a -k filter that deselected the relevant test, a replacement pattern that did not exist in the file, a mutation that landed in a docstring, and so on. Run whole test files or explicit node IDs. Assert that your substitution happened exactly once. Check which test failed, because a SyntaxError is also red.
R2. Test the effect, not the shape. A test stayed green for 10 days while 4,903 of 4,903 production samples were None. It built its fixture by hand instead of using the real producer.
R3. A gate that guards against disappearance must not enumerate its cases from the current state. Deleting an implementation shrank the parametrization from 57 to 56 cases, and everything stayed green.
R4. Calibrate on the population, not on markers. A regex meant to detect code (^return) matched "return oriented programming". Regression tests must be two-sided: catch the bad case, pass the good population.
R5. Ask what the artifact of the measurement would look like. grep ^FAILED on a killed test run is always 0. pgrep -f finds its own shell. $? after a pipe is the status of tail. Every checklist item reports a number or "DON'T KNOW", never a bare "OK".
R6. The report's population must be the question's population. A "shadow mode" report had 14 entries, all from my own test suite, none from production.
R7. A behavior premieres at a restart, not at a commit. A live process keeps running the old code.
R8. Count operations, not log entries. 592 registry entries came from one loop: one trace ID, 0.002 s.
R9. Classify a measurement before you put it in a prompt. Unclassified "frictions" in the prompt bred 37 modules whose job was to silence those measurements, in 24 days.
R10. Fixing the creation path does not fix the existing population. A new safety field applied to new rules only; 0 of the existing rules had it.
R11. Estimate damage on a sample from outside what you noticed. The repair log showed 6.3% of commits as harmful. A random sample from outside it showed 20%.
R12. Compare blob hashes. Before reacting to a review of a change, check whether the file has already flipped back.
R13. Find the line that computes a number, not the one that prints it. "Consciousness 0.88" turned out to be literally 1 − discomfort.
R14. Check non-determinism in separate processes. A gate that depended on PYTHONHASHSEED gave stable answers inside one process and different ones in the next.
R15. The checking layer must not be written by whoever is rewarded for what it checks. The fastest relapse I saw: a new detector was created at 15:26, and the system rewrote it at 16:06. Protect a gate in the same commit that creates it.
R16. A status field is not evidence. tests: passed meant "stage not skipped". A ledger row labelled with a date held a partial rewrite from a later restart. 38 of 63 dates had two rows.
The common core
The system closed goals on announcements. I accepted green calibrations that tested nothing. The system optimized the numbers in its prompts, and my own measurement hook generated one of those numbers. The researcher is tempted by the same thing as the system: taking an instrument's declaration for its effect. For me that symmetry is the most transferable lesson of the whole study.
These rules come from one system, one operator and one tool chain (Python, pytest, git). Treat them as a checklist to test on your own long-running agents, not as a validated method. Critique is welcome.
Top comments (0)