Originally published on hexisteme notes.
I keep a decision journal for stock and crypto trades. Every thesis I write down — "buy X because Y, and here's exactly how I'll know I'm wrong" — has to survive one more step before it's allowed to become an actual trade gate: an outside model has to attack it. Not the model that wrote the thesis. A different one.
The reason for that rule is a specific embarrassment. I'd written an INTC thesis with six falsifiers I called "machine-checkable" and was pretty pleased with it. I sent it to an outside reviewer anyway, on principle, and it came back scored 2 out of 10 with four concrete structural problems: two of the falsifiers only measured whether the trade had made money, which is tautological, not predictive; a "down" signal built on a competitor's stock moving +10% turned out to be useless because that competitor moves with my target on sector beta, not against it, so the signal couldn't actually fire in the direction I'd assigned it; a "flat" condition depended on "no major news," which is a judgment call, not a check; and the base rate I'd leaned on was pooled across 45 unrelated tickers, which quietly erases the one thing that made this ticker's situation unusual in the first place. None of that was something I was going to catch by rereading my own thesis. It's the kind of flaw that survives self-review precisely because self-review is being done by the same reasoning that produced the flaw.
So I made outside adversarial review mandatory: no new thesis gets to graduate into a real pre-trade gate until an external model has scored it, and either passed it (5 or higher out of 10) or forced concrete new falsifiers into it in response to what it found. In the case above, the review's two sharpest objections turned directly into two new checkable conditions — a relative-strength-ratio falsifier and a close-vs-gap-close falsifier — replacing vague ones with things a script can actually evaluate. That's the whole value of the rule: it's not about the score, it's about whether the criticism was specific enough to change the thesis.
The part that broke: a review gate that named its reviewer
Here's the part I didn't get right the first time. The rule that mandates outside review also had to say which outside model to call, because someone has to write the actual API call. So the first version of the rule named one directly, with a named backup for emergencies. Both were spelled out as literal version strings, sitting in the rule text and in the code that consumed it, the same way you'd hardcode a package version or a base image tag.
At the time that felt like the responsible thing to do — pin something so specific there's no ambiguity about what runs. What it actually did was smuggle an expiration date into a safety gate, and nobody wrote that expiration date down anywhere.
The audit that found two dead pins
Months later, during an unrelated pass where I was auditing every standing rule for internal consistency, I went back to check this one. The free tier both pinned names ride on had died — throttled to limit:0, confirmed with an HTTP 429 on 2026-07-30. Both pins were effectively dead: the gate was one call away from silently failing. The gate would have just started failing every call it made, indistinguishable at a glance from a transient network hiccup, on a thesis-review step that nobody was watching closely enough to notice going quiet.
That's the shape of the trap, and it's a different shape from a reference that never worked in the first place — that one was broken from day zero. This one worked, then rotted. This failed slow: the pin was correct on the day it was written, kept being correct for a while, and then stopped being correct underneath code that had no mechanism for finding out.
Why swapping the model name isn't the fix
The tempting fix is: pick a name that's less likely to rot, put it in the rule, move on. I don't think that's actually a fix, because it doesn't change what's structurally wrong — it just moves the same failure to a later date with a different name attached. The actual defect was that the rule pinned an identity when what it needed was a requirement.
So that's what changed. The rule no longer names a model at all. It states a requirement — "the strongest currently-registered model that isn't the same family as the one being reviewed" — and resolves that requirement with a live list-models call at the moment the gate actually runs, not at the moment someone wrote the rule. The routing table also grew a second leg: a fallback path through a different vendor's tooling entirely, for when the primary review tool itself is unreachable. And the record schema stopped storing the model name the rule intended to call and started storing whichever model ID was actually invoked — a trail you can audit after the fact instead of a promise you have to hope was kept.
The other half of the fix matters just as much: if resolution fails — nothing satisfies the requirement, or the resolved call errors out — the gate has to say so out loud, something like "review not performed: reason," rather than let the thesis through on the technicality that nothing technically failed. A gate whose dependency can die silently and still show green is worse than no gate, because it costs you the discipline of remembering to check it yourself.
The reason this can't just be dropped
It would be simpler to skip outside review altogether and trust my own second look. I don't trust that, based on watching it happen: the same model, given the same context, will hand back a materially different verdict on roughly one out of every five re-asks. That's not a small number — it's close to a coin flip on the margin, and it's exactly the instability that makes a model's self-review of its own reasoning worthless as a check on that reasoning. The outside reviewer isn't a nice-to-have layer of extra rigor; it's the only part of the system actually doing the job the gate exists for. Which is precisely why it can't also be the part of the system quietly allowed to stop existing.
The general shape of it
None of this is specific to reviewing trading theses with language models. It's the same failure as a CI pipeline pinned to a container tag that its registry later garbage-collects, a service hardcoded against a specific vendor API version that eventually gets sunset, or a dependency pinned to a version resolved through an internal mirror that goes away without anyone updating the lockfile. Grepping the codebase for the pinned string wouldn't have caught any of this, either — the string is still there, spelled correctly, syntactically fine to read. Only a live call tells you whether what it points to is still breathing. Anywhere a build step, a gate, or a reviewer resolves an external catalog identity — a model version, a package version, an API version header, a preview channel — that resolution failing needs to fail the gate, not silently skip it. And it's worth separating this from a related but different trap: a reference surviving past the point its target was deprecated is a story about copies outliving their original; this one is the opposite direction — the original moved on, and the copy pinned inside a rule file just never found out.
More notes at hexisteme.github.io/notes.
Top comments (0)