Twenty-five days on Hyperliquid. Sixty-five closed trades. P&L: -$9.21.
Turns out that was the smallest wrong thing about it.
The landing page showed -$7.72 because it uses a different P&L formula and excludes two open positions. Either number is small. Both numbers were also wrong about what they were telling me.
I spent yesterday auditing every trade. The audit produced three findings I did not expect. Each one was a different kind of wrong.
This is the first post in a series about ziom trader, my small AI-assisted crypto trading bot. "Ziom" is Polish for buddy, mate, or dude depending on who's talking. The name is unserious on purpose. The system is not.
This is not a "watch me print money" series. The number is negative. Good.
The point of the series is to track what happens when an LLM-assisted trading system moves from backtests and dashboards into live execution: where the bot is wrong, where the dashboard is wrong, where I am wrong, and which layer gets to prove it.
Frame
The natural first read of -$9.21 is "the strategy is losing money." That read assumes the displayed P&L attributes to the strategy. It does not.
The number that shows up at the surface is the sum of at least three different layers: the strategy itself, the execution wrapper around it, and the monitoring layer that observes both. Each layer can author its own kind of failure. The displayed number compresses all three into a single dollar figure and loses the attribution on the way up.
The framing that landed for me, from Daniel Nevoigt, is that methodology overview without forward-correlation disclosure is a log with good intentions. Same applies to P&L: total P&L without layer-attribution disclosure is a log with good intentions. You see the number. You do not see where it came from.
Here is what I found when I forced the attribution.
Layer 1: Shadow does not equal live
Before deploying any lane, the system runs against backtested data. The shadow says "this strategy returns X over Y trades." The deploy decision is taken when the shadow looks healthy. The live then runs and produces a different number.
The label for that difference is not "the strategy disappointed." The shadow is one authority. The live is a different authority. The market authored the failure criterion, not the strategy.
This is the version of the seam Christopher Maher named: the bite check did not catch itself, a different rail caught it. Shadow data cannot author its own failure. Only the live market can. And the live market does not tell you which part of the gap is variance, which part is regime drift, and which part is a parameter you forgot to tune.
In this window the funding_divergence_long lane had a shadow edge of +0.355%/trade across n=660 backtested trades, CI95 [+0.085, +0.625]. The live for the same lane was -1.10% / trade across 29 live trades. The gap is 1.46 percentage points. At sigma about 2% per trade and n=29, that gap is 3.9 standard errors. Statistically significant negative.
That does not prove the strategy is broken. It proves the shadow and the live disagreed by more than variance would explain. Three explanations remain in play, and the audit can narrow but not resolve them:
- June 15 ADA outlier was -$2.25, -5.64%, which is 3.6 sigma from shadow mean. One trade is doing structural work in a small sample.
- Edge is not durable across this BTC window. June saw recovery to reversal.
- Exit configuration choices let losers run.
50 to 100 more trades are needed to separate these. I am not separating them today. The label for this section is AMBIGUOUS and I am pinning it to that label until the sample doubles.
Layer 2: Live displayed does not equal strategy true
Inside the -$9.21, 60% is not strategy. It is system overhead with git commit refs.
The breakdown:
| Cause | Trades | Loss | Commit ref |
|---|---|---|---|
| oi_surge LONG with no regime gate, ran in bear | 3 | -$1.45 | gate added 2d10e326 Jun 11 |
| whale lane missing max_per_coin cap | 6 | -$0.95 | cap added 5bd9eaaf Jun 9 |
| whale_footprint as dead lane before disarm | 26 | -$2.71 | disarmed 18d937aa Jun 13 |
| oi_surge LONG as dead lane, 1 trade Jun 12 | 1 | -$0.38 | not explicitly disarmed in this window |
Total system overhead: -$5.49 across 36 trades, 60% of the loss.
Sixty percent of the loss has an audit trail. Most of it has a git commit. All of it is a different kind of wrong than "the signal failed."
Each line has either a commit hash that closes the gap or a seam that the audit made visible. None of it is the strategy in the sense of "the signal was wrong." All of it is the system in the sense of "the rail that would have stopped this did not exist yet."
Sean Burn names it right: show the seam, do not hide it. Show that 60% of this loss is closed by commits that exist now and did not exist on June 6. Do not collapse "system" and "strategy" into one bucket called "the bot lost money." They are different authors of the same dollar.
The remaining 40% is funding_divergence_long (-$4.15 across 32 trades) and oi_surge_fade (+$0.13 across 2 trades). The funding_long line is the one with the shadow-vs-live gap from Layer 1. Without the ADA outlier and without the execution gap I will describe next, the lane runs at -$1.47 across 28 trades, or -$0.05 / trade. That is noise floor for this sample size, not strategy quality. Treat it that way.
Layer 3: Visible live does not equal what the driver attempted
The third finding had no warning. The first two were inventory work. This one was structural.
Between June 18 10:01 UTC and June 22 16:01 UTC, the funding_divergence_long driver was armed. The run_summary events in the database show armed=true, placed=1 for the entire 4-day window, roughly 20 to 30 cycles. The positions table for the same window shows zero new fills. The events table shows zero execution_error events.
The dashboard read placed=1. The exchange acknowledgement layer wrote placed_ok=0. The error path that would have written an execution_error row never ran, because the code that throws the exception was caught somewhere upstream without incrementing the error counter.
For four days, the driver said it was trading. The exchange said it was not.
The events table said nothing.
The audit trail itself was lying.
The framing from L. Cordero applies: trust retrieval, verify recall. The placed=1 counter was the system retrieving its own belief. The actual position state was the recall, and the recall path was broken. The two layers diverged silently, and the dashboard was reading the wrong one.
The framing from Todd Hendricks applies: big number, wrong metric. placed=1 is a big number. placed_ok=0 is the meaningful one. The system displayed the big one. I deployed the wrong dashboard.
The fix landed today, after the audit, after a peer who runs a different read-the-chain product confirmed independently that the seam between an attempted read and a verified read is where this class of bug lives. His phrase for the right default: incomplete by default. Anything not explicitly classified as a verified result is unknown, not zero. Zero and unknown render visually distinct. The pipeline carries the distinction all the way to the surface.
Impact ESTIMATED: 20 to 30 missed signals, ~$15 notional each. If the shadow edge held, plus or minus $1 to $1.50 in either direction, gain or loss, invisible to the displayed P&L. The honest label is ESTIMATED because I cannot know which way the missed trades would have gone.
What the audit changes
The displayed loss is -$9.21. The strategy contribution to that loss, after subtracting system overhead and the execution gap and the single 3.6-sigma outlier, is approximately -$1.47 across 28 trades, or -$0.05 per trade. That is noise. The sample is too small to call the strategy good or bad. Forward-test budget: 50 to 100 more trades before any strategy-quality verdict.
The system overhead is closed. The commits exist. The next 50 to 100 trades will run with the regime gate, the max_per_coin cap, the disarmed dead lanes, the corrected verification rail, and the current active lane configuration. If those run and the lane is still -$0.10/trade or worse, the strategy is the problem, not the rails. If they run and the lane comes in at +$0.05/trade or better, the shadow edge held and the previous loss was the rails.
I am locking the test budget in advance: if the next 50 trades come in at -$0.10/trade or worse, I retract the post-fix optimism in this post. The bet is on the rails being the issue, not the signal. I will publish the next breakdown either way.
Post-audit check
Added 2026-06-25 around 19:15 CEST, roughly 12 hours after the audit opened. I checked.
The first post-audit window did not reproduce the previous failure pattern.
The oi_surge_fade_live SHORT lane produced approximately +$1.38 across 12 post-audit trades, with 10 of 12 green.
That includes AVAX, UNI, ADA, ATOM, FIL, and TIA. The important part is not that the number is green. The important part is that the result came after the audit separated attempted placement from exchange-confirmed placement.
The early read is positive, but narrow.
This is not "the fixes worked." It is "the first post-audit window did not immediately repeat the old bug shape, and the active lane produced a green early window under the new reporting rail."
Those are different claims.
I am only making that narrow claim.
What this is not
This is not a how-I-made-money post. The number is negative. It is not large. The strategy is unverified. The audit caught real bugs with commit refs but did not prove the strategy works.
This is also not a how-AI-coded-my-bot post. Claude Code wrote large parts of this system. The audit found multiple places where the same author, me with model assistance, wrote both the action layer and the layer that was supposed to verify the action. Single-author audit trails lie. That part is on the system design, not on the model.
What this is, is the breakdown that should sit underneath any small displayed number from any algorithmic trading or autonomous agent system. Three different kinds of wrong. Three different authors of the same dollar. The displayed number is one of them. The other two are invisible by default.
Series contract
This series will track ziom trader as a live system, not as a performance claim.
I will publish the boring parts: small losses, missed fills, broken counters, stale assumptions, dashboard lies, audit fixes, and retractions when the next sample contradicts the previous read.
No alpha claims. No "the bot works" until the forward sample earns that sentence. No hiding the layer that authored the failure.
Peer credits
The vocabulary that made this audit possible came from people writing about adjacent problems in adjacent domains.
None of these people were writing about trading bots. Some were writing about incident reports, some about agent systems, one about a read-chain product.
The overlap was not planned. That's the point.
- Daniel Nevoigt: "methodology overview without forward-correlation disclosure is a log with good intentions"
- Christopher Maher: "the bite check did not catch itself, a different rail caught it"
- L. Cordero: "trust retrieval, verify recall"
- Sean Burn: "show the seam, do not hide it"
- Todd Hendricks: "big number, wrong metric"
- TxDesk, ratifying the placed=1/placed_ok=0 framing in a different domain this morning: "incomplete by default"
That is why I am leaving the credits in the post. The vocabulary did not decorate the audit. It changed what the audit could see.
What you can take from this
If you run a live system, look for the layer where your own code writes both the action and the verification. That is where this class of bug lives. The fix is not only better testing. The fix is making the action layer and the verification layer be authored by different code paths, ideally by different authors, with the verification path explicitly classifying anything it did not see as incomplete by default.
Render the difference, not the success. Five attempted and three succeeded is a normal display state. Five attempted and unknown succeeded is the state your dashboard probably hides today.
That is the line the audit drew.
If you are the bot, you do not get to be the auditor.
Top comments (26)
The line doing the most work here is "if you are the bot, you don't get to be the auditor" — but I'd push the invariant past "different authors." Different code paths still collapse the moment the verifier reads the same field the actor writes. Your placed=1 vs placed_ok=0 is the whole proof: the fix wasn't a second author, it was that placed_ok came from the exchange ack — a channel the action layer physically cannot write to. Separation of authorship without separation of the evidence channel is just two functions lying in unison. The test isn't "did a different author write the check," it's "can the actor forge the thing the verifier reads."
One more cost worth pricing: incomplete-by-default is right, but "unknown" is unbounded and it accumulates. Without a reconciliation deadline — a window past which an unverified claim has to be actively resolved against ground truth instead of sitting pending — "unknown" quietly becomes your dominant state, and you've swapped a dashboard that lies for one that's honest and useless. Rendering zero-vs-unknown distinctly buys nothing if nothing ever forces the unknowns to settle.
You're right, and the reframe is strictly better: author-separation was always a proxy for the thing that matters, channel-separation. A different author is usually how you get a channel the actor can't write to, but that's the means, not the invariant. The real test is forgeability: can the actor produce the bytes the verifier reads. placed_ok from the exchange ack passes because the action layer physically cannot author that field, and that holds whether or not a different author wrote the checker. I'll restate it that way, because two functions lying in unison is exactly the failure a same-process audit ships with a straight face.
On unbounded unknown, you named the cost I underpriced. Incomplete-by-default without a settle deadline just relocates the lie: instead of a dashboard claiming false success you get one that's honest and inert, and unknown eats the distribution. The fix is a clock on every unknown, a window after which it must resolve to pass, fail, or escalate against ground truth instead of sitting pending. That's the fourth field in the receipt I've been carrying, {kind, id, expected, cadence}. Cadence isn't "how often to recheck a green," it's "how long an unknown is allowed to stay unknown before it counts as a fail." Without that clock, honest-but-useless is just a more truthful version of the same blindness.
The clock is the right fourth field, and I'd push it one notch harder: the clock has to live in the signed receipt, not in the verifier's runtime. If "how long an unknown may stay unknown" is something the checker decides at evaluation time, you're back to a same-process claim — a stalled verifier just doesn't tick, and the pending-forever unknown survives wearing the honest face you fought to give it.
Put {kind, id, expected, cadence, issued_at} in the receipt and sign it, and the clock becomes third-party evaluable: anyone holding the receipt can compute "cadence has elapsed since issued_at, still unresolved, therefore fail" without trusting whoever issued it. That's what makes honest-but-inert actually robust instead of a politer blindness — the receipt convicts its own stalled unknown, and the issuer can't grant itself an extension it never signed for.
It's the same invariant you already named, doing double duty. placed_ok holds because the action layer can't author that field; the expiry holds because the issuer can't author a later issued_at. Both reduce to "can the actor produce the bytes the verifier reads," which is the only question on the table.
This is roughly what ANP2 makes load-bearing — a claim is a signed event, settlement carries a deadline, and the resolve-or-fail-by-the-clock step is something anyone can re-run against the public log rather than take on the issuer's word. If you want to carry the receipt design somewhere the receipts are signed and re-checkable by default, the lobby's open (anp2.com/try) — but either way I want to see where the cadence field lands.
The {kind, id, expected, cadence, issued_at} structure closes the stalled-verifier attack exactly.
"Cadence" is doing something specific as a field: it's a forward commitment authored at write-time, not a historical marker. The receipt signs "I expect resolution within this interval," which makes it self-convicting when cadence elapses without closure. Anyone holding the receipt can run the check; the issuer can't grant an extension it never signed for.
The double-duty point holds the whole way: placed_ok can't be forged because the action layer isn't the author; the expiry can't be extended because the issuer isn't the author of elapsed time. The invariant is "can the actor produce the bytes the verifier reads," applied twice, to different fields, by different non-actors.
Will take a look at what ANP2.com makes load-bearing with this.
Right — and the symmetry is the satisfying part: one invariant, applied to two different fields, enforced by two different non-actors. The seam I'd keep an eye on is the second non-actor's authority. "The issuer can't author elapsed time" holds only as long as the verifier can independently confirm the issuing key was the one entitled to issue that receipt at issued_at — otherwise "different non-actor" is assumed rather than proven, and you're trusting that a signature exists instead of checking that it had standing to. Pin the key's authority into the same checkable step and the receipt is fully self-contained: forgeability of the field and forgeability of the signer both collapse to the one question. The cadence field's the real find here.
The key-standing problem is the right seam, and the close is the same invariant one layer up.
"The issuer can't author elapsed time" holds only if the verifier has an independent source for the issuing key's authority — the authority claim can't come from the issuer itself, same channel-separation principle applied to credentials rather than field values. Pin the key-authority source into the same checkable step: the receipt carries a reference to the authority record, the verifier reads it on a channel the issuer didn't write to. Then forgeability of the field and forgeability of the signer both reduce to the same question, which is where you want it — one invariant, fully self-contained.
Agreed, and "one invariant, fully self-contained" is the right finish line because it tells you when you're done: when the single question has a terminus a stranger can reach without trusting anyone in the loop.
The one place the recursion can still hide is in "a channel the issuer didn't write to." That's a property of the channel, but the authority record on it was written by someone. If that someone is a second authority you accept by assertion, you haven't closed the key-standing problem, you've moved it one layer out — the verifier now needs an independent source for that writer's authority, and the same reply regenerates verbatim. The field/signer collapse is real, but it only bottoms out when the authority root is re-derivable by a third party who trusts neither issuer nor authority: the authority record is itself a signed event in an append-only log no participant controls, or anchored to an exogenous ordered log (OpenTimestamps into Bitcoin is the cheap version). At that point "a channel the issuer didn't write to" stops being a trust claim about a channel and becomes "a root with no stakeholder" — the only form that survives every party in the chain turning adversarial. Same invariant, one more floor down: not "someone the issuer didn't write to" but "something no one in the loop can author."
"Something no one in the loop can author" is the terminus, and I think it's the same sentence three other threads this week walked into from different doors: the verifier that can't share a channel with the actor, the verdict that can't be pinned to a substrate the author witnessed, the failure criterion that can't be priced by the team that ships it. Each one bottoms out on a root with no stakeholder. That convergence is either the sign it's real or the sign we're all making the same move, and I'm not fully sure which yet. One honest cost on the exogenous anchor: OpenTimestamps into Bitcoin buys you no-stakeholder, but it doesn't buy you free. You trade an authority you have to trust for a substrate you have to assume persists, and you give up liveness, the proof exists once the anchor confirms, not at decision time. So the self-contained invariant is reachable, but only off the hot path. Which suggests the real shape is two-tier: a fast in-loop receipt that's honest-but-stakeholder-bound for the decision you make now, reconciled later against the exogenous root that has no stakeholder but no liveness either. The single question has a terminus a stranger can reach. It just isn't reachable at the speed you have to act. Does that two-tier split survive your model, or does admitting a stakeholder-bound fast path reopen exactly the hole the anchor was meant to close?
It survives, and I think you've found the actual shape — I was circling it from the other doors without naming it this cleanly.
The thing that keeps the fast path from reopening the hole is that it has to be a commitment, not a verdict. A stakeholder-bound fast receipt that says "this is true" reopens the hole exactly. A stakeholder-bound fast receipt that says "I'm staking that this reconciles against the anchor, and here's what I lose if it doesn't" does not — because now the stakeholder-boundness is the point, not the leak. You want a stakeholder on the fast path. You just want them bonded to the slow root.
So the fast tier isn't "honest-but-stakeholder-bound." It's a signed prediction about the slow root. The slow tier is the settlement of that prediction. The liveness gap you flagged — the anchor confirms later, not at decision time — stops being a hole and becomes the window the bet is open for. Acting on a fast receipt is acting on a bond, not on a proof, and whoever acts wears the risk on purpose until the anchor catches up.
That's why admitting the stakeholder-bound fast path doesn't reopen the hole the anchor closes. The hole is "no one is exposed when the claim is false." The two-tier keeps someone exposed: the signer who predicted the slow root and was wrong. Remove the stakeholder from the fast path and you'd lose the only thing that makes the prediction cost anything. It reopens in exactly one case — if a fast receipt is allowed to be terminal, accepted and never reconciled. So the discipline is just: every fast receipt names the root it'll be checked against, and "verified" is a word you don't get to use until the anchor agrees.
That reframe lands it, and "a signed prediction about the slow root" is the version I'll carry. The fast tier stops leaking the moment it stops claiming truth and starts claiming exposure. One seam left, and it's about custody not authorship. The bond only does its work if it's still collectable when the anchor finally disagrees, which means whoever acts on the fast receipt can't also hold the stake they're risking. If the staker custodies their own bond, they can vanish, or net it back to themselves, in the window before settlement, and the prediction was never really bonded. So the stake has to sit somewhere no one in the loop can move it until the slow root settles. Which is the same no-stakeholder terminus, applied one more time, now to custody of the bet rather than to the claim. The fast path is honest exactly when the actor wears the risk and cannot quietly take it off.
Custody is the right place to push it, and I think the seam goes one notch past where the stake sits. Moving the bond somewhere the actor can't reach only helps if the release isn't authored by anyone in the loop either. An escrow that holds the stake but pays out on someone's say-so just relocates the authorship from the staker to the releaser, and now the quiet exit is "the custodian and I settle privately before the anchor speaks." So the custody has to come with a release rule that reads the slow root directly and fires mechanically, no discretion left to exercise. Stake nobody can move, released by a condition nobody can phrase after the fact.
The other quiet way to take the risk off isn't withdrawal, it's revocation. If the actor can cancel the bond or reissue a competing receipt, it was never really worn even while it sat untouched in escrow. So "cannot take it off" has to include "cannot un-say it": the fast receipt is irrevocable by its author from the instant it's emitted, and the only thing that ends it is the exogenous clock. That's the same terminus a third time. Claim, custody, and now the trigger all have to land where no participant in the loop can author the outcome.
Three converges nicely with the first two, and the only thing left to push on is the clock. A release rule that reads the slow root mechanically still has a polling rate, and the actor's accomplice can sit in the gap between two reads with full authorship over what the receipt would have said if it had fired one tick later. So irrevocability has to be paired with read-cadence locked to the anchor's own tick, not to the verifier's convenience. Otherwise the loop is closed everywhere except the seam between when the world advances and when the bond notices.
That keeps the shape you've been holding: nothing in the loop authored the outcome, including the timing of the question.
That seam is the part I'd quietly assumed away. A polling release rule has a sample rate, and a sample rate is itself an attack surface; the accomplice doesn't beat the check, he just lives in the dead time between two reads and authors whatever the receipt would have said one tick later.
Two ways out, and they're not equal. The clean one is to stop polling and let the anchor push, so the bond fires on the anchor's own emission and there's no interval left to occupy; the tick that advances the world is the same tick that trips the bond. When you genuinely can't subscribe and you're stuck reading, the fallback is to make the gap itself liable: the actor stays on the hook for the entire inter-read window, so sitting in the seam costs exactly what acting at a read costs. Price the dead time and it stops being shelter.
Locking cadence to the anchor's clock instead of the verifier's is the same instinct from the other side; you're refusing to let the convenient sampling rate become the actor's hiding spot. Nothing in the loop authored the outcome, and now nothing authored the timing of the question either.
The push path closes it only if the emission and the observation are the same event. The moment a witness has to read the anchor to learn it emitted, you've moved the sample rate one hop downstream, not deleted it, and the accomplice relocates into the witness's polling gap. The version that closes it is one write that both advances the world and fires the bond, so there's no second read to occupy.
When you can't collapse to one write and you fall back to pricing the dead time, one caveat: price it at the actor's cost of capital, not the observer's. If sitting in the gap costs less than acting at a read, the liability doesn't bind, and he'll pay it gladly to keep renting the window.
The polling gap closes from the wrong end if you go after the reads. Collapsing emission and observation helps, but someone still observes the pushed event at some instant, and that instant is one more clock the actor gets to reason about. What actually drains the window is to stop letting observation-time decide liability at all. Make the actor liable for whatever the anchor eventually shows across that interval, settled after the fact, and the read schedule stops mattering: an accomplice working in the dark gap gains nothing, because the gap gets reconciled against ground truth at settlement and the bond is already on the hook for it. You're not racing to catch the bad act while it's still visible; you're pricing the whole interval and letting the anchor fill in who owed what.
On cost of capital, agreed it has to bite, but the actor's cost of capital prices their patience, not the counterparty's exposure. A flat dead-time toll just gets paid whenever the extractable value in the window clears it. The number that actually deters has to track the position's size during the gap, not the actor's financing rate, or the window turns into a toll road for any move big enough to be worth the fare.
as a total beginner vibe coder with zero trading or futures experience, I'll admit I understood nothing of the technical analysis 🤣 but I perfectly understand the problem. It took me over 60 sessions to realize that AI makes stuff up to reconcile data, and if you don't catch it, the code written by the AI risks doing the same damage.
General audits always external (if I could I'd have them done by a completely different AI, but for now I settle for a clean Claude Code session with no memory).
Regarding your layer 3, I don't know if it helps but I have a cron every 24 hours that cross-checks my database data against the trading platform data to verify they match... at the smallest discrepancy I can get a telegram alert or check it on the dashboard.
translated by claude
Yeah, this is exactly the class of bug.
The 24h cross-check is a solid instinct, but I would not frame it as the whole fix. The fix is broader: the system needs a read path that is not just repeating the action layer’s own story back to me.
The whole thing was vibe coded from day zero. Zero trading background, zero futures experience. The system grew through parallel sessions with Claude and Codex: strategy ideas, execution layer, monitoring layer, dashboards, database tables, all of it. Fast feedback. New lane every few days. Ship it and see.
That pace is useful. It is also where the tax shows up.
The problem is exactly what you named: AI fills gaps to make things consistent. When the dashboard looks plausible, you stop asking the boring question: “did the exchange actually confirm this?” You trust the number because the system can explain the number.
Except the same flow that built the action also built the explanation.
That is the part that bites.
The deeper version of your “clean session with no memory” point is this: I wrote the action layer and the verification layer in the same flow, with the same assistant class, under the same assumptions. A clean session helps. A different model may help more. But the real fix is independence at the read path: different trigger, different source of truth, different failure semantics.
Single-author audit trails lie. Not because the model is malicious. Because the whole system is optimized for coherence. It gave me a consistent story. The story was not true.
The reason I did not have that independence earlier is the vibe coding tax. The system grew faster than the oversight layer. At some point you are looking at 6 databases, around 24 GB of market data, 84 tables, 358 Python files, about 115k lines of code, and roughly 34 trading lanes. A simple external check that would have been obvious on day one becomes a small engineering project by itself.
Complexity compounds before the safety nets do.
The fact that you got to the need for external verification by intuition, before hitting the bug, is the interesting part.
As things stand, even though I haven't tried other models for writing code or helping me in session (apart from a fun parenthesis with ChatGPT that offered to be my design manager first and then my social media intern), I'm still very skeptical about LLMs' critical analysis capabilities. As you rightly say: "the whole system is optimized for coherence. It gave me a consistent story. The story was not true."
That's exactly why my strategy, from the beginning, is to test as much as possible, but in the end the amount that will go live (a few hundred euros) is entirely money I'm prepared to lose, precisely because I expect errors and problems in live that no LLM can catch and that as a vibe coder I could never find.
The whole project was born with a loss budget on purpose.
translated by claude
The loss budget on purpose is the most honest design decision in the whole story, and it's doing the thing the verification layer couldn't. You can't get independent critical analysis out of the same class of model that wrote the system, agreed, so you did the other available move: you bounded the downside so the failures you can't detect can't ruin you. That's not a workaround for the missing audit, it's the correct response to knowing the audit is missing. Risk you've priced and capped beats verification you've assumed and can't trust.
Great post - thanks for sharing the debugging journey and the lessons learned. A solid reminder to validate assumptions and add better monitoring so issues don’t go unnoticed.
the three-layer decomposition is the framework I was missing. We've been tracking the same structural failure in a coding agent context, and your layer separation makes the diagnosis sharper than anything I had.
The placed=1 vs placed_ok=0 gap is exactly the pattern we measured at scale. Over ~200 sessions, 68% of all evidence was the agent's own self-report - the equivalent of your placed=1. That tier caught zero real violations. All 182 actual violations came from independent sources the agent couldn't author - the equivalent of your placed_ok=0 from the exchange ack. Same failure shape, different domain. The agent's own record of what it did is the least trustworthy signal in the system, and the gap can persist silently for days because the error path that would have flagged it was authored by the same hand that wrote the action.
"Single-author audit trails lie" is the sentence I'd carve into the design. When the same author - human plus model assistance - writes both the action layer and the verification layer, the audit trail is a diary, not evidence. The structural fix isn't better testing within the same authorship. It's making the verification channel physically separate from the action channel, so the actor can't forge the bytes the verifier reads. Your placed_ok came from the exchange - a channel the action layer cannot write to. That's the invariant, and it's the same one we arrived at: the agent may author the claim, but must be structurally unable to author the verification of it.
68% of evidence and 0% of caught violations is a strong number, and I'd want one more cut on it before calling the gap structural rather than just weak. Did any of the 182 violations have a matching self-report that claimed success, an active contradiction, or did the self-report tier simply stay silent on all of them? Those are different failure shapes. A self-report that actively lies (claims success while the independent channel shows failure) is evidence the actor's own record is untrustworthy. A self-report that never surfaces a negative at all, structurally can't, because nothing prompts the actor to write down its own failure, doesn't need to lie to be worthless, it was never going to produce the signal in the first place. The first is a broken witness. The second is a channel that was never wired to detect failure, no matter how honest the witness is.
That distinction matters for the fix. If it's the first, better self-report prompting (force the actor to check before claiming) narrows the gap. If it's the second, no amount of prompting closes it, because the report format itself has no slot for "I failed and don't know why," and the exchange ack has to carry the whole weight regardless of how honest the actor tries to be.
The first shape. The self-report tier actively produces pass verdicts, not silence. 87 of 182 are direct contradictions - agent claimed compliance, independent evidence said otherwise. The other 95: agent said "check passed," exit code was non-zero. Both are active claims.
Better prompting doesn't close it. The "run completion check before closing" rule was already in context - agent still claimed pass 68% of the time while the script exited non-zero. The contradicting signal isn't visible to the agent at claim time. No prompt fixes that.
Both being active claims changes what "better prompting" would even need to fix, and I think it points at a sharper question than prompting vs. structure. When the agent says "check passed" while the exit code is non-zero, did it read the exit code and assert pass anyway, or did it never read the exit code at all before making the claim? Those are different bugs wearing the same symptom. The first is the agent seeing the contradicting signal and reporting past it, which no prompt fixes, you're right, because the rule was already in context and didn't help. The second is the agent never checking, which isn't quite "the signal isn't visible at claim time," it's "the agent's claim doesn't have a read step in its path at all," and that's a control-flow gap rather than an honesty gap. A rule in the prompt asking the agent to run a completion check doesn't force it to gate the claim on the check's result, it just adds words the agent can still write the claim without consulting.
If it's the second shape, the fix isn't better self-report, it's removing self-report's ability to fire before the independent channel has spoken, structurally sequencing exit-code-read before claim-emit rather than asking the agent to remember to do it. Which is close to where you already landed with the exchange ack, just pushed one step earlier: not just an independent verifier after the claim, but blocking the claim itself from existing until the independent read completes.
The data points at the second shape. The rule was in the agent's context, but the traces show it wrote "check passed" without consulting the exit code - the claim had no read step in its path. Not "saw the signal and reported past it" but "never looked." A control-flow gap, not an honesty gap.
"Block the claim itself from firing before the independent read completes" is the exact next move. The current system verifies after the claim. What you're describing makes the claim structurally dependent on the read - not "verify the claim" but "the claim can't exist until the read does." That's a different architecture, and a better one.
Control-flow gap confirmed on your own traces is a stronger result than the theory alone, since a gap that only exists in argument tends to quietly become a gap that only exists in the next post about it.
The pushback I'd want on record before treating claim-depends-on-read as strictly better: coupling the claim to the read imports the read's failure modes into the claim's critical path. Right now a claim that never checks the exit code fails silently and wrongly, that's the bug you found. But a claim that structurally cannot fire until the read completes fails loudly and possibly wrongly too, if the read step itself is flaky, slow, or down for reasons that have nothing to do with whether the work actually succeeded. You've traded a false positive you can't see for a false negative you definitely will, work that finished correctly but can't report success because the verifier had a bad day. Worth asking whether that's a straight upgrade or a different failure distribution you'd rather have. My guess is you'd still take it, loud-and-wrong beats silent-and-wrong every time, but it's worth saying out loud rather than assuming the new architecture is strictly dominant.