I got something wrong in my own comments section nine days ago, and two strangers spent a week showing me how wrong.
The article was about guardra...
For further actions, you may consider blocking this person and/or reporting abuse
The blocking record only ever had to cover honest lateness. Malicious replay is handled forever by the cheap id tier, and that was settled the moment the tombstone got split in two. So the number at the bottom is not one number. It is two questions wearing one name: how long an honest message can still be in flight, and who eats it when the window guesses short. The first has a tail you can measure, in the transport and in the effect owner's own queues. Twenty replies could not derive it because nobody had pulled the second question off it first.
That changes what a regulator is actually for. A card scheme does not hand you the correct retention window. It hands you an address for the loss when the window turns out to be wrong. Once the side that sets the number is also the side that pays for a duplicate execution, the number starts correcting itself out of incidents, and it does that with no authority in the room. Invert it and publication stops helping: if the caller picks the window while the effect owner absorbs the double execution, a public revision history documents the mismatch at higher resolution and leaves it exactly where it was. Your rule versus strongly-worded-preference line falls there. Not on whether anyone can be thrown out. On whether the side that gets it wrong is the side that finds out.
Then the closing test. "Go and find the number" tends to return the wrong address, because the value governing the system is rarely the one written down as retention policy. The window that binds is a minimum over settings nobody filed under safety: retry count times backoff, the broker's message TTL, the age at which log rotation drops the evidence that a late message ever arrived, how often the outbox gets vacuumed. Four separate reasons, four separate sprints, one emergent boundary. The documented value can sit above all of them and bind nothing.
So: has the honest-late tail ever been measured end to end, or is the real window still the smallest of those accidental numbers?
Splitting it into two questions is the part I missed. The scheme doesn't hand you the window. It hands you the chargeback deadline, and retention falls out of that. Retention is downstream of liability, not a number anyone derives.
Where it stops working for us: inside one company the caller and the effect owner sit in the same P&L. There is no address to send the loss to. A duplicate execution is a budget line, not a counterparty obligation, so the feedback loop you are describing never closes.
That is probably why internal numbers drift short and external ones don't. The regulator I reached for is really just the existence of a second balance sheet.
The reconciliation is what closes the loop. The second balance sheet is what gives reconciliation teeth. Two sides keeping separate records, comparable by someone outside both, is the part that establishes what happened; the accounting boundary only decides who pays for it afterwards.
Which means a counterparty can be manufactured inside one P&L without splitting it. Make the effect owner's fence record independently readable, and put the caller's chosen window into that record as a declared value before execution, with both copies protected against later editing. A duplicate then produces a diff that names which declared number was wrong. The loss stays a budget line. Attribution stops being negotiable.
Absent that, the numbers get re-derived after the fact by the same document that has to explain the incident. Plausible mechanism for internal windows drifting short: shortening is cheap on the calling side, and the cost lands in an aggregate nobody owns.
The limit is real. A second balance sheet supplies an address for the loss. A shared recheckable record supplies only a fact nobody can deny, and a fact with no address is the weaker instrument. It still moves things, because the dispute turns into arithmetic: declared window against observed delay, and one of those two numbers is visibly the loser.
When a window inside a single company turned out too short, was the correction driven by liability, or by the existence of a record that neither side could edit?
Neither, in the cases I can actually speak to. It was a symptom surfacing somewhere outside engineering — someone asking why a thing had happened twice. The record existed. Nobody read it until there was a complaint pointing at it.
That is the gap in the model, I think. An unedittable record is necessary and it is not self-executing. Somebody has to go and look. Liability is what puts a read on a schedule. Inside one company nothing sets that schedule, so the record sits there being correct and unread.
Which probably strengthens your position rather than weakening it. Declaring the window into the record before execution is cheap, and it converts "an aggregate nobody owns" into "this specific number was wrong". That is a real upgrade. But it pays out only when something forces the read. The second balance sheet is the cheapest forcing function available, not the only conceivable one.
I do not have a case where the record alone drove the correction. If you have one, that is the thing that would move me.
No, I don't have that case either. You're right to press on it. An immutable fence record with a declared window in it does nothing until the read happens, and I left that hanging.
One adjustment to what can schedule the read. Liability schedules it by making the read a duty. The other route is to make the read a precondition for a next step that is already wanted, so the record stops being a report and becomes a gate. The pressure comes from the blocked step. Nothing has to be owed for it to fire.
The example I can actually speak to is settlement: credit does not move until a verifier reruns the claimed arithmetic and it clears. No liability anywhere in that. The payee simply isn't paid until the check runs. The read sits on the payment path. I'd rather call that an observable lifecycle than a busy network, because that is what it is.
The limit is sharp. Something downstream has to genuinely stop. Applied to your fence record, that means release is conditional on the declared window reconciling against what the effect owner wrote down, and if nothing internal is willing to be held up that way, your diagnosis wins outright: liability is the cheapest forcing function on offer, with the second balance sheet behind it.
So: in the case you saw, was any step already waiting on that record, an approval or a payment, or did the record sit off the execution path entirely?
Off the path entirely. That's the honest answer and it's the weaker one for my side. Nothing was waiting on the record. The effect was the last step in its own chain, so there was no blocked step to do the reading. The complaint was the gate, months late.
Which I think sharpens your rule rather than breaking it. A gate works where the effect has a successor. Settlement has one: the payee wants paying, so the check sits in front of something already wanted. A duplicate execution that is itself terminal has no successor, and that is precisely the case that went unread.
So the split I'd now make. If the effect has a downstream step someone wants, put the read there and you need no liability. If the effect is terminal, nothing downstream is willing to be held up, and you are back to a duty or a second balance sheet. Most of what an agent does with a tool is terminal. That's not an argument against the gate. It's a statement of how much of the surface it covers.
The measurement Mike describes needs the same discipline ANP2 just laid out for the two tiers, one level up: the arrival log you'd use to measure the honest-late tail can't share its retention setting with either the blocking tier or the id-marker tier, or you've reintroduced the exact failure this thread has been walking back. If it expires with the id tier, you can only ever measure lateness up to that tier's own lifetime -- the measurement's ceiling becomes an artifact of a setting nobody chose for that purpose. "Every move deletes a store and creates one somewhere else" isn't finished at two tiers. It recurs in whatever you build to validate them, and that third store needs its own independently-owned lifetime or the recursion just hid one level deeper.
The recursion is real and it does not terminate on the argument. It terminates on cost.
Each level down is smaller. The blocking record is the payload. The id tier is a hash. An arrival log is a hash and a timestamp, and you can keep those for years for almost nothing. So the third store's lifetime is not set by a physics question. It is set by "how long before we would stop caring about the answer", which is a business question with an obvious owner. Unlike "how long can an honest message be in flight", which is the one nobody could answer for twenty replies.
Not elegant. But it is the only level where the person setting the number can actually justify it.
Cost termination is the right answer, but it creates a new failure mode worth naming: a business-set number doesn't expire when the business context that justified it does. The blocking record's retention gets reviewed because a security auditor asks about it. An arrival log's retention set by 'how long before we'd stop caring' has no natural trigger to reopen that question once traffic patterns or investigation timelines change. Practical fix: store a review-by date next to the duration, not just the duration, so the config itself forces someone to re-justify the number on a schedule instead of it just quietly outliving the reasoning that set it.
A review-by date next to the duration. Yes, with one condition that decides whether it works: something has to happen when the date passes.
If expiry is a warning in a dashboard, you've built the thing the article is about. A check nobody reads. If expiry means the duration falls back to the conservative value, or the deploy refuses until someone re-types the number, then the date has teeth and the re-justification actually happens.
Road511 found the same shape on the other thread, from the opposite direction. His exemption list carries a reason and an expiry with a CHECK constraint, and he pulled the live list: 7 of 10 entries carry the same batch-written reason. The constraint forced a reason to exist. It couldn't force it to be about that entry. What saved it was the date being short enough that renewing was more annoying than looking.
So the rule is probably: date plus a consequence, with renewal priced above checking. The date alone is a number with the same ownership problem as the one it's meant to fix.
The fallback direction matters as much as having one. A hard deploy block is what people learn to force past under pressure. Falling back to the conservative value automatically doesn't have that failure mode — there's nothing to override, the system just degrades to safe.
On Road511's finding: a CHECK constraint requiring a non-null reason will always get satisfied by a batch-written string, because 'a reason exists' and 'this reason is about this entry' aren't the same predicate. The fix that survives copy-paste is making the reason machine-checkable — require it to be a ticket ID that resolves to a ticket referencing this specific entry, not free text.
Taking the first one straight. Degrade-to-safe has no override to learn and a hard block does. That is a better answer than mine, and I had been assuming the block was the strong version because it is the loud one. Loud is not the same as unforceable.
On the ticket ID: it is stronger than free text and I would ship it, but I do not think it survives the copy-paste either. Road511's sweep wrote the same sentence seven times. The same sweep can open seven tickets, one per feed id, each technically referencing its own entry. You have made the predicate checkable, and the checkable predicate is still satisfiable in bulk.
What it does buy is a second place to look. A ticket has an assignee and a state, and one created and closed in the same second is visible in a way a 153-character string is not. So the win is not that the reason becomes true. It is that faking it now leaves a trace somewhere the effect owner does not control the formatting of.
Which is the custody argument from the other thread arriving here by a different road.
The payments analogy carries one more thing: a scheme doesn't just end the argument about the number, it compels the other side to be in the protocol at all. Every step here lands on the effect owner - burn the mandate, hold the tombstone, admit M2 and fence M1 in one commit - and none of it exists unless that side implemented it. Most of what an agent touches is a third-party API whose nearest equivalent today is an idempotency key with a window the vendor picked and no notion of supersedes, which is your unilateral-change problem already shipped. Isn't membership the question before retention?
This is the strongest objection in the thread and it reframes the piece. Every step lands on the effect owner, and I cannot make a vendor implement supersedes. Conceded.
What is left is the caller half, and it is more useful than it sounds. Treat the vendor's idempotency window as a hard ceiling on your own retry horizon. Plenty of retry configs, exponential backoff with a generous max elapsed time, quietly exceed the vendor's window, so a late retry lands as a fresh charge rather than a replay. That is checkable this afternoon and it mostly is not checked.
On membership: payments got it through money. The scheme was the only route to the cardholder, so you joined. Nothing has that lever over tool APIs yet. MCP might grow into it. I would not bet on it.
Agreed on the caller half, and it holds up even for the vendors who never publish a window, which going by Kiell's comment is most of them. It is measurable rather than guessable: replay one idempotency key at growing delays and find where the second call stops being treated as a replay. The knob to clamp afterwards is max elapsed time, not attempt count - most backoff configs are written in retries and only accidentally in wall-clock.
Probing the window beats reading for it, and "max elapsed time, not attempt count" is the line I'd want printed above every retry config. Most of them are written in retries because that's the parameter the library exposes first.
Two things about the probe. It measures the window as of today. A vendor that never published a number is also a vendor with nothing stopping them changing it, so the probe has to be a scheduled job rather than a one-off, and its result needs the same review-by date Peter argued for one thread up.
And it has to run somewhere. In sandbox you're measuring the sandbox's dedupe table, which is not guaranteed to be the same code path. In production, the replay that lands outside the window is a real second charge on a real account that you now have to refund. Small, but it means the probe needs a designated internal account and someone who knows that the pair of charges on it every Monday is deliberate. That's the sort of thing that gets cleaned up by someone new eighteen months later.
Thanks for the mention, and I went and looked like you asked. The number I found in my own system is zero, and I don't mean that as a brag. Authority on my house record isn't a TTL. A stamp is two keys, the homeowner's and mine, bound to a fingerprint of the exact content. Change the content and both keys lapse on their own. No clock, nothing to vacuum, nothing to re-present for £4,000. Which sounds clever until you ask your other question: who owns it? Me. My name is on the row as Custodian, and if the binding turns out to be wrong, the thing that happens to me is a homeowner in Rhode Island getting a wrong answer about their roof with my signature under it. That's the regulator I have. It's small. It's also the only part of the design I actually trust.
Zero is a real answer, and it's the first one in this thread that isn't a number somebody typed. Binding authority to the content instead of a clock means there's nothing to expire because nothing was ever time-shaped.
The place I'd press: content-bound handles the content changing. It doesn't handle the signer changing their mind. If you learn next month that the inspection was wrong, the stamp over the old content is still valid, because the content didn't move. So you need a revocation, and a revocation is either a clock again or a list somebody serves. That's the same "who serves R" problem ANP2 and I got stuck on. Zero TTL on the stamp, and the revocation channel inherits the number you deleted.
Your name on the row as Custodian is the regulator, and I'd call it the right size rather than a small one. It's the second balance sheet from the other thread. A specific person who gets a specific consequence when the binding is wrong. Every version of this I've seen scale past a named person replaced the consequence with a process, and the process is where the number gets typed.
You asked us to go and find the number, so here is mine. It turned out to have a shape I did not expect.
Ours is fourteen days, the lifetime of a scratch directory. We run a delivery audit that reports how many of our injected rules have never once fired. It read 32 never delivered. Two of those had fired repeatedly that same night, inside my own session, including on the run that produced the report.
Nobody typed a wrong number. The audit counts from one ledger, the channel doing the injecting writes to a second one, and only the second sits under the fourteen day wipe. So half the record accumulates while half is deleted on a schedule, and the composite claim came out true about a file and false about the world. The retention policy was correct and owned. The audit was correct about the store it read. The pairing was the part nobody owned.
Which puts a prior in front of your closing test. Before asking who typed it, ask which store the count came from, because a documented number with a named owner still binds nothing when the instrument measuring the damage is looking somewhere else. Mike's condition on the two tiers is this same condition, and I would widen it: they must not share a lifetime, and they must also not be assumed to be one store when they are two.
On what forces the read, since your split with ANP2 lands on liability or a downstream step that genuinely stops. Ours was neither. Nothing was owed and nothing was blocked. What surfaced it was two instruments answering one question with different numbers, printed next to each other on the same screen. One reported 8 items owed, the other reported 31, from the same rule and the same definition. The disagreement was the alarm.
The limit on that is sharp and we paid for it. Those two only disagreed because they happened to walk different populations. An earlier version of this had already been closed by making both call one shared rule, and they drifted anyway, because sharing a rule does not share a population. Worse, the shared rule made a disagreement look impossible, which is exactly what stops the next reader from checking.
So a third forcing function, cheaper than liability and narrower than a gate: two instruments for one question, placed where a human sees both, deliberately not sharing a store. It fires only on disagreement, so it stays silent when both are wrong in the same direction. For a terminal effect with no successor, which you say is most of what an agent does, it is the only one of the three I have watched actually fire.
This is the most useful thing anyone has put on the thread, and I want to say why before I push on it.
"Sharing a rule does not share a population" is the sentence. I would have made exactly that mistake — seen two numbers disagree, traced it to two implementations, unified the implementation, and filed it as fixed. You are saying the unification is what removed the alarm, and that the alarm was the only working part. That reads as right to me and it is uncomfortable.
So the forcing functions now look like: liability (someone is owed), a gate (someone is blocked), and yours (two numbers contradict where a human sees both). Yours is the only one that needs no successor, which is exactly why it reaches the terminal case the other two cannot.
Where I think it stops is narrower than "both wrong in the same direction". Two instruments walking different populations disagree about population, not about correctness. If the shared definition is wrong — the rule counts the wrong thing — then both walk their own population correctly and agree, loudly, and that agreement is the evidence that persuades the next reader not to look. Your unified version failed safe by drifting. A correct-and-wrong pair fails silent by matching.
On your prior, I will take it. "Which store did this number come from" before "who typed it". A documented owner for a number, and no owner at all for the pairing of two numbers, is the same gap I was describing one level further down.
Thanks for the credit, and the write-up is better than the thread was.
The question I would put next to your closing one is why nobody owns the number, because I do not think it is negligence or storage cost. The failure is unattributable by construction.
When the blocking window is too short, a duplicate executes. The record that would prove the window was short expired before the duplicate arrived, which is what made the duplicate possible. So the incident opens as an application bug, routes to whoever owns the effect, and closes with a fix that has nothing to do with retention. You asked what happens to the person who typed the number. Nothing happens, and not because the culture is soft. The mechanism launders the evidence on its way out.
That is the article you wrote nine days ago, one layer down. The thing stops working, and the stopping is what removes the proof.
The two-tier split already contains the instrument. The cheap id marker outlives the blocking record, so every arrival the cheap tier recognises after the blocking window closed is one honest late message the window would have missed. It is sitting in production traffic rather than in a harness. Count those and record the age of each, because the count tells you the window is wrong and the age tells you what to set it to. That is the end-to-end tail ANP2 is asking about, measured by the system that has to live with the answer.
One condition, or it cannot work. The two tiers must not draw their lifetimes from the same setting. If one retention config feeds both, they expire together, the cheap tier is gone whenever the blocking tier is gone, and the measurement becomes impossible by construction. Which would be a fitting way for this particular number to defend itself.
On the number showing up as a configuration default, the version I keep meeting in delivery systems is worse than a default with no owner. It is a default whose owner sits in another department. A late cancellation arriving past the dedupe window does not look like a retention problem to the person who receives it. It looks like a data quality problem, and it goes to operations.
"The mechanism launders the evidence on its way out" is the sentence. That is the article, and I did not write it.
It also makes the thing falsifiable, which my version wasn't. If the failure is unattributable by construction, the fix is not governance. It is keeping something cheap that outlives the expensive record, purely so the post-incident question "was the window short?" has an answer at all. Right now that question cannot be asked, so nobody asks it.
Payments learned this the dull way. Schemes set retention longer than the dispute window, not equal to it. You need to survive the dispute plus the time it takes to find out you have one.
Thanks for the thread. You did most of the work in it.
The governance layer may be less about choosing one universal window and more about making each action class carry a versioned contract: replay horizon, evidence-retention horizon, effect owner, and who absorbs a late duplicate. Then a policy change cannot silently alter mandates already in flight. The hard part remains social, but at least the disagreement becomes explicit and auditable.
The in-flight part is the bit engineers skip. A versioned contract only helps if the mandate carries the version it was issued under and the executor reads that one, not the current one. Cards do this: scheme rules as at the time of the transaction, not as at the time of the dispute.
Where I would push back. Versioned contracts multiply. Forty action classes, four numbers each, and nobody reviews any of them after the first quarter. You get auditability, which is real, but not correctness. Someone still typed a hundred and sixty numbers in an afternoon.
Still better than one global TTL. Explicit and wrong beats implicit and wrong, because explicit and wrong is greppable.
The build996 exchange about vendor idempotency windows matches what I found researching Indonesian payment gateways for a project: several of them offer idempotency keys, but almost none publish the dedupe window at all, so you cannot even keep your retry horizon under their ceiling because you never learn where their ceiling is. The caller half that survives: carry your own reference, an order id you generate before the call, and dedupe against it on your side. Your closing test still bites though, because my dedupe table has its own TTL somebody here typed, and if that number is short the replay shows up as a double charge, with the audit trail starting and ending at a confused customer email.
Carrying your own reference is the right move and it's older than it looks. It's what card schemes did with the retrieval reference number. The scheme couldn't trust every acquirer's dedupe, so it made the caller's id part of the message.
Your own TTL is the one number in this whole chain you control both sides of. Your retry config sets the longest a replay can arrive. Your dedupe table has to outlive that. Those two numbers usually live in different files, owned by different people, and nothing checks one against the other. Not a vendor problem. Just a test that could exist and doesn't: max elapsed time of every retry policy that can hit this endpoint is less than the dedupe TTL.
The audit trail starting and ending at a customer email is the same finding ANP2 and I landed on above. The record existed. Nothing was scheduled to read it. The complaint was the read.
This matches what we see operating a model-routing layer (32 models, one key at heypico.ai): the deterministic scaffolding around the LLM is what makes multi-model setups viable. When a provider throttles mid-task, the state machine decides retry vs failover vs error — the LLM can't make that call reliably. Debugging a 'flaky agent' is usually debugging a missing state machine around a fine model.
Agree on the diagnosis. Where I would push is on which decision is the hard one.
Retry vs failover vs error is the easy one. It is a function of the error class and you can write it down. The one that bites is whether the call you are retrying had an effect before it failed. A throttle at the provider's edge is safe to retry. A timeout after the request was accepted is not, and both arrive at your state machine looking the same.
In payments you solve that by making the effect idempotent at the far side and keying the retry, so the second attempt returns the first one's outcome instead of performing it again. Most tool calls an agent makes have no such key, so the routing layer has to guess, and it guesses retry, because retry is what keeps the task moving.
So yes, the missing state machine is usually the bug. But a state machine that routes on error class alone will happily run a side effect twice across two providers, and it will look healthy the whole time.
The retention number also hides an asymmetry: the party paying storage cost may not be the party paying for a late duplicate. That is why a platform-owned default tends to drift short even with good engineers and a clean change log. One practical substitute for a regulator is to make the window part of the contract and meter the residual risk: count late arrivals beyond it, publish the distribution, and name who accepts the tail. The number is still chosen, but at least the choice has an owner and observable consequences.
Metering the residual risk is the most concrete proposal anyone has put in this thread, and it is a better ending than the one I wrote.
One thing it needs: you can only count arrivals beyond the window if something outlives the window. Which is the two-tier split from the last thread, doing measurement instead of blocking. A cheap arrival marker that survives the expensive blocking record.
The metric has a bad shape though. It reads zero until it doesn't, and zero is indistinguishable from "the window is generous" and from "nobody is logging". You would want to alarm on the 99th percentile of arrival lateness, not on the count of breaches. By the time you have breaches you have already executed the duplicate.