An AI asks to write something, and a person has to approve it.
There are two kinds of "expired" in that sentence, and they are not the same thing:
- should the conversation keep waiting for the approval, or move on without it
- can this row still be decided at all, or is it dead
We used one timeout to answer both. What that produced was a list showing "waiting on you" for a row where clicking was guaranteed to fail.
This piece is about splitting those into two windows with one function each, and about the single point where a decision has to be arbitrated.
1. Why they look like one question
Both read as "how long does a confirmation last", and once there is one number, the instinct is one column.
But they answer different questions, and different things consume the answers:
| The question | Who consumes it | What a wrong answer costs |
|---|---|---|
| should the conversation keep waiting? | the one stream that is waiting | waiting a little too long, or moving on a little early |
| can this row still be decided? | the data layer and every decision entry | a click that cannot execute, or a row that is dead and still actionable |
A wrong answer to the first costs experience. A wrong answer to the second costs consistency. And when they share a value, the second question's answer gets dragged around by the first one's needs.
2. The scene: computed in two places, held together by a comment
This happened to us.
The offline window's boundary was computed twice: the read path compared timestamps in memory, and the maintenance sweep compared them in SQL.
What kept the two in agreement was a comment in the code, saying roughly that the predicate here matches the one over there.
A comment is not a predicate. The day the two disagree, you get a row like this:
the list shows "waiting on you", while the sweep has already expired it.
So the click fails, necessarily. And it is not an occasional failure: the list manufactures it, by presenting a dead row as a pending task.
What makes this hard to find is that it looks like "the click sometimes fails", while the cause is two places holding different definitions of the same instant.
3. Fix one: two windows, and one function per boundary
There are two halves to the fix.
First, admit they are two windows.
- the in-conversation wait is short, 60 seconds by default, and its expiry only ends that wait so the conversation can continue. It does not touch the data.
- the offline window is long, 24 hours by default, and the row stays decidable inside it. Only its expiry flips the row to timed out.
Second, every boundary is computed by exactly one function.
The read path and the sweep both call it and get the same instant. So "shown as actionable" and "already expired" become structurally impossible to be true at once.
The point is not that the arithmetic is right. The point is that there is one place doing it. Two places and all you have left is a comment promising they agree.
4. Fix two: one arbitration point for decisions
Once there are two windows, there are two decision entries: the button in the conversation, and the pending list afterwards. They can race.
So the second thing is to make them contend for the same thing.
What we do is a conditional update in the database, where the condition includes "the current status is still pending". Then:
- the update touches one row, so this decision stands and it executes
- the update touches zero rows, so somebody already decided, and this one does not execute
Why not read, then decide? Because there is a gap between the read and the write, and both decisions can read "still pending". That is a decision placed before the action without the arbitration being placed before the action. A conditional update is atomic: whoever changes the row first owns the decision.
Two entries, one arbitration point. That is the whole of this section.
5. The second scene: one "we don't know" read as "yes"
Another one of ours.
A decision has to be written before it counts. What if that write throws?
The old behavior swallowed the error and released the in-memory wait anyway. Which gave this chain:
- the database hiccups, the write fails
- the in-memory wait is released, so the tool runs
- the row is still pending
- the next decision, in the conversation or from the pending list, sees "still pending" and runs it a second time
A "we do not know whether the decision landed" was read as a "yes".
The fix is plain: without evidence that the decision reached the database, do not release. Fail this one and let the person retry, rather than translating uncertainty into execution.
One thing worth being precise about: for external tools with no idempotency key, that second run is a real replay and the target really does receive it twice. On that path, failing closed is not caution. It is required.
6. A boundary this design does not cross
The arbitration point guarantees at most one decision.
It does not guarantee the tool runs at most once. If the same request gets resent, by a model retry or a restart, that is a different problem and it has to be solved at a different layer.
The two must be kept apart: one is "the same decision cannot be taken twice", the other is "the same call cannot be sent twice". Rolling them into a single sentence about "we guarantee it executes once" is overclaiming.
We put that in the result messages and in the docs rather than leaving it to be discovered.
7. A check you can run yourself
Four questions.
- How many questions does your timeout answer? If one value decides both "does the UI keep waiting" and "can this row still be used", that is a shared value.
- How many implementations does the boundary have? Two is one comment away from drifting.
- What arbitrates a racing decision? An in-memory check is not arbitration. A conditional update, or any atomic compare-and-swap, is.
- What is the default when a decision fails to persist? If the next step after a failed write is "execute anyway", the default is allow.
Closing
Back to the opening: one timeout cannot answer two questions.
What we do: split it into two windows, 60 seconds of in-conversation waiting and 24 hours of decidable pending; compute each boundary in exactly one function; make every decision contend for one conditional update; and release nothing without evidence that the decision landed.
The mechanism above is taken from KeelBase, an open-source runtime that sits between an AI agent and a business system and re-checks identity, authorization, scope, risk and confirmation state before a tool actually runs.
Repository (a Gitee mirror is the faster route from inside China):
Two things stated plainly. The boundary above: we guarantee at most one decision, not at most one execution. And a premise of the whole design. It assumes decisions can be persisted. If the database is unavailable we refuse rather than allow, which for some systems means confirmation stops working entirely. That is deliberate. If confirmation means "nothing runs that was not confirmed", then when nothing can be confirmed, nothing should run.
Drafted with an AI assistant. The system, the positions and the mistakes are mine.
Top comments (20)
The "we don't know read as yes" scene is the one worth underlining. On the GUI side, where an agent's "send" is a real click in a real app, there is no idempotency key at all, so a released wait after a failed write means the message really goes out twice. Failing closed is the only safe default there.
Two things that might fit your model:
Does a late approval in KeelBase execute right away, or wait until the conversation picks it back up?
Right away. The conditional update is the decision, and the execution happens in the same request, so a row approved from the pending list runs at that moment rather than when the conversation resumes.
The conversation is deliberately not woken. Unblocking it would let the original stream execute the same tool a second time, because that branch only knows approve. So the stream waits out its own window and reports a timeout, and the cost is recorded in the code: the tool ran, and the agent was told it did not. Your second bullet names what that costs, and we do not have the notification it asks for.
How much it costs depends on the tool. A retry of the same call in the same conversation hits the idempotency key and gets the recorded result back. A tool outside that path, an external one with no key or anything that clicks a real interface, has nothing to hit, so the retry is a second execution. That is your first paragraph arriving through a different door.
On binding to what the person saw: the row already carries the exact arguments, and both execution paths run the arguments from the row rather than from wherever the retry came from, so a retry with different arguments is a different token and never reaches this one. A hash inside the conditional update would be defence in depth against a later change to that invariant. I would take it for that reason rather than because the hole is there today.
On the GUI surface I can only agree from the outside. We do not cover it, which is why your paragraph is the right argument to make there.
Thanks, that answers it, and the reason not to wake the stream makes sense: approve-only branch plus a live stream is exactly how you get the second execution.
One cheap way to shrink that cost without waking anything: when the wait times out, return "pending, not confirmed" instead of a plain timeout, plus the row id. An agent that reads "not run" retries; one that reads "pending, check row X" usually asks first. For the keyless tools that is the difference between one send and two.
Agreed on the hash: an invariant guard, not a fix for a hole.
That one is cheap enough to take, and it is cheaper than it looks from outside. A timeout here already returns an explanation alongside the error, with both windows in it, so the shape exists. What is wrong is the sentence at the top, which reads as "not run", and what is missing is the handle.
The handle is already in scope where the message is built, so this is passing a value through rather than plumbing a new one. One wrinkle you cannot see from the post: at run level the identifier is the run, not a confirmation row, so the sentence should not pretend the two are the same object.
Whether it actually stops the retry, I do not know and would not claim. "Pending, check row X" making an agent ask is a plausible mechanism, and the only way I would learn it here is to count it: the same tool in the same conversation, called again within some minutes of a timeout, measured before and after the change.
Agreed on keeping the run and the row as separate things. Saying "run R timed out, its confirmation is still pending" is honest, and it still gives the agent something to quote back when it asks.
For the count, I'd match on the same tool plus the same arguments hash within a few minutes of a timeout. A retry with different arguments is usually the agent changing its plan, not a double send, so leaving those out keeps the before/after number clean.
That phrasing is better than mine and I will use it.
On the count, your key is right and we happen to already compute it: the key that decides idempotency here hashes the user, the conversation, the tool and the sorted arguments, so matching on tool plus argument hash is the same shape. My first version of the metric was missing that dimension.
What I would change is where the counter sits. After a timeout nothing ran, so there is no recorded result to replay, and a retry looks like the first execution of that call rather than a replay of it. So the count belongs at confirmation creation: the same key asking for a second confirmation inside the window. That is the number that will move if the wording changes the behaviour.
Agreed, that's the better spot: after a timeout there is nothing to replay, so the confirmation request is the first place the duplicate becomes visible.
One split I'd keep inside that count: a second request for the same key while the first confirmation is still pending, versus one after the first expired or was declined. The first is the model retrying blind and should drop when the wording changes. The second can be a legitimate re-ask, so mixing them would hide the effect you are trying to measure.
The split is right, and it needs no new state here. The row already carries a status, so the count can look at the previous row for that key and bucket it by whether it was still pending or already terminal.
The pending bucket is the one that should move. It also catches a concurrent duplicate, which I would keep in the same bucket since both are "asked again before anyone answered". The terminal bucket is different behaviour and I would leave it out of this number rather than dilute it.
Agreed on leaving terminal out of this number. I would still keep it as its own small counter though: a rise in "asked again after a decline" points to a different bug, the agent not treating a no as final, and it is easy to miss if nothing counts it. Two numbers, two meanings, neither diluting the other.
Two counters it is. The second one is worth keeping separate, and our side has one wrinkle that decides what it can honestly mean.
In the confirmations this counter looks at, a decline comes from one of two places. One is the button in the conversation, and the model on that stream is told "User declined the operation". The other is the pending list after the conversation has moved on. That path deliberately does not wake the conversation, so by the time the model asks again it has been told the wait timed out, not that the answer was no. Both paths write the same status and the same decided timestamp, and nothing else on the row records which one it was.
So "declined, then asked again" would mix a model that heard a refusal with one that never heard back. The first is the defect you named. The second is the late-decision gap we were discussing, appearing in the same bucket.
Keeping them apart needs one more recorded fact: whether the refusal reached the conversation. After that the counter moves for one reason, and it is the reason worth watching.
That wrinkle is worth fixing at the source, not just in the counter. If a list-path decline is never told to the conversation, the next request for the same key could return it: "this was declined from the pending list at T" instead of opening a new confirmation. Then the model hears the no on its next try, the second bucket mostly empties into the first, and the flag you'd record is also the thing that closes the gap.
You are right that the source is better than the counter, and it fixes something the counter cannot. Today every request opens a new confirmation. Nothing looks back at the last one for the same key, so the model re-asks and the person is asked twice for a call they already refused.
Two things we would have to get right. The lookup has to use the same canonical key as the idempotency path, because we store the arguments as raw JSON and the key sorts them first, so two spellings of one call are not equal on the row. And returning the old refusal forever would make "no" permanent for that call: if the person changes their mind and asks for it, the same conversation, tool and arguments would keep getting the old decline back. Return it once, or until the person asks again, and let the next request open a fresh confirmation.
If the lookup covers every out-of-band refusal, the flag stops being needed, which is the part of your message I like most: a new row after a decline would then only be possible when the refusal had already been delivered.
I would still keep the late-approval notice separate. Your fix reaches the model when it asks again, but an agent that never asks again still ends its turn believing nothing happened. So I would take this as the cheaper half of the fix, not the whole of it.
Agreed on both. "Once" is easy to make concrete: when the lookup hands back the old refusal, stamp that row as delivered, and only undelivered refusals are returned. After that the same call opens a fresh confirmation, so a changed mind costs one extra ask, never a permanent no. Reusing the idempotency key for the lookup also means the two paths can't disagree about what "the same call" is.
And yes, the late-approval notice is the other half: this only helps an agent that asks again.
Agreed, and the stamp belongs on both paths, not only on the lookup. A decline clicked in the conversation reaches the model through the stream, but the row does not record that either. If only the lookup stamps, the next ask after a conversational decline gets the old refusal handed back once more. That costs a changed mind one extra ask, and it hides the very case the second counter exists for, because the re-ask no longer creates a row.
Reusing the idempotency key is the right call. The row already carries the four parts it needs, user, conversation, tool and arguments, and the arguments only need the same canonicalization the key applies. The stamp itself is an annotation column rather than a new status: the row stays declined, and the lookup only hands back the ones not yet delivered.
And yes, the late-approval notice is the other half. An agent that never asks again still ends its turn believing the wait expired, and no lookup can reach that one.
Good catch, I only had the lookup in mind. Stamping at whichever point the refusal reaches the model, stream or lookup, keeps the second counter honest, and an annotation column instead of a new status is the cleaner way to do it. I think that covers the re-ask side; the late-approval notice is what's left.
Agreed. The re-ask side is settled as a shape, and none of it is built here yet: there is no delivery column on the row today.
The late-approval half is the one I would not leave as an open question. It is the case where the agent never asks again, so no lookup can reach it.
I'd include the offline deadline in the same conditional update that checks whether the row is pending. Otherwise an approval arriving after 24 hours could still win before the maintenance sweep changes the status, even with both paths computing the same boundary. A test that pauses the sweep, advances past the deadline, and races the two approval entries would expose that gap. I'd also test the exact deadline instant so the list, decision entry and sweep agree on whether equality is still actionable.
You are right, and the window is wider than a race. The sweep runs hourly, so an approval landing anywhere inside that hour still wins. The conditional update checks the token, the owner and pending, and nothing about the deadline. The deadline is enforced only afterwards, by the sweep that flips the row.
Which sharpens your second point. Wiring the deadline into the condition has to use the same inclusive edge the list uses, or the fix creates the disagreement it was meant to remove: at exactly the cutoff, the list offers the row and the decision would refuse it.
Both tests are the ones I would want, and the first is the only way to see this at all. With the sweep running, the gap is under an hour and nobody notices. With it paused, two entries race on a row that should already be dead.
One thing that does not need fixing, for what it is worth: the in-conversation path is already safe, because its timer drops the pending entry and a late click returns false there. The hole is only on the out-of-band path.
The distinction between “still waiting” and “still actionable” is easy to miss until it causes a real bug. I like that the solution separates the two timeouts instead of trying to make one value handle both responsibilities.
The split came out of a bug, not a design session. The list kept offering a row as still actionable after the sweep had already killed it, so the click was a guaranteed failure. Two values were the only way to make the list and the sweep agree on the same instant.
The part I would not call finished is the case where the decision arrives late and the conversation never hears it. That one needs a message, not a value.