The Lock That Looked Safe But Guarded Nothing

Our AI DM bot had a bug that only showed up if you were fast. Send one message, wait a beat, send another and you'd sometimes get the same confirmation back three or four times, word for word. Slow typers never saw it. Fast ones almost always did. Because fast, back-to-back messages are rare in aggregate, this had been quietly happening in production for a while before anyone connected the dots.

The bug

Here's the classic trigger: a user sends their phone number, then a few seconds later, before the bot has replied to the first message, sends their email address.

Each inbound message re-triggered the function that starts (or resumes) an AI run. If a run was already in progress for that conversation, the second message didn't wait for it. It either kicked off a second parallel run or overwrote the first run's input with the newer message. So a two-message burst usually turned into two runs stepping on each other's context.

The only thing standing in the way was a database advisory lock, acquired with a single one-shot call. The problem: that kind of lock is scoped to the transaction that acquires it and the call itself completes (and releases the lock) in milliseconds. It runs and finishes long before the actual work (loading the conversation, calling the LLM, sending a reply) even starts. So it looked like mutual exclusion in the code, but by the time it mattered, the lock was already gone. It guarded nothing.

Result: two flavors of the same failure. Sometimes both runs generated the exact same reply and both sent it, byte-identical duplicates. Sometimes they got semi-serialized but each still generated its own answer to the same question, so the bot re-answered the same thing two or three times in slightly different words.

Why "just add a debounce" doesn't fully fix it

The obvious fix is: wait a couple seconds after the last message before doing anything, in case more messages are coming.

That helps, but only for messages that arrive before generation starts. Generation itself (calling the LLM, running tools, sending the reply) takes several seconds, sometimes closer to ten. A debounce does nothing once that clock has started. If a third message arrives mid-generation, you're back to the original problem, just shifted a few seconds later. Either it spawns a second run in parallel (duplicate replies again) or it gets blocked with no clean way to retry, which can wedge the conversation: the bot answers the first two messages but never acknowledges the third until the user says something else.

There's also a subtler trap in where you store the debounce timer. Our first attempt stored it inside the same blob of conversation state that gets fully rewritten at the end of every run. Since a run takes several seconds to finish, by the time it rewrote that blob, it clobbered the timer's own bookkeeping, a classic lost update. The practical symptom was almost funny: the bot would ask the same follow-up question twice, because its own "have I already handled this" state kept getting reset out from under it.

The actual fix: three pieces working together

Everything below lives on one row per conversation, so a burst of messages from the same person always lands on the same shared state. That's what makes the counters below safe to reason about.

1. A version counter and an "answered up to" watermark.
Every genuine inbound message atomically bumps a counter. Separately, a second number records the last counter value that actually got a reply sent for it. "Is there anything unanswered right now?" stops being a fuzzy question about timestamps and becomes plain integer subtraction: counter minus watermark. If it's zero, everything's been answered. If it's not, something's still owed a reply. This sidesteps almost every race you'd otherwise hit trying to compare "when did this arrive" against "when did that get answered."

2. A debounce window, stored in its own dedicated place (not inside the state blob that gets rewritten every run, see above). Every new message extends the window by a couple of seconds, but never past a hard cap from when the window first opened, so a user who never stops typing can't stall the reply forever. While the window is open, the system does nothing but extend it and return immediately. It explicitly does not start generating yet. That fast, do-nothing return turns out to be important: it's what lets the rest of a real burst actually get recorded before anything else happens.

3. A session-scoped lease with an expiry. This is what actually replaces the broken advisory lock. Instead of a one-shot RPC, it's a compare-and-swap update: "set this lease to my ID, but only if no one currently holds it (or their lease has expired)." Whoever wins that update is the only one allowed to generate a reply; whoever loses it exits immediately without touching the LLM. Because it has an expiry, a run that crashes mid-flight doesn't leave the conversation stuck. The lease frees itself after a timeout and someone else can pick it up.

How a burst actually flows through this

1. Message arrives, bumps the counter, opens or extends the debounce window, and returns immediately with no generation yet.
2. When the window finally closes, try to acquire the lease.
   • Lost it? Something else is already handling this conversation. Back off and let it finish.
   • Won it? Check the counter vs. the watermark. Nothing unanswered? Release the lease and stop, no reply needed.
   • Something unanswered? Gather every message since the last reply, fold them into one prompt ("here's everything you haven't addressed yet") and generate a single reply covering all of it.
3. Right before sending, one last check: did anything new arrive while this was generating? If yes, skip the send entirely and don't advance the watermark. The next cycle will pick up everything, including this run's contribution, in one clean pass, instead of sending a reply that's already stale.
4. Either way, release the lease.

Edge cases worth calling out

• A run crashes mid-generation, holding the lease. The lease's expiry reclaims it automatically; nobody has to notice and manually intervene.
• A slow, late cleanup call from a timed-out run. It's only allowed to release a lease it still actually owns, so it can't accidentally steal the lease from whoever picked things up after it timed out.
• A message lands right at the edge of the debounce window. The window's cap is anchored to when it first opened, computed in the same atomic step that extends it, so a late arrival can extend the window but can never push the hard cap itself further out.
• A stale wake-up fires after everything's already been answered. The counter-vs-watermark check catches this and it's a silent no-op instead of an unnecessary reply.
• A newer message arrives mid-generation. Covered above. The pre-send check catches it and defers to the next cycle instead of sending something already outdated.

---

Have you hit a similar "the lock released before it mattered" bug? I'd genuinely like to hear how you caught it. These are easy to ship and easy to miss in code review, since the code reads like it's correct.