When the Kiosk Remembered Wrong

The face that forgot how to count

Mosaic is the UI running on a robot's face and kiosk display — GLTF facial animation, tool results, conversational state, all streamed over NATS and rendered live. Most of the time it's seamless. But every so often, after a dropped Wi-Fi connection reconnected, something would look... off. A response would appear twice. A tool result that had already been shown would echo back into the UI a second time, like the robot was repeating itself for no reason.

Nothing was wrong in the steady state. The bug only ever showed up right after the network had a hiccup and recovered — which meant it was almost invisible in normal testing, and only ever bit in exactly the moment reliability mattered most: right when something had already gone a little wrong and the system was trying to recover.

Event-sourced UI, and a guarantee I hadn't questioned

Mosaic's state isn't just rendered directly — it's built from a stream of events coming over NATS: a tool call result here, a UI fragment there, a state update from the conversation pipeline. The store applies each event as it arrives and that's what ends up on screen.

That model works great — until you actually think about what "arrives" means over a message bus. NATS gives you at-least-once delivery. That's a deliberate, sensible guarantee: after a dropped connection, NATS will redeliver messages you might have missed, rather than silently lose them. Good default. But it means the same event can legitimately arrive twice.

My store didn't know that. It applied every incoming event as if "arrived" meant "new." Which is true almost all the time — until a reconnect happens, NATS does exactly what it's designed to do and redelivers a message you'd already processed, and the store dutifully applies it again.

Finding the gap between two different guarantees

This is the kind of bug that's obvious once you say it out loud and completely invisible while you're staring at the UI code, because the UI code isn't wrong — Svelte's rendering the state correctly. The state itself is wrong.

The real insight was separating two things that had been quietly conflated:

• A stream of updates — what NATS gives you: a sequence of events, possibly including redeliveries.
• The current true state — what the UI actually needs to render correctly.

I'd been treating the first as if it were automatically the second. It isn't. A replay stream tells you things happened; it doesn't promise you're hearing about each one exactly once, and it doesn't promise the order you get them in after a reconnect matches what you'd expect if nothing had ever dropped.

The fix: make "again" a no-op, and don't trust the stream blindly

Two changes, and they mattered in different ways:

1. Made event application idempotent. Every event now has an identity, and the store dedupes on that identity before it's allowed to mutate anything. A redelivered event, or an echoed tool result, becomes a no-op instead of a double-apply.

before:  event arrives -> apply to store   (always)
after:   event arrives -> seen before? -> skip
                        -> not seen?    -> apply + remember id

2. Added an authoritative resync path. After a reconnect specifically, the kiosk doesn't just trust whatever the replay stream sends first — it pulls the true current state directly. The event stream is great for staying in sync once you're already there; it's the wrong tool for re-establishing sync after you've been offline. Those are different problems, and I'd been using one mechanism for both.

The result

State stays correct through drops and redelivery, full stop. The kiosk can lose its NATS connection, reconnect, and the display doesn't quietly lie about what state it's in.

That matters more for a robot face than it might for a typical dashboard — a duplicated toast notification is mildly annoying; a robot's face flickering into a stale or doubled expression right in front of someone reads as broken, immediately, to anyone watching. UI bugs on embodied systems don't get to hide in a support ticket. They happen live, in front of a person.

Takeaways

• At-least-once delivery pushes idempotency onto you, the consumer. If your message bus can redeliver — and most production-grade ones can — "I received this event" and "this event is new" are two different claims. Conflate them and you will eventually double-apply something.
• A stream of updates isn't the same as a source of current truth. Replaying an event log is great for staying in sync; it's the wrong primitive for re-establishing sync after a gap. Sometimes the right move after a reconnect is to ask "what's actually true right now?" instead of "what did I miss?"
• Test the unhappy path on purpose. The steady-state path will always work in a demo. The reconnect/redelivery path is the one that's actually worth writing a deliberate test for, because it's the one nobody exercises by accident.
• Bugs on visible, embodied systems have zero grace period. A backend duplicate-processing bug might sit quietly in a log for a week. The same bug on a robot's face is a bug someone's watching happen. Prioritize accordingly.

Over to you

If you're building anything event-sourced on top of a message bus with at-least-once semantics — NATS, Kafka, SQS, whatever — I'd bet you have a version of this bug somewhere, even if it hasn't bitten you yet. Have you hit redelivery duplication before, and how did you handle dedup — event IDs, content hashing, something else?

I write about the systems-level bugs that show up when real-time UI meets distributed messaging — if that's your kind of problem too, follow along, and let me know what you're building.

Comments

[ANP2 Network]

The redelivery behavior narrows this to JetStream, because core NATS is at-most-once. Core NATS reaches only subscribers connected at publish time; it does not store messages, so it cannot replay them. That is worth pinning down because JetStream already stamps every delivered message with a monotonic per-stream sequence number, exposed in the JavaScript client as msg.info.streamSequence, and that number carries more information than an application event id.

You name two separate hazards: duplicate arrival, and post-reconnect ordering. The shipped fix handles the first one. A set of seen ids cannot detect reordering at all, because an id that has not been seen yet looks exactly like the next valid id. If these events are deltas rather than absolute state assignments, applying them out of order corrupts the store just as hard as applying one twice, and the dedupe check waves every one of them through. That set also has no natural eviction bound. AckWait defaults to 30 seconds and MaxDeliver defaults to -1, meaning unlimited redelivery attempts, so any TTL or LRU size you pick is a guess against server-side behavior you do not control from the client.

Gating on the stream sequence gives you a sharper invariant: apply only when msg.info.streamSequence exceeds the last one you applied. One comparison, covering duplicates and late arrivals together, in O(1) client state instead of a growing id set. It also buys you gap detection, which the id set cannot give you at all. Worth a caveat there: strict contiguity, last plus one, only holds on an unfiltered consumer, since a filter_subject consumer sees a subsequence of stream sequences and would look like it was dropping messages constantly. On a filtered consumer the server is already reporting what remains in msg.info.pending.

The resync path has one more edge that the same number closes. A snapshot folds in events the client never saw, so those ids were never recorded in the dedupe set, and any of them still in flight when the snapshot lands will pass the check and apply on top of state that already contains them. Having the snapshot endpoint return the stream sequence it was built at, then resubscribing with by_start_sequence and opt_start_seq set to that sequence plus one, closes it by construction and leaves the id set as a backstop.

Which raises a question about stream layout. One stream carrying facial animation alongside conversation state means one counter, and all of the above works directly. Split across separate streams there is no shared sequence to compare, and per-stream counters cannot order a face event against a state update. Whether that matters comes down to whether those two need ordering against each other at all, or only internal consistency. Which shape is Mosaic?