The broker may bring the same message three times. The consumer punches its id and does the work in one transaction, so the second and third visits find the ticket already punched.
👋 Hi, I'm Anton - a software engineer. These days I work on a platform: contracts, generated code, data paths and the tooling around them. Running notes live on my GitHub: github.com/brilliant-almazov.
The once-per-message consumer is built into the shared library as one of its thirteen generic routines, and a handful of services use that runtime. I describe the mechanism and its shape; I did not measure how many duplicates it has caught.
This is part 4 of the series on what a shared platform does for every service, told as a zoo. Maybe you already do it better, maybe you'd do it another way - this is how it is going, and where it stops.
In real life
The zoo has a turnstile, and Nora stands next to it with a hole punch. Every visitor has a ticket; Nora punches it, the turnstile turns, the visitor goes in. Simple.
Then Biscuit arrives. Biscuit is the zoo's courier dog, and he is very, very reliable - so reliable that when he is not sure a ticket got through, he brings it again. "Me again! Same ticket!" It is the third time this hour.
The first time, Nora punches the ticket and lets Biscuit in: "Punched. Welcome!" The second time, she looks at the ticket and sees the hole: "Already punched." Biscuit is a little offended - "But I ran so fast!" - but the zoo does not let one ticket in twice.
The last panel is the part that matters. Pip insists that the punch and the turnstile happen in one move. If Nora punched a ticket and the turnstile jammed, the visitor would be locked out with a used ticket. If the turnstile turned and Nora forgot to punch, the same ticket would come in again tomorrow. Never one without the other.
At least once is a promise about delivery, not about work
Message brokers promise to deliver at least once. That is a good promise: a message is not lost when a consumer dies halfway. It also means that a consumer will, sooner or later, see the same message twice - after a redelivery, a reconnect, a restart in the wrong moment. If the work is "add a record" or "send a notice", twice is a bug.
Each consumer used to guard against that on its own, or not at all. The shared library now has a generic consumer routine that does it the same way everywhere: receive a message once per x-message-id.
The punch and the entry, in one transaction
The routine keeps a small inbox table of message ids. For each message it inserts the id and, in the same transaction, runs the handler. As a form - the names are illustrative:
BEGIN;
INSERT INTO inbox (message_id) VALUES ($1) ON CONFLICT DO NOTHING;
-- 0 rows: seen before -> acknowledge, skip the work
-- 1 row : new -> do the work here, in this transaction
COMMIT; -- the punch and the work land together, or neither does
The order inside the transaction is the whole design. If the handler fails, the transaction rolls back and the id is not stored - the next delivery is treated as new, which is right, because the work did not happen. If the handler succeeds, the id and the work commit together, and every later delivery hits the conflict and is skipped. There is no moment in which the ticket is punched but the visitor is outside, or the visitor is inside with an unpunched ticket.
| Delivery | Inbox insert | Work | Result |
|---|---|---|---|
| first | 1 row | done, same transaction | committed together |
| first, handler fails | rolled back | not done | next delivery is new |
| second, same id | 0 rows | skipped | acknowledged |
| third, same id | 0 rows | skipped | acknowledged |
One of thirteen routines
The once-per-message consumer is not a separate library. It is one of thirteen generic routines in the library's scenario runtime - the next-but-one part of this series is about all thirteen. A service that consumes messages does not write the inbox, the conflict rule or the transaction; it binds its handler to the routine.
What is left for the service is the part only the service knows. In the shape of the code, with the names shortened:
// a consumer in a service - a form
inbox.Once(ctx, msg.ID, func(ctx context.Context) error {
return orders.Apply(ctx, msg) // the work, inside the routine's transaction
})
The handler runs inside the routine's transaction instead of opening its own, which is how the punch and the work end up in one commit. A handler that opened a second connection and wrote there would quietly step outside the turnstile - that is the one rule a service has to keep, and it is the kind of rule a review can see.
There is a small cost, and it is worth naming. The inbox is a table that grows with every message, and it needs the same care as any growing table. The routine owns the punch; how long used tickets are kept is not part of what I describe here, and I do not claim it is solved.
What went wrong
The failure this routine answers is the quiet kind: duplicates that nobody notices until two notices go out, or a counter is one too high. Every consumer that guarded itself did it slightly differently, and a consumer that did not guard itself did not look any different from the outside - until a redelivery.
The second lesson is about where the punch lives. A common shortcut is to check "have I seen this id?" in one step and do the work in another. Between the two steps, a second delivery can pass the same check. Inserting the id and doing the work in one transaction removes that gap instead of narrowing it.
What I do not claim
- Not exactly-once delivery. The broker still delivers at least once; the work happens once.
- Not that every consumer in every service uses the routine. A few services do.
- Not a count of duplicates caught. I did not measure it.
- Not that this covers effects outside the database transaction. A call to another system inside the handler is still the handler's problem.
- Not the exact names. The SQL and the Go above are forms that show the shape.
- Not a retention rule for the inbox. It grows, and that is a separate decision.
What of this is method
- Punch and entry in one transaction. The id and the work commit together, or neither.
- Accept at least once; make the work once. Do not fight the broker's promise.
- Write the guard once, bind it everywhere. A routine in the library, not a habit in each consumer.
Zoo keeping, built in - Part 4. Next: Feeding By Sectors.
If your consumers are idempotent, where does the punch live - in the same transaction as the work, or next to it? If you rely on the broker for exactly once, what happens on a redelivery after a restart? If you had a duplicate in production, how did you find it? How many times did your last message come through the turnstile?








Top comments (0)