DEV Community

Cover image for One Ticket, Punched Once
Anton Brilliantov
Anton Brilliantov

Posted on

One Ticket, Punched Once

The broker may bring the same message three times. The consumer punches its id and does the work in one transaction, so the second and third visits find the ticket already punched.


👋 Hi, I'm Anton - a software engineer. These days I work on a platform: contracts, generated code, data paths and the tooling around them. Running notes live on my GitHub: github.com/brilliant-almazov.

The once-per-message consumer is built into the shared library as one of its thirteen generic routines, and a handful of services use that runtime. I describe the mechanism and its shape; I did not measure how many duplicates it has caught.

This is part 4 of the series on what a shared platform does for every service, told as a zoo. Maybe you already do it better, maybe you'd do it another way - this is how it is going, and where it stops.


In real life

The zoo has a turnstile, and Nora stands next to it with a hole punch. Every visitor has a ticket; Nora punches it, the turnstile turns, the visitor goes in. Simple.

Then Biscuit arrives. Biscuit is the zoo's courier dog, and he is very, very reliable - so reliable that when he is not sure a ticket got through, he brings it again. "Me again! Same ticket!" It is the third time this hour.

The first time, Nora punches the ticket and lets Biscuit in: "Punched. Welcome!" The second time, she looks at the ticket and sees the hole: "Already punched." Biscuit is a little offended - "But I ran so fast!" - but the zoo does not let one ticket in twice.

A four-panel comic: Biscuit at the turnstile with the same ticket for the third time this hour; Nora punches it the first time and Biscuit is delighted; the next time Nora says already punched and Biscuit says but I ran so fast; Pip says punch and open in one move, Pickle asks never just one of them

The last panel is the part that matters. Pip insists that the punch and the turnstile happen in one move. If Nora punched a ticket and the turnstile jammed, the visitor would be locked out with a used ticket. If the turnstile turned and Nora forgot to punch, the same ticket would come in again tomorrow. Never one without the other.

At least once is a promise about delivery, not about work

Message brokers promise to deliver at least once. That is a good promise: a message is not lost when a consumer dies halfway. It also means that a consumer will, sooner or later, see the same message twice - after a redelivery, a reconnect, a restart in the wrong moment. If the work is "add a record" or "send a notice", twice is a bug.

Each consumer used to guard against that on its own, or not at all. The shared library now has a generic consumer routine that does it the same way everywhere: receive a message once per x-message-id.

Three tiles with faces: the broker, bored; the inbox, proud; the handler, happy; labels read x-message-id and same transaction; pills read built, and used by a few services

The punch and the entry, in one transaction

The routine keeps a small inbox table of message ids. For each message it inserts the id and, in the same transaction, runs the handler. As a form - the names are illustrative:

BEGIN;
INSERT INTO inbox (message_id) VALUES ($1) ON CONFLICT DO NOTHING;
-- 0 rows: seen before  -> acknowledge, skip the work
-- 1 row : new          -> do the work here, in this transaction
COMMIT;   -- the punch and the work land together, or neither does
Enter fullscreen mode Exit fullscreen mode

A dark card with the inbox transaction as a form; the insert with ON CONFLICT DO NOTHING is highlighted; a pill reads built

The order inside the transaction is the whole design. If the handler fails, the transaction rolls back and the id is not stored - the next delivery is treated as new, which is right, because the work did not happen. If the handler succeeds, the id and the work commit together, and every later delivery hits the conflict and is skipped. There is no moment in which the ticket is punched but the visitor is outside, or the visitor is inside with an unpunched ticket.

Delivery Inbox insert Work Result
first 1 row done, same transaction committed together
first, handler fails rolled back not done next delivery is new
second, same id 0 rows skipped acknowledged
third, same id 0 rows skipped acknowledged

An animation: the broker delivers a message, the inbox punches the new id and the handler does the work; the same id arrives a second and a third time and is skipped at the inbox; a pill reads handled once

One of thirteen routines

The once-per-message consumer is not a separate library. It is one of thirteen generic routines in the library's scenario runtime - the next-but-one part of this series is about all thirteen. A service that consumes messages does not write the inbox, the conflict rule or the transaction; it binds its handler to the routine.

What is left for the service is the part only the service knows. In the shape of the code, with the names shortened:

// a consumer in a service - a form
inbox.Once(ctx, msg.ID, func(ctx context.Context) error {
    return orders.Apply(ctx, msg) // the work, inside the routine's transaction
})
Enter fullscreen mode Exit fullscreen mode

The handler runs inside the routine's transaction instead of opening its own, which is how the punch and the work end up in one commit. A handler that opened a second connection and wrote there would quietly step outside the turnstile - that is the one rule a service has to keep, and it is the kind of rule a review can see.

There is a small cost, and it is worth naming. The inbox is a table that grows with every message, and it needs the same care as any growing table. The routine owns the punch; how long used tickets are kept is not part of what I describe here, and I do not claim it is solved.

A waffle of thirteen cells: one teal cell with a check - consume, punched once; twelve indigo cells - the other twelve routines; Pip says one of thirteen, the same for all; a pill reads built

Bars: files about 120, commits since October about 15, services importing it 4; Pickle says four services, one punch; a pill reads counted, rounded

Before and after: at least once - 2 effects for one message delivered twice; by hand, each consumer guards itself - and punched once - 1 effect per x-message-id; 1 transaction for the punch and the work

What went wrong

The failure this routine answers is the quiet kind: duplicates that nobody notices until two notices go out, or a counter is one too high. Every consumer that guarded itself did it slightly differently, and a consumer that did not guard itself did not look any different from the outside - until a redelivery.

The second lesson is about where the punch lives. A common shortcut is to check "have I seen this id?" in one step and do the work in another. Between the two steps, a second delivery can pass the same check. Inserting the id and doing the work in one transaction removes that gap instead of narrowing it.

Service tiles with faces: the broker happy, the inbox proud, the handler happy; a repeat bored, the books happy; pills read built, and used by a few services

What I do not claim

  • Not exactly-once delivery. The broker still delivers at least once; the work happens once.
  • Not that every consumer in every service uses the routine. A few services do.
  • Not a count of duplicates caught. I did not measure it.
  • Not that this covers effects outside the database transaction. A call to another system inside the handler is still the handler's problem.
  • Not the exact names. The SQL and the Go above are forms that show the shape.
  • Not a retention rule for the inbox. It grows, and that is a separate decision.

What of this is method

  • Punch and entry in one transaction. The id and the work commit together, or neither.
  • Accept at least once; make the work once. Do not fight the broker's promise.
  • Write the guard once, bind it everywhere. A routine in the library, not a habit in each consumer.

Zoo keeping, built in - Part 4. Next: Feeding By Sectors.

If your consumers are idempotent, where does the punch live - in the same transaction as the work, or next to it? If you rely on the broker for exactly once, what happens on a redelivery after a restart? If you had a duplicate in production, how did you find it? How many times did your last message come through the turnstile?

Top comments (0)