If one service has to tell ten others that an order was placed, calling them one after another makes every order wait for the sum of all ten calls, so the slowest service sets the pace for everyone. Publish/subscribe lets each subscription process events independently, so the publisher can continue without waiting for subscriber work.
Prerequisites {#prerequisites}
A worker that gets a job every 5 ms and needs 18 ms per job has an offered load of ρ = λE[S] = 18/5 = 3.6, or 360% of one worker's capacity. The worker stays busy and its backlog grows while arrivals continue. The same queueing logic sits behind latency versus throughput.
Definitions {#definitions}
Definition
Publisher: the service that announces an event (here, "order placed") without knowing who will act on it.
Definition
Topic: a named channel the publisher sends each event to once. The broker makes the event available to every matching subscription.
Definition
Subscription: an independent delivery stream and backlog for a consuming service, which can share processing among multiple consumer instances. In Kafka, the equivalent position is tracked per partition within each consumer group.
Definition
Backlog: in our lab, the largest number of events that were placed but not yet started by a given party.
How publish/subscribe works {#how}
- The publisher sends the event to a topic and gets an acknowledgement from the broker. It does not wait for any subscriber.
- The broker makes the event available to each matching subscription; storage depends on the implementation. Google Cloud Pub/Sub delivers each message to every subscription attached to the topic (overview). RabbitMQ routes it through a fanout exchange to every bound queue, and in the tutorial a message sent before any queue is bound is discarded. Kafka keeps one log per partition, and each consumer group tracks its own offset in it (docs).
- Each subscriber pulls or receives from its own subscription at its own speed and acknowledges what it finished.
With direct calls the publisher pays for every subscriber on every event. With a topic it pays for one publish, and the broker does the fan-out. Our lab assumes that publish costs 0.05 ms plus 0.002 ms per subscription, so 0.052 ms at one subscriber and 0.07 ms at ten.
Why direct calls pile up: the arithmetic {#math}
With direct calls in sequence, the publisher's service time per event is the sum over subscribers of round trip plus work:
S_direct = sum over i of (rtt + work_i)
= 10 x 0.3 ms + nine fast jobs (1 to 2 ms) + one slow job (about 51 ms)
= 67.97 ms per event (lab output, seed 61)
arrivals every 5 ms -> offered load ρ = 67.97 / 5 = 13.6
At 13.6 the publisher's queue grows by about 12.6 events for every one it finishes. A fast subscriber receives event n only after the publisher reaches it, so it inherits the slow one's delay. Even without the slow subscriber, ten fast calls took 18 ms per event, an offered load of 3.6, and fast jobs averaged 6,502.54 ms.
With pub/sub the publisher's service time is the publish cost alone, 0.07 ms against a 5 ms gap, an offered load of about 0.014, so it never builds a backlog. Each fast subscriber is busy about 30% of the time (1 to 2 ms of work every 5 ms), which keeps its wait close to its own work time.
Worked example {#example}
1,000 orders, one every 5 ms -> the last one arrives at 4,995 ms
direct: publisher starts one order every 67.97 ms (average; seeded work varies per event)
by 4,995 ms it has started floor(4,995 / 67.97) + 1 = 74 orders
1,000 - 74 = 926 orders placed but not started (lab: 926)
order n waits about n x (67.97 - 5) ms before its calls start
average over n = 0..999: about 499.5 x 62.97 = 31,453 ms,
plus its place in the call sequence (lab: 31,483.88 ms)
pub/sub: fast completion ≈ 0.07 ms publish + 0.3 ms hand-off + 1.5 ms mean work = 1.87 ms
slow subscriber: first start at 0.37 ms, then about 51.5 ms per event,
so by 4,995 ms it has started about (4,995 - 0.37) / 51.5 + 1 ≈ 98 events
(98 in the seeded run); 1,000 - 98 = 902 waiting (lab: 902)
Assumed by hand: round trip 0.3 ms, work 1 to 2 ms, the +50 ms on subscriber six, the hand-off 0.3 ms and the publish cost. Derived by the simulation: every backlog, wait and finish time.
Code {#code}
cd labs/sd-61
node run.mjs # Node.js 22, no packages, no network, under a second on a virtual clock
Compare the two 10 yes rows: pub ms/event 67.97 against 0.07, fast avg ms 31483.88 against 1.87, and the slow subscriber's backlog, 926 against 902. The slow subscriber finished 16,449.28 ms earlier with pub/sub, because its work overlapped with the other subscribers instead of waiting behind their calls.
Failure modes and edge cases {#failure}
- The slow consumer's backlog grows. Pub/sub moved the 50 ms problem into the invoice service's own queue. Watch each subscription's oldest unacknowledged message age and message count, and add consumers where messages can be processed independently (the next part, competing consumers); a single ordered key remains limited by its serial processing rate. Or make the work cheaper.
- Isolation depends on broker capacity: growing queues can exhaust shared resources and throttle publishing. RabbitMQ, for example, blocks publishers when its memory alarm fires. Monitor broker resource alarms alongside subscription backlog.
- Check retention and overflow policies. Configured expiry (RabbitMQ TTL, for example) or storage limits can remove unread events; alert before the applicable limit.
- Redelivery. With RabbitMQ manual acknowledgements, closing a channel or connection requeues its unacknowledged deliveries; another consumer may receive them. Durability and acknowledgement mode are separate settings (RabbitMQ).
- Ordering. Publish related events in causal order and configure the broker's ordering guarantee. In Google Pub/Sub, use the same ordering key and region, and enable subscription ordering (docs). Preserve that order when processing messages concurrently.
- Hidden coupling. If checkout needs the invoice service's answer before replying to the user, it is a request, and that call should stay direct.
Trade-offs {#tradeoffs}
| Option | Good for | Costs you | Use when |
|---|---|---|---|
| Direct calls in sequence | Simple code, immediate errors | Publisher time is the sum of every call | One or two fast, critical callees |
| Direct calls in parallel | Lower latency than sequence | Still waits for the slowest; failure handling per callee | The caller truly needs every answer now (our lab did not test this) |
| Publish/subscribe | Many independent reactions to one event | A broker to run, redelivery, backlog and broker resources to monitor | Subscribers can act later and at their own speed |
When you split a monolith into services, this choice comes up early. Services that keep direct calls find each other through service discovery.
Common mistakes {#mistakes}
Expecting pub/sub to speed up a slow consumer. Scale or fix that consumer separately.
Sharing one subscription between two different services, which splits messages between them. Give each service its own.
Running without an alert on oldest message age per subscription.
Writing the dedupe record separately from the business change. For database effects, insert the event ID under a unique constraint in the same transaction as the business change, then acknowledge after commit. A separate dedupe write leaves a crash window.
In an interview {#interview}
Ask whether the caller needs each service's answer before it replies. If not, publish once to a topic with one subscription per service; if it does, keep that call direct.
Practice {#practice}
BeginnerWhat does a topic do that a queue does not?
It makes each message available to every subscription; a queue gives each message to one consumer.
IntermediateDirect calls take 18 ms per event and events arrive every 5 ms. What happens?
Offered load is 3.6, so the publisher's backlog grows by about 2.6 events per event served while arrivals continue.
IntermediateWhy did fast jobs wait 31 s with direct calls?
They queued behind earlier orders that each took 67.97 ms.
AdvancedWhy did the slow subscriber finish earlier with pub/sub?
Only its own work limited it.
AdvancedHow can a slow subscriber still slow the publisher?
Its backlog fills shared broker memory or disk, and the broker throttles publishing.
AdvancedWhen should you keep a direct call?
When the caller cannot answer its own user without the callee's result.
Cheat sheet {#cheat}
| Term or formula | Meaning |
|---|---|
| S_direct = sum (rtt + work_i) | Publisher time per event with sequential calls |
| offered load ρ = λE[S] = S / arrival gap | Above 1, the backlog grows while arrivals continue |
| Topic | Publish once; every matching subscription can read the event |
| Subscription | An independent delivery stream and backlog for a consuming service, which can share processing among multiple consumer instances |
| At-least-once | Duplicates possible; dedupe in the same transaction |
| Oldest message age | The per-subscription backlog alert |
Further reading {#reading}
- Google Cloud Pub/Sub overview and ordering guide: subscriptions, ordering keys and their requirements.
- Apache Kafka documentation: logs, partitions and consumer groups with their own offsets.
- RabbitMQ Publish/Subscribe tutorial, plus the memory alarm and acknowledgement pages.
This article describes general messaging patterns and public documentation; it does not describe any company's private systems, and the diagrams are conceptual. Every number comes from labs/sd-61 (seed 61), a virtual-time simulation with hand-picked costs. It shows the shape of the effect, not broker benchmarks.
Top comments (0)