DEV Community

Vishwa
Vishwa

Posted on Originally published at Medium

Pub/Sub for Event-Driven Orders: One Order, Many Independent Reactions

Publish order_placed once; email, inventory and analytics each get their own copy and their own retries

A customer taps "Place order". Behind that tap, several things should happen: a confirmation email goes out, inventory is updated, and the order shows up in the analytics reports. The customer only cares that the order went through.

The problem

The simple way is for the order service to call each one in turn:

place_order()
  -> call email service      (wait)
  -> call inventory service  (wait)
  -> call analytics service  (wait)
  -> return "order placed"
Enter fullscreen mode Exit fullscreen mode

This breaks in a few ways:

  • Slow: the customer waits for every call, including ones they do not care about.
  • Fragile: if the email service is down, does the order fail? If you add a retry loop, you are now writing a scheduler inside the order service.
  • Tightly coupled: adding a fourth reaction, like a loyalty-points service, means changing and redeploying the order service.

The solution: one topic, one subscription per reaction

With Pub/Sub, the order service publishes one order_placed event to a topic and returns. Each reacting service has its own subscription to that topic. Pub/Sub puts a copy of every message in every subscription.

order-service --publish--> topic: order_placed
                              |-- email-sub      -> email worker
                              |-- inventory-sub  -> inventory worker
                              |-- analytics-sub  -> analytics pipeline
Enter fullscreen mode Exit fullscreen mode

The key idea: each subscription has its own backlog, its own acknowledgements and its own retries. That gives three useful properties.

Failures stay local. If the email worker is failing, its messages wait and retry in email-sub. Inventory and analytics keep going, untouched.

Slow consumers do not lose data. If the inventory worker is slow, its backlog grows. Nothing is dropped; it catches up when it can.

Adding a reaction is a new subscription. The loyalty-points service subscribes to order_placed. The order service does not change.

Retries and the dead-letter topic

Pub/Sub does the retrying, so no scheduler is needed. If a worker does not acknowledge a message, Pub/Sub redelivers it with exponential backoff. After a maximum number of attempts (for example, 5), the message moves to that subscription's dead-letter topic, where someone can inspect it, fix the cause and republish.

Push or pull

  • Pull: the worker asks for messages and acknowledges with an explicit ack(). It pulls only what it can handle, which is the most direct form of rate limiting. Good for heavy or batch workers.
  • Push: Pub/Sub POSTs each message to a URL, such as a Cloud Run service. A 2xx response is the ack. Rate control is indirect, through Cloud Run max instances and Pub/Sub backing off on errors.

For this example, I would use push to Cloud Run for email and pull for the analytics pipeline, which can feed Dataflow and then BigQuery.

Duplicates are normal

Pub/Sub delivers at least once, so the same message can arrive twice. Every consumer must be idempotent. The inventory worker can record the order_id it has processed and skip a repeat. The analytics pipeline can drop duplicates using a key such as order_id plus table plus timestamp.

A smaller version of this pattern

In VJourney, my personal travel-companion project, notifications to a trip host when someone joins are delivered asynchronously through a Redis Stream consumer group. It is the same idea at a smaller scale: the request finishes quickly, and a separate consumer handles the follow-up work.

Trade-offs and limits

  • Not everything should be async. Checking that stock exists before confirming the order must happen inside the order transaction. Pub/Sub is for reactions after the fact.
  • Pick the right tool. Pub/Sub is for broadcasting an event to many subscribers. A command to one target with a delay ("remind in 30 minutes") fits Cloud Tasks better. If you need to replay a long history of events as a core feature, a log like Kafka fits better.
  • Create subscriptions first. A message published to a topic with no subscription is dropped.
  • Watch the dead-letter topics. A DLQ nobody reads is just a quiet place to lose orders. I would alert on its size.
  • Same subscription vs different subscriptions. Several pods on one subscription share the work. Different services need different subscriptions to each get a copy. Mixing these up is a common bug.

Takeaway

Publish the event once; each consumer gets its own subscription.

Retries, backlogs and dead-letter topics are per subscription, so failures stay local.

Delivery is at least once, so every consumer must be idempotent.

Top comments (0)