A discussion about event-driven architecture caught my attention because the replies disagreed on something more useful than Kafka versus a database. People were describing different problems with the same word: event.
One team needs to run a slow task after a transaction. Another needs to tell several independent systems that a business fact has changed. A third needs an audit history it can replay. All three may produce a message, but they have different requirements for ownership, ordering, and recovery.
I would start by naming the failure that the current design cannot handle. If that failure is a web request waiting on slow work, a database-backed job may be enough. If independent consumers need a durable record of a business change, publishing an event may be worth the extra machinery. That gives the team a reason for the extra component.
A slow operation is not automatically an event
Imagine an application that accepts an order and generates a PDF receipt. The user needs to know whether the order was accepted. They do not need to hold the connection open while the PDF renderer runs.
The first design I would test is simple:
The order and job are written in one database transaction. If the transaction rolls back, neither exists. If the renderer fails, the job remains visible for a retry or investigation. The application can show the user an order state and a separate receipt state. There is no broker to operate merely because work happens later.
That job still needs engineering. Give it a stable ID, an attempt count, a timeout, and a clear terminal failure state. Decide how a worker claims a job and how another worker takes over after a crash. PostgreSQL documents SKIP LOCKED as useful for avoiding contention among consumers of a queue-like table, while warning that it gives an inconsistent view of rows. That is a tool for workers, not a way to make arbitrary reporting queries faster.
A database queue has limits. Heavy fan-out, many unrelated consumers, a large backlog, or retention far beyond the operational life of a job can make the table and its cleanup awkward. Those are measurable reasons to consider messaging infrastructure. “We want async” is not yet one.
Publish facts when other owners need them
Now add a finance service, a customer notification service, and a fulfillment system. Each has its own release schedule and its own idea of what to do when an order is paid. The order service should not need to call all three during the payment request and wait for their health before it can record payment.
OrderPaid can be a useful event here. It says something that has happened, with an order ID, event ID, and version of the contract. The consumers decide what that fact means for their own systems. This is a better reason for an event than the mere existence of multiple functions in one application.
The awkward part is the boundary between the order database and the broker. This code is unsafe:
The process can die between those lines. Reversing the lines creates the opposite problem: a consumer may act on an event for a payment that never committed. AWS describes the transactional outbox pattern for this dual-write problem. Store the business update and an outbox record in the same transaction; a separate publisher sends committed records to the broker.
The outbox improves the handoff. It does not make the rest of the system atomic. The publisher can send an event and crash before marking it delivered. The consumer must cope with that event arriving again. A stable event ID and a consumer-side record of processed IDs are more useful than a diagram that promises “exactly once” without saying where the guarantee ends.
For example, a notification consumer can store OrderPaid:event-123 before or with the state change that schedules an email. If it sees the same event again, it can skip the duplicate scheduling step. A payment consumer has a higher bar: it must establish whether an external charge already succeeded before retrying. Its idempotency boundary is the payment provider and the local record together, not only the broker.
Decide what the user must know now
Asynchrony changes the product contract. Suppose checkout returns “payment complete” immediately, but fulfillment might reject the order ten minutes later because stock ran out. That may be a valid business process, but the UI and support team need to know what “complete” meant.
For each step, I would write down three states:
| Question | Example answer |
|---|---|
| What is committed before the response? | Payment authorization and order record. |
| What can finish later? | Receipt generation and loyalty points. |
| What happens if later work fails? | Retry, visible pending state, then support action. |
If the next operation must succeed before the user can proceed, a synchronous API call may be easier to reason about. An event can still be emitted afterward for observers. If the user can wait, an accepted response plus a status endpoint can make the delay explicit. AWS's asynchronous communication guidance describes that claim-check approach: acknowledge the request, return an identifier, and let the client retrieve the later result.
The delay has a user-facing cost: the text on a screen, the retry a user may press, and the question a support engineer has to answer.
Delivery guarantees do not finish the design
The team needs to decide what duplicates or ordering would do to the business operation before selecting a broker. A standard Amazon SQS queue, for example, delivers messages at least once. The same message may be received again. Its standard queue type also gives best-effort ordering.
Those properties are fine for many tasks. A consumer updating a current projection can ignore an event it has already processed. A consumer applying account balance deltas cannot casually apply the same delta twice. It may also need per-account ordering. Picking a FIFO queue can help with ordering and broker deduplication, but it does not remove the need to reason about retries across external systems and the full business workflow.
I would ask the following before choosing the transport:
- Is this a command for one worker, or a fact several owners may consume?
- What state is committed when the message is created?
- Can the same message be processed twice without damage?
- Must messages be ordered globally, per entity, or not at all?
- How long must a consumer be able to recover after an outage?
- Who notices a stuck consumer, and how is the backlog cleared?
The answers narrow the choice considerably. A one-owner receipt job does not need the same infrastructure as a company-wide change stream. A durable business event deserves more thought than a callback disguised as a topic.
Operate the failure path before scaling the happy path
Events make producers and consumers independent in useful ways. They also make a transaction harder to follow across time. An API request may be complete while the business process is still pending in another service. A dashboard that shows only broker throughput cannot tell you whether orders are actually progressing.
I would give each event a correlation ID, a producer timestamp, a contract version, and an owning service. Then I would measure the delay from publication to the consumer's business outcome, not merely the time until a message is read. A dead-letter queue is a holding area for work that failed repeatedly. It is not a recovery plan until someone owns inspection, correction, and replay.
Changes to the event contract need similar care. If consumers deploy independently, a producer cannot assume everyone upgrades on the same day. Add fields in a compatible way, document their meaning, and test old consumers against new payloads. When the meaning of a fact changes, a new event version may be clearer than silently reusing an old name.
This is the cost side of the architecture decision. If a system has one application and one database, a job table may give the team all the separation it needs. If several autonomous systems depend on the same business fact and must recover independently, the broker, outbox, consumer idempotency, and operational tooling earn their place.
My rule is to write down the failure and the recovery path first. Once those are concrete, the decision between a worker, an API call, and an event is usually much less mysterious.
Top comments (0)