Event-Driven Pipelines Fail in Five Predictable Ways — and How to Design Against Them

Event-driven architecture is having another moment — the topic sits at #7 on daily.dev Popular this morning — and for good reason: it is how most serious automation actually moves. A scraper finishes, an event fires, a scorer picks it up, a writer stores the result, a human reviews the edge cases. Nobody polls anybody. The system reacts. That is also exactly why event-driven systems fail in ways that are hard to see coming: the failure is not a crash, it is a message delivered twice, a message arriving late, a message nobody can replay, and the damage only surfaces hours later in data nobody trusts anymore.

We build pipelines like this for a living — lead ingestion, enrichment, scoring, and unattended workflows that move data between systems — and the five failure modes below are the ones we design against first, before any feature work. They are not exotic. They are the default behaviour of queues, brokers, and at-least-once delivery, and every one of them has bitten a team that assumed "the queue handles it."

Our how we work rule applies here in full: automate aggressively, but keep the blast radius small. In event-driven terms, that means every consumer must be safe to run twice, every message must carry enough context to be understood alone, and a human must be able to stop, inspect, and replay the flow. Our dark software factories piece made the same argument for lights-out systems; this piece is the event-driven version of it.

1. Duplicates are the default, not an accident

Most messaging systems guarantee at-least-once delivery. That is a polite way of saying: your consumer will see the same message twice, and it is your problem, not the broker's. Retries after a timeout, a consumer crash after processing but before acknowledging, a broker redelivery — all of them produce a second copy of a message your system already acted on. If acting twice means charging twice, emailing twice, or scoring a lead twice and double-counting it, you have a correctness bug, not a performance problem.

The fix is idempotency, applied at the point where the side effect happens — not in a comment, not in a hope. Every message carries a stable event ID, and the consumer records that ID in the same transaction as the work it does. The second delivery finds the ID already recorded and skips the work.

-- The processed-events table is the whole trick.
-- Record the event ID in the SAME transaction as the side effect.
CREATE TABLE processed_events (
  event_id TEXT PRIMARY KEY,
  consumer TEXT NOT NULL,
  processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

def handle_lead_scored(event):
    with db.transaction() as tx:
        inserted = tx.execute(
            "INSERT INTO processed_events (event_id, consumer) "
            "VALUES (%s, 'lead-scorer') ON CONFLICT DO NOTHING RETURNING event_id",
            (event.id,),
        )
        if not inserted:
            return  # duplicate delivery: work already done, skip silently
        tx.execute(
            "UPDATE leads SET score = %s WHERE id = %s",
            (event.score, event.lead_id),
        )

Two details matter. First, the dedupe record and the side effect must commit together; recording the ID in a separate step reopens the exact crash window you were closing. Second, the event ID must be stable across redelivery — a UUID generated by the consumer on receipt is useless, because the redelivered copy gets a different one. The producer assigns the ID, and it travels with the event forever.

The same discipline applies to outbound calls: when a consumer calls a payment API, an email API, or another service, it should pass an idempotency key derived from the event ID, so a retried call is recognised by the downstream service too. Idempotency is not one layer; it is a property you push through every boundary a message crosses.

2. Order is not guaranteed, so stop designing as if it were

A queue that delivers messages in order under normal conditions will not deliver them in order under failure conditions. A retried message arrives late. A consumer that processes messages in parallel finishes them out of order. A partitioned topic delivers one partition's backlog after another's live stream. If your consumer assumes "update A always arrives before update B," it will eventually apply B, then A, and end up with stale state presented as current.

The practical response is to make consumers order-tolerant rather than order-dependent:

• Carry a version with every event. A monotonically increasing version per aggregate (or a timestamp from the producer's clock, with its known skew limits) lets the consumer reject stale writes: apply only if event.version > current.version.
• Model state as a state machine, not a sequence of overwrites. A lead that has moved to "contacted" should not move back to "new" because a delayed event arrived. Transitions that are illegal from the current state get logged and dropped, not applied blindly.
• Keep the event, not just the result. When the consumer stores the event log alongside the current state, a later investigation can reconstruct what actually happened, in what order, instead of guessing from the final row.

There is a legitimate exception: some flows genuinely need ordering, such as a sequence of edits to the same document. For those, use a single partition key per aggregate so all of its events land in one ordered stream, and accept the throughput ceiling that comes with it. Ordering is a feature you pay for with parallelism. Do not pay for it everywhere, and do not assume it anywhere you did not pay.

3. One poison message can stall everything behind it

Every pipeline eventually receives a message it cannot process: a payload that fails validation, a record that triggers an unhandled edge case, a downstream service that rejects one specific input. What happens next decides whether you have an incident or a footnote.

The naive design retries forever, immediately. The message fails, goes back to the front of the queue, fails again, and the consumer never reaches the healthy messages behind it. Worse, if the failure is a downstream timeout, the retry loop becomes a self-inflicted denial of service against the very service that is struggling.

The standard shape, and the one we use, has four parts:

1. Bounded retries with exponential backoff and jitter. A handful of attempts, each waiting longer than the last, with random jitter so a fleet of consumers does not retry in lockstep. Backoff without jitter just moves the thundering herd to a later timestamp.
2. A dead-letter queue (DLQ). After the final attempt, the message moves to a separate queue instead of blocking the main one. The main flow keeps moving; the bad message waits somewhere it can be inspected.
3. An alert on DLQ depth, not on individual failures. One poison message is routine. Ten in an hour is a signal — a broken producer deploy, a schema change, a downstream outage. Alert on the rate, and page on the trend.
4. A replay path out of the DLQ. A dead letter that can only be deleted is a data-loss decision made under pressure. The DLQ needs a tool that requeues a message after a fix, with the original event ID intact so idempotency still protects the replay.

import random, time

def retry_with_backoff(attempt, base_seconds=1.0, cap_seconds=300.0):
    # Exponential backoff + full jitter: spreads retries instead of
    # synchronising a whole fleet onto the same second.
    delay = min(cap_seconds, base_seconds * (2 ** attempt))
    time.sleep(random.uniform(0, delay))

The design question to ask before shipping any consumer is not "does it handle success?" but "what happens to the tenth consecutive failure, and who finds out?" If the answer is "it retries forever and nobody is paged," the consumer is not finished.

4. Schemas change, and every consumer breaks at once

An event schema is a contract with every consumer, and contracts get renegotiated constantly: a new field appears, a field changes type, a field disappears because a producer team decided it was redundant. The failure mode is rarely dramatic. It is a consumer that silently drops a field it no longer recognises, or crashes on a payload shape it never expected, or — worst — interprets a changed field under its old meaning.

The defences are boring and they work:

• Additive changes only, by default. New fields are optional. Old fields are never renamed or retyped in place; they are deprecated, then removed after every consumer has migrated. A breaking change is a new event version, not an edit.
• Version the event envelope, not just the payload. Every event carries event_type and event_version at the top level, so a consumer can route v1 and v2 to different handlers instead of guessing from the payload shape.
• Tolerant readers. Consumers read the fields they need and ignore the rest. A consumer that fails on an unknown field has coupled itself to every future producer change.
• Contract tests between producer and consumer. Before a producer ships a schema change, a test replays a sample of real events through every consumer's parser. This catches the rename that "nobody uses" — which is always used by somebody.

{
  "event_id": "evt_01J9Q2X...",
  "event_type": "lead.scored",
  "event_version": 2,
  "occurred_at": "2026-10-26T01:30:00Z",
  "producer": "lead-scorer",
  "data": {
    "lead_id": "lead_88412",
    "score": 73,
    "score_version": "v3"
  }
}

Note what the envelope carries that the payload does not: identity, type, version, time, and origin. A message without those five things cannot be debugged, replayed, or routed by anyone except the team that wrote it. That is not a message; it is a rumour.

5. If you cannot replay it, you cannot recover from it

The previous four failure modes all end in the same place: something went wrong, and now you need to reprocess a slice of history. A consumer deployed with a bug scored a thousand leads wrong. A downstream outage lost a day of writes. A schema migration misclassified a batch. In every case, the question is the same: can you take the original events and run them through the fixed consumer again?

Most systems cannot, because they treated the queue as a transport and threw the events away after delivery. The queue is not storage. Retention expires, DLQs get purged, and the original input is gone. Recovery then means reconstructing data from side effects — the current database rows, the sent emails, the logs — which is guesswork dressed as engineering.

The fix is to treat the event log as the system of record, or at least as a durable archive:

• Persist events before, or alongside, processing. An append-only event store — even a plain table with the envelope and payload — turns "reprocess last Tuesday" from impossible into a query.
• Replay is a first-class operation, not a script written during an incident. The replay tool takes a time range or a set of event IDs, feeds them through the current consumer, and leans on idempotency so replaying an already-processed event is a no-op. Replaying without idempotency just doubles the damage.
• Correlation IDs tie the story together. One ID, generated at the entry point, travels through every event a request produces. When a lead's score looks wrong, one query shows every event in its chain instead of a grep across five services' logs.

This is the same argument our LeadOps build notes made from the other side: a pipeline you cannot inspect is a pipeline you cannot trust. Observability for event-driven systems is not dashboards of queue depth. It is the ability to answer, for any single record, "what happened to it, in what order, and can I run it again?"

The pre-flight checklist

Before a new consumer ships, it should pass five checks — one per failure mode above:

1. Duplicates: can this consumer run twice on the same event with no visible difference? (Idempotency key + same-transaction dedupe.)
2. Order: does it reject stale writes and illegal transitions instead of applying them?
3. Poison: after N failures, does the message leave the main queue, page someone on trend, and stay replayable?
4. Schema: can the producer add a field tomorrow without deploying this consumer?
5. Replay: can you reprocess last week's events through today's consumer and get the same result?

A consumer that fails any one of these is not "mostly done." It is a future incident with a deployment date. Event-driven architecture rewards the teams that treat these five checks as the definition of done — and it punishes everyone else on exactly the schedule nobody planned for.

FAQ

Is event-driven architecture always the right choice?
No. If a flow is a simple request-response with one producer and one consumer and no need for audit history, a direct call is simpler, easier to debug, and easier to reason about. Events earn their complexity when there are multiple consumers, when the history itself has value, or when producers and consumers must evolve independently.

Does exactly-once delivery solve the duplicate problem?
No product offers true end-to-end exactly-once processing once side effects leave the system — a consumer can always crash after acting but before acknowledging. Exactly-once within a single broker is a transport guarantee, not a business guarantee. Idempotent consumers are still required.

How many retries are enough?
There is no universal number. Transient failures (timeouts, brief downstream outages) usually clear within a handful of attempts over minutes. Permanent failures (bad payloads, validation errors) never clear, no matter how many retries. The design goal is not a magic count; it is bounded retries plus a DLQ plus an alert, so both kinds of failure end up somewhere visible.

Should every event be stored forever?
Not necessarily forever, but for longer than your longest recovery window. If your incident response ever needs "last month's events," retention must cover last month. Storage is cheap; reconstruction from side effects is not.

References

• Martin Kleppmann, Designing Data-Intensive Applications — chapters on message brokers, log-based messaging, and exactly-once semantics (the standard treatment of at-least-once delivery and ordering).
• Chris Richardson, Transactional Outbox pattern: https://microservices.io/patterns/data/transactional-outbox.html
• Apache Kafka documentation, delivery semantics: https://kafka.apache.org/documentation.html
• Stripe API documentation, idempotency keys: https://docs.stripe.com/api/idempotency
• Botslave, "Dark Software Factories: What Runs Lights-Out and What Breaks Without a Human in the Loop": https://botslave.com/blog/dark-software-factories-lights-out
• Botslave, "LeadOps Autopilot: Sourcing and Scoring US Clinic Leads from Public Data": https://botslave.com/blog/leadops-autopilot-clinic-lead-scoring
• Botslave, "How we work": https://botslave.com/how-we-work

Originally published on botslave.com. Botslave is an automation studio — see what we build and how we work.