The Two-Line Bug That’s Probably Hiding in Your Event-Driven System

Vamsi T — Thu, 30 Jul 2026 16:43:44 +0000

A while back I was staring at a distributed sync issue that made no sense. The database had the right record. The API had returned 200. The background job logs showed “completed.” And yet, somewhere downstream, the data was just... wrong.

Turns out the bug wasn’t a bug at all. It was this:

await database.SaveChangesAsync(); await messageBus.Publish(accountUpdatedEvent);

Two lines. Looks harmless. It’s the default way almost everyone writes event-driven code, and it’s quietly one of the riskiest patterns in distributed systems.

The gap between “saved” and “told everyone”

Updating your database and publishing an event are two separate operations, on two separate systems, with no shared transaction. That gap between them — sometimes milliseconds, sometimes a full crash window — is where things go wrong:

The database commit succeeds, but the broker call fails—or the process dies before publishing. Retrying the entire request may duplicate business operations, while a process crash may leave no durable record that an event is still waiting to be published. 0

Or the reverse: the event fires before the commit, the transaction then rolls back, and now some downstream consumer has already acted on data that technically doesn’t exist — a “ghost event.”

Neither of these shows up as an obvious error. The request succeeded. The logs are clean. It’s the business process that’s broken, weeks later, in a report nobody trusts anymore.

And that’s before you add the usual distributed-systems tax on top: retries producing duplicate events, events arriving out of order and clobbering newer state with older state, or — my personal favorite — two regions endlessly re-publishing the same change back and forth at each other because nobody tagged the event with where it came from.

None of this is exotic. It’s what happens by default once you have more than one service and a message broker.

The fix is almost boring: write the event to the same place as the data

The transactional outbox pattern solves the core problem with one idea: don’t publish the event directly. Write it to an outbox table, in the same database transaction as the business update.

Begin transaction Update Account Insert Outbox record Commit

Now the two things you actually care about — “did the data change” and “will everyone find out about it” — either both happen or neither does. A separate background publisher reads pending outbox rows and pushes them to the broker, retrying independently of the original request. If the broker is down for ten minutes, that’s a publisher problem, not a data-integrity problem.

It also gives you something you didn’t have before: a queryable audit of exactly what got published, when, and whether it’s still pending. In a regulated environment that alone is worth the extra table.

What it doesn’t fix

This is the part people skip, and it’s the part that matters:

Consumers still need to be idempotent. With a reliable publisher and retry policy, the outbox supports at-least-once publication, the publisher can crash after sending but before marking the row done, and you’ll get a duplicate. Your consumers need to dedupe by event ID, ideally in the same local transaction as the change they’re applying.

Ordering isn’t automatic. You still need aggregate version numbers or sequence checks if a consumer might see v8 before v7.

Poison events still need a dead-letter path, or one bad payload will sit retrying forever.

The outbox closes the gap between committing data and creating the event. It doesn’t remove the rest of distributed-systems reality — it just narrows the failure surface to a place you can actually monitor.

The metric that actually matters

Most teams monitor “is the broker up” and “is the consumer running.” Those are necessary but they miss the point. The alert that actually tells you something is:

The oldest unpublished event has been pending for more than 30 minutes.

That’s a business-risk signal, not an infrastructure signal. It tells you synchronization is falling behind before someone downstream notices their data is stale.

Do you even need this?

Not every event deserves an outbox. Cache invalidation, UI activity pings, best-effort analytics — if losing one occasionally is a shrug, direct publishing is fine.

But if the event represents state someone will act on — order status, payment confirmation, audit trail, a sample or record moving to the next stage of a regulated workflow — the cost of inconsistency is almost always higher than the cost of one extra table and a background job.

The line I keep coming back to when reviewing event-driven designs: publishing an event isn’t the end of a transaction. It’s the start of a business process that now lives outside your database’s guarantees. Worth designing for on purpose, not by accident.

How are you handling the dual-write problem today—transactional outbox, change data capture, retries, or reconciliation? I would be interested to hear what has worked in real production systems.

DEV Community: Vamsi T

The Two-Line Bug That’s Probably Hiding in Your Event-Driven System