Local Transactions Aren't Enough: The Dual-Write Problem in Microservices

Our system shared a lot of state with an external BPMN process engine. The engine decided which flow a user should go through, and to make that decision it needed the user and the user type to exist on both sides: in our database and in the engine.

That "on both sides" turned out to be harder than it sounds. We'd create a user in our own database, then push it to the engine, and the two would quietly drift apart. Sometimes the engine already had the user and rejected our save with an "already exists" error. Sometimes our write succeeded and the push didn't, so the engine never heard about a user we thought we'd shared. Two systems, no shared transaction, and no reliable answer to a simple question: does the other side actually know about this user?

The dual-write problem

What we hit has a name: the dual-write problem. It shows up any time a single operation has to write to two systems that don't share a transaction. In our case that was our PostgreSQL database and the process engine, but it's the same story with a database and Kafka, a database and a search index, or any two systems you update together.

Here's why it's unavoidable. A database transaction can only protect writes inside that database. When your method does two things:

@Transactional
public void createUser(User user) {
    userRepository.save(user);        // writes to our database
    engineClient.registerUser(user);  // writes to the engine
}

the @Transactional annotation only covers the first line. The engine call is a network request to a completely separate system, and it has no idea our database transaction exists. So the two writes can't succeed or fail together. One of them can land while the other doesn't.

And once you accept that, you realise there's no ordering that saves you. Whichever line you put first, there's a moment between the two where a crash, a timeout, or a network blip leaves the two systems disagreeing. The gap is small, but in production small gaps get hit thousands of times.

The fixes that don't work

Before reaching for a pattern, we tried the obvious things. Each one looks reasonable and each one leaves a hole.

Database first, then the engine. Save the user locally, then push to the engine. If the engine call fails, we've got a user in our database the engine never heard about. The two are already out of sync, and now we need cleanup logic to find and retry those orphaned users.

Engine first, then the database. Flip the order. Now if our local save fails after the engine already accepted the user, the engine knows about a user our own system has no record of. Same problem, opposite direction.

Wrap both and roll back manually. "Fine, if the second write fails, I'll undo the first." This is where people write a try/catch that deletes the user when the engine call throws. But the compensating delete can fail too, or the service can crash between the failed engine call and the cleanup. You're now writing error-handling for your error-handling, and there's still a gap.

The pattern underneath all three is the same: as long as the two writes are independent, there is always a moment where one has happened and the other hasn't. You can shrink that window, but you can't close it. What we actually need is a way to make "save the user" and "remember to tell the engine" part of the same transaction, so they can never come apart.

The fix: the transactional outbox

The idea is small once it clicks. Instead of calling the engine inside your method, you write a row into an outbox table in your own database, in the same transaction as the user. Two writes, one database, one transaction. They commit together or not at all.

CREATE TABLE outbox (
    id           BIGSERIAL PRIMARY KEY,
    event_type   VARCHAR(100) NOT NULL,   -- e.g. "USER_REGISTERED"
    payload      JSONB        NOT NULL,   -- the user data to send
    status       VARCHAR(20)  NOT NULL DEFAULT 'PENDING',
    created_at   TIMESTAMP    NOT NULL DEFAULT now()
);

Now the method changes. There's no engine call here anymore, just two local writes that live or die together:

@Transactional
public void createUser(User user) {
    userRepository.save(user);
    outboxRepository.save(
        new OutboxEntry("USER_REGISTERED", toJson(user))
    );
}

That's the whole trick. If this transaction commits, both the user and the intent to notify the engine are safely stored. If anything throws, both roll back, and there's no half-done state to clean up. The engine hasn't been called yet, but we've made a durable promise to call it, and that promise is sitting in the same database as the data it's about.

The network call still has to happen, of course. It just doesn't happen here. We've turned an unreliable "write to two systems at once" into a reliable "write to one system," and moved the risky part somewhere we can retry it safely.

Getting it to the engine: the relay

The outbox table is now a to-do list of things that need to reach the engine. Something has to read that list and actually make the calls. That something is the relay.

The simplest version is polling. A scheduled job wakes up every few seconds, grabs the pending rows, sends each one to the engine, and marks it done:

@Scheduled(fixedDelay = 5000)
public void publishOutbox() {
    for (OutboxEntry entry : outboxRepository.findByStatus("PENDING")) {
        engineClient.register(entry.getPayload());
        entry.setStatus("SENT");
        outboxRepository.save(entry);
    }
}

This is what I built first, and it worked. It isn't the fanciest option, but it's easy to reason about and easy to debug, which counts for a lot in production. A more advanced setup would drive the relay from the database changelog instead of polling (change data capture, with a tool like Debezium), and that's the direction I'd take it next. But polling every few seconds was more than enough to keep the two sides in sync, and I'd rather ship the simple version that works than the clever one that might.

There's one catch, and it's exactly where our original problem came from. The relay is at-least-once: if the service restarts after the engine call but before the row is marked SENT, that row gets sent again. So the engine gets the same user twice and throws the "already exists" error.

In a perfect world you'd fix this on the engine side, make it treat a repeat registration as a no-op. But I couldn't touch the engine; it's an external system I don't control. So I handled it on my side instead: the relay treats "already exists" as a success. If the engine says the user is already there, that's fine, that's the state I wanted anyway. I mark the row SENT and move on.

It's not the textbook answer, but it's the realistic one when you own only one side of the integration.

When you actually need this

The outbox pattern isn't free. You're adding a table, a relay, and a bit of operational surface to keep an eye on. So it's worth being honest about when it earns that cost.

You need it when two systems must agree and you can't lose the update between them: a user that has to exist on both sides, an order that must reach billing, a record that has to land in another service no matter what. Anywhere a lost write leaves two systems quietly disagreeing, the outbox is the pattern that closes the gap.

You can skip it when the second write doesn't have to be reliable. If it's fine to miss the occasional update, or you can recompute the other side later, a direct call with some retries is simpler and good enough. Not every integration deserves an outbox, and reaching for it everywhere just moves the complexity around.

For us it was the right call because the whole system depended on both sides knowing about the same users. Once the outbox was in place, the "already exists" errors and the silent drift just stopped. The two sides stayed in sync, and I stopped getting paged about users that existed in one place but not the other.