Splitting one database across two services deletes the free ACID transaction you never thanked it for. A saga is the textbook fix - a chain of local writes with compensations. But most of the time you don't need a saga at all. You need idempotency, a correct ordering, and one rule you break at your peril: never call a service from inside a transaction.
👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go. This is Part 6 of a series about carefully breaking a large PHP monolith into Go microservices while it keeps serving a real business. The seam I keep coming back to is a rule-set write that now spans two databases; the earlier parts walked through the silent 200 OK, who owns identity, the cascade that timed out, and flipping the master without a flag day. Part 5 - The hard part is the wire - was about inter-service communication and partial failure. This part is about what partial failure does to a write. The running notes live on my GitHub: github.com/brilliant-almazov. No hype, just the real work.
Here's the provocation, stated plainly because it's the whole point: most of the time "we need a saga" is really "we need idempotency." Reaching for a saga framework when a re-send would do is one of the most common over-engineering traps in a migration. I'll show you the two real mechanisms in my system - one mature, one being born - and exactly why neither is a saga, even though one of them is saga-shaped.
Where this sits in the migration
I keep re-drawing this map because every part of the series is one step along it, and the distributed-transaction problem is a step you arrive at, not one you start with. At Stage 0 there is no such problem - one database hands you ACID for free. It's born at Stage 1, the moment identity moves to a second database and a single logical write starts spanning two of them. This part lives at Stage 2, where a second engine runs in parallel and every write is a cross-service write:
Stage 0 Monolith only — one DB, one ACID transaction. No distributed write.
Stage 1 Identity extracted — a second DB appears; the write now spans two. <-- the split
Stage 2 Write logic extracted — two engines, every write cross-service. <-- THIS ARTICLE
Stage 3 Prove the two engines agree.
Stage 4 Flip, then delete — the data consolidates again.
The whole article is about that Stage-2 line: one logical write, two databases, a network in between, and no BEGIN that spans both.
The primer: what ACID gave you, and what splitting the DB takes away
Inside one database, a transaction is a gift you stop noticing. BEGIN, write three tables, COMMIT - and either all of it is durable or none of it is. Atomicity, consistency, isolation, durability, for free. You lean on it without a thought.
Now put the write across two services, each owning its own database. The monolith persists the rule data in its store; the rule-set store registers that rule set's identity - a content hash - in its store. One logical operation, two databases, two processes, a network in between. The shared transaction is simply gone. There is no BEGIN that spans both. If the second write fails after the first commits, nothing rolls the first one back for you. That's the concrete stake, not an abstraction.
Three tools get reached for here, and they're worth separating in one paragraph. Two-phase commit (2PC) tries to keep the illusion of one transaction across both stores with a coordinator and a prepare/commit handshake - it's slow, it blocks, and it fails badly when the coordinator dies mid-flight; I use it nowhere. A saga drops the illusion: it's a sequence of local transactions, each committing on its own, with a compensating action for each step so that if step three fails you run "undo step two, undo step one" forward. A transactional outbox is humbler still: it makes only the intent to call the other service part of your local transaction, then delivers that intent after commit, retrying until it sticks. Same family, wildly different weight. The engineering is in picking the lightest one that's actually correct.
weight what it does do I use it?
2PC heavy one txn across both stores; blocks; dies badly → nowhere
SAGA medium local txns + one compensation per step → only if a step is irreversible
OUTBOX light record intent in the local txn, deliver after → yes, the mature mechanism
The default should be the lightest row you can make correct - and, as the rest of this article argues, an idempotent far side pushes almost everything down to the OUTBOX line.
Mechanism 1: the transactional outbox (the mature one)
The monolith's side of this write is the grown-up version, and it's an outbox - at-least-once delivery, no compensation, no 2PC.
The trick is smaller than it sounds. There's an outbox table. An ORM lifecycle listener enqueues an outbox row inside the same unit-of-work as the domain write, so the rule-set row and the outbox row hit the same COMMIT. The only thing that's transactional is "did I durably record the intent to publish this?" - and that's a purely local fact, so ACID still covers it.
┌─ ONE local COMMIT ───────────────────┐
│ write rule_set row (the data) │ same unit-of-work,
│ write outbox row (the intent) │ one atomic COMMIT
└───────────────────┬───────────────────┘
│ committed
▼
drain loop polls Pending rows (100 / 5s)
▼
gRPC register(payload) ──► content-addressed store (idempotent upsert-by-hash)
│
success → markSent failure → recordError → stays Pending → retried
The call to the other service happens only on the far side of that COMMIT, in the drain loop - never inside the transaction that wrote the data.
declare(strict_types=1);
// The mapping LAYER: turn a domain entity into a transport message. One instance
// method behind an interface — never a static ::forRuleSet() on the message itself.
interface RuleSetMessageMapperInterface
{
public function map(RuleSet $ruleSet): OutboxMessage;
}
// A Doctrine onFlush listener. It writes the outbox row in the SAME
// unit-of-work as the domain entity, so intent and data share one COMMIT.
final readonly class RuleSetOutboxListener
{
public function __construct(
private RuleSetMessageMapperInterface $mapper,
private OutboxFactoryInterface $outbox,
) {}
public function onFlush(OnFlushEventArgs $args): void
{
$em = $args->getObjectManager();
$uow = $em->getUnitOfWork();
foreach ($uow->getScheduledEntityInsertions() as $entity) {
if (!$entity instanceof RuleSet) {
continue;
}
// entity -> message (the mapping layer) -> outbox row.
$message = $this->mapper->map($entity);
$row = $this->outbox->enqueue($message);
// Persist in THIS flush → shares the domain write's COMMIT.
$em->persist($row);
$uow->computeChangeSet(
$em->getClassMetadata(OutboxRow::class),
$row,
);
}
}
}
A separate drain loop then does the actual delivery: poll Pending rows in batches (100 every 5s in my case), re-envelope each onto the message bus, and let the handler make the real gRPC call. A failure is recorded back onto the row, never swallowed - and retried on the next tick.
declare(strict_types=1);
// The handler makes the real call. Failure is recorded, not lost; the row
// stays Pending and is retried. At-least-once delivery is the guarantee.
final readonly class OutboxDrainHandler
{
public function __construct(
private RuleSetStoreClientInterface $store,
private OutboxRepositoryInterface $rows,
) {}
public function handle(OutboxRow $row): void
{
try {
// Idempotent upsert-by-hash on the far side: a duplicate send
// dedups to the same content-addressed id. So re-sending is safe.
$this->store->register($row->payload());
$this->rows->markSent($row);
} catch (StoreUnavailable $e) {
$this->rows->recordError($row, $e->getMessage()); // retried later
}
}
}
Here's why this is enough, and why no saga is hiding underneath it. The far side is the content-addressed store: register is an idempotent upsert-by-hash. Send the same rule set twice and the second call dedups to the very same id - a no-op. So "at-least-once" - the outbox's honest guarantee - is not a compromise, it's exactly right, because a duplicate has no effect worth undoing. There is no compensation because there is nothing to compensate. No 2PC because you never needed two databases to agree in one breath - you needed one of them to remember, and the other to be safe to retry.
Mechanism 2: the saga-shaped write that isn't a saga
The new service - the rule-set-markup service, now live in parallel with the monolith and writing straight into its database - has the write that looks like it wants a saga. It doesn't have an outbox. It does a sequential dual-store write, store-first, with no compensation. And it's correct.
The order is the whole design:
- Call the identity store synchronously, outside any local transaction, to resolve id/hash for every item, keyed by a
correlation_id. - Guard that every computed item got a correlation match - if one is missing, error out. Never a silent skip. (Part 1 of this series is a 2,000-word monument to what a silent skip costs.)
- Then open exactly one local transaction that writes the scope links plus an audit row, and commit.
- After commit, fire a notify/refill so the changed configs get recomputed.
1 resolve identity ── OUTSIDE any tx ──► content-addressed store (retry dedups)
│
2 guard: every computed item matched? ── no ──► error out (never a silent skip)
│ yes
3 BEGIN ─ write scope links + audit row ─ COMMIT (exactly one local tx)
│ committed
4 notify / refill ── AFTER commit, never before ──► recompute changed configs
The store call sits before the transaction and the notify sits after it - so at no point is a network call trapped inside an open BEGIN. That placement is the entire safety argument, and the next section is why.
// Persister owns the dual-store write. The executor is a struct FIELD, never an
// argument. RunPersist reads as four ordered steps — it never sees BEGIN/COMMIT.
type Persister struct {
store IdentityStore // exactly one method: Resolve
exec Executor // the pool by default; a tx after withTx
notify Notifier
}
func (p *Persister) RunPersist(ctx context.Context, req Request) error {
// 1. Resolve identity OUTSIDE any transaction — content-addressed, so a
// retry dedups to the same id: safe to repeat, safe to orphan.
ids, err := p.store.Resolve(ctx, req.CorrelationID, req.Items)
if err != nil {
return fmt.Errorf("resolve identity: %w", err)
}
// 2. Guard: every computed item MUST have a match. No silent skip.
for _, item := range req.Items {
if _, ok := ids[item.CorrelationID]; !ok {
return fmt.Errorf("no identity for %s", item.CorrelationID)
}
}
// 3. ONE local tx — its lifecycle lives inside RunInTx, not here. This method
// never holds a pgx.Tx; it just hands work to a tx-scoped Persister.
if err := p.RunInTx(ctx, func(ctx context.Context, tx *Persister) error {
return tx.writeScope(ctx, req, ids)
}); err != nil {
return err
}
// 4. Notify AFTER commit — never before.
return p.notify.Refill(ctx, req.CorrelationID)
}
// RunInTx encapsulates BEGIN/COMMIT/ROLLBACK. It opens a tx, runs fn against a
// tx-scoped Persister, and commits — or rolls back on any error. No pgx.Tx ever
// leaves this method, and no caller ever passes one.
func (p *Persister) RunInTx(ctx context.Context, fn func(context.Context, *Persister) error) error {
tx, err := p.exec.Begin(ctx)
if err != nil {
return fmt.Errorf("begin: %w", err)
}
if err := fn(ctx, p.withTx(tx)); err != nil {
if rbErr := tx.Rollback(ctx); rbErr != nil && !errors.Is(rbErr, pgx.ErrTxClosed) {
return errors.Join(err, rbErr)
}
return err
}
return tx.Commit(ctx)
}
// withTx returns a scoped Persister whose executor IS the tx — carried as a
// field, never passed as an argument anywhere.
func (p *Persister) withTx(tx pgx.Tx) *Persister {
scoped := *p
scoped.exec = tx
return &scoped
}
Consistency here is best-effort-sequential, leaning entirely on the store's content-addressed idempotency. Walk the failure cases and you'll see why that's fine:
- Local tx fails after the store insert. The store row is now orphaned - but it's immutable and content-addressed, so a retry of the whole operation dedups to that exact same row. Orphaned-but-immutable is harmless; a garbage-collector can sweep unreferenced hashes later. Nothing to compensate.
- The post-commit notify partially fails. The data is persisted but not yet recomputed. That's a staleness bug, not a corruption bug - a re-run of the refill fixes it, and it's idempotent too. Still nothing to compensate.
This is the live example of the thesis. We didn't need a saga. We needed idempotency on the far side, a correct ordering (store first, tx second, notify last), and a correlation key to tie computed items to persisted identities. A saga would have added a compensation framework to undo things that are, by construction, safe to leave alone.
The invariant: never call a service from inside a transaction
Every design decision above bends around one hard-won rule, and it's the spine of this article:
Never call a microservice from inside a database transaction. Only after a successful
COMMIT.
I learned it the way these things get learned. A dual-write was once placed, by mistake, inside a BEGIN...COMMIT. It read fine in review. But think about what a rollback means there: the local transaction aborts and unwinds - while the gRPC call to the downstream service has already happened and cannot be un-happened. The other service is now acting on a rule set that your database just erased. That's a phantom: a fact you shipped to the world about data that no longer exists on your side. No status code reveals it. The two systems simply, quietly, disagree - the exact failure class this whole series keeps circling.
INSIDE a transaction:
BEGIN
gRPC call to downstream ─────► already happened, cannot be un-happened
ROLLBACK ◄───── your DB erases the row
──────────────────────────────────────────────────────────────────────
result: the downstream service now acts on data that no longer exists
on your side = a PHANTOM. No status code ever reveals it.
There's a nastier variant that makes "I'm safely past the transaction" a lie. A nested "transactional" wrapper in most ORMs is not a real transaction - it's a savepoint. So code that thinks it committed may only have released a savepoint while an outer transaction is still open, still capable of rolling everything back including the state you called the service about. "Past the transaction" has to mean the outermost COMMIT returned, not "my inner transactional() block ended."
This one rule is why both mechanisms are shaped the way they are. The outbox exists precisely so intent is recorded transactionally and the call is made after commit, by a separate drainer. The new service calls the identity store before its transaction (so the call isn't inside it) and notifies after commit (same reason). Two different mechanisms, one invariant, obeyed from opposite directions.
When a saga is actually right - and when it's over-engineering
So when do you reach for a real saga? The test is simple and it's about the far side, not your side.
If the downstream step is immutable and content-addressed - idempotent - you don't need a saga. You need an outbox plus idempotent retry. There is nothing to compensate, because a re-send is a no-op and an orphan is inert. Both of my mechanisms live here.
A saga earns its weight when a step has a real, irreversible side effect that must be semantically undone. Money was captured. An email was sent. An order was placed with an external vendor. You cannot ROLLBACK a charged card, so you design a forward compensation - issue a refund - that offsets the first action. The whole decision is this one fork:
Is the downstream step idempotent?
(content-addressed / safe to re-send, an orphan is inert)
│ │
YES NO — a real, irreversible side effect
▼ ▼
outbox + idempotent retry write a compensation (a genuine saga step)
nothing to compensate forward-undo: refund the charge, void the order
── both my mechanisms live here ── ── the rule-set write has NO step like this ──
A genuine saga step looks like this:
// A REAL saga step: a side effect with no rollback, so you write the undo.
type Step interface {
Do(ctx context.Context) error
Compensate(ctx context.Context) error
}
// Capturing a payment moves money; there is no ROLLBACK for that. Compensate
// issues a refund — a forward action that offsets Do. The rule-set write has
// NO step like this: a re-send is a no-op, an orphan is inert. Nothing to
// compensate, so there is no saga — idempotency replaces the whole machine.
Put those side by side and the rule falls out: if you can make the far side idempotent, do that instead of writing compensations. Compensations are code you have to get right for every failure branch, forever. Idempotency is a property you establish once. Reaching for the saga when idempotency would do isn't sophistication - it's building a rollback engine for actions that don't need rolling back.
And the pressure to get this wrong only grows. As more of the monolith's write path moves into the rule-set-markup service - the batch upsert, the whole CLIENT ⊃ PROJECT ⊃ CONFIG cascade - a single user write fans out into more cross-service steps: resolve identity → persist locally → notify → refill. More steps feels like more reason to reach for a saga. The discipline is to keep resisting until one of those steps grows a genuinely irreversible side effect. Until then, ordering plus idempotency plus a correlation key is not the poor man's saga - it's the correct design, and it's less code.
AI as a multiplier, and what it can't decide for you
I lean hard on AI coding assistants for work like this, and my take hasn't shifted across six parts: AI amplifies good engineers and exposes weak ones. It's a multiplier, not a crutch.
Ask one to "add a saga for a distributed transaction" and it will happily generate a clean, plausible orchestrator with compensations for every step - including the steps that never needed one. It produces the heavier correct-looking answer fast, because that's what the phrase asked for. What it won't do is stop and tell you the far side is content-addressed, so the entire compensation layer is dead weight and an outbox with idempotent retry is both simpler and more correct. It won't feel the scar of a phantom shipped on a rollback, so it won't insist the service call move outside the transaction. Point a multiplier at the judgment - is this step actually irreversible? is the far side idempotent? is my COMMIT the real one or a savepoint? - and it makes the right, lighter design cheap to build. Point it at the vocabulary - "saga," "distributed transaction" - and it'll help you over-engineer faster than you can review it.
This is Part 6 of a series
The seam that started with a silent 200 OK keeps teaching the same lesson from new angles: correctness across two databases is about identity, ordering, and idempotency far more often than it's about clever machinery.
- Part 5 - The hard part is the wire: partial failure, sync vs async, idempotency across the network.
- Part 6 - this one: the transaction you lost when you split the DB, why most sagas are really idempotency in disguise, and the one rule - never call a service inside a transaction - that both mechanisms bend around.
What's next - Part 7: when the flows genuinely do get long-running and offline - multi-step, spanning minutes or hours, surviving process crashes - you stop hand-rolling a saga engine and let a durable-execution system be the orchestrator. I'll walk through modeling the long cutover as a workflow with Temporal, where it earns its weight, and - just as important - where it absolutely does not.
If you build serious backends - Symfony, Go, or the messy space between a monolith and its microservices - follow along. And if you're mid-migration reaching for a saga right now: is your far side idempotent? If it is, you might be about to build a rollback engine you'll never fire. I'd genuinely like to compare notes.
Top comments (0)