If you INSERT an order and then publish to Kafka, you have two systems and a crash window, not an atomic write. A transactional outbox stores the event in the same database transaction as the order. Delivery is a later, retryable job. That is the interview answer.
I treat this as a Node.js drill. Five assertions. If one of them is hand-wavy, the answer is not ready.
What is the interviewer actually asking?
"How do you publish an event when an order is created?" is a trap if you stop at await kafka.send(...).
They want you to name the dual-write problem. Confluent's write-up is the cleanest public version: you must update two independent systems, a database and a broker, and a failure between those two writes leaves them inconsistent. Reverse the order and you just swap "lost event" for "phantom event." Wrap the Kafka call in a database transaction and it still does not roll back the message that already left the process.
The follow-up is always the same: without two-phase commit.
Microsoft's Cosmos DB guidance says the same thing in vendor language: save the business object and its events in one transaction, then let a separate worker publish. The request path stops owning the broker.
How does a dual-write fail in code?
The store below is an in-memory stand-in for Postgres. begin / commit / rollback copy the maps. That is enough to make the crash window visible.
class Store {
constructor() {
this.orders = new Map();
this.outbox = [];
this.nextId = 1;
this.tx = null;
}
begin() {
this.tx = {
orders: new Map(this.orders),
outbox: this.outbox.map((row) => ({ ...row })),
nextId: this.nextId,
};
}
commit() {
this.orders = this.tx.orders;
this.outbox = this.tx.outbox;
this.nextId = this.tx.nextId;
this.tx = null;
}
rollback() {
this.tx = null;
}
view() {
return this.tx ?? this;
}
insertOrder(payload) {
const id = String(this.view().nextId++);
this.view().orders.set(id, { id, ...payload });
return id;
}
insertOutbox(event) {
this.view().outbox.push({
id: event.id,
type: event.type,
payload: event.payload,
published: false,
lockedBy: null,
});
}
}
Naive create: commit the order, then publish. Crash after commit.
function dualWriteCreateOrder(store, broker, { crashAfterCommit }) {
store.begin();
const orderId = store.insertOrder({ sku: "SKU-1", qty: 1 });
store.commit();
if (crashAfterCommit) return { orderId, published: false };
broker.publish({ id: `order-${orderId}`, type: "order.created", orderId });
return { orderId, published: true };
}
Assertion one: the order row exists, the broker is empty. That is a lost order.created. Shipping never starts. Support gets a ticket that looks like a "Kafka issue."
Flip the order of operations and you get the other inconsistency.
function phantomPublishThenRollback(store, broker) {
store.begin();
const orderId = store.insertOrder({ sku: "SKU-1", qty: 1 });
broker.publish({ id: `order-${orderId}`, type: "order.created", orderId });
store.rollback();
return orderId;
}
Assertion two: the order is gone, the broker still has the event. Downstream just charged a card for a row that never committed.
Retries do not close this. An in-memory retry dies with the process. A durable retry is another write, which is the original problem again. Confluent walks through both anti-patterns; I will not pretend a try/catch around kafka.send is a design.
How do you commit the event with the order?
Write both rows in one transaction. Do not talk to the broker on the request path.
function outboxCreateOrder(store, payload) {
store.begin();
const orderId = store.insertOrder(payload);
store.insertOutbox({
id: `order-${orderId}`,
type: "order.created",
payload: { orderId },
});
store.commit();
return orderId;
}
Assertion three: after create, the order and the outbox row exist, and the broker is still empty. That emptiness is the point. The request returned 201 because Postgres committed, not because Kafka acknowledged.
If create rolls back, both rows vanish together. There is no phantom event to unwind.
How does the relay claim rows without blocking?
A separate process reads unpublished rows and publishes them. Two workers must not take the same row, and they must not sit in a lock-wait queue either.
Postgres documents this as FOR UPDATE SKIP LOCKED. The docs are blunt: skipping locked rows is an inconsistent view of the table, "not suitable for general purpose work," and specifically for "multiple consumers accessing a queue-like table." That sentence is the interview soundbite.
The in-memory version is a locked-by field. Same contract.
function claimSkipLocked(store, workerId, limit) {
const claimed = [];
for (const row of store.outbox) {
if (claimed.length >= limit) break;
if (row.published || row.lockedBy) continue;
row.lockedBy = workerId;
claimed.push(row);
}
return claimed;
}
function relay(store, broker, workerId, { failAfterPublish = false } = {}) {
const claimed = claimSkipLocked(store, workerId, 10);
for (const row of claimed) {
broker.publish({ id: row.id, type: row.type, ...row.payload });
if (failAfterPublish) {
row.lockedBy = null;
continue;
}
row.published = true;
row.lockedBy = null;
}
return claimed.map((row) => row.id);
}
Assertion four: two workers each claim one of two pending rows. The IDs differ. Nobody blocked.
In real SQL the claim and the status update belong in one statement, usually a CTE SELECT ... FOR UPDATE SKIP LOCKED feeding an UPDATE ... RETURNING. If you SELECT, close the transaction, then UPDATE, you rebuilt the race with extra steps.
CDC (Debezium tailing the WAL) is the other relay. Polling vs CDC is a latency/ops trade-off, not a different guarantee. Both are at-least-once.
Why is the consumer still responsible for duplicates?
Publish, then crash before published = true. The next relay pass publishes the same id again.
function consumeIdempotent(inbox, event, sideEffects) {
if (inbox.has(event.id)) return "duplicate";
sideEffects.push(event);
inbox.add(event.id);
return "applied";
}
Assertion five: the broker has two copies of the same event id. The inbox applies it once.
That is not a bug in the outbox. Microsoft's Cosmos write-up says the change feed is at-least-once and the consumer dedupes by event id. Same rule on Kafka. If you tell the interviewer "exactly-once," they will ask where the dedupe table lives. Put the inbox write in the same transaction as the side effect, or a crash between "I processed it" and "I recorded that I processed it" duplicates the side effect.
The event id is the idempotency key. Do not hash the payload. Two legitimate order.updated events can look similar.
How do you narrate this in the room?
I use a four-beat answer and stop.
- Dual-write: two systems, no shared transaction, crash window either way.
- Outbox: event row in the same commit as the order. Request path does not call Kafka.
- Relay: poller or CDC. Concurrent workers use
SKIP LOCKED, notFOR UPDATEthat waits. - Delivery: at-least-once. Idempotent consumer with a durable inbox. Event sourcing or "listen to yourself" if you do not have local transactions.
Trade-offs I expect to be asked:
- Latency. The event is not on the broker when the HTTP handler returns. Say "eventual, bounded by poll interval or CDC lag," not "immediate."
-
Ordering. Publishing by
id/ insert order preserves causality inside one aggregate. Do not promise global order across partitions. - Retention. Mark published or delete. Deleting shrinks the table; keeping rows helps replay. Either is fine if you can explain vacuum and the unique event id.
- When it is overkill. A monolith that sends email inside the request and can tolerate a missed email does not need this. A payment capture that must notify ledger and fraud does.
Once the five assertions pass, the remaining work is saying those trade-offs under follow-up pressure. I rehearse that part with aceround.app — AI interview assistant, because the code is the easy half.
FAQ
Does a transactional outbox give exactly-once delivery?
No. It makes the intent to publish atomic with the business write. The relay is at-least-once. Exactly-once end-to-end is outbox plus an idempotent consumer.
Can I skip the outbox table and use Kafka transactions?
Kafka transactions coordinate produces and offsets inside Kafka. They do not enlist your Postgres INSERT. You still have two systems.
Is SKIP LOCKED enough without idempotency?
No. It stops two live workers from grabbing the same unpublished row. It does not stop a row from being published twice after a crash between publish and ack.
When would you pick CDC over a polling relay?
When you already run Debezium (or Cosmos change feed) and you care about lower lag without hammering SELECT. Polling is the honest default: one extra table, one worker, obvious metrics.
Sources: Confluent on the dual-write problem, Microsoft's Cosmos DB transactional outbox, PostgreSQL SELECT locking / SKIP LOCKED.
Written with AI assistance, then checked against a local Node.js run of the five assertions and the vendor pages above.
Top comments (0)