DEV Community

Cover image for The Outbox Pattern Is Not Enough
Ramesh Yara
Ramesh Yara

Posted on • Originally published at Medium

The Outbox Pattern Is Not Enough

#kafka #distributedsystems #microservices #springboot

The textbook version of the transactional outbox is tight. You save the domain entity and an outbox row in one local transaction. A background scheduler picks up PENDING rows and publishes them to Kafka. You never publish inside the request thread — no dual-write, no atomicity breach. The pattern closes the consistency gap.

Then you load-test it.

I ran 1,000 authenticated requests through my event-driven platform in 70 seconds. The gateway returned 201 for every one of them. The outbox absorbed every row. The consumer drained everything. By every visible metric the system looked healthy. Underneath that health, I found three production-grade problems the textbook never mentioned.

What a correct implementation looks like

Before the problems, the shape of the solution. The outbox publisher runs on a @Scheduled virtual-thread worker:

@Scheduled(fixedDelay = 5000)
@Transactional
public void publishPendingEvents() {
    List<OutboxEvent> batch = outboxRepository
        .findTop20ByStatusOrderByCreatedAtAsc(OutboxStatus.PENDING);

    for (OutboxEvent event : batch) {
        event.setStatus(OutboxStatus.PROCESSING);
        outboxRepository.save(event);

        try {
            kafkaTemplate.send(event.getTopic(), event.getPayload()).get();
            event.setStatus(OutboxStatus.PUBLISHED);
        } catch (Exception e) {
            event.incrementRetryCount();
            if (event.getRetryCount() >= MAX_RETRIES) {
                event.setStatus(OutboxStatus.FAILED);
            } else {
                event.setStatus(OutboxStatus.PENDING);
            }
        }
        outboxRepository.save(event);
    }
}
Enter fullscreen mode Exit fullscreen mode

This is correct. The PROCESSING state prevents another scheduler instance from claiming the same row. The retry cap prevents infinite cycling. The PENDING fallback on transient errors gives the event another chance. The dual-write problem is genuinely closed.

Here is what that correctness does not cover.

Gap 1: Your throughput ceiling is a config line

fixedDelay = 5000 means the scheduler runs every 5 seconds. findTop20 means it picks up 20 rows per cycle.

Maximum publish throughput: 20 events ÷ 5 seconds = 4 events per second.

That number does not appear in your unit tests. It does not appear in your monitoring unless you specifically look for it. It is a ceiling determined by two config values chosen without measurement.

During the 1,000-event baseline run, the gateway processed ~14.3 requests per second. The publisher was draining at 4 per second. The backlog grew to 720 rows before the burst ended and the scheduler caught up. outbox_oldest_pending_age_seconds — the gauge that measures the age of the oldest PENDING row — peaked at 191 seconds.

[outbox burst profile]
t=0s    → 1,000 requests fire at 14.3 req/s
t=70s   → all 1,000 return HTTP 201; 720 rows queued
t=70s+  → publisher drains at 4 ev/s (~3 minutes to clear)
         outbox_oldest_pending_age_seconds peak: 191s
t=250s  → backlog reaches 0; gauge returns to 0
Enter fullscreen mode Exit fullscreen mode

outbox_oldest_pending_age_seconds — a 200-request parallel burst. Age climbs to 110s as inbound rate outpaces the publisher, then drops to zero. The 07-17 baseline (1,000 requests at 14.3 req/s) peaked at 191s and took 3 minutes to drain.

The system worked correctly. No events were lost. No data was corrupted. But the freshness SLO — "events delivered within 30 seconds of creation" — was structurally impossible to meet at any input rate above 4 req/s. The ceiling isn't a bug. It's a design constant hiding in plain sight.

The honest version of a freshness SLO for this system is: "Events are delivered within 30 seconds, given input rates below 4 req/s." That constraint belongs in your SLO catalog, not buried in two config lines.

# SLO catalog entry — what honest capacity accounting looks like
freshness_slo:
  target: 99.9% of events published within 30s of creation
  constraint: input rate ≤ 4 ev/s (fixedDelay=5000ms × batchSize=20)
  at_higher_rates: freshness degrades proportionally; availability unaffected
Enter fullscreen mode Exit fullscreen mode

Gap 2: The alert you write will fire for the wrong reason

The natural monitoring instinct for the outbox is an age threshold:

- alert: OutboxBacklogAgeHigh
  expr: outbox_oldest_pending_age_seconds > 60
  for: 5m
  labels:
    severity: page
Enter fullscreen mode Exit fullscreen mode

During the 720-row burst, outbox_oldest_pending_age_seconds crossed 191 seconds. The threshold is 60 seconds. Two rules went pending.

Neither fired.

The for: 5m clause — which distinguishes a transient burst from a sustained incident — held. The burst resolved in under 5 minutes. Both rules sat pending through the whole event and silently cleared when the backlog drained.

That is the correct behavior. But it is only correct if you understand why the window exists.

The naive response to a pending alert is to shorten the window. Drop for: from 5 minutes to 30 seconds to "catch problems faster." What you actually get is pages for every deployment spike, every cold-start burst, every Schema Registry restart. The alert stops being a signal and becomes noise that engineers learn to dismiss — which is worse than no alert at all.

The for: duration is where you encode your operational definition of incident. The 720-row burst at 191 seconds is a load event: the system is processing work faster than it can publish, and it will self-resolve when the input rate drops. A broker offline for 7 minutes is an incident: the backlog grows indefinitely and the oldest-age gauge only increases.

You cannot calibrate that boundary by intuition. You need to run a burst that is definitively not an incident — a known-finite load, system otherwise healthy — and measure the oldest-age peak. Then set for: so that peak does not fire. In this system: 191s peak, 5-minute for: window, zero false positives across every deployment and cold-start since.

Gap 3: The terminal state is invisible by design

This is the one that costs you.

When the publisher exhausts its retry budget, the row becomes FAILED and the scheduler never touches it again. Your outbox_oldest_pending_age_seconds goes back to zero — there are no more PENDING rows to report age for. Your backlog count goes to zero. Your age alert stays silent.

The event is gone. No notification was sent. No audit trail was written downstream. Nothing alerted.

This happened on my platform. During the cluster's first night, Schema Registry took approximately four minutes to become ready after a pod restart. The outbox publisher started immediately and encountered Error registering Avro schema for every publish attempt. After five retries, five rows were marked FAILED permanently.

The Grafana dashboard showed a healthy system. The Prometheus alerts list showed no active rules. The outbox_failed gauge was plotted on the business dashboard and showing 5. No one had wired a rule to it.

Left: MySQL terminal FAILED rows. Right: Grafana outbox panel — failed=5, no active alert. The "looks healthy" beat.

Those rows sat there for 24 hours.

The gauge without an alert is a decoration. The outbox_failed metric existed. It had a panel. It had a y-axis label. It had no operational consequence. The fix wasn't adding a new gauge — it was adding the rule that should have been there from the start:

- alert: OutboxPublishTerminalFailure
  expr: outbox_failed > 0
  for: 2m
  labels:
    severity: page
    slo: outbox_integrity
  annotations:
    runbook: https://docs.internal/runbooks/outbox-failed
    summary: "Terminal FAILED outbox rows detected — events permanently blocked"
Enter fullscreen mode Exit fullscreen mode

The for: 2m gives the scheduler one extra cycle to confirm before paging. The severity is page because FAILED is a permanent state — unlike PENDING, it never self-heals. The runbook names the re-drive query: update rows to PENDING, identify the root cause (Schema Registry, broker auth, schema incompatibility), resolve it, let the publisher retry.

Live-fire verified against a real condition: the FAILED rows described above triggered OutboxPublishTerminalFailure after 2 minutes. Alertmanager delivered [active] OutboxPublishTerminalFailure | severity=page | slo=outbox_integrity. Rows deleted. Alert auto-resolved in 30 seconds.

Alertmanager: OutboxPublishTerminalFailure ACTIVE — severity=page, slo=outbox_integrity, namespace=microservices. The pager delivery moment.

The operational layer, assembled

Three concrete additions on top of a correct outbox implementation:

1. Name the throughput ceiling in your SLO catalog.
fixedDelay and batchSize are not internal implementation details. They are your freshness SLO's capacity constraint. Make them visible, version them, and review them when input load changes.

2. Calibrate the age alert against measured burst data.
Run a known-safe burst. Record the oldest-age peak. Set for: so the burst doesn't page. Re-run the calibration whenever you change the batch size or schedule interval.

3. Alert unconditionally on FAILED, with severity page.
The terminal state has exactly the property that makes it most dangerous: it looks like a healthy system. Wire the rule, write the runbook, and treat every FAILED row as a lost event until the runbook says otherwise.

The numbers

Everything above came from a 70-second load run on a JWT-authenticated, fully containerized platform:

Metric Value
Requests 1,000 POST /api/v1/users
HTTP 201 1,000 (100%)
Availability SLI 1.0
Latency SLI 0.999
p99 gateway / user-service 186ms / 177ms
Peak backlog 720 rows
Oldest-age peak 191 seconds
Publisher rate ~4 ev/s
Consumer drain rate ~150 ev/s
False-positive alerts during burst 0

Terminal failure was live-fired separately: synthetic FAILED row inserted directly into the outbox table → OutboxPublishTerminalFailure pending at t=0, FIRING at t=2m → Alertmanager severity=page delivered → row deleted → alert auto-resolved in 30 seconds.

Key takeaways

  • The outbox closes the dual-write problem; it opens an operational one. "Works correctly" and "is observable and bounded" are different properties.
  • Your throughput ceiling is determined by two config values that most implementations never name. Instrument it, measure it, put it in your SLO catalog.
  • The for: duration is where you define "incident." Shortening it to catch faster is how alerts become noise. Calibrate it against real burst data, not intuition.
  • A gauge without an alert is a decoration. The terminal FAILED state has exactly the property that makes it most dangerous: it looks like a healthy system.
  • FAILED rows never self-heal. Alert on them unconditionally, with severity page, and with a runbook that names the re-drive query.
  • Honest SLOs require measurement. A freshness target that doesn't name its throughput constraint is either vacuous or untested.

The full implementation — outbox publisher, status lifecycle, PENDING/FAILED gauges, alert rules, and runbooks — is in the open-source repository:

https://github.com/Rummy43/ai-microservices-platform

What's the throughput ceiling of your outbox implementation, and have you measured it?

Top comments (0)