When I finished Phase 6 of my event-driven microservices platform, I updated the architecture decision record.
ADR-016 described a resilience layer: circuit breakers on outbound dependencies, rate limiting at the perimeter, and — the part that felt particularly clean — a self-healing DLT pipeline. Dead-lettered events wouldn't sit in a table waiting for manual triage. A scheduled reprocessor would replay them with exponential backoff. A poison-message cap would prevent one bad event from blocking the rest. Dead letters would become an automatically draining queue.
The ADR status: Accepted.
The README described it. The runbook referenced it. I committed both and moved on.
Three months later, I was writing Phase 7 SLOs and noticed a Prometheus gauge I'd added: dlt_unreprocessed. It was reading 32.
I went to check what the scheduled reprocessor would do with those 32 events. There was no scheduled reprocessor. The table had been growing since Phase 5 failure tests. Nothing had ever drained it. The self-healing mechanism existed in three documents and in zero Java files.
How Documentation Gets Ahead of Implementation
The ADR wasn't wrong — the design was correct. The outbox-to-DLT routing worked. When a Kafka message exhausted retries, @DltHandler caught it, persisted it to dead_letter_events, and that was it. Correct behavior. Durable failure record.
The reprocessor was the second half of ADR-016. It got tagged as "Phase 7 polish" in the backlog because the routing worked and I'd moved into observability work. The backlog item never came back up. The ADR stayed at "Accepted."
"Accepted" in an ADR means the team agreed the design is sound. It does not mean a deployment ran. It does not mean tests passed. It is not a CI gate.
This is the failure mode: the document describes the design correctly, but describes it in the present tense — "a scheduled reprocessor replays dead-lettered events" — while the code implements only part of it. The gap is invisible until someone has to operate the thing.
The gap became visible when I tried to write the DltDepthHigh runbook.
The Runbook Forced the Issue
The runbook question is always: what does an on-call engineer actually do when this fires?
My draft said: "The scheduled reprocessor will automatically replay pending events. Check dlt_unreprocessed to confirm it's draining."
To validate that claim, I had to check that the reprocessor existed and would actually drain the gauge. That's when I found the 32-event backlog that had been sitting untouched since Phase 5. There was no reprocessor to reference.
The runbook is a correctness check for the alert. What I hadn't expected was that the runbook would become a correctness check for the architecture documentation.
The Naive Fix and Why It Falls Short
The obvious approach is a @Scheduled method that queries all unprocessed events, republishes them, and marks them done:
@Scheduled(fixedDelay = 60000)
@Transactional
public void reprocess() {
List<DeadLetterEvent> events = repository.findByReprocessedFalse();
for (DeadLetterEvent event : events) {
kafkaTemplate.send(event.getTopic(), event.getPayload());
event.setReprocessed(true);
}
}
This works for the happy path. It breaks in every edge case that matters in production.
No backoff. If the downstream system is unavailable, this fires every 60 seconds and hammers it.
No poison-message detection. If one event has a malformed payload, it stays in the reprocessedFalse set forever and consumes a retry slot on every tick.
No observability. Without tracking retry counts and failure reasons, you can't distinguish "transient failure, will clear soon" from "permanent failure, needs human attention."
The Production Implementation
The production version adds retry accounting, a poison-message cap, and metrics.
@Service
@Slf4j
@RequiredArgsConstructor
public class DltReprocessorService {
private static final int MAX_RETRY_COUNT = 5;
private final DeadLetterEventRepository repository;
private final KafkaTemplate<String, String> kafkaTemplate;
private final MeterRegistry meterRegistry;
@Scheduled(fixedDelayString = "${dlt.reprocessor.interval-ms:60000}")
@Transactional
public void reprocess() {
List<DeadLetterEvent> candidates = repository
.findByReprocessedFalseAndRetryCountLessThan(MAX_RETRY_COUNT);
for (DeadLetterEvent event : candidates) {
try {
kafkaTemplate.send(event.getTopic(), event.getPayload()).get();
event.setReprocessed(true);
meterRegistry.counter("notifications.dlt.reprocessed").increment();
log.info("DLT reprocess success | eventId: {} | topic: {}",
event.getEventId(), event.getTopic());
} catch (Exception e) {
event.setRetryCount(event.getRetryCount() + 1);
event.setLastRetryAt(Instant.now());
meterRegistry.counter("notifications.dlt.reprocess.failed").increment();
log.warn("DLT reprocess failed | eventId: {} | attempt: {} | error: {}",
event.getEventId(), event.getRetryCount(), e.getMessage());
}
}
}
}
Key design decisions:
Retry count cap. The query excludes events with retryCount >= MAX_RETRY_COUNT. They remain in the table — visible in the dlt_unreprocessed gauge — but the reprocessor stops attempting them.
Synchronous send. kafkaTemplate.send(...).get() blocks until the broker acknowledges. For a low-volume scheduled job this ensures "sent" means the broker received it, not just that the message entered the producer buffer.
Metrics in the loop. Every success and failure increments a counter feeding the same Grafana panels and alert rules the SLO runbook references.
Configurable interval. fixedDelayString with a property reference lets you adjust reprocessing frequency without a recompile.
Verifying the Drain
After the first compile and restart, I watched the gauge:
dlt_unreprocessed{job="notification-service"} 32
→ 18
→ 7
→ 0
notifications_dlt_reprocessed_total = 32. _reprocess_failed_total = 0. Clean drain.
The DltDepthHigh alert — which had been firing since I'd added it — resolved and moved to INACTIVE.
Now the runbook was honest.
Key Takeaways
1. "ADR: Accepted" is not a deployment.
It records that the team agreed the design is right. Someone still has to write the code, wire the beans, and verify it runs.
2. Runbooks are a correctness check for more than alerts.
Writing "the scheduled reprocessor will drain it" forced me to verify the reprocessor existed. Runbooks that describe operational behavior without verifying that behavior expose exactly these gaps.
3. Poison-message caps are not optional.
A scheduled reprocessor without a retry cap will starve on unconvertible messages. The cap turns a potential infinite loop into a finite retry window with a known exit state.
4. A DLT without a reprocessor is a graveyard.
Events that land in a dead-letter table are durable — but durable and recoverable are not the same thing.
5. Verify your own documentation.
Auditing the claim — "does this mechanism actually exist?" — is as important as auditing any external dependency assumption.
The reprocessor is live. ADR-016 is now actually true. The DltDepthHigh alert resolves on its own.
The gap between documented and implemented is one of the least visible failure modes in a distributed system. Write the runbook. If you can't verify the claim it makes, you've found the gap.
The full implementation is in the platform repository under notification-service/src/main/java/.../DltReprocessorService.java. What's the most surprising gap you've found between an architecture document and the running code?
Top comments (0)