Who Did This? Identity Across Async Boundaries

You put a lot of work into authentication. A gateway validates the Keycloak JWT, maps realm roles to authorities, checks that the caller is allowed. By the time a request reaches your service, you know exactly who is calling.

Then the request crosses into async land, and all of that evaporates.

This is the story of the point where identity quietly disappears in an event-driven system, why the dead-letter queue is the worst possible place for it to disappear, and how I made the acting user as durable and replay-safe as the event itself.

The flow everyone believes is fine

The platform is a set of Spring Boot services: an API gateway in front, a user-service on MySQL, a notification-service on PostgreSQL, and Kafka carrying events between them. A user is created, an event is published, a notification is sent.

Authentication is handled at the edge. The gateway is an OAuth2 Resource Server; it validates the token once and propagates the caller's identity downstream as headers:

// api-gateway — IdentityPropagationFilter (@Order(2), after security)
IdentityContext identity = identityContextExtractor.extract(jwt);

// Always set all three headers (empty when absent) to mask any spoofed values.
enrichedRequest.putHeader(IdentityHeaders.USER_NAME,  nullToEmpty(identity.username()));
enrichedRequest.putHeader(IdentityHeaders.USER_EMAIL, nullToEmpty(identity.email()));
enrichedRequest.putHeader(IdentityHeaders.USER_ROLES, identity.rolesAsString(DELIM));

One detail here matters more than it looks. The headers are always overwritten, even when a claim is absent. If a client tries to inject X-User-Name: admin on the inbound request, the gateway stomps it with the validated value (or empty). Downstream trust in those headers is only safe because the perimeter guarantees they cannot be forged. Miss that, and you've built an impersonation API.

So far, so good. The synchronous hop carries identity. The problem starts one line later.

The hidden failure: the thread boundary

I don't publish to Kafka inside the request. I use the transactional outbox pattern: the request persists the user and an outbox row in one local transaction, and a separate scheduled poller publishes to Kafka afterward. (Why: a direct publish inside the request can commit the DB row and then lose the event if the broker call fails — the dual-write problem. The outbox closes that gap.)

That decoupling is correct for delivery. It is also exactly where identity dies.

The outbox publisher runs on a scheduled thread, not the request thread. SecurityContextHolder, ThreadLocal, MDC — every ambient place you might have stashed "who is calling" — are all empty by the time the poller runs. There is no request. There is no token. There is nothing to read.

So the event goes out anonymous. The consumer logs anonymous. The notification is written anonymous. And when an event exhausts its retries and lands in the dead-letter queue — the one moment you will urgently want to know who triggered it — the dead-letter row is anonymous too. Provenance vanishes at precisely the point forensics begins.

The naive fixes (and why each one fails)

The tempting answers all fail at the same boundary:

"Just log the username in the controller." You can — but the log line you care about is the publish, which happens later, on another thread, after the HTTP response has already returned. The controller log tells you a request arrived; it can't attribute the event that failed twenty seconds later.

"Stash it in MDC / a ThreadLocal and read it in the publisher." The publisher isn't on your thread. MDC is thread-scoped; the scheduled poller starts with a clean, empty context. You'll read null every time. Worse, under Virtual Threads with pooled carriers, a stale ThreadLocal is a correctness hazard — you can leak one request's identity onto another's event.

"Re-read the JWT in the publisher." There is no request in scope and no token to re-validate. The authentication event is long over.

Every naive fix assumes identity lives in ambient thread state. Across an async boundary, ambient state is exactly what you don't have.

The production implementation: persist identity with the event

If the publish is decoupled from the request in time and thread, then identity has to travel the same way the event does — as data, not as ambient context.

So I capture the actor on the request thread and persist it onto the outbox row, inside the same transaction as the entity write:

// user-service — OutboxEventService (runs on the request thread)
IdentityContext actor = IdentityContextHolder.get().orElse(null);
String traceId = MDC.get(CorrelationConstants.TRACE_ID);

OutboxEvent outboxEvent = OutboxEvent.builder()
        .eventId(...).aggregateType("USER").eventType("USER_CREATED")
        .payload(objectMapper.writeValueAsString(payload))
        .status(OutboxEventStatus.PENDING)
        .actorUsername(actor != null ? actor.username()      : null)
        .actorEmail(   actor != null ? actor.email()         : null)
        .actorRoles(   actor != null ? actor.rolesAsString() : null)
        .traceId(traceId)
        .build();

outboxEventRepository.save(outboxEvent);   // same TX as the user write

Now identity is as durable as the event. It survives a crash, a restart, a redeploy — because it's a committed row, not a value on a dying thread.

The scheduled publisher later rehydrates that context from the row and rides it onto the message as Kafka headers — supplied explicitly, never read from MDC:

// user-service — EventPublisher (runs on the scheduled outbox thread)
public void publishUserCreatedEvent(UserCreatedEvent event, String traceId, IdentityContext actor) {
    ProducerRecord<String, Object> record = new ProducerRecord<>(TOPIC, event.getId().toString(), event);
    addHeader(record, KAFKA_TRACE_ID_HEADER, traceId);
    if (actor != null) {
        addHeader(record, KAFKA_USER_NAME,  actor.username());
        addHeader(record, KAFKA_USER_EMAIL, actor.email());
        addHeader(record, KAFKA_USER_ROLES, actor.rolesAsString());
    }
    kafkaTemplate.send(record);
}

The consumer does the mirror operation: read the headers, rebind identity to the consumer thread and MDC so every log line is attributed — and, critically, clear it in a finally block so nothing bleeds into the next message on a reused thread:

// notification-service — KafkaConsumerService
IdentityContext actor = restoreIdentityContext(userNameHeader, userEmailHeader, userRolesHeader);
try {
    notificationService.sendWelcomeNotification(event, topic, partition, offset, attempt);
} finally {
    clearIdentityContext();          // MDC.remove(...) + IdentityContextHolder.clear()
    MDC.remove(CorrelationConstants.TRACE_ID);
}

A single immutable IdentityContext record is the canonical "who is acting" shape at every hop — serialized to HTTP headers at the gateway, to a DB row in the outbox, to Kafka headers on publish, back to MDC on consume. It's deliberately transport-agnostic, so a sync→async hop never has to re-derive trust from the token.

Here's the whole path, including the failure branch:

[HTTP request thread]                    [scheduled thread]           [consumer thread]
Gateway (validate JWT)                    Outbox Publisher             @KafkaListener
   │ X-User-* headers (overwritten)          │ read actor from row        │ read actor from headers
   ▼                                          ▼                            ▼ bind to MDC
user-service ──save user + outbox row──▶ outbox_events ──Kafka headers──▶ process + log (attributed)
             (one transaction: entity                                     │
              + actor_username/email/roles)                               │ retries exhausted
                                                                          ▼
                                                             @DltHandler → dead_letter_events
                                                             (actor_* columns persisted)

The payoff: the dead-letter queue keeps its memory

The reason all of this is worth it lives in the @DltHandler. When every retry is exhausted, the failed event is persisted for triage with the originating actor attached:

deadLetterEventRepository.save(DeadLetterEvent.builder()
        .eventId(event.getEventId().toString())
        .lastError(errorMessage)
        .actorUsername(actor.username())
        .actorEmail(actor.email())
        .actorRoles(actor.roles().isEmpty() ? null : actor.rolesAsString())
        .build());

A dead letter without an actor is a mystery ticket. A dead letter with one is an incident you can route, reproduce, and explain. The schema changes that back this are explicit migrations — Liquibase actor_* + trace_id on outbox_events (MySQL), Flyway actor_* on dead_letter_events and notification_log (PostgreSQL) — not ddl-auto guesses.

Verification: prove it survives the failure path

Assertion isn't evidence. I verified the branch that matters — the failure one — by fault injection: force the notification step to fail so an event exhausts its four attempts and dead-letters, then read the audit table.

SELECT event_id, actor_username, actor_email, last_error
FROM   dead_letter_events
ORDER  BY failed_at DESC;
-- actor_username / actor_email populated on the failed row ✅

The identity is there, on a failed event, written by a background thread that never saw the original request. That's the whole thesis in one row.

A subtle bonus: because identity rides Kafka headers as data, it survives even where distributed tracing currently doesn't. On this platform the OTel agent doesn't yet emit Kafka spans (a gap I wrote about separately), so the async trace edge is blind — yet the actor still crosses it, because header-borne data doesn't depend on span instrumentation. Two independent propagation mechanisms; the durable one keeps working when the automatic one has a hole.

Key takeaways

• Authentication is not propagation. Validating a token proves a request is allowed now; it says nothing about who's behind the event that fails an hour later on another thread.
• Across an async boundary, identity must be data, not ambient context. ThreadLocal/MDC/SecurityContext are all empty on the publisher and consumer threads. Persist the actor with the event.
• Attach identity in the same transaction as the write. That's what makes it as durable and replay-safe as the event — surviving crashes, retries, and redeploys.
• The dead-letter queue is where provenance matters most and is usually lost. Persisting actor_* on the DLT row turns an anonymous mystery into a routable incident.
• Trust in propagated headers requires an unspoofable perimeter. Unconditionally overwrite X-User-* at the gateway, or you've built impersonation-as-a-service.
• Clear context in finally. Under Virtual Threads and pooled consumers, a stale ThreadLocal is a cross-request correctness bug, not just untidy.

The platform is open source — gateway, both services, the outbox, and this identity path end to end: https://github.com/Rummy43/ai-microservices-platform

When one of your events lands in the dead-letter queue tonight, can you tell who triggered it?

Originally published on Medium.