DEV Community: Ramesh Yara

Who Did This? Identity Across Async Boundaries

Ramesh Yara — Sun, 09 Aug 2026 18:20:05 +0000

You put a lot of work into authentication. A gateway validates the Keycloak JWT, maps realm roles to authorities, checks that the caller is allowed. By the time a request reaches your service, you know exactly who is calling.

Then the request crosses into async land, and all of that evaporates.

This is the story of the point where identity quietly disappears in an event-driven system, why the dead-letter queue is the worst possible place for it to disappear, and how I made the acting user as durable and replay-safe as the event itself.

The flow everyone believes is fine

The platform is a set of Spring Boot services: an API gateway in front, a user-service on MySQL, a notification-service on PostgreSQL, and Kafka carrying events between them. A user is created, an event is published, a notification is sent.

Authentication is handled at the edge. The gateway is an OAuth2 Resource Server; it validates the token once and propagates the caller's identity downstream as headers:

// api-gateway — IdentityPropagationFilter (@Order(2), after security)
IdentityContext identity = identityContextExtractor.extract(jwt);

// Always set all three headers (empty when absent) to mask any spoofed values.
enrichedRequest.putHeader(IdentityHeaders.USER_NAME,  nullToEmpty(identity.username()));
enrichedRequest.putHeader(IdentityHeaders.USER_EMAIL, nullToEmpty(identity.email()));
enrichedRequest.putHeader(IdentityHeaders.USER_ROLES, identity.rolesAsString(DELIM));

One detail here matters more than it looks. The headers are always overwritten, even when a claim is absent. If a client tries to inject X-User-Name: admin on the inbound request, the gateway stomps it with the validated value (or empty). Downstream trust in those headers is only safe because the perimeter guarantees they cannot be forged. Miss that, and you've built an impersonation API.

So far, so good. The synchronous hop carries identity. The problem starts one line later.

The hidden failure: the thread boundary

I don't publish to Kafka inside the request. I use the transactional outbox pattern: the request persists the user and an outbox row in one local transaction, and a separate scheduled poller publishes to Kafka afterward. (Why: a direct publish inside the request can commit the DB row and then lose the event if the broker call fails — the dual-write problem. The outbox closes that gap.)

That decoupling is correct for delivery. It is also exactly where identity dies.

The outbox publisher runs on a scheduled thread, not the request thread. SecurityContextHolder, ThreadLocal, MDC — every ambient place you might have stashed "who is calling" — are all empty by the time the poller runs. There is no request. There is no token. There is nothing to read.

So the event goes out anonymous. The consumer logs anonymous. The notification is written anonymous. And when an event exhausts its retries and lands in the dead-letter queue — the one moment you will urgently want to know who triggered it — the dead-letter row is anonymous too. Provenance vanishes at precisely the point forensics begins.

The naive fixes (and why each one fails)

The tempting answers all fail at the same boundary:

"Just log the username in the controller." You can — but the log line you care about is the publish, which happens later, on another thread, after the HTTP response has already returned. The controller log tells you a request arrived; it can't attribute the event that failed twenty seconds later.

"Stash it in MDC / a ThreadLocal and read it in the publisher." The publisher isn't on your thread. MDC is thread-scoped; the scheduled poller starts with a clean, empty context. You'll read null every time. Worse, under Virtual Threads with pooled carriers, a stale ThreadLocal is a correctness hazard — you can leak one request's identity onto another's event.

"Re-read the JWT in the publisher." There is no request in scope and no token to re-validate. The authentication event is long over.

Every naive fix assumes identity lives in ambient thread state. Across an async boundary, ambient state is exactly what you don't have.

The production implementation: persist identity with the event

If the publish is decoupled from the request in time and thread, then identity has to travel the same way the event does — as data, not as ambient context.

So I capture the actor on the request thread and persist it onto the outbox row, inside the same transaction as the entity write:

// user-service — OutboxEventService (runs on the request thread)
IdentityContext actor = IdentityContextHolder.get().orElse(null);
String traceId = MDC.get(CorrelationConstants.TRACE_ID);

OutboxEvent outboxEvent = OutboxEvent.builder()
        .eventId(...).aggregateType("USER").eventType("USER_CREATED")
        .payload(objectMapper.writeValueAsString(payload))
        .status(OutboxEventStatus.PENDING)
        .actorUsername(actor != null ? actor.username()      : null)
        .actorEmail(   actor != null ? actor.email()         : null)
        .actorRoles(   actor != null ? actor.rolesAsString() : null)
        .traceId(traceId)
        .build();

outboxEventRepository.save(outboxEvent);   // same TX as the user write

Now identity is as durable as the event. It survives a crash, a restart, a redeploy — because it's a committed row, not a value on a dying thread.

The scheduled publisher later rehydrates that context from the row and rides it onto the message as Kafka headers — supplied explicitly, never read from MDC:

// user-service — EventPublisher (runs on the scheduled outbox thread)
public void publishUserCreatedEvent(UserCreatedEvent event, String traceId, IdentityContext actor) {
    ProducerRecord<String, Object> record = new ProducerRecord<>(TOPIC, event.getId().toString(), event);
    addHeader(record, KAFKA_TRACE_ID_HEADER, traceId);
    if (actor != null) {
        addHeader(record, KAFKA_USER_NAME,  actor.username());
        addHeader(record, KAFKA_USER_EMAIL, actor.email());
        addHeader(record, KAFKA_USER_ROLES, actor.rolesAsString());
    }
    kafkaTemplate.send(record);
}

The consumer does the mirror operation: read the headers, rebind identity to the consumer thread and MDC so every log line is attributed — and, critically, clear it in a finally block so nothing bleeds into the next message on a reused thread:

// notification-service — KafkaConsumerService
IdentityContext actor = restoreIdentityContext(userNameHeader, userEmailHeader, userRolesHeader);
try {
    notificationService.sendWelcomeNotification(event, topic, partition, offset, attempt);
} finally {
    clearIdentityContext();          // MDC.remove(...) + IdentityContextHolder.clear()
    MDC.remove(CorrelationConstants.TRACE_ID);
}

A single immutable IdentityContext record is the canonical "who is acting" shape at every hop — serialized to HTTP headers at the gateway, to a DB row in the outbox, to Kafka headers on publish, back to MDC on consume. It's deliberately transport-agnostic, so a sync→async hop never has to re-derive trust from the token.

Here's the whole path, including the failure branch:

[HTTP request thread]                    [scheduled thread]           [consumer thread]
Gateway (validate JWT)                    Outbox Publisher             @KafkaListener
   │ X-User-* headers (overwritten)          │ read actor from row        │ read actor from headers
   ▼                                          ▼                            ▼ bind to MDC
user-service ──save user + outbox row──▶ outbox_events ──Kafka headers──▶ process + log (attributed)
             (one transaction: entity                                     │
              + actor_username/email/roles)                               │ retries exhausted
                                                                          ▼
                                                             @DltHandler → dead_letter_events
                                                             (actor_* columns persisted)

The payoff: the dead-letter queue keeps its memory

The reason all of this is worth it lives in the @DltHandler. When every retry is exhausted, the failed event is persisted for triage with the originating actor attached:

deadLetterEventRepository.save(DeadLetterEvent.builder()
        .eventId(event.getEventId().toString())
        .lastError(errorMessage)
        .actorUsername(actor.username())
        .actorEmail(actor.email())
        .actorRoles(actor.roles().isEmpty() ? null : actor.rolesAsString())
        .build());

A dead letter without an actor is a mystery ticket. A dead letter with one is an incident you can route, reproduce, and explain. The schema changes that back this are explicit migrations — Liquibase actor_* + trace_id on outbox_events (MySQL), Flyway actor_* on dead_letter_events and notification_log (PostgreSQL) — not ddl-auto guesses.

Verification: prove it survives the failure path

Assertion isn't evidence. I verified the branch that matters — the failure one — by fault injection: force the notification step to fail so an event exhausts its four attempts and dead-letters, then read the audit table.

SELECT event_id, actor_username, actor_email, last_error
FROM   dead_letter_events
ORDER  BY failed_at DESC;
-- actor_username / actor_email populated on the failed row ✅

The identity is there, on a failed event, written by a background thread that never saw the original request. That's the whole thesis in one row.

A subtle bonus: because identity rides Kafka headers as data, it survives even where distributed tracing currently doesn't. On this platform the OTel agent doesn't yet emit Kafka spans (a gap I wrote about separately), so the async trace edge is blind — yet the actor still crosses it, because header-borne data doesn't depend on span instrumentation. Two independent propagation mechanisms; the durable one keeps working when the automatic one has a hole.

Key takeaways

Authentication is not propagation. Validating a token proves a request is allowed now; it says nothing about who's behind the event that fails an hour later on another thread.
Across an async boundary, identity must be data, not ambient context. ThreadLocal/MDC/SecurityContext are all empty on the publisher and consumer threads. Persist the actor with the event.
Attach identity in the same transaction as the write. That's what makes it as durable and replay-safe as the event — surviving crashes, retries, and redeploys.
The dead-letter queue is where provenance matters most and is usually lost. Persisting actor_* on the DLT row turns an anonymous mystery into a routable incident.
Trust in propagated headers requires an unspoofable perimeter. Unconditionally overwrite X-User-* at the gateway, or you've built impersonation-as-a-service.
Clear context in finally. Under Virtual Threads and pooled consumers, a stale ThreadLocal is a cross-request correctness bug, not just untidy.

The platform is open source — gateway, both services, the outbox, and this identity path end to end: https://github.com/Rummy43/ai-microservices-platform

When one of your events lands in the dead-letter queue tonight, can you tell who triggered it?

Originally published on Medium.

Your Service Map Is Lying

Ramesh Yara — Sun, 09 Aug 2026 18:10:22 +0000

You attach the OpenTelemetry Java agent, point it at a collector, and within minutes Grafana is drawing a service map you never drew. A box for each service, arrows between them, latency on every edge. It feels like magic, and — more dangerously — it feels complete. "The agent traces everything" is the sentence repeated in every onboarding doc.

This is the story of the moment that sentence stopped being true on my platform, why I'm glad it did, and the difference between a system that is working and a system you can actually see.

The flow everyone trusts

The platform is an event-driven set of Spring Boot services: an API gateway in front, a user-service backed by MySQL, a notification-service backed by PostgreSQL, and Kafka carrying events between them. A user is created, an event is published, a notification is sent.

I didn't want to draw that topology. A hand-drawn architecture diagram is documentation that drifts — true the day you commit it, slightly wrong a month later, actively misleading after a quarter. I wanted the dependency graph generated from live traffic, so it would always reflect what the system actually does.

Grafana Tempo does exactly this. Its service-graphs processor reads matched client/server span pairs out of trace data and emits a metric — traces_service_graph_request_total — that Grafana renders as a node graph. No edge is ever wired by hand. The topology is derived, continuously, from real spans.

The edge that wasn't there

I generated the graph and the synchronous edges lit up immediately:

api-gateway → user-service
user-service → MySQL
notification-service → PostgreSQL

Then I looked for the one edge I actually cared about — user-service → notification-service, the asynchronous hop over Kafka.

It wasn't there.

The naive conclusion (and why it's wrong)

The tempting read is immediate and obvious: the async hop is broken. The event isn't getting across. Go debug the consumer.

So I checked. And the consumer was completely fine. notification-service had consumed every event and written every corresponding row to PostgreSQL. Its database edge was lit. Liveness was perfect; the feature worked end to end.

That is the trap. A missing edge looks exactly like a broken feature, and the instinct is to go "fix" something that was never broken. The defect wasn't in the message path at all — it was in the observability of the message path. Those are two different failure domains that happen to render identically on a dashboard.

So I stopped trusting the picture and went to the source of truth: the trace store.

Evidence over assertion

Two TraceQL queries settled it.

{ span.messaging.system = "kafka" }
→ 0 results

{ resource.service.name = "user-service" } && { resource.service.name = "notification-service" }
→ 0 traces

Zero Kafka messaging spans anywhere in the system. Zero traces spanning both services. The agent — this build, under this Spring Boot version — simply was not instrumenting the Kafka client. Nothing errored. No warning was logged. The map wasn't wrong about the data; the data was never produced.

This is the part worth sitting with: a green dashboard would have let me believe the chain was fully traced. The absence of red is not the presence of coverage.

How the graph is actually built

The pipeline that produces the map is worth seeing, because one of its links is also a silent path you have to consciously turn on:

OTel Java Agent (spans)
        │
        ▼
Tempo  (service-graphs processor)
        │  remote_write
        ▼
Prometheus  (--web.enable-remote-write-receiver)
        │  query
        ▼
Grafana  (Tempo data source → serviceMap → nodeGraph)

Three deliberate decisions shaped it:

Generated edges, not declared ones. Tempo reads client/server span pairs and emits the edge metric. The graph is a test, not a drawing — it fails when reality diverges from expectation, which is exactly what makes it valuable.

One processor, scoped on purpose. I enabled service-graphs only — deliberately not span-metrics. The latter generates RED/latency series and would have inflated cardinality for a deliverable that was strictly about topology. Minimal blast radius, single responsibility.

A push boundary you must open explicitly. Tempo remote-writes its generated metrics into Prometheus — the inverse of every other component, which Prometheus scrapes. That requires flipping Prometheus into a receiver with --web.enable-remote-write-receiver. Forget it, and the metrics silently never land. Same lesson as the Kafka gap, one layer down: the data paths that fail quietly are the ones nobody turned on.

One missing edge, two real causes

There's a subtlety the graph forced me to articulate. Even once the agent emits Kafka spans, the user-service → notification-service edge would still not look like a normal synchronous call — because my publish doesn't happen on the request thread.

I use the transactional outbox pattern: the request persists the user and an outbox row in one local transaction, and a scheduled poller publishes to Kafka afterward. So the Kafka branch roots its own trace, off the poll cycle, structurally separate from the originating HTTP request. That's the outbox decoupling working as designed — not a propagation bug.

So a single missing edge had two distinct, both-legitimate explanations: a real instrumentation gap (no Kafka spans) and a real design boundary (async decoupling). Conflating them would have been the naive read. The generated graph didn't just find a bug — it made an architecture decision legible.

Making the map legible, not just generated

There was one more way the graph was technically correct but practically dishonest: both databases first rendered as a single node named localhost. The agent labels an uninstrumented peer by its host, and both databases live on the loopback interface — so MySQL and PostgreSQL collapsed into one vertex. A heterogeneous-persistence design rendered as if it shared one database.

The fix is a peer-service mapping on the agent:

-Dotel.instrumentation.common.peer-service-mapping=\
  localhost:3306=MySQL,localhost:5432=PostgreSQL,\
  localhost:8085=SchemaRegistry,localhost:8180=Keycloak

Now each dependency resolves to a real, named vertex, and the Schema Registry shows up as its own node. I confirmed it the same way I confirmed everything else — in the backend, not the UI:

traces_service_graph_request_total{server="MySQL"}        → present
traces_service_graph_request_total{server="PostgreSQL"}   → present
traces_service_graph_request_total{server="localhost"}    → 0 (gone)

A graph that says localhost is generated; a graph that says MySQL and PostgreSQL is legible. The labels are what turn telemetry into topology.

Key takeaways

Auto-instrumentation is a claim, not a guarantee. "The agent traces everything" holds until a version boundary quietly says otherwise. Verify spans exist in the backend; never infer coverage from the absence of errors.
A missing edge is not a missing feature. Liveness and observability are independent failure domains. A system can be correct and unobservable at the same time — the most dangerous state, because it looks fine.
A generated graph is a test, not a decoration. Its entire value is that it can fail when reality diverges. A hand-drawn diagram cannot fail; that's precisely why it's worthless as a safety net.
Silent data paths must be consciously enabled. Remote-write receivers, peer mappings, opt-in instrumentation — the telemetry that "just never lands" is the telemetry nobody turned on. Make the gaps loud.
Name your peers. Topology you can't read is only half-built; labels are the difference between a generated graph and a useful one.
Verify with queries, not screenshots. Green is the absence of evidence, not evidence of absence.

The platform is open source — gateway, two services, Kafka, the full observability stack, and this service-graph setup: https://github.com/Rummy43/ai-microservices-platform

What's the most expensive silent gap you've found by verifying instead of trusting the dashboard?

Originally published on Medium.