DEV Community: Ramesh Yara

The Outbox Pattern Is Not Enough

Ramesh Yara — Mon, 17 Aug 2026 21:04:50 +0000

The textbook version of the transactional outbox is tight. You save the domain entity and an outbox row in one local transaction. A background scheduler picks up PENDING rows and publishes them to Kafka. You never publish inside the request thread — no dual-write, no atomicity breach. The pattern closes the consistency gap.

Then you load-test it.

I ran 1,000 authenticated requests through my event-driven platform in 70 seconds. The gateway returned 201 for every one of them. The outbox absorbed every row. The consumer drained everything. By every visible metric the system looked healthy. Underneath that health, I found three production-grade problems the textbook never mentioned.

What a correct implementation looks like

Before the problems, the shape of the solution. The outbox publisher runs on a @Scheduled virtual-thread worker:

@Scheduled(fixedDelay = 5000)
@Transactional
public void publishPendingEvents() {
    List<OutboxEvent> batch = outboxRepository
        .findTop20ByStatusOrderByCreatedAtAsc(OutboxStatus.PENDING);

    for (OutboxEvent event : batch) {
        event.setStatus(OutboxStatus.PROCESSING);
        outboxRepository.save(event);

        try {
            kafkaTemplate.send(event.getTopic(), event.getPayload()).get();
            event.setStatus(OutboxStatus.PUBLISHED);
        } catch (Exception e) {
            event.incrementRetryCount();
            if (event.getRetryCount() >= MAX_RETRIES) {
                event.setStatus(OutboxStatus.FAILED);
            } else {
                event.setStatus(OutboxStatus.PENDING);
            }
        }
        outboxRepository.save(event);
    }
}

This is correct. The PROCESSING state prevents another scheduler instance from claiming the same row. The retry cap prevents infinite cycling. The PENDING fallback on transient errors gives the event another chance. The dual-write problem is genuinely closed.

Here is what that correctness does not cover.

Gap 1: Your throughput ceiling is a config line

fixedDelay = 5000 means the scheduler runs every 5 seconds. findTop20 means it picks up 20 rows per cycle.

Maximum publish throughput: 20 events ÷ 5 seconds = 4 events per second.

That number does not appear in your unit tests. It does not appear in your monitoring unless you specifically look for it. It is a ceiling determined by two config values chosen without measurement.

During the 1,000-event baseline run, the gateway processed ~14.3 requests per second. The publisher was draining at 4 per second. The backlog grew to 720 rows before the burst ended and the scheduler caught up. outbox_oldest_pending_age_seconds — the gauge that measures the age of the oldest PENDING row — peaked at 191 seconds.

[outbox burst profile]
t=0s    → 1,000 requests fire at 14.3 req/s
t=70s   → all 1,000 return HTTP 201; 720 rows queued
t=70s+  → publisher drains at 4 ev/s (~3 minutes to clear)
         outbox_oldest_pending_age_seconds peak: 191s
t=250s  → backlog reaches 0; gauge returns to 0

outbox_oldest_pending_age_seconds — a 200-request parallel burst. Age climbs to 110s as inbound rate outpaces the publisher, then drops to zero. The 07-17 baseline (1,000 requests at 14.3 req/s) peaked at 191s and took 3 minutes to drain.

The system worked correctly. No events were lost. No data was corrupted. But the freshness SLO — "events delivered within 30 seconds of creation" — was structurally impossible to meet at any input rate above 4 req/s. The ceiling isn't a bug. It's a design constant hiding in plain sight.

The honest version of a freshness SLO for this system is: "Events are delivered within 30 seconds, given input rates below 4 req/s." That constraint belongs in your SLO catalog, not buried in two config lines.

# SLO catalog entry — what honest capacity accounting looks like
freshness_slo:
  target: 99.9% of events published within 30s of creation
  constraint: input rate ≤ 4 ev/s (fixedDelay=5000ms × batchSize=20)
  at_higher_rates: freshness degrades proportionally; availability unaffected

Gap 2: The alert you write will fire for the wrong reason

The natural monitoring instinct for the outbox is an age threshold:

- alert: OutboxBacklogAgeHigh
  expr: outbox_oldest_pending_age_seconds > 60
  for: 5m
  labels:
    severity: page

During the 720-row burst, outbox_oldest_pending_age_seconds crossed 191 seconds. The threshold is 60 seconds. Two rules went pending.

Neither fired.

The for: 5m clause — which distinguishes a transient burst from a sustained incident — held. The burst resolved in under 5 minutes. Both rules sat pending through the whole event and silently cleared when the backlog drained.

That is the correct behavior. But it is only correct if you understand why the window exists.

The naive response to a pending alert is to shorten the window. Drop for: from 5 minutes to 30 seconds to "catch problems faster." What you actually get is pages for every deployment spike, every cold-start burst, every Schema Registry restart. The alert stops being a signal and becomes noise that engineers learn to dismiss — which is worse than no alert at all.

The for: duration is where you encode your operational definition of incident. The 720-row burst at 191 seconds is a load event: the system is processing work faster than it can publish, and it will self-resolve when the input rate drops. A broker offline for 7 minutes is an incident: the backlog grows indefinitely and the oldest-age gauge only increases.

You cannot calibrate that boundary by intuition. You need to run a burst that is definitively not an incident — a known-finite load, system otherwise healthy — and measure the oldest-age peak. Then set for: so that peak does not fire. In this system: 191s peak, 5-minute for: window, zero false positives across every deployment and cold-start since.

Gap 3: The terminal state is invisible by design

This is the one that costs you.

When the publisher exhausts its retry budget, the row becomes FAILED and the scheduler never touches it again. Your outbox_oldest_pending_age_seconds goes back to zero — there are no more PENDING rows to report age for. Your backlog count goes to zero. Your age alert stays silent.

The event is gone. No notification was sent. No audit trail was written downstream. Nothing alerted.

This happened on my platform. During the cluster's first night, Schema Registry took approximately four minutes to become ready after a pod restart. The outbox publisher started immediately and encountered Error registering Avro schema for every publish attempt. After five retries, five rows were marked FAILED permanently.

The Grafana dashboard showed a healthy system. The Prometheus alerts list showed no active rules. The outbox_failed gauge was plotted on the business dashboard and showing 5. No one had wired a rule to it.

Left: MySQL terminal FAILED rows. Right: Grafana outbox panel — failed=5, no active alert. The "looks healthy" beat.

Those rows sat there for 24 hours.

The gauge without an alert is a decoration. The outbox_failed metric existed. It had a panel. It had a y-axis label. It had no operational consequence. The fix wasn't adding a new gauge — it was adding the rule that should have been there from the start:

- alert: OutboxPublishTerminalFailure
  expr: outbox_failed > 0
  for: 2m
  labels:
    severity: page
    slo: outbox_integrity
  annotations:
    runbook: https://docs.internal/runbooks/outbox-failed
    summary: "Terminal FAILED outbox rows detected — events permanently blocked"

The for: 2m gives the scheduler one extra cycle to confirm before paging. The severity is page because FAILED is a permanent state — unlike PENDING, it never self-heals. The runbook names the re-drive query: update rows to PENDING, identify the root cause (Schema Registry, broker auth, schema incompatibility), resolve it, let the publisher retry.

Live-fire verified against a real condition: the FAILED rows described above triggered OutboxPublishTerminalFailure after 2 minutes. Alertmanager delivered [active] OutboxPublishTerminalFailure | severity=page | slo=outbox_integrity. Rows deleted. Alert auto-resolved in 30 seconds.

Alertmanager: OutboxPublishTerminalFailure ACTIVE — severity=page, slo=outbox_integrity, namespace=microservices. The pager delivery moment.

The operational layer, assembled

Three concrete additions on top of a correct outbox implementation:

1. Name the throughput ceiling in your SLO catalog.
fixedDelay and batchSize are not internal implementation details. They are your freshness SLO's capacity constraint. Make them visible, version them, and review them when input load changes.

2. Calibrate the age alert against measured burst data.
Run a known-safe burst. Record the oldest-age peak. Set for: so the burst doesn't page. Re-run the calibration whenever you change the batch size or schedule interval.

3. Alert unconditionally on FAILED, with severity page.
The terminal state has exactly the property that makes it most dangerous: it looks like a healthy system. Wire the rule, write the runbook, and treat every FAILED row as a lost event until the runbook says otherwise.

The numbers

Everything above came from a 70-second load run on a JWT-authenticated, fully containerized platform:

Metric	Value
Requests	1,000 POST `/api/v1/users`
HTTP 201	1,000 (100%)
Availability SLI	1.0
Latency SLI	0.999
p99 gateway / user-service	186ms / 177ms
Peak backlog	720 rows
Oldest-age peak	191 seconds
Publisher rate	~4 ev/s
Consumer drain rate	~150 ev/s
False-positive alerts during burst	0

Terminal failure was live-fired separately: synthetic FAILED row inserted directly into the outbox table → OutboxPublishTerminalFailure pending at t=0, FIRING at t=2m → Alertmanager severity=page delivered → row deleted → alert auto-resolved in 30 seconds.

Key takeaways

The outbox closes the dual-write problem; it opens an operational one. "Works correctly" and "is observable and bounded" are different properties.
Your throughput ceiling is determined by two config values that most implementations never name. Instrument it, measure it, put it in your SLO catalog.
The for: duration is where you define "incident." Shortening it to catch faster is how alerts become noise. Calibrate it against real burst data, not intuition.
A gauge without an alert is a decoration. The terminal FAILED state has exactly the property that makes it most dangerous: it looks like a healthy system.
FAILED rows never self-heal. Alert on them unconditionally, with severity page, and with a runbook that names the re-drive query.
Honest SLOs require measurement. A freshness target that doesn't name its throughput constraint is either vacuous or untested.

The full implementation — outbox publisher, status lifecycle, PENDING/FAILED gauges, alert rules, and runbooks — is in the open-source repository:

https://github.com/Rummy43/ai-microservices-platform

What's the throughput ceiling of your outbox implementation, and have you measured it?

Who Did This? Identity Across Async Boundaries

Ramesh Yara — Sun, 09 Aug 2026 18:20:05 +0000

You put a lot of work into authentication. A gateway validates the Keycloak JWT, maps realm roles to authorities, checks that the caller is allowed. By the time a request reaches your service, you know exactly who is calling.

Then the request crosses into async land, and all of that evaporates.

This is the story of the point where identity quietly disappears in an event-driven system, why the dead-letter queue is the worst possible place for it to disappear, and how I made the acting user as durable and replay-safe as the event itself.

The flow everyone believes is fine

The platform is a set of Spring Boot services: an API gateway in front, a user-service on MySQL, a notification-service on PostgreSQL, and Kafka carrying events between them. A user is created, an event is published, a notification is sent.

Authentication is handled at the edge. The gateway is an OAuth2 Resource Server; it validates the token once and propagates the caller's identity downstream as headers:

// api-gateway — IdentityPropagationFilter (@Order(2), after security)
IdentityContext identity = identityContextExtractor.extract(jwt);

// Always set all three headers (empty when absent) to mask any spoofed values.
enrichedRequest.putHeader(IdentityHeaders.USER_NAME,  nullToEmpty(identity.username()));
enrichedRequest.putHeader(IdentityHeaders.USER_EMAIL, nullToEmpty(identity.email()));
enrichedRequest.putHeader(IdentityHeaders.USER_ROLES, identity.rolesAsString(DELIM));

One detail here matters more than it looks. The headers are always overwritten, even when a claim is absent. If a client tries to inject X-User-Name: admin on the inbound request, the gateway stomps it with the validated value (or empty). Downstream trust in those headers is only safe because the perimeter guarantees they cannot be forged. Miss that, and you've built an impersonation API.

So far, so good. The synchronous hop carries identity. The problem starts one line later.

The hidden failure: the thread boundary

I don't publish to Kafka inside the request. I use the transactional outbox pattern: the request persists the user and an outbox row in one local transaction, and a separate scheduled poller publishes to Kafka afterward. (Why: a direct publish inside the request can commit the DB row and then lose the event if the broker call fails — the dual-write problem. The outbox closes that gap.)

That decoupling is correct for delivery. It is also exactly where identity dies.

The outbox publisher runs on a scheduled thread, not the request thread. SecurityContextHolder, ThreadLocal, MDC — every ambient place you might have stashed "who is calling" — are all empty by the time the poller runs. There is no request. There is no token. There is nothing to read.

So the event goes out anonymous. The consumer logs anonymous. The notification is written anonymous. And when an event exhausts its retries and lands in the dead-letter queue — the one moment you will urgently want to know who triggered it — the dead-letter row is anonymous too. Provenance vanishes at precisely the point forensics begins.

The naive fixes (and why each one fails)

The tempting answers all fail at the same boundary:

"Just log the username in the controller." You can — but the log line you care about is the publish, which happens later, on another thread, after the HTTP response has already returned. The controller log tells you a request arrived; it can't attribute the event that failed twenty seconds later.

"Stash it in MDC / a ThreadLocal and read it in the publisher." The publisher isn't on your thread. MDC is thread-scoped; the scheduled poller starts with a clean, empty context. You'll read null every time. Worse, under Virtual Threads with pooled carriers, a stale ThreadLocal is a correctness hazard — you can leak one request's identity onto another's event.

"Re-read the JWT in the publisher." There is no request in scope and no token to re-validate. The authentication event is long over.

Every naive fix assumes identity lives in ambient thread state. Across an async boundary, ambient state is exactly what you don't have.

The production implementation: persist identity with the event

If the publish is decoupled from the request in time and thread, then identity has to travel the same way the event does — as data, not as ambient context.

So I capture the actor on the request thread and persist it onto the outbox row, inside the same transaction as the entity write:

// user-service — OutboxEventService (runs on the request thread)
IdentityContext actor = IdentityContextHolder.get().orElse(null);
String traceId = MDC.get(CorrelationConstants.TRACE_ID);

OutboxEvent outboxEvent = OutboxEvent.builder()
        .eventId(...).aggregateType("USER").eventType("USER_CREATED")
        .payload(objectMapper.writeValueAsString(payload))
        .status(OutboxEventStatus.PENDING)
        .actorUsername(actor != null ? actor.username()      : null)
        .actorEmail(   actor != null ? actor.email()         : null)
        .actorRoles(   actor != null ? actor.rolesAsString() : null)
        .traceId(traceId)
        .build();

outboxEventRepository.save(outboxEvent);   // same TX as the user write

Now identity is as durable as the event. It survives a crash, a restart, a redeploy — because it's a committed row, not a value on a dying thread.

The scheduled publisher later rehydrates that context from the row and rides it onto the message as Kafka headers — supplied explicitly, never read from MDC:

// user-service — EventPublisher (runs on the scheduled outbox thread)
public void publishUserCreatedEvent(UserCreatedEvent event, String traceId, IdentityContext actor) {
    ProducerRecord<String, Object> record = new ProducerRecord<>(TOPIC, event.getId().toString(), event);
    addHeader(record, KAFKA_TRACE_ID_HEADER, traceId);
    if (actor != null) {
        addHeader(record, KAFKA_USER_NAME,  actor.username());
        addHeader(record, KAFKA_USER_EMAIL, actor.email());
        addHeader(record, KAFKA_USER_ROLES, actor.rolesAsString());
    }
    kafkaTemplate.send(record);
}

The consumer does the mirror operation: read the headers, rebind identity to the consumer thread and MDC so every log line is attributed — and, critically, clear it in a finally block so nothing bleeds into the next message on a reused thread:

// notification-service — KafkaConsumerService
IdentityContext actor = restoreIdentityContext(userNameHeader, userEmailHeader, userRolesHeader);
try {
    notificationService.sendWelcomeNotification(event, topic, partition, offset, attempt);
} finally {
    clearIdentityContext();          // MDC.remove(...) + IdentityContextHolder.clear()
    MDC.remove(CorrelationConstants.TRACE_ID);
}

A single immutable IdentityContext record is the canonical "who is acting" shape at every hop — serialized to HTTP headers at the gateway, to a DB row in the outbox, to Kafka headers on publish, back to MDC on consume. It's deliberately transport-agnostic, so a sync→async hop never has to re-derive trust from the token.

Here's the whole path, including the failure branch:

[HTTP request thread]                    [scheduled thread]           [consumer thread]
Gateway (validate JWT)                    Outbox Publisher             @KafkaListener
   │ X-User-* headers (overwritten)          │ read actor from row        │ read actor from headers
   ▼                                          ▼                            ▼ bind to MDC
user-service ──save user + outbox row──▶ outbox_events ──Kafka headers──▶ process + log (attributed)
             (one transaction: entity                                     │
              + actor_username/email/roles)                               │ retries exhausted
                                                                          ▼
                                                             @DltHandler → dead_letter_events
                                                             (actor_* columns persisted)

The payoff: the dead-letter queue keeps its memory

The reason all of this is worth it lives in the @DltHandler. When every retry is exhausted, the failed event is persisted for triage with the originating actor attached:

deadLetterEventRepository.save(DeadLetterEvent.builder()
        .eventId(event.getEventId().toString())
        .lastError(errorMessage)
        .actorUsername(actor.username())
        .actorEmail(actor.email())
        .actorRoles(actor.roles().isEmpty() ? null : actor.rolesAsString())
        .build());

A dead letter without an actor is a mystery ticket. A dead letter with one is an incident you can route, reproduce, and explain. The schema changes that back this are explicit migrations — Liquibase actor_* + trace_id on outbox_events (MySQL), Flyway actor_* on dead_letter_events and notification_log (PostgreSQL) — not ddl-auto guesses.

Verification: prove it survives the failure path

Assertion isn't evidence. I verified the branch that matters — the failure one — by fault injection: force the notification step to fail so an event exhausts its four attempts and dead-letters, then read the audit table.

SELECT event_id, actor_username, actor_email, last_error
FROM   dead_letter_events
ORDER  BY failed_at DESC;
-- actor_username / actor_email populated on the failed row ✅

The identity is there, on a failed event, written by a background thread that never saw the original request. That's the whole thesis in one row.

A subtle bonus: because identity rides Kafka headers as data, it survives even where distributed tracing currently doesn't. On this platform the OTel agent doesn't yet emit Kafka spans (a gap I wrote about separately), so the async trace edge is blind — yet the actor still crosses it, because header-borne data doesn't depend on span instrumentation. Two independent propagation mechanisms; the durable one keeps working when the automatic one has a hole.

Key takeaways

Authentication is not propagation. Validating a token proves a request is allowed now; it says nothing about who's behind the event that fails an hour later on another thread.
Across an async boundary, identity must be data, not ambient context. ThreadLocal/MDC/SecurityContext are all empty on the publisher and consumer threads. Persist the actor with the event.
Attach identity in the same transaction as the write. That's what makes it as durable and replay-safe as the event — surviving crashes, retries, and redeploys.
The dead-letter queue is where provenance matters most and is usually lost. Persisting actor_* on the DLT row turns an anonymous mystery into a routable incident.
Trust in propagated headers requires an unspoofable perimeter. Unconditionally overwrite X-User-* at the gateway, or you've built impersonation-as-a-service.
Clear context in finally. Under Virtual Threads and pooled consumers, a stale ThreadLocal is a cross-request correctness bug, not just untidy.

The platform is open source — gateway, both services, the outbox, and this identity path end to end: https://github.com/Rummy43/ai-microservices-platform

When one of your events lands in the dead-letter queue tonight, can you tell who triggered it?

Originally published on Medium.

Your Service Map Is Lying

Ramesh Yara — Sun, 09 Aug 2026 18:10:22 +0000

You attach the OpenTelemetry Java agent, point it at a collector, and within minutes Grafana is drawing a service map you never drew. A box for each service, arrows between them, latency on every edge. It feels like magic, and — more dangerously — it feels complete. "The agent traces everything" is the sentence repeated in every onboarding doc.

This is the story of the moment that sentence stopped being true on my platform, why I'm glad it did, and the difference between a system that is working and a system you can actually see.

The flow everyone trusts

The platform is an event-driven set of Spring Boot services: an API gateway in front, a user-service backed by MySQL, a notification-service backed by PostgreSQL, and Kafka carrying events between them. A user is created, an event is published, a notification is sent.

I didn't want to draw that topology. A hand-drawn architecture diagram is documentation that drifts — true the day you commit it, slightly wrong a month later, actively misleading after a quarter. I wanted the dependency graph generated from live traffic, so it would always reflect what the system actually does.

Grafana Tempo does exactly this. Its service-graphs processor reads matched client/server span pairs out of trace data and emits a metric — traces_service_graph_request_total — that Grafana renders as a node graph. No edge is ever wired by hand. The topology is derived, continuously, from real spans.

The edge that wasn't there

I generated the graph and the synchronous edges lit up immediately:

api-gateway → user-service
user-service → MySQL
notification-service → PostgreSQL

Then I looked for the one edge I actually cared about — user-service → notification-service, the asynchronous hop over Kafka.

It wasn't there.

The naive conclusion (and why it's wrong)

The tempting read is immediate and obvious: the async hop is broken. The event isn't getting across. Go debug the consumer.

So I checked. And the consumer was completely fine. notification-service had consumed every event and written every corresponding row to PostgreSQL. Its database edge was lit. Liveness was perfect; the feature worked end to end.

That is the trap. A missing edge looks exactly like a broken feature, and the instinct is to go "fix" something that was never broken. The defect wasn't in the message path at all — it was in the observability of the message path. Those are two different failure domains that happen to render identically on a dashboard.

So I stopped trusting the picture and went to the source of truth: the trace store.

Evidence over assertion

Two TraceQL queries settled it.

{ span.messaging.system = "kafka" }
→ 0 results

{ resource.service.name = "user-service" } && { resource.service.name = "notification-service" }
→ 0 traces

Zero Kafka messaging spans anywhere in the system. Zero traces spanning both services. The agent — this build, under this Spring Boot version — simply was not instrumenting the Kafka client. Nothing errored. No warning was logged. The map wasn't wrong about the data; the data was never produced.

This is the part worth sitting with: a green dashboard would have let me believe the chain was fully traced. The absence of red is not the presence of coverage.

How the graph is actually built

The pipeline that produces the map is worth seeing, because one of its links is also a silent path you have to consciously turn on:

OTel Java Agent (spans)
        │
        ▼
Tempo  (service-graphs processor)
        │  remote_write
        ▼
Prometheus  (--web.enable-remote-write-receiver)
        │  query
        ▼
Grafana  (Tempo data source → serviceMap → nodeGraph)

Three deliberate decisions shaped it:

Generated edges, not declared ones. Tempo reads client/server span pairs and emits the edge metric. The graph is a test, not a drawing — it fails when reality diverges from expectation, which is exactly what makes it valuable.

One processor, scoped on purpose. I enabled service-graphs only — deliberately not span-metrics. The latter generates RED/latency series and would have inflated cardinality for a deliverable that was strictly about topology. Minimal blast radius, single responsibility.

A push boundary you must open explicitly. Tempo remote-writes its generated metrics into Prometheus — the inverse of every other component, which Prometheus scrapes. That requires flipping Prometheus into a receiver with --web.enable-remote-write-receiver. Forget it, and the metrics silently never land. Same lesson as the Kafka gap, one layer down: the data paths that fail quietly are the ones nobody turned on.

One missing edge, two real causes

There's a subtlety the graph forced me to articulate. Even once the agent emits Kafka spans, the user-service → notification-service edge would still not look like a normal synchronous call — because my publish doesn't happen on the request thread.

I use the transactional outbox pattern: the request persists the user and an outbox row in one local transaction, and a scheduled poller publishes to Kafka afterward. So the Kafka branch roots its own trace, off the poll cycle, structurally separate from the originating HTTP request. That's the outbox decoupling working as designed — not a propagation bug.

So a single missing edge had two distinct, both-legitimate explanations: a real instrumentation gap (no Kafka spans) and a real design boundary (async decoupling). Conflating them would have been the naive read. The generated graph didn't just find a bug — it made an architecture decision legible.

Making the map legible, not just generated

There was one more way the graph was technically correct but practically dishonest: both databases first rendered as a single node named localhost. The agent labels an uninstrumented peer by its host, and both databases live on the loopback interface — so MySQL and PostgreSQL collapsed into one vertex. A heterogeneous-persistence design rendered as if it shared one database.

The fix is a peer-service mapping on the agent:

-Dotel.instrumentation.common.peer-service-mapping=\
  localhost:3306=MySQL,localhost:5432=PostgreSQL,\
  localhost:8085=SchemaRegistry,localhost:8180=Keycloak

Now each dependency resolves to a real, named vertex, and the Schema Registry shows up as its own node. I confirmed it the same way I confirmed everything else — in the backend, not the UI:

traces_service_graph_request_total{server="MySQL"}        → present
traces_service_graph_request_total{server="PostgreSQL"}   → present
traces_service_graph_request_total{server="localhost"}    → 0 (gone)

A graph that says localhost is generated; a graph that says MySQL and PostgreSQL is legible. The labels are what turn telemetry into topology.

Key takeaways

Auto-instrumentation is a claim, not a guarantee. "The agent traces everything" holds until a version boundary quietly says otherwise. Verify spans exist in the backend; never infer coverage from the absence of errors.
A missing edge is not a missing feature. Liveness and observability are independent failure domains. A system can be correct and unobservable at the same time — the most dangerous state, because it looks fine.
A generated graph is a test, not a decoration. Its entire value is that it can fail when reality diverges. A hand-drawn diagram cannot fail; that's precisely why it's worthless as a safety net.
Silent data paths must be consciously enabled. Remote-write receivers, peer mappings, opt-in instrumentation — the telemetry that "just never lands" is the telemetry nobody turned on. Make the gaps loud.
Name your peers. Topology you can't read is only half-built; labels are the difference between a generated graph and a useful one.
Verify with queries, not screenshots. Green is the absence of evidence, not evidence of absence.

The platform is open source — gateway, two services, Kafka, the full observability stack, and this service-graph setup: https://github.com/Rummy43/ai-microservices-platform

What's the most expensive silent gap you've found by verifying instead of trusting the dashboard?

Originally published on Medium.