DEV Community

Cover image for Every Verdict Carries A Trace
Anton Brilliantov
Anton Brilliantov

Posted on

Every Verdict Carries A Trace

#testing #observability #devops #platformengineering

Every Verdict Carries A Trace - the cover: a verdict row with a 32-character trace id

Every Verdict Carries A Trace

A readiness pass on the stand that started with less than half green, the defects it caught before production, telemetry built from almost nothing, and a QA gate that rejects any verdict without a trace id.

👋 Hi, I'm Anton - a software engineer. These days I work on a platform: contracts, generated code, data paths and the tooling around them. Running notes live on my GitHub: github.com/brilliant-almazov.

The readiness pass, the fixes, the case file, the verdict row and the gate are real and checked by QA. The platform's telemetry contract is in review. Three matrices and a provider stub are proposals; two reds are still open.

This is part 6 of the series on testing cross-zone, cross-datacenter and cross-provider behaviour on a laptop. Maybe you already do it better, maybe you'd do it another way - this is how it is going, and where it stops.

Less than half green

Part 5, How Others Test Cross-Zone Locally, ended on a point from Jepsen: a cut is not a test, the check of the result is. This part is about our check - what a verdict looks like, and what makes it trustworthy.

It started with a readiness pass. Before any failure is injected, a stand has to be ready: services up, routes answering, logins working. QA counted the readiness checks on the local stand. At first, less than half were green.

Two halves: the first pass, mostly red - registration 500, admin UI 502, admin auth let a fake user in; after the fixes, most of the admin block green - registration accepted, admin UI 200, no cookie now 401

That is not a bad number. It is the reason a local stand exists: everything red there was red before production, not in it.

What the stand caught before production

Three of the reds are worth telling.

Registration returned 500. A step in the image build stripped developer tools to keep the image small, and took with it a package that the service needed at run time. The fix was one line. Registration in the monolith is now accepted, and the confirmation mail is caught by a local mail catcher instead of going anywhere.

A strip of three panels: clean-up, an image step strips dev tools; a package the runtime needed went too; registration 500; then one line, fixed, the mail lands in the local catcher

The admin was unreachable. Its router was on the wrong port, and the admin UI returned 502. Both are fixed; the UI answers 200.

The admin let in a fake developer user. Without a session cookie, the admin's auth check fell through to a development user instead of refusing. It now answers 401. I describe it only at this level, and it is closed.

None of the three would have been found by unit tests. Each needed the whole thing up, wired the way production is wired.

Telemetry from almost nothing

The stand's observability was almost empty, and a verdict without telemetry is an opinion. What was added:

  • env and service labels on every metric series, with a switch;
  • Loki and a log shipper;
  • tracing in the environment router, with traceparent written to the access log;
  • a check command that says whether all of the above is there.

A flow: env and service labels, Loki and a shipper, router tracing with traceparent, a check command; a pill reads built

Every call now carries a trace id from the environment router, exported over OTLP to the trace store. Two services still write no spans of their own, so their part of a trace is a gap; that is open.

The platform side

The platform change that makes this uniform is in review, not shipped. By design:

  • one environment name, APP_ENV, feeds the env label of metrics, the path in the configuration and secret store, and deployment.environment.name of traces;
  • identity labels service, env, instance, version are set once, at the metrics exporter, so every series carries them without any service adding them;
  • generated dashboards get env and service selectors first;
  • a log contract for every line: 
{
  "timestamp": "<ISO 8601>",
  "message": "<text>",
  "service": "<service>", "env": "<env>", "instance": "<instance>", "version": "<version>",
  "trace_id": "<from the context>", "span_id": "<from the context>"
}
Enter fullscreen mode Exit fullscreen mode

Written to stdout, never sampled.

A dark JSON card with one log line by the contract: timestamp, message, service, env, instance, version, trace id and span id from the context; the trace id line highlighted; a pill reads in review

A case is a file

A test case on the stand is one YAML file. It says what is tested, how badly a failure hurts, how it behaves in regression, and how to undo it:

id: SC-<NNN>
title: <what is checked>
weight: red | orange | yellow
regression: must-pass | may-drift | known-broken
surface: <host variable>
account: <role>
preconditions: [<fixture>]
steps:
  - { method: POST, path: "<host>/<path>", body: <verbatim> }
expectation: { status: <code>, error_class: <machine reason>, fields: [...], read_after_write: true }
rollback: snapshot → act → restore
Enter fullscreen mode Exit fullscreen mode

A declarative runner executes the file and writes every request and response to an io.jsonl journal.

A dark card with the case form: id, title, weight, regression mode, surface, account, preconditions, steps, expectation, rollback; the expectation line highlighted; a pill reads built

The regression field is the honest one. known-broken says out loud that a case is red on purpose, so a red run is not a surprise and a green one is.

A verdict is a row

The result of a case is one row, and the row has a fixed shape:

| NN · case | ✅ ❌ 🟡 ⏸ | "<status> — what came, what was expected, where the decision is wrong" | trace_id <32 hex> |
Enter fullscreen mode Exit fullscreen mode

The third column is a sentence with three parts, not a status code. A partial result carries its share: "N of M = X %". The fourth column is the trace.

A dark card with three verdict rows: a pass, a fail with what came and what was expected, a partial with N of M; each ending in a trace id; the fail row highlighted

No trace, no verdict

The QA gate rejects a verdict row without a trace_id. Not a warning - the row does not go in.

An animation: a verdict row without a trace id slides into the gate and is rejected in red; then the same row with a 32-character trace id slides in and is accepted in green

Where a service gives no trace at all, the row may carry an X-Request-Id with a reason - and that gap is closed later by tracing at the proxy, not by writing more text. The rule is the point: anyone reading a verdict can open the exact request that produced it.

One more piece is planned: the scenarios stay declarative YAML on the same runner, and a separate trace-analysis tool, in Rust with Polars, pulls the traces from the trace store, checks the expectations, flags anomalies - spans that hang, retries without a bound, errors hidden behind a 2xx - and hands the agent only a compact summary.

Images and data

Two smaller things keep the stand honest. Each service's image source is chosen per service, dev or tag:<version>, built read-only from a sibling clone, with one extra layer for stand fixes on top.

The data is synthetic, generated from the archives of earlier runs - no production dataset. The generator is deterministic and idempotent: a repeat run adds nothing.

Two halves: a production dataset - none; generated from earlier runs, deterministic, a repeat run adds nothing

Proposals

Three forms from QA are proposals, not built:

  • a role × flow matrix - roles root, admin, specialist, user, no token and a broken token, against login, token, the webhook-to-cache binding, reading the main project, writing, and admin; each cell holds the status the design expects - 200, 403 with a machine reason, 401 - and a trace;
  • a daemon × queue failure matrix - a row per service, daemon (from its manifest), queue and action (stop, kill, delay, restart); recorded: what piled up, what was lost (any loss is ❌), dead-letter and retries per the contract, time to drain. No thresholds - the owner set none; facts only;
  • a local provider stub - network aliases on the providers' hostnames, which are literals in config and code; TLS from a local CA mounted in the trust store; daemons on an internal network with no way out; provider callbacks posted from the stub to the service's webhook. Its acceptance line: "went to the stub, 0 outbound connections", with the stub's counter and a trace.

Three dashed tiles: role by flow, expected status and a trace per cell; daemon by queue, piled up, lost, DLQ, drain time; provider stub, aliases, local CA, no way out, 0 outbound; a pill reads a proposal

Still red

Two reds are open, named as open:

  • the binding from a webhook through the monolith to the cache is not measured yet;
  • two services write no spans of their own.

Three tiles: webhook to monolith to cache, not measured, open; two services without spans, open; the admin dev user, closed

What I do not claim

  • Not that the stand is all green. It started less than half green; the admin block is mostly green now, two reds are open.
  • Not that the platform's telemetry contract is shipped. It is in review.
  • Not that the matrices or the provider stub exist. They are proposals.
  • Not that every service traces itself. Two do not yet.

What of this is method

  • Ready before broken. A readiness pass comes before any chaos.
  • A verdict is a sentence and a trace. What came, what was expected, where the decision is wrong - and the request that proves it.
  • No trace, no verdict. The gate refuses, not the reviewer.

Cross-anything on a laptop - Part 6. Next: Clouds On A Laptop.

If your test reports link to traces, who checks the link is there? If you mark cases known-broken, how long do they stay that way? If you found a defect only once the whole stand was up, what was it? What would your QA gate refuse first?

Top comments (0)