Placement In Every Trace

Four variables from the topology - provider, region, site, zone - become trace attributes, metric labels and log fields, so a chaos run can be read by where things ran. Metadata only; it routes nothing.

---

👋 Hi, I'm Anton - a software engineer. These days I work on a platform: contracts, generated code, data paths and the tooling around them. Running notes live on my GitHub: github.com/brilliant-almazov.

The four variables are built and set on every copy. The platform change that turns them into telemetry is in review: unit tests are green, the full CI run is still pending. Nothing below is shipped.

This is part 4 of the series on testing cross-zone, cross-datacenter and cross-provider behaviour on a laptop. Maybe you already do it better, maybe you'd do it another way - this is how it is going, and where it stops.

---

Why placement belongs in telemetry

Part 3, A Planet On A Laptop, placed every copy of a service on a planet and gave the network between them distance. That opens a question the old telemetry cannot answer: when a zone goes down during a run, which spans were in it?

A trace that says "the gateway called the store, and it was slow" is half a fact. The other half is where each end was. Without it, a chaos run produces a pile of traces that cannot be sorted by the one thing the run changed.

Four variables

Placement already tells every copy where it is. Each gets four environment variables:

APP_PROVIDER=prov-a
APP_REGION=eu-frankfurt
APP_SITE=dc-a1
APP_ZONE=dc-a1-z1

They come from the deterministic placement of part 3 - round-robin over providers, datacenters, zones - so the same declaration always puts the same copy in the same place. That part is built.

Three signals

The platform reads the four variables once at start and writes them into all three signals:

Variable	Trace attribute	Metrics	Logs
APP_PROVIDER	cloud.provider	a platform_info label	a field on every line
APP_REGION	cloud.region	a platform_info label	a field on every line
APP_SITE	a site attribute	a platform_info label	a field on every line
APP_ZONE	cloud.availability_zone	a platform_info label	a field on every line

On a span's resource it looks like this:

{
  "service.name": "<service>",
  "deployment.environment.name": "<env>",
  "cloud.provider": "prov-a",
  "cloud.region": "eu-frankfurt",
  "cloud.availability_zone": "dc-a1-z1",
  "<site attribute>": "dc-a1"
}

With that in every span, a trace of a request that crossed zones shows the crossing. The waterfall below is the picture we want to read after a run:

What the convention has, and what it does not

Three of the four names are not ours. OpenTelemetry's semantic conventions define a cloud resource with cloud.provider - the name of the cloud provider - cloud.region - the geographical region within a provider - and cloud.availability_zone - the zone where the resource runs (OpenTelemetry semconv). Using them means any backend that knows the convention can group by them without being taught.

Two caveats, both from the same page. The cloud attributes are marked Development, not stable. And there is no attribute for a site - a datacenter inside a region. That one is ours, and it is named plainly as a site attribute until the convention has something better.

Continent is not on the list at all. It is an attribute of the stand's topology, not of a running copy: QA's proposal keeps it there.

Metadata only

This is the part most likely to be misread, so it gets its own section. The four variables change nothing about where a request goes. No router reads them, no client prefers a closer copy, no load balancer weights by zone.

Placement-aware routing is a real design and a different one. Mixing it into a telemetry change would make every run answer two questions at once. This change answers one: where did it run.

One environment name

The same change fixes a smaller thing that kept getting in the way. An environment had several names in several places. Now one variable, APP_ENV, feeds all three:

• the env label on every metric;
• the path in the configuration and secret store;
• deployment.environment.name on every trace.

A run on qa-1 can no longer write metrics as qa, read secrets of dev, and trace as something else.

What it is for

The point of all this is reading a chaos run by where things ran: filter the traces of a "zone down" scenario by cloud.availability_zone, and see who noticed and who did not. That reading needs the scenarios to run, and they are planned. QA's proposal already lists the placement attributes a run report should carry: deployment.environment.name, cloud.provider, cloud.region, site, cloud.availability_zone.

What I do not claim

• Not that this is shipped. It is in review; the full CI run is pending.
• Not that the cloud attributes are stable. The convention marks them Development.
• Not placement-aware routing. This is metadata only.
• Not a chaos run read by zone. The runs are planned.

What of this is method

• Placement is a fact about every signal. Traces, metrics and logs carry the same four fields.
• Use the convention where it exists. Name your own only where it does not.
• Metadata is not routing. One change, one question.

---

Cross-anything on a laptop - Part 4. Next: How Others Test Cross-Zone Locally.

If you already put cloud.availability_zone on spans, what do you put for a site? If you route by zone, how do you keep it apart from telemetry? If you have one environment with three names, which one wins? Which question would you ask first of a trace from a chaos run?