DEV Community: Anton Brilliantov

How Others Test Cross-Zone Locally

Anton Brilliantov — Fri, 06 Nov 2026 05:04:07 +0000

How Others Test Cross-Zone Locally

tc netem, Toxiproxy, Pumba, Chaos Mesh, LitmusChaos, zone labels in kind and k3d, Jepsen and Testcontainers: what each gives, a short config form for each, and an honest comparison with what we built.

👋 Hi, I'm Anton - a software engineer. These days I work on a platform: contracts, generated code, data paths and the tooling around them. Running notes live on my GitHub: github.com/brilliant-almazov.

Every claim about a tool below links to its own documentation, read for this article. Our side of the comparison is what part 3 built; our failure-scenario runs are still planned.

This is part 5 of the series on testing cross-zone, cross-datacenter and cross-provider behaviour on a laptop. Maybe you already do it better, maybe you'd do it another way - this is how it is going, and where it stops.

Five layers

Part 4, Placement In Every Trace, made every span say where it ran. Before going further with our own emulation, it is worth looking at what the world already uses. The tools sort into layers by where they act:

Layer	Tools	Acts on
kernel	tc netem	packets on a device
TCP proxy	Toxiproxy, Testcontainers' Toxiproxy module	one connection through a proxy
container	Pumba	a container's namespace, from outside
Kubernetes	Chaos Mesh, LitmusChaos, zone labels in kind and k3d	pods selected by labels
history	Jepsen	what the system did, checked afterwards

The lower the layer, the more it can do and the less it knows about your system. The higher, the more it knows and the more it needs installed.

tc netem

Everything network-shaped in this list ends at the kernel's netem sooner or later. It adds delay, jitter and loss to packets leaving a device (tc-netem(8)):

tc qdisc change dev eth0 root netem delay 100ms 20ms distribution normal
tc qdisc change dev eth0 root netem loss 0.3% 25%

The man page has one warning worth repeating: for TCP performance tests, netem should sit on the ingress of the receiver, not on the sender. On its own, netem knows nothing about which peer is which; that is up to the filters around it.

Toxiproxy

Toxiproxy is "a TCP proxy to simulate network and system conditions for chaos and resiliency testing" (Toxiproxy). The client connects to the proxy instead of the real service, and toxics are added to the proxy over an HTTP API on port 8474:

toxiproxy-cli create -l localhost:26379 -u localhost:6379 redis
toxiproxy-cli toxic add -t latency -a latency=1000 redis

The toxics are latency, down, bandwidth, slow_close, timeout, reset_peer, slicer, limit_data and packet_loss. That is a richer vocabulary than netem has, at one price: the client must be pointed at the proxy.

Pumba

Pumba does chaos and network emulation for containers on Docker, containerd and Podman (Pumba). It runs tc from a helper that joins the target's namespace, so the target image needs nothing:

pumba netem --duration 5m delay --time 3000 mydb
pumba iptables --duration 2m loss --probability 0.1 myapp

That is the same trick as our sidecar in part 3, packaged as a tool.

Chaos Mesh and LitmusChaos

In Kubernetes, network chaos becomes a resource. Chaos Mesh's NetworkChaos supports netem, delay, loss, duplicate, corrupt, partition and bandwidth, selects pods by label and has a direction of to, from or both (Chaos Mesh):

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata: { name: delay-example }
spec:
  action: delay
  mode: one
  selector: { namespaces: [default], labelSelectors: { app: frontend } }
  delay: { latency: '20ms', jitter: '5ms', correlation: '0' }

LitmusChaos, a CNCF Incubating project, describes itself as "a Cloud-Native Chaos Engineering Framework with cross-cloud support" (Litmus). Its pod-network-latency experiment starts tc with netem inside the target, takes NETWORK_LATENCY, JITTER and DESTINATION_IPS or DESTINATION_HOSTS to aim at one peer, and can use Pumba as its library (pod-network-latency).

Zones in kind and k3d

Zones on a laptop cluster are node labels. kind takes them in the cluster config (kind); k3d takes them on the command line with a node filter (k3d):

# kind
nodes:
  - role: worker
    labels: { topology.kubernetes.io/zone: a }
  - role: worker
    labels: { topology.kubernetes.io/zone: b }

# k3d
k3d cluster create --agents 3 \
  --k3s-node-label "topology.kubernetes.io/zone=a@agent:0" \
  --k3s-node-label "topology.kubernetes.io/zone=b@agent:1" \
  --k3s-node-label "topology.kubernetes.io/zone=c@agent:2"

Then topologySpreadConstraints spread the pods. DoNotSchedule tells the scheduler not to place a pod that would break the skew; ScheduleAnyway places it anyway, preferring nodes that reduce the skew (Kubernetes):

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector: { matchLabels: { app: my-app } }

Labels and spread give zones as a scheduling fact. They give no distance: two "zones" on a laptop are as close as two processes.

Jepsen

Jepsen is at a different layer altogether. It "aims to improve the safety of distributed databases, queues, consensus systems" (Jepsen). While a test runs operations against a system, a nemesis process introduces faults, and a checker then analyses the recorded history for correctness (jepsen-io/jepsen). Network manipulation in its tests is typically done with iptables.

What Jepsen adds is not a better way to cut a network. It is a rigorous answer to "was the result correct", which every tool above leaves to you.

Testcontainers

For tests in code, Testcontainers wraps Toxiproxy as a module. A ToxiproxyContainer joins the same network as the service under test, a proxy is created in front of it, and toxics are added from the test (Testcontainers):

proxy.toxics().latency("latency", ToxicDirection.DOWNSTREAM, 1_100).setJitter(100);
proxy.toxics().bandwidth("CUT_CONNECTION_DOWNSTREAM", ToxicDirection.DOWNSTREAM, 0);
proxy.toxics().bandwidth("CUT_CONNECTION_UPSTREAM", ToxicDirection.UPSTREAM, 0);

A connection cut is bandwidth zero in both directions; removing the toxics restores it.

Side by side

	What the world gives	What we built
latency and loss	netem directly; Pumba, Chaos Mesh, Litmus on top of it; Toxiproxy per connection	the same netem, but every rule derived from one topology file, per pair of sites
zones	node labels in kind and k3d, spread constraints	the same in k3d, plus compose with a subnet per site and zone
continents and providers	not a concept in any of these tools	levels of the declaration, with link kinds derived from them
partitions	Chaos Mesh partition, Jepsen's nemesis, Toxiproxy down	iptables DROP on both sides, each cut remembering its cause
checking the result	Jepsen's checker on the history	QA thresholds declared; the runs are planned
what we lack	bandwidth limits, a Jepsen-grade history check	—

The honest summary: we did not invent a new way to delay a packet. Under the hood it is netem and iptables, like most of the list. What is ours is the model above them - one declaration of a planet, from which every rule, every placement and every chaos action is derived. None of the tools above has a continent or a provider as a concept.

What we would borrow

Two things, both from the top of the list. Toxiproxy's vocabulary - slow_close, reset_peer, limit_data - names failures that netem cannot express. And Jepsen's stance: a chaos run without a strict check of the result is a demonstration, not a test. Both are ideas, not plans yet.

What I do not claim

Not a ranking of these tools. Each is good at its own layer.
Not that our emulation does what Jepsen does. It does not check histories.
Not that any of our scenarios has run. They are planned.
Not completeness. These are the tools I read for this part, not all of them.

What of this is method

Know the layer of a tool. Kernel, proxy, container, cluster, history - each sees a different system.
Read the documentation, not the reputation. Every claim here has a link.
A cut is not a test. The check of the result is the test.

Cross-anything on a laptop - Part 5. Next: Every Verdict Carries A Trace.

If you use Toxiproxy in tests, which toxic found the most? If you run Chaos Mesh or Litmus, do you run it locally or only on a shared cluster? If you have run Jepsen, what did the checker catch that nothing else would? Which of these would you add to a laptop first?

Placement In Every Trace

Anton Brilliantov — Thu, 05 Nov 2026 09:35:07 +0000

Placement In Every Trace

Four variables from the topology - provider, region, site, zone - become trace attributes, metric labels and log fields, so a chaos run can be read by where things ran. Metadata only; it routes nothing.

The four variables are built and set on every copy. The platform change that turns them into telemetry is in review: unit tests are green, the full CI run is still pending. Nothing below is shipped.

This is part 4 of the series on testing cross-zone, cross-datacenter and cross-provider behaviour on a laptop. Maybe you already do it better, maybe you'd do it another way - this is how it is going, and where it stops.

Why placement belongs in telemetry

Part 3, A Planet On A Laptop, placed every copy of a service on a planet and gave the network between them distance. That opens a question the old telemetry cannot answer: when a zone goes down during a run, which spans were in it?

A trace that says "the gateway called the store, and it was slow" is half a fact. The other half is where each end was. Without it, a chaos run produces a pile of traces that cannot be sorted by the one thing the run changed.

Four variables

Placement already tells every copy where it is. Each gets four environment variables:

APP_PROVIDER=prov-a
APP_REGION=eu-frankfurt
APP_SITE=dc-a1
APP_ZONE=dc-a1-z1

They come from the deterministic placement of part 3 - round-robin over providers, datacenters, zones - so the same declaration always puts the same copy in the same place. That part is built.

Three signals

The platform reads the four variables once at start and writes them into all three signals:

Variable	Trace attribute	Metrics	Logs
`APP_PROVIDER`	`cloud.provider`	a `platform_info` label	a field on every line
`APP_REGION`	`cloud.region`	a `platform_info` label	a field on every line
`APP_SITE`	a site attribute	a `platform_info` label	a field on every line
`APP_ZONE`	`cloud.availability_zone`	a `platform_info` label	a field on every line

On a span's resource it looks like this:

{
  "service.name": "<service>",
  "deployment.environment.name": "<env>",
  "cloud.provider": "prov-a",
  "cloud.region": "eu-frankfurt",
  "cloud.availability_zone": "dc-a1-z1",
  "<site attribute>": "dc-a1"
}

With that in every span, a trace of a request that crossed zones shows the crossing. The waterfall below is the picture we want to read after a run:

What the convention has, and what it does not

Three of the four names are not ours. OpenTelemetry's semantic conventions define a cloud resource with cloud.provider - the name of the cloud provider - cloud.region - the geographical region within a provider - and cloud.availability_zone - the zone where the resource runs (OpenTelemetry semconv). Using them means any backend that knows the convention can group by them without being taught.

Two caveats, both from the same page. The cloud attributes are marked Development, not stable. And there is no attribute for a site - a datacenter inside a region. That one is ours, and it is named plainly as a site attribute until the convention has something better.

Continent is not on the list at all. It is an attribute of the stand's topology, not of a running copy: QA's proposal keeps it there.

Metadata only

This is the part most likely to be misread, so it gets its own section. The four variables change nothing about where a request goes. No router reads them, no client prefers a closer copy, no load balancer weights by zone.

Placement-aware routing is a real design and a different one. Mixing it into a telemetry change would make every run answer two questions at once. This change answers one: where did it run.

One environment name

The same change fixes a smaller thing that kept getting in the way. An environment had several names in several places. Now one variable, APP_ENV, feeds all three:

the env label on every metric;
the path in the configuration and secret store;
deployment.environment.name on every trace.

A run on qa-1 can no longer write metrics as qa, read secrets of dev, and trace as something else.

What it is for

The point of all this is reading a chaos run by where things ran: filter the traces of a "zone down" scenario by cloud.availability_zone, and see who noticed and who did not. That reading needs the scenarios to run, and they are planned. QA's proposal already lists the placement attributes a run report should carry: deployment.environment.name, cloud.provider, cloud.region, site, cloud.availability_zone.

What I do not claim

Not that this is shipped. It is in review; the full CI run is pending.
Not that the cloud attributes are stable. The convention marks them Development.
Not placement-aware routing. This is metadata only.
Not a chaos run read by zone. The runs are planned.

What of this is method

Placement is a fact about every signal. Traces, metrics and logs carry the same four fields.
Use the convention where it exists. Name your own only where it does not.
Metadata is not routing. One change, one question.

Cross-anything on a laptop - Part 4. Next: How Others Test Cross-Zone Locally.

If you already put cloud.availability_zone on spans, what do you put for a site? If you route by zone, how do you keep it apart from telemetry? If you have one environment with three names, which one wins? Which question would you ask first of a trace from a chaos run?

A Planet On A Laptop

Anton Brilliantov — Wed, 04 Nov 2026 09:20:26 +0000

A Planet On A Laptop

Continents, providers, datacenters and zones declared in one file, turned into netem and iptables rules inside every container, broken at every level on purpose - and a measured round trip that came out close to what was declared.

The topology emulation is built and has unit tests; placement and links were applied on a laptop and a round trip was measured. The failure scenarios on top of it are written and not run - the owner postponed them.

This is part 3 of the series on testing cross-zone, cross-datacenter and cross-provider behaviour on a laptop. Maybe you already do it better, maybe you'd do it another way - this is how it is going, and where it stops.

One file

Part 2, Four Failure Domains, ended with a ladder: zone, datacenter, provider, continent. This part puts that ladder on a laptop.

Everything starts from one declaration file. It is switched on by an environment variable, and only in the high-availability profile - a normal developer environment does not pay for it. The form, with cities and round trips as an example:

continents:
  eu:
    providers:
      prov-a:
        dc-a1: { city: Frankfurt, region: eu-frankfurt, zones: [dc-a1-z1, dc-a1-z2] }
        dc-a2: { city: Amsterdam, … }
      prov-b:
        dc-b1: { city: Paris, … }
  us:
    providers:
      prov-b:
        dc-b2: { city: Virginia, … }
links:
  intra_dc: { rtt_ms: 0.5, jitter: 0.1, loss: 0 }
  pairs:
    dc-a1~dc-a2: { rtt_ms: 11, jitter: 1, loss: 0 }
    dc-a1~dc-b2: { rtt_ms: 87, jitter: 3, loss: 0.1 }
clients:
  client-eu: { continent: eu, dc: dc-a1, rtt_ms: 20, jitter: 2 }
placement: {}   # override <service|cluster>: [zone, …]

Where the numbers come from

The round trips in the file are not guesses. Each is the median of public ping data between the two cities, cross-checked against a second public source. The declaration says what the internet between those cities looks like, not what we wish it did.

The kind of every link is derived from where its ends are, in the order from part 2: intra-dc, intra-metro, intra-continent, inter-continent. Continent ranks above provider; provider is only same or cross.

Placement is deterministic: copies are dealt round-robin over providers, then datacenters, then zones. Three copies land in three datacenters; a fourth goes to the other continent. Each copy is told where it is through four variables - APP_PROVIDER, APP_REGION, APP_SITE, APP_ZONE - and part 4 is about what happens to them next.

From a declaration to rules

The declaration becomes a model, and the model becomes rules inside every container.

The rules are applied from a sidecar that joins the container's network namespace - so no service image needs tc, iptables or extra capabilities:

docker run --rm --net container:<name> --cap-add NET_ADMIN nicolaka/netshoot \
  sh -c '
    tc qdisc add dev eth0 root handle 1: prio bands <one per peer>
    tc qdisc add dev eth0 parent 1:<band> netem delay <rtt/2> <jitter> loss <loss>
    tc filter add dev eth0 parent 1: protocol ip u32 match ip dst <peer subnet> flowid 1:<band>
  '

A prio qdisc gives one band per peer; netem in the band adds half the round trip, the jitter and the loss; a u32 filter sends traffic for the peer's subnet into its band. The mirror rule on the other side adds the other half, so the round trip equals the declared one. The netem syntax is the kernel's own (tc-netem(8)); the sidecar image is netshoot.

The rules are re-applied on every up, start and restart, because a recreated container comes back with a fresh namespace and none of them.

A cut is a DROP

A link cut is not a stopped container. It is a rule on both sides:

iptables -I OUTPUT -d <peer subnet> -j DROP

Both ends stay alive and both believe they are fine; only the path between them is gone. That is the case a stopped container never shows.

Compose: one network, many subnets

In compose, the whole planet is one docker network. Each datacenter gets its own /20, each zone its own /24, and every container a static address in a generated overlay.

A network per datacenter looked cleaner and was rejected: docker's DNS is per network, so service names would stop resolving across datacenters. One network with subnets keeps every name working and still gives every link its own address range to match on.

k3d: a node per zone

In k3d, each zone is one agent node, labelled with topology.kubernetes.io/region and topology.kubernetes.io/zone plus provider and datacenter labels. The spread is two constraints with different strictness:

topologySpreadConstraints:
  - { maxSkew: 1, topologyKey: <datacenter label>,        whenUnsatisfiable: DoNotSchedule }
  - { maxSkew: 1, topologyKey: topology.kubernetes.io/zone, whenUnsatisfiable: ScheduleAnyway }

Across datacenters the scheduler must spread; across zones it should (Kubernetes). The same netem and iptables rules run inside the node containers.

Why not a real VPN

In production the links between datacenters are a VPN. WireGuard was considered and rejected: it would need NET_ADMIN and route changes in every service image. The emulation gives the links the latency and loss of a VPN without touching a single service.

Breaking it at every level

Every level of the ladder has its own action:

Action	What it does
`zone-down` / `dc-down` / `provider-down` and `-up`	stop and start the members; rules re-applied on start
`dc-isolate` / `dc-heal`	DROP from one datacenter to every other
`continent-isolate` / `continent-heal`	the same, for a continent
`link-down` / `link-up`	cut one pair
`link-degrade`	`tc qdisc change … netem` with a given rtt, loss and jitter
`link-restore`	back to the declared values

A scenario can aim at a role, instance: leader, rather than at a name. Every run records which latency variant it used, --latency none or --latency declared, so a report always says whether the network was there.

A cut remembers why

Cuts overlap. A link can be cut on its own, and a datacenter at one end can be isolated on top. Healing the datacenter must not quietly heal the link.

So every cut remembers its cause - link, datacenter or continent - and healing removes only the cuts of that cause. The state lives in a JSON file per environment.

Measured

Placement and links were applied on a laptop. A ping across a declared 11 ms link came out slightly above what was declared: the minimum almost exact, the average a little higher. Inside one zone, near zero.

The earlier store failover runs of Seven Ways To Break It On Purpose were done without any of this latency. That is worth saying out loud: their results hold for a flat network, and the next runs are the first with distance in them.

Written, not run

The scenarios are written. None has run - the owner postponed them:

a zone down under load, in compose and in k3d;
the zone of a database leader down;
a broker's or a cache's datacenter isolated;
a provider down;
one link down; one link degraded to a high round trip with loss; the transatlantic link degraded;
a continent cut off; an even split.

QA has declared what a pass would mean. It is a proposal, agreed with DevOps and postponed by the owner, and these numbers live only here, in the labelled block:

# HA thresholds - a proposal, agreed with DevOps, postponed by the owner
rtt_per_declared_link: ±20 %
leader_election:       ≤ 35 s
replication_lag:       { within_continent: ≤ 1 s, across: ≤ RTT×3 + 1 s }
rejoin:                ≤ 60 s
broker_elections:      recorded only
split_brain:           always ❌

They are declared, not met.

Not emulated

Bandwidth limits, BGP and anycast, a provider's DNS failing, clock skew between datacenters, and how operator-managed StatefulSets spread in k3d. Each is a known gap, not a forgotten one.

What I do not claim

Not that any failure scenario has run on this topology. They are written and planned.
Not that the thresholds are met. They are a proposal.
Not that a laptop network is a planet. The measured round trip came close to declared, slightly above, and that is all that was measured.
Not bandwidth, BGP, provider DNS or clock skew. They are not emulated.

What of this is method

Declare the planet once. Rules, placement and chaos all read one file.
Rules in the namespace, not in the image. A sidecar keeps every service untouched.
Cuts remember their cause. Healing one failure must not hide another.

Cross-anything on a laptop - Part 3. Next: Placement In Every Trace.

If you emulate latency with netem, do you set it per peer or per container? If you use a chaos tool for partitions, how does it heal overlapping cuts? If you run k3d with zone labels, what spreads and what does not? What is the first link on your planet you would cut?

Four Failure Domains

Anton Brilliantov — Tue, 03 Nov 2026 07:25:34 +0000

Four Failure Domains

Zone, datacenter, provider, continent: what a system feels at each level - latency, loss, a split, the whole level gone - and what the clouds themselves say about it.

This part is conceptual. The ordering of link kinds at the end is built; the failure-scenario runs across these levels are planned, not run.

This is part 2 of the series on testing cross-zone, cross-datacenter and cross-provider behaviour on a laptop. Maybe you already do it better, maybe you'd do it another way - this is how it is going, and where it stops.

One ladder

An availability zone, a datacenter, a provider and a continent sound like four separate topics. They are one ladder. Each step is a failure domain one size larger than the last, and each step up changes two things at once: how far apart the pieces are, and what can go wrong between them.

The value of seeing it as a ladder is that one design can cover all four. Copies of a service are placed on the ladder; the network between them is described by the ladder; failures are cut at a step of the ladder. Part 1, One Login For Every Stand, gave every local environment one way in. This part is the map of where the copies behind that door can live, and what each place costs.

Zone

A zone is the smallest domain that a cloud promises to keep independent. AWS describes an Availability Zone as one or more discrete data centers with separate and redundant power, networking and connectivity, far enough apart to avoid correlated failures and close enough for synchronous replication with single-digit millisecond latency (AWS).

For a system, a zone means:

latency - near zero; nobody designs around it;
loss of a zone - the copies in it are gone; a quorum spread over three zones survives one.

The second line is the whole reason for three. Two copies of three is a majority; one of two is not.

Datacenter and metro

The next step is a whole site, or a metro of sites. Latency is small but no longer zero, and it is the first step where a timeout chosen on a laptop can be wrong in production. Synchronous replication is still possible inside a metro; it is simply slower.

A datacenter lost is a zone lost, only bigger: everything in it at once, including the things that were assumed to be "elsewhere" because they were in another zone of the same site.

Provider

A provider is a different kind of step. It is not further away so much as differently owned. Everything a provider runs - its API, its DNS, its network, its control plane - can fail together, and when it does, every site of that provider goes with it.

For a link between two places, the provider question has only two answers: the same provider or a different one. That is the whole axis.

Continent

The top step is the one physics decides. Between continents the round trip is many times larger than inside a region, and no configuration makes it smaller. Google's guidance says it plainly: communication within regions will always be cheaper and faster than communication across regions (Google Cloud).

At this step synchronous replication stops being a design choice. The main risk becomes the split: a cut between two continents leaves two halves that both believe they are the system.

What a system feels

Put the four steps side by side and the effects line up:

Level	Latency	Loss on the link	A split	The whole level gone
zone	near zero	rare	unlikely	copies in it lost; a quorum of three zones holds
datacenter / metro	small	rare	possible	everything in the site at once
provider	depends on placement	depends on placement	possible	every site of that provider, its API and DNS
continent	many times larger	more likely	the main risk	half the system on the far side

The latency column reads as words, not numbers, on purpose. The order matters more than any single figure, and the order is the same in every cloud.

What the clouds say

Three providers, one picture:

AWS - Regions are isolated and independent from other Regions; this separation limits service failures to a single Region, and resources do not exist in another Region unless replicated (AWS).
Google - zones are designed to minimize correlated failures of power, cooling and networking; the recommendation is to deploy across multiple zones and multiple regions (Google Cloud).
Azure - a zone is a logical group of physically separate datacenters, the target round trip between zones is under about two milliseconds, and zones do not protect against the loss of a whole region (Azure).

None of them says "zones are enough". All three say: know which step of the ladder each copy is on.

A ladder a tool can read

For a test, the ladder has to become data. In our topology emulation, the kind of every link is derived from where its two ends are, in a fixed order:

intra-dc  <  intra-metro  <  intra-continent  <  inter-continent
provider:  same | cross            (continent ranks above provider)

Continent ranks above provider: two sites of different providers on one continent are closer than two sites of one provider on different continents. The provider is only ever "same" or "cross". That ordering is built; how it is used is part 3.

On a laptop

On a laptop each step of the ladder is a network: a subnet for a zone, a larger one for a site, a rule for every link between them. The failures become commands: take a zone down, isolate a site, cut a continent off. The runs of those scenarios are planned - the owner postponed them - so this part stops at the map.

What I do not claim

Not that any of these failures has been run across the levels. They are planned.
Not exact latencies for any level. The order is the point; the numbers belong to each cloud's documentation.
Not that a laptop network behaves like a cloud network. It is an emulation, and part 3 says how close it got.

What of this is method

One ladder, four steps. Zone, site, provider and continent are one design, not four.
Know the step of every copy. A quorum is only as good as where its members are.
Make the ladder data. A test tool can only break what it can read.

Cross-anything on a laptop - Part 2. Next: A Planet On A Laptop.

If you spread copies across zones, do you know which site each zone is in? If you run on two providers, what fails together that you thought would not? If you have lived through a continent split, which side kept writing? Which step of the ladder does your system assume will never fail?

One Login For Every Stand

Anton Brilliantov — Mon, 02 Nov 2026 06:39:01 +0000

One Login For Every Stand

One self-hosted Supabase auth stand for every local environment: one door, four roles, a token audience that had to become a string, and the three answers the gateway gives.

The auth stand, its four roles, its webhooks and the gateway's answers have been checked by QA. How the monolith picks the stand up was established by reading its code; the switch between local and production auth is planned.

This is part 1 of the series on testing cross-zone, cross-datacenter and cross-provider behaviour on a laptop. Maybe you already do it better, maybe you'd do it another way - this is how it is going, and where it stops.

Why one login

Before anything can be broken across zones, something has to log in. Every local environment needs users, tokens, roles and the webhooks that production's auth fires on every change. The easy answer is a copy of the auth stack inside every environment. That answer gives as many auth stacks as environments, each slightly different, and each one a place where a token is shaped a little unlike production.

The answer we took is the opposite: one shared, self-hosted auth stand, and every local environment logs in against it. One place shapes tokens, one place holds the keys, one place fires the webhooks. When something about a token is wrong, it is wrong once and fixed once.

The stand is built on the official Supabase self-hosting setup with docker compose (Supabase docs). Nothing in it is our own auth code. What is ours is the door, the keys, the roles and how environments are told apart.

One door

The stand has exactly one door, and it is Traefik. The API gateway of the Supabase stack - Kong, in the version we run - stays inside, with no published port.

Kong cannot simply be dropped. Rest, storage and realtime are reached through it, and without Kong inside, the sb_ API keys are rejected. So the design keeps the stack whole and changes only what faces the outside: one router in front, everything else private.

The stand now sits behind the shared edge router under a *.localhost name, the same way every other local service will. QA checked it there: the JWKS endpoint answers 200, and a token is issued with 200 for each of the four roles.

Keys and four roles

Tokens are signed with ES256 and carry a kid. The key id is what lets a verifier pick the right public key from a JWKS without a shared secret, and what lets keys rotate without everyone being told at once.

Roles come from a token hook. It issues four of them, each with the user_role claim the services read:

Role	`user_role`	Checked
root	root	token and refresh
admin	admin	token and refresh
specialist	specialist	token and refresh
user	user	token and refresh

All four were checked by QA end to end: a token issued, used, and refreshed.

An audience that had to be a string

One fix in the stand is worth its own section, because it is the kind of thing that only shows up when two systems meet.

The JWT standard allows aud to be a string or an array of strings. A token from the stand arrived with aud as an array. The monolith expected a string. In normal mode it cast the array to the literal text Array and compared that; in debug mode it threw.

The stand now forces the audience to a single string, and every check QA runs asserts it. The monolith also verifies tokens offline, against the exact issuer - another thing QA checked - so a token from the wrong issuer fails without a network call.

The lesson is not about Supabase. A local stand is where two parts of a system agree on a format for the first time outside production. Better there than in production.

Webhooks as in production

Production's auth fires database webhooks on its tables, and services depend on them. A local stand without them would be a stand that logs in but never tells anyone.

So the stand has database webhooks on all auth and public tables, as production does. Because one stand serves many environments, each webhook is routed to the environment it belongs to.

What the gateway answers

The gateway is the first thing that reads a token. It now gives exactly three answers, each with a trace:

200  token valid             trace_id=<32 hex>
401  token_missing           trace_id=<32 hex>
401  token_invalid           trace_id=<32 hex>

How the gateway gets there I know by the code, not from a measurement. It takes the auth key only as an inline JWKS, with no network fetch; it picks the issuer by the exact iss; it maps the four roles with a hierarchy, root above admin above specialist above user. A token whose key is missing is a 401. A write needs one more thing: the user's identity chain in the cache.

The two 401s are different on purpose. "No token" and "a token that does not verify" are different defects with different owners, and the machine reason says which one it is without anyone opening a log.

The monolith side

This part I know from reading code, not from a run, and I say it that way.

The monolith supports more than one auth integration. A second integration, local, is picked up as a mounted configuration file - no code change. Its JWT secret is a JWKS rather than a shared secret, which is what makes ES256 with kid work. The webhook address for that integration is a route plus a hashid of the integration's id.

# a second integration as a mounted file - established by reading code
integration: local
jwt_secret: <JWKS of the auth stand>
webhook:    <route>/<hashid of the integration id>

What is next

Two things are not done:

a switch, ENV_AUTH=local|production, so an environment chooses where it logs in with one variable - planned;
production-shaped names <service>.<env>.localhost for the other services behind the one edge router, several environments at once - in progress. The auth stand already lives under such a name.

What this buys a team

One stand instead of one per environment: fewer copies, one place where token shape is decided, and defects of the kind the array audience was found before production. The cost is a shared dependency - if the stand is down, every local environment is logged out together. For a laptop that is a fair trade.

With AI coding agents, the plumbing - a token hook, a router rule, a webhook route - is quick to write. Deciding that there is one stand, not many, is the part that is not typing.

What I do not claim

Not that the monolith side or the gateway's key handling was run. Both were established by reading code.
Not that the switch between local and production auth exists. It is planned.
Not that every service already has a .localhost name. Only the auth stand does; the rest are in progress.
Not that a shared stand fits every team. It is a shared dependency.

What of this is method

One stand, not many. A format decided once is a format that cannot drift.
One door. Keep the vendor's stack whole; change only what faces outside.
Two kinds of 401. A machine reason names the defect before anyone opens a log.

Cross-anything on a laptop - Part 1. Next: Four Failure Domains.

If you run a copy of auth inside every environment, how do you keep them alike? If you share one auth stand, what happens to everyone when it is down? If you have been bitten by an array aud, where did it show up first? Which part of your auth would you rather find broken on a laptop?

Anatomy of 32 Days

Anton Brilliantov — Sun, 01 Nov 2026 06:00:16 +0000

Every number here was cheap to collect and late to arrive. That gap is the whole story.

👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services. The seven parts before this one each took one angle on a single write path that left the monolith. This one is the ledger: the same 32 days read as one timeline, with every figure in one place and a link to the part that unpacks it. Running notes live on my GitHub: github.com/brilliant-almazov.

This is how I do it right now, with the price attached - maybe you already do it better, maybe you'd do it another way.

The path in one paragraph

A user saves a set of matching rules; every domain under that account gets marked by them. Four participants: the Go API gateway as the only door from outside, the rule-set-markup service which takes the write and folds the cascade, the domain-rule-map service which owns rule-set identity and recalculates the map, and PostgreSQL behind them - the monolith's database for the rule data, the identity service's own for identity.

One user action, four processes, two databases. Accepting that as working took 32 days, and this is what was inside them.

What "32 days" actually contains

The headline number is the least useful one in the set, so it is worth breaking before anything is built on it.

  DAYS   32 ┤████████████████████████████████  the acceptance timeline
            │        ▼ −21  automated verdicts only: pass/fail per case, no numbers, no traces
         11 ┤███████████
            │        ▼ −2   two days with no verdict at all — the wrong thing was under test
          9 ┤█████████
            │        ▼ −2   days inside the detailed window with no runs
          7 ┤███████  ◄ days with detailed, traced runs
  ──────────┴──────────────────────────────────────────────────────
  MINUTES inside those 7 days — 985 total = 16 h 25 min
    day 1 ██████████████████ 178      day 5 ██████████████████ 179
    day 2 ███████████████    150      day 6 ███████            77
    day 3 ██████████         102      day 7 ████████           89
    day 4 █████████████████████ 210   ◄ one run alone = 120

34 runs across the period, 28 of them numbered. Duration was recorded for 29 of the 34: mean 34 minutes, longest 2 hours, shortest 2 minutes. So the honest version of the headline is not "32 days of testing" - it is 16 hours 25 minutes of measured work, spread over 7 days, inside a 32-day window.

I am starting here because every ratio later in this article has one of these denominators in it, and the wrong denominator turns a modest finding into a dramatic one for free.

Three windows, three instruments

The 32 days are not one measurement. They are three, taken with different instruments, and no line crosses between them honestly.

  window 1   before day 0        no acceptance artefacts exist at all
                                 behaviour reconstructed from migrations, commits, docs
  window 2   day 0 … day 21      automated verdicts per case, per deploy tag
                                 pass / fail, and nothing else — no timings, no traces
  window 3   day 21 … day 32     traced runs: timings, state dumps, statement text
                                 the numbered defect registry starts on day 25

Which produces the reading rule I put on every chart in this material:

Comparing across those windows compares methods of observation, not states of the system.

Two more caveats travel with it, and they are in every part of this series for the same reason: round-trip time to the environment sits at a steady 220-260 ms regardless of endpoint, so it is never part of anyone's work; and a share is never published without its absolute value - 440.0 ms · 61 %, never 61 % on its own.

Day one of the detailed window: three objects, everything passes

For the first weeks the checks ran against a playground of three objects. The runs were honest - real requests, real assertions, real data written and read back - and everything passed.

Then a run went across everything the environment held:

  playground              3 objects        everything passes
  full sweep            276 objects        the active-set registry is EMPTY for 210 of them
  later sweeps          392 objects        392 of 392 covered; 129 map rows of 2 329
                                           carry neither rule set nor priority (5.5 %)

Nothing was fixed between the first two lines. The system was in the same state; what changed is that it could be asked - the services had grown read endpoints that describe state: the input snapshot, the rule set in effect, the pointer to the active set, the stored map page by page.

That is Part 2 - A Service That Cannot Explain Itself Can Only Be Guessed At. It is the earliest thing in this timeline in the causal sense: nothing else here was findable before those four reads existed.

The argument about who was slow

The next thing the traced window settled was an argument that had been running on opinions. A single write, end to end: 121 spans, four services, 1 256.9 ms on the client's stopwatch. The blame sat on the write service, because that is the address on the envelope.

Read as nesting rather than as a sum:

  client wait                                       1 256.9 ms   100 %
  ├── transport, client ↔ environment                  244.9      19.5 %   not ours
  └── four services at work                          1 012.0      80.5 %
      └── the suspected service, own zone                87.4       7.0 %
          ├── own work                                    1.4
          └── waiting on PostgreSQL                      86.0      98.4 % of its zone

87.4 ms of 1 256.9 - 7.0 %, and 98.4 % of that is waiting on the database. The code itself accounts for 1.4 ms.

The cheapest confirmation in the whole set came from a request that was rejected as invalid: server work 6.6 ms, database 0.0 ms, client stopwatch 231.8 ms. Nothing computed, nothing stored, and 97.2 % of the call still happened outside our code.

And when the shape of one read was fixed, the same three-build comparison showed why the unit matters: database round trips went 92 to 7, database time 440.0 to 31.8 ms - a factor of 14 - while the network never moved. By share alone the network went from a third of the call to 88 % of it and reads as a catastrophe. In milliseconds, everything we own fell and the rest stood still.

Full breakdown, including the zone we still cannot measure: Part 1 - Most of Your Latency Is Not Your Code.

The account that did not fit

The same instrument answered a question nobody had asked before shipping: how much work is one caller allowed to order?

One object addressed directly is 121 spans and 53 database round trips, 11 of them repeats. One account addressed at the top, with 366 children below it, is 10 822 spans - roughly 30 spans per child, derived by dividing one measured total by one measured count.

The deadline on that call is a constant 20 seconds. The work behind it is a line. On one observed run the caller was cut off at 20.26 s and the work carried on for another 8.7 s, committing successfully at 28.96 s:

  0 s                                  20.26 s               28.96 s
  ├───────────────────────────────────────┤───────────────────┤
  │  work in progress                     │  8.7 s more work  │
                                          │
                          caller is told: the call failed
                                                              │
                                          database holds: the write landed

Around 253 children is where the endpoint stops fitting, and how many accounts sit above that line is a product fact nobody had written down. In aggregate it showed up as deadline failures on 13 of 76 objects, then 27 of 76 after a change meant to help. The whole slope, and why optimisation moves the crossing point rather than removing it, is Part 5 - The Client Decides How Much Work You Do.

The value nobody owned

The longest-lived defect of the period had no code to blame. A rule is saved, the response is 200, the read shows the new set - and the engine keeps marking by the old one. Eight rounds thirty seconds apart, four minutes of the two answers disagreeing, no error anywhere.

Six iterations, the record of the whole timeline. The analysis that explained it took an afternoon:

the value lives in four carriers across two databases, and there is no foreign key between them - there cannot be, they belong to different services;
the right to write it had moved seven times in 48 days;
in one of those states, two days long, it belonged to nobody;
and the state was produced by two commits, same day, two repositories, each correct on its own, which together formed a circle: a freshly written set has no map rows, so it can never become active.

No review inside either repository could have caught it: each diff was right, and the contradiction lived on the seam. Part 3 - Nobody Owned the Value is that afternoon written out, including the detector that catches the class forever: two reads and an equality check, run on all 392 objects.

The contract that could not say zero

While that ran, a second class was producing numbers of its own. A priority that was stored, correct and impossible to present, because the strongest priority in this system is written as 0 and on the wire 0 looks exactly like a field nobody set. Both sides were locally correct: the producer omits zero values by the rules of the format, the consumer treats an absent field as unset.

Eight of the 26 numbered defects were defects of contract shape - the largest single class in the registry, and not one of them was a computation error. The same field's own history: removed from the contract, put back about 45 days later, and pinned down with an explicit meaning for zero on day 91.

What made it observable was not an assertion on a response - every response was correct throughout - but a count over stored state: map rows carrying no priority at all, 27 to 41 to 0.

And its quieter twin: when two rules tie, which wins? Nobody had ever decided. The behaviour was deterministic - first match in array order - and it had held as a norm for 17 months as a property of how a loop was written. Part 6 - Zero Is Not Absent, and Nobody Wrote the Order Down.

The baseline that moved

The method for accepting the new write path was parity: run both implementations on the same input, compare field by field. That is a sound method exactly once - when the reference stands still.

On one day the same input went through nine iterations, and the size of the cascade the two sides produced went 2, then 258, then 2, then 8 rows. The reference was changing its own definition mid-day, and it was internally inconsistent too: one identifier came back as a number in one field and as a string in another. Three times out of nine the new implementation turned out to be the correct one.

Underneath sat the same failure in three different disguises:

a 502 returned after the row was committed;
a 200 completed on a row that had been silently deleted;
a 504 at 20.26 s on a write that went through.

None of them is a lie by the transport. With two storages and no shared transaction, a response code reports that the call arrived and its own transaction closed - and nothing about what is in the database. Then the parity work was erased in one line by a correct decision: the old write endpoint would be removed entirely. Part 4 - Parity Against a Moving Baseline has the full accounting, including the one thing that was missing - the date the comparison would expire.

The ledger

Everything above, as counts.

  26 numbered defects
     16   closed as real defects            (by subtraction from the four rows below)
      5   declared normal behaviour
      3   still open at the end of the period
      2   withdrawn as false

  39 defect-iterations between those 26 numbers      longest single defect: 6 iterations
   8 false alarms in total, of which one is a false PASS - a check that ran zero comparisons

Two rows are worth more than the rest.

Three still open at the end of the period, and their wording is the giveaway: rule order is not provable; cursor paging says "more" and returns no token; an asynchronous rebuild lands in tens of seconds - or does not. None of them is "the code is wrong". All three are "the system cannot demonstrate the property being asked about".

One false PASS. A probe sent a single value to an endpoint that requires a list, got a 422, took the empty decision list as agreement and printed PASS 24, FAIL 3. Two builds were accepted on a line that had performed zero comparisons. A false FAIL creates work. A false PASS removes the check and leaves the report looking identical.

What a document would have caught

This is the number I keep from the whole period, and the one I am least comfortable with.

   26 numbered defects
   ████████████████████████████████████████████  26
             │
             ▼  step 1 — the specification names WHO WRITES this value
   ██████████████████████████████                18  left
             │
             ▼  step 0 — endpoint and fields taken from the contract, not from memory
   ████████████████                              13  left
             │
             ▼  steps 2/3/4/6 — actual measurement against a live system
   ██                                             2  ◄ one conditional, one unobservable

24 of 26 - 92 % - would have been removed by acceptance written before the code, and 21 of 39 defect-iterations die on one question: who is allowed to write this value? That question is answered by reading a document. It is an estimate over the registry rather than an experiment, and I publish it as one - but even halved it says the same thing.

What it cost, and the number I never took

The bill is not "we wrote buggy code". It is the cost of finding out what was wanted:

  requirements with two defensible readings            23
  reversals of decisions already implemented            7      in 5 of 6, the code matched
                                                               the specification of its day
  iterations spent finding out rather than fixing   ~13 of 27  (estimated over intervals)
  iterations of briefs written and discarded            15     within a day
  brief files accumulated in the repository            289     137 of them in 33 numbered sets
  write-ups of my own acceptance mistakes               28     5 of them repeated, 1–6 days apart
  ─────────────────────────────────────────────────────────
  days with a measurement of money or tokens             0

The last row is the one I would fix first if I were starting again. Every duration in this article was recorded; not one day carries a figure for what the automated work cost in tokens or money. That means everything here is a claim about what the arrangement found, and nothing here is a claim about whether it was worth it. Those are different claims and I can only make the first.

The row above it is the second-cheapest lesson: 5 of 28 mistakes repeated within one to six days, every one of them already written up in prose. A rule that lives only in a postmortem is re-derived by whoever reads it next, which is to say never - and an automated executor never reads it at all. What stopped the repeats was moving each rule into the two places that are read on every run: the brief, as a numbered constraint, and the check, as an assertion that fails.

The arrangement that produced most of the numbers above - one automated pass pointed at the output of another, told to instrument the zone with no spans and report back - is Part 7 - One Executor Checking Another Executor's Work.

Seven readings, one per part

If the timeline collapses to seven sentences, these are the seven:

A service is judged by its own span, not by the client's stopwatch. 87.4 ms of 1 256.9 - and 97.2 % of a rejected call is not our code. -> Part 1
A service that cannot state what its answer should be can only be guessed at. Three objects passed; 276 objects had 210 empty registries, and nothing had changed in between. -> Part 2
A value with no named owner does not have a bug - it has an unfinished analysis. Four carriers, two databases, seven moves of the write right in 48 days, two of those days belonging to nobody. -> Part 3
A comparison is a test only while the reference stands still. Nine parity runs in one day, a cascade of 2, 258, 2 and 8 rows for the same input, and a comparison with no expiry date. -> Part 4
A synchronous answer over work the caller sizes is a promise the caller gets to break for you. A constant 20-second deadline against a cost that is linear in 366 children. -> Part 5
A field that cannot separate "absent" from "zero" is two facts sharing a slot. Eight of 26 defects were contract shape; a tie-break rule held as a norm for 17 months because it lived in a loop. -> Part 6
A second automated pass earns its cost only if it can add a measurement the first one never had. 92 database round trips on a call that needs 7, reported as 200 OK in every column. -> Part 7

And the one that only appears when they are read together: the split did not create these defects, it made them expressible. A single process can hold an unstated ownership rule, an unstated tie-break and an unstated capacity limit indefinitely, because there is no boundary at which the omission has to be declared. Two services cannot.

What none of this proves

Being explicit, because a ledger invites conclusions it cannot support.

It does not say microservices are worse than a monolith, or better. It does not say anything got faster after the split - the instruments changed inside the window, and the earlier stretch has no numbers to compare against. It does not price anything: no money, no person-hours, no token counts. It does not say anything about the reliability of anyone's production system; every figure here comes from a test environment.

What it does support is narrower: on this path, in this period, 24 of 26 defects were answerable before anything ran, and the four questions that would have answered them fit on one page.

The multiplier line

Nine parity runs in a day, a sweep of 392 objects per build, a trace folded into a table and diffed against yesterday's - all of that was cheap because it was automated, and none of it would have been affordable by hand. What automation never supplied was a single one of the sentences that were missing: who may write this value, what absent means for this field, how large a caller is allowed to be, which side of a comparison is entitled to be right. Fast iteration multiplies the definitions that exist in writing. Where a definition is missing, it multiplies the circling instead - and produces, very precisely and very confidently, 39 iterations of a description where four written sentences would have done.

Anatomy of a service split - Part 8. That closes this set. The obvious hole in it is the last row of the bill: 985 minutes of recorded duration and not one day with a cost figure attached. The next thing I measured was exactly that - what the automated work itself bills, per session, in tokens and in money, before any of it produces a line of output.

If you do this better, tell me which of your defects you can honestly say a document would have caught. If you have been through this, what did your own ledger look like when you counted the iterations rather than the tickets? If you see it differently, say where writing the requirement down first costs more than it returns. How is it solved on your side, and what broke there?

One Executor Checking Another Executor's Work

Anton Brilliantov — Wed, 28 Oct 2026 02:14:53 +0000

A second pass earns its cost only if it can add a measurement the first pass never had.

👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services. This series is about one write path that left the monolith and what it took to accept it as working. This last part is about the arrangement that produced most of the numbers in the other six: one automated executor pointed at the output of another, told to instrument what was not instrumented, run the measurement and report what it saw. Running notes live on my GitHub: github.com/brilliant-almazov.

This is how I do it right now, with the price attached - maybe you already do it better, maybe you'd do it another way.

The path in one paragraph

Work on that path ran in iterations: a brief describing what to change, an executor producing the change, a run against a live environment, a verdict. This part is about the second executor in that sentence - the one whose whole job is to look at what came out of the first, and whose findings are the only reason several of the numbers in this series exist.

What the second pass actually is

Not a review of the diff. A review of the diff is cheap and mostly useless here: every change in this period was locally correct in its own repository, and I have a whole part of this series about two individually correct commits producing one defect.

The second pass is five steps, in this order:

  1  read what the other pass produced          the change, and the claim it makes
  2  find the zone that has no numbers          which span would have to exist to check that claim
  3  carry the instrumentation there            usually into a neighbouring service, not this one
  4  run the measurement on a live environment  same input, before and after, same day
  5  report two kinds of finding                "this is what it takes to run it at all"
                                                "this looks wrong - here is the number, take a look"

Step 3 is the one that makes the arrangement worth anything, and it is the one that gets skipped. A pass that only measures where measurement already exists will confirm whatever the first pass believed. The finding below existed because the executor was told, in the brief, that adding spans in a service it was not otherwise touching was in scope.

The zone with no spans

The claim under check was ordinary: a read endpoint had been reworked and was, by the response, fine. Status 200, correct data, content assertions passing on every object in the sample.

What did not exist was any number underneath that verdict. The service's own spans said "the call took this long"; nothing said what the call had been doing. So the second pass added, in the service that answers that read, the one field that turns a duration into a finding: the statement text on every database span, with a repeat count.

Then it ran the same request - two domains, one account - against three consecutive builds on the same day:

  build                          A          B          C
  ─────────────────────────────────────────────────────────
  database round trips          92          7          7      ◄ the shape
  database time, ms          440.0       37.2       31.8      ◄ 14× less
  server root, ms            474.7       76.2       59.1      ◄ ours, 8× less
  client wall, ms            715.9      324.0      433.1
  network, ms                242.1      250.1      381.3      ░ not ours, never moved
  response code                200        200        200      ← unchanged throughout

Ninety-two round trips on a call that needs seven. Nothing about that is visible in a response code, a status page, an error rate or a duration alone. It is visible in exactly one place - the count and the text of the statements - and that place did not exist until a pass was sent to create it.

Two things I take from that run, and keep:

The unit of the finding is round trips, not milliseconds. 92 → 7 is a fact about the shape of the work. 440.0 → 31.8 ms is what the shape happened to cost on that day, on that data volume, on that machine.
A share without its absolute value is not publishable. The network's share of the call went from 33.8 % to 88.0 % across those three builds and reads like a catastrophe; in milliseconds it never moved, and everything we own fell. I write 440.0 ms · 61 %, never 61 %.

The two kinds of finding, and why the difference matters

Everything the second pass reported fell into one of two boxes, and I ask for them separately now:

"This is what it takes to run at all." Missing configuration, a wrong assumption about a field, an endpoint that needs a list where a single value was sent. Mechanical, verifiable, and safe to act on immediately.
"This looks wrong - here is the number, take a look." 92 statements where 7 would do. A resolver called once per rule. A wait with no span around it. These are suspicions with evidence attached, not verdicts.

The second box is the valuable one and it is the one that must not be auto-applied. A suspicion is a claim about intent - somebody meant this call to cost one round trip per request, not one per rule - and intent is not in the trace. What the pass supplies is the number that makes the conversation short. What it cannot supply is the decision that the number is wrong.

I have seen the failure mode of skipping that distinction, in this same period: a probe read a value from a listing endpoint that quietly ignored its filter, printed a confident finding, and produced two phantom defects - one of them filed as a comment on another team's ticket before being withdrawn. The executor was not sloppy. It reported what it measured, with the same formatting and the same confidence it uses when it is right.

The instruction surface is where the cost sits

The cheap part of this arrangement is the running. The expensive part is the writing.

  brief files in the repository        289
  of them, in 33 numbered sets         137
  ─────────────────────────────────────────
  what a brief has to contain          the exact surface, the exact request shape,
                                       what counts as a normal empty, where artefacts go,
                                       and an explicit ban on concluding from one field

That is the honest bill for "the executor did it". Two hundred and eighty-nine files of instruction accumulated behind a period whose code changes would fit in a much smaller number. None of it is boilerplate: a brief that does not say what a normal empty result looks like will get a normal empty result reported as a failure, and I have the reports to prove it.

What it buys is repeatability. A brief that is complete enough to hand to an executor is, by construction, complete enough to hand to a person, to a pipeline, or to yourself in three weeks. The instruction surface is the durable artefact here; the run is disposable.

Defects do not die on the first pass

The other thing a second pass changes is the arithmetic of a defect's life. Twenty-six numbered defects over the period, and 39 defect-iterations between them - a defect on average outlived more than one pass, and the worst of them outlived six:

  iterations   0    1    2    3    4    5    6
               ├────┼────┼────┼────┼────┼────┤
  D-01         ████████████████████████████▌  6   a rule written, the engine kept the old set
  D-04         ███████████████████████▌       5   map row carried no priority (+ a regression)
  D-11         ██████████████▌                3   priority 0 arrived as null on the wire
  false ✗      ██████████████▌                3   no defect at all - measured the wrong source
  D-14         █████████▌                     2   timeout on a third of the objects
  D-16         █████████▌                     2   map row did not name its set - came back later
  eight more   ████▌                          1   each
  ──────────── still open at the end of the period ────────────
  D-23         █████████▶                    ≥2   rule order not provable
  D-25         ████▶                         ≥1   cursor paging says "more", returns no token
  D-26         ▶                             ≥0   async rebuild lands in tens of seconds - or not

The line worth staring at is the one labelled false ✗: three iterations spent on something that was never a defect, because a probe read the wrong source and reported it with full confidence. Three iterations is the same cost as the third-worst real defect in the set.

So the second pass is not a heroic single sweep that closes things. It is a loop that has to survive being wrong about its own findings, which is why every finding it reports carries the measurement that produced it rather than the conclusion.

A rule written only into prose comes back

I keep a write-up for every mistake I make in acceptance work. Over this period that is 28 of them, which is a number I am fine publishing because the interesting part is not the count.

The interesting part is that 5 of those mistakes repeated, and the gap between the first occurrence and the repeat was one to six days:

reading a configuration's rule set from an endpoint that does not authoritatively answer it - came back after six days, four iterations burnt;
writing a request from memory instead of from the contract - three separate times;
writing to a production environment from a state that had been read wrongly - four days later, with 17 rules wiped and a four-minute gap in the data;
misreading an auth 401 as a defect - five days later, one ticket filed and withdrawn;
overwriting a shared artefact with a blind write - the same day.

Every one of those already had a write-up. The write-up did nothing. A rule that lives in prose is re-derived by whoever reads it next, which is to say never, and an executor never reads it at all.

What actually stopped the repeats was moving the rule into the two places that are read on every run: the brief (as a numbered constraint, with the ban stated as a ban) and the check (as an assertion that fails, not a note that warns). That is the same movement as freezing requirement tests before code, applied to my own process rather than to the product.

Closing the loop, and the steps that stay manual

Right now the handovers between passes are mine. I take the analysis, I decide it is worth a run, I hand it to the run, I read the report, I decide what it means. The obvious next step is to close that: one orchestrating pass that fans out analysis, instrumentation and measurement, and returns a report instead of returning to me three times on the way.

Three steps I am deliberately not closing:

what "correct" means - the decision that 92 round trips is wrong and 7 is right is a product decision, and the trace does not contain it;
writing to a live environment - a write from a wrongly-read state has already cost me 17 rules and four minutes of data; it stays behind a person on purpose;
accepting a defect as a defect - because the cheapest failure mode in the whole set is a confident, well-formatted, entirely fictional finding.

Calling those three "ritual" is fair, and I keep them anyway. Not because automation cannot perform them - it can perform all three - but because the cost of being wrong on them is asymmetric, and the cost of doing them by hand is a few minutes per iteration.

What I never measured

The honest hole in this part, in one line: there is not a single day in this period with a measurement of what the automated work cost. No token counts, no money, no per-iteration accounting. I have 985 minutes of run durations and I have zero currency.

That means everything above is a statement about what the arrangement found, and nothing above is a statement about whether it was worth it. Those are different claims and I can only make the first one. The second needs a measurement I did not take, and taking it costs nothing next to the thing being measured - which is exactly why not having it is embarrassing rather than defensible.

The multiplier line

The second pass produced findings the first pass could not have produced, and it did it in an afternoon: the missing spans, the run, the comparison, the report. What it did not produce was the instruction that sent it into a service it was not otherwise touching, or the judgement that a correct 200 response can still be a defect. Those came out of a brief - one of 289 - written by a person who had already been wrong enough times to know which zone had no numbers in it. Fast execution multiplies the definition of "done" that it is given; where that definition is a response code, it multiplies green.

Anatomy of a service split — Part 7. That closes this set. The remaining piece is the whole ledger in one place: 32 days, 34 runs, 26 numbered defects and every figure from these seven parts, read end to end as one timeline rather than seven angles on it.

If you do this better, tell me how you decide which zone gets instrumented before a run. If you have been through this, what did your second pass find that the first one had reported as green? If you see it differently, say where a second automated pass costs more than it returns. How is it solved on your side, and what broke there?

A Service That Cannot Explain Itself Can Only Be Guessed At

Anton Brilliantov — Mon, 26 Oct 2026 21:53:44 +0000

If a service cannot state what its answer should be, the only way to check it is to guess.

👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services. This series is about one write path that left the monolith and what it took to accept it as working. This part is about the thing that made checking it possible at all: read endpoints whose only job is to describe state. Running notes live on my GitHub: github.com/brilliant-almazov.

This is how I do it right now, with the price attached - maybe you already do it better, maybe you'd do it another way.

The path in one paragraph

A user saves a set of matching rules; every domain under that account gets marked by them. Four participants: the Go API gateway as the only door, the rule-set-markup service which takes the write and folds the cascade, the domain-rule-map service which owns rule-set identity and recalculates the map, and PostgreSQL behind them - the monolith's database for the rule data, the identity service's own for identity.

We knew our input exactly: the rules the user sent. What nobody could state was the expected output: given this input and this account, what should the resulting map contain, and how would we know the answer was complete rather than merely non-empty?

What a three-object test hides

The first weeks of checking ran against a playground of three objects. Everything passed. The runs were honest, the assertions were real, and the conclusion was worthless - three objects cannot show you anything that depends on the shape of the data.

Then a run went across everything the environment held:

  playground              3 objects        everything passes
  full sweep            276 objects        active-set registry EMPTY for 210 of 276
  later sweeps          392 objects        392 of 392 covered, 129 map rows of 2 329
                                           carrying neither rule set nor priority (5.5 %)

Nothing was fixed between the first two lines. The system was in exactly the same state; what changed is that it could now be asked. 210 of 276 is not a defect that appeared - it is a defect that was already there, in production data, invisible because no endpoint would tell us.

The endpoints that made the difference

Four reads, none of them user-facing features, all of them ordinary functionality the services already had internally:

the input snapshot for an account - which domains are in scope at all, and how many.
the rule set in effect, with its rules - what is supposed to be applied.
the pointer to the active set - which set the engine believes is current.
the stored map, page by page - what was actually written, per row.

With those four, checking stops being an opinion. You read the snapshot and the rules, compute the expected result locally, then compare it to the stored map and to what the engine answers live:

  snapshot            8 693 domains in scope
  rule set               87 rules
  expected, computed     87 marked
  stored map             87 marked
  engine answer          87 marked
  ───────────────────────────────────
  mismatches              0            ← a verdict, not an impression

This is the whole argument of this part. The expected result is not something a checker invents; it is something the system has to be able to state. If the only endpoint is "do the work", then every verdict about it is a guess dressed as a test.

Two ways this goes wrong when the surface is missing

Both of these are mine, both cost real days.

A green result that never ran. A probe sent a single-value field to an endpoint that requires a list of ids. The endpoint answered 422; the probe took the empty decision list as "nothing disagreed" and printed PASS 24, FAIL 3. Two builds were accepted on a check that had performed zero comparisons. A false FAIL creates work; a false PASS silently removes the check.

The rule that came out of it is one line: an empty selection is a FAIL, not a PASS, and every probe carries a counter of comparisons actually made, non-zero by construction.

The right-looking read of the wrong thing. The set attached to a configuration was read from a listing endpoint that quietly ignores the filter it was given - a request for one account returned a hundred rows belonging to other accounts. The probe printed a confident, entirely fictional finding. Five days later the same misreading produced two phantom defects, one of which was filed as a comment on another team's ticket before being withdrawn.

That one costs a section in every spec since, in two halves:

positive definition - which endpoint returns this value, by contract;
negative definition - which reads do not prove it, and why.

This is functionality, not scaffolding

The tempting way to read all this is "add some debug endpoints". That framing is why they never get built.

Every one of those four reads is capability the service already has. Resolving the active pointer, listing rules, paging the map - it does all of it internally on every request. The only question is whether it is exposed. A service that keeps it inside is not simpler; it is a service whose behaviour can only be inferred.

Where the endpoint lives is a detail, and the detail is smaller than it looks: it can sit behind the same gateway as everything else, under an operator role, under normal rights, or on a separate admin surface. What matters is that the capability is presentable, and that it is treated as part of the product rather than as something added while chasing a bug. The same reads that let a checker compute an expectation are what an operator wants during an incident and what a support answer needs when someone asks why one account looks different.

Two design notes that came out of using them in anger:

Read endpoints must page and count honestly. A page of five rows out of eighty-seven that says "there is more" while giving no cursor to continue with is not a diagnostic - it's another guess. That was its own defect and it is still open.
Absence must be answerable. An account with no data should return an empty, successful answer, not an error. Modelling "nothing here" as a failure means every sweep has to guess which failures are real - we had objects answering three different codes for one state.

And the reason to treat these as requirements rather than nice-to-haves: everything the sweep found came out of these reads, not out of running the feature.

Today an executor, tomorrow the pipeline

Right now the sweep is driven by an automated executor: it reads the four endpoints, computes the expectation, compares, and writes a report with the numbers and the trace contents attached. That is a stage, not a destination. The same four reads and the same comparison belong in a pipeline against a stand - a scheduled run, or one run per deploy of the services involved.

Conceptually nothing changes between those columns; only the thing holding the clipboard does. Which is exactly why it is worth building the surface first and arguing about the runner later - the surface is the durable part, and it is useful to a human during an incident whether or not anything automated ever consumes it.

What it costs

Being honest about the price, because there is one:

Endpoints have to be maintained. A read that describes internal state drifts with the internals; a stale describe-endpoint is worse than none, because it is believed.
They widen the surface. Rights, rate limits, and "who may see how many rows of whose data" all have to be answered - the same answers as for any other endpoint, but they must actually be given.
Computing the expectation locally is real work. It means writing down the rule of the system in a second place, which is a second thing to keep true.

Against that: the sweep of 276 that found 210 empty registries took minutes once the reads existed. Before they existed, the same finding was unavailable at any price.

The multiplier line

An executor turned a two-hour manual sweep into a repeatable one, and made "run it again on the next build and print the difference" cost nothing. It could not invent the four reads, and it could not tell a genuine mismatch from a probe reading the wrong endpoint - it printed both with equal confidence. What made the automation worth anything was the surface underneath it and the definition of where each value legitimately comes from. Fast is only useful once correct is defined.

Anatomy of a service split — Part 2. Next: the part before all of this - the analysis that was not done. One value with four homes and no owner, and how long it takes to find that out when nobody wrote down who is allowed to write it.

If you do this better, tell me what your services expose for checking their own state. If you have been through this, what did your first full sweep turn up that the sample never did? If you see it differently, say where a describe-your-state endpoint costs more than it returns. How is it solved on your side, and what broke there?

Most of Your Latency Is Not Your Code

Anton Brilliantov — Sun, 25 Oct 2026 17:13:50 +0000

A service is judged by its own span, not by the client's stopwatch.

👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services. This series is about one write path that left the monolith and what it took to accept it as working. This part is about latency: where it actually sits, and how long we argued about it before anyone measured. Running notes live on my GitHub: github.com/brilliant-almazov.

This is how I do it right now, with the price attached - maybe you already do it better, maybe you'd do it another way.

The path in one paragraph

One feature: a user saves a set of matching rules, and every domain under that account gets marked according to them. Four participants. The Go API gateway is the only door from outside. The rule-set-markup service takes the write, folds the cascade and persists it. The domain-rule-map service owns rule-set identity and does the recalculation, with a worker process behind a queue. Storage is PostgreSQL - the monolith's database for the rule data, the domain-rule-map service's own for identity.

One user action, four processes, two databases. When it is slow, everyone in the chain has an opinion about whose fault it is, and none of the opinions cost anything to hold.

The call everyone argued about

A single write, traced end to end: 121 spans, four services, 1 256.9 ms as seen by the client.

The blame went to the write service, because that is the one the request is addressed to. Here is the same call as nesting, which is the only shape that answers the question:

client wait                                          1 256.9 ms   100 %
├── transport, client ↔ environment                     244.9      19.5 %   not ours
└── gateway root                                      1 012.0      80.5 %
    ├── auth, rate limit, identity, cache                 3.7
    ├── own work + 4 SQL + one catalog call              47.4
    └── batch write call                                950.9
        ├── hop, gateway → write service                  7.9              not ours
        └── rule-set-markup                              943.0
            ├── own work                                   1.4
            ├── validate + gather, 5 SQL                  32.3
            ├── identity calls → domain-rule-map          76.7
            ├── its own transaction, 10 SQL               54.4   (SQL 53.7 · work 0.7)
            └── recalculation → domain-rule-map          775.6
                └── worker: canonicalise                  350.9
                    ├── SELECT dictionary rows            129.6
                    └── INSERT dictionary rows            112.5

The service under suspicion owns three rows of that tree: its own work, its validate/gather step and its transaction. 87.4 ms out of 1 256.9 - 7.0 % of what the client waited. And inside those 87.4 ms, 98.4 % is waiting on PostgreSQL; the code itself accounts for 1.4 ms.

Two things worth saying out loud about that tree:

Never sum nested spans. 943.0 already contains 775.6, which already contains 350.9. Adding them up is how a 1.2-second call becomes "three seconds of work".
The whole call cost 53 database round trips, 11 of them repeats of a query already made in the same call. That count, not the milliseconds, is the actionable number - and I'll come back to it.

The rejected request that settled the zone question

Arguments about "the network is fine, it's your service" ended with the cheapest measurement in the whole set: a request that was rejected as invalid.

POST … → 422 Unprocessable Entity
  server work    6.6 ms
  database       0.0 ms
  client wall  231.8 ms
  ─────────────────────
  225.2 ms, or 97.2 % of the call, is not the service

The server parsed a body, failed validation and answered in 6.6 ms. The client saw 231.8 ms. Nothing was computed, nothing was stored, and 97 % of the call still happened outside our code.

That is not a defect and it is not fixable by us - the environment sits at a steady 220-260 ms regardless of endpoint or payload size, confirmed on a rejection and on a full run. It only becomes a lie when someone quotes the client's stopwatch as a service metric.

So the reading rule for the whole series, written down after this:

A service is judged by its root and server spans. wall describes the trip, not the work.

Three builds, one input

The same read request - two domains, one account - run three times on three consecutive builds of the matching service, same day:

client wall: 715.9 → 324.0 → 433.1 ms
server root: 474.7 → 76.2 → 59.1 ms
database total: 440.0 → 37.2 → 31.8 ms
database queries: 92 → 7 → 7
network: 242.1 → 250.1 → 381.3 ms, i.e. a share of 33.8 % → 77.2 % → 88.0 %

Read the network share alone and the story is "it got dramatically worse: the network went from a third of the call to almost all of it". Read the row above it and the story is the opposite: the part we own fell by a factor of eight, and the part we don't own never moved.

  wall         715.9 ╲___324.0 ╱‾433.1     the trip bounced around
  server root  474.7 ╲______76.2 ╲_59.1    ◄ what we own — fell monotonically
  database     440.0 ╲______37.2 ╲_31.8    ◄ 14× drop
  queries         92 ╲_________7 ─── 7     ◄ the shape of the query is what got fixed
  network      242.1 ─────250.1 ╱‾381.3    ░ not ours — our fix did not move it

Hence the second rule, which I now apply to every chart in this material: a share is never published without its absolute value. 440.0 ms · 61 %, never 61 %. A percentage of a moving denominator is the easiest honest-looking way to say the wrong thing.

What build A was actually doing

Build A was not slow because the database was slow. It was slow because of the shape of the work:

one compile step held 411.1 ms - 91 % of the server window;
inside it, a per-rule resolver ran 86 times;
each of those read the rule-type catalog: 88 identical SELECT … ORDER BY id in one call.

The response was correct the whole time. Status 200, right data, PASS on content - and a class of defect that no response code can express. It was caught in one call for exactly one reason: the spans carried the query text, so 88 identical statements were visible as 88 identical statements, not as "the database was busy".

The fix moved it to 7 queries. And with the database gone from the picture, the slowest remaining participant became the network - which is the correct end state, not a regression.

The other direction: when the wait has no name

The opposite case, from the write path, same period: recalculation calls started failing on a 20-second deadline - first on 13 of 76 objects, then, after a change meant to help, on 27 of 76.

The instinct was to look at the write transaction. The transaction took 14.8 ms.

The time was in two spans nobody had thought about:

queue wait: 64-84 s - work sitting in a queue before anyone started it;
one canonicalisation lookup: 32.3 s.

Neither is "the service being slow". Both are the service waiting, and the only reason they were nameable is that somebody had instrumented the wait before it became a problem. The four places a request sits still - acquiring a connection, waiting for a worker, handing work over, waiting for the answer - are almost never covered by a span, which is why they show up as an unexplained gap.

What the traces still do not show

Honest inventory, because a measurement set with unknown holes in it is worse than one with known holes.

transport, client ↔ environment - measured: steady 220-260 ms, not ours.
hop between services - measured as a gap: 0.8-2.4 ms per transition, 7.9 ms on the write call.
the service's own work - measured: its own spans, per service.
database - measured: per query, with the statement text and a repeat count.
gRPC encode / decode - not measured: unknown, probably small, but not a measured zero.

The gap is real: between-service time is derived from where the child span starts, so serialisation, deserialisation and whatever the client library does around them are folded into one number that nobody has split. On the numbers above it cannot be large. That is a bound, not a measurement, and I keep it in the list rather than pretending it is zero.

Why this is an SLO conversation, not a performance one

Once the zones are separated, the chain stops being a mystery and starts being arithmetic:

  chain response ≈ the slowest participant + everything spent between them

That reframes every argument we had:

a service's target belongs to its own span, because that is the only part its team can move;
the door's target is the sum plus the hops, and it is the only number a user experiences;
the environment gets an explicit line - 220-260 ms of it here - so it is never mistaken for someone's fault;
and the sensible unit for a per-service budget is often not milliseconds but round trips: 53 database calls with 11 repeats, or 92 queries where 7 will do.

None of that needs new infrastructure. It needs the zones to be separable in the data you already collect - and one line in the report saying which zone each number belongs to.

The multiplier line

An automated executor made this cheap: run, pull the trace, fold it into a table, run again on the next build, print the difference. What it did not do was decide what counts as ours. That distinction - our span, the environment, the database, the unmeasured gap - is a judgement, and it is the whole value of the exercise. Speed of measurement amplifies whoever set the definitions; it does not supply them.

Anatomy of a service split — Part 1. Next: the same run produced numbers nobody could have predicted from the code, because the services had endpoints that could describe their own state. That is the next part - what a service has to expose before anyone can check it.

If you do this better, tell me how you draw the line between your zone and the environment. If you have been through this, what did your unexplained gap turn out to be? If you see it differently, say where per-zone budgets cost more than they return. How is it solved on your side, and what broke there?

Parity Against a Moving Baseline

Anton Brilliantov — Sat, 24 Oct 2026 15:28:23 +0000

Comparing a new implementation against an old one is only a test while the old one stands still.

👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services. This series is about one write path that left the monolith and what it took to accept it as working. This part is about checking a migration while both sides keep moving - and about response codes that describe the wire rather than the storage. Running notes live on my GitHub: github.com/brilliant-almazov.

This is how I do it right now, with the price attached - maybe you already do it better, maybe you'd do it another way.

Where the boundary was drawn

The migration moved computation out of the monolith before it moved data. For a stretch of about seven weeks that produced a split that looks reasonable in a diagram and is very hard to check:

  the monolith                              the extracted service
  ├── works out the composition             ├── owns the identity of a rule set
  ├── asks the service for identity  ────►  ├── answers over the wire
  └── mirrors the row into its own DB       └── knows nothing about the mirror

                    no shared transaction between them

One operation - work out what the composition should be, and save it - now spans two processes and two databases with no shared transaction. Over 49 days that boundary produced four tasks of exactly one shape: the monolith computed the right thing and something else got stored. In the two weeks right after ownership moved, the old system took 18 more edits on this subject alone.

The response code describes the wire

The first thing a parity check leans on is the response. Three cases from this period, all of them real:

502 after commit - the row was written; the caller retried a completed write.
200 completed - the row was silently deleted; the run counted it as success.
504 at 20.26 s - the write went through, and the cascade kept working 8.7 s after the answer.

None of these is a lie by the transport. A code honestly reports that the call arrived, the body parsed and the request's own transaction closed. With two storages and no shared transaction, that is all it reports. So the rule that went into the acceptance instructions and has stayed there:

Assert on the persisted read-back, not on the write response. A green answer is not a green database.

The baseline was being fixed at the same time

Now the harder problem. The new write path was accepted by comparing it against the old one - byte for byte, field for field. That is a sound method exactly once: when the reference is frozen.

Ours was not. On one day the same input went through nine iterations, and the size of the cascade the two sides produced went:

  run 1 ─────► 2 rows
  run 4 ─────► 258 rows        same input
  run 6 ─────► 2 rows
  run 9 ─────► 8 rows          settled — for that day

And the diff itself, one object, both sides in the same run:

  the old implementation                        the new implementation
  {                                             {
    "rule_set": { "id": 1806235194000891392 },    "rule_set": { "id": "1806235194000891392" },
    "scope_rule_set_id": "9412008365512340992",   "scope_rule_set_id": "9412008365512340992",
    "priority": null,                             "priority": 0,
  }                                             }

Two differences, two entirely different causes. The identifier being a number in one field and a string in another is the old side being inconsistent with itself. The null against 0 is the contract: on the wire, zero and absent are the same thing, so a real value of zero cannot be told from no value at all.

A machine can produce that diff in a second. What it cannot produce is the sentence "the right-hand side is correct here and the left-hand side is correct there".

Every one of those runs was honest work. The swing came from the reference itself changing definition mid-day: what counts as a sibling in the cascade, whether inactive objects belong in it, what the scope of a client-level write is. And the reference was internally inconsistent too: the same identifier came back as a number in one field of one response and as a string in another.

Three times out of nine, the new implementation turned out to be the correct one. Which is the real lesson of the day: when you name a reference, you are also declaring it correct - and that declaration needs to be true.

A green suite that could not fail

The same period had a quieter version of the same mistake. The write path had a switch selecting which implementation runs. Production ran the new one. The test environment's config file still pointed at the old one.

So the suite was green, permanently, and green meant nothing: it exercised a code path production had stopped using. One flag, set differently in two places, for weeks. A switch that selects an implementation belongs in the test matrix, not in an environment file.

And then the comparison stopped existing

The end of the parity story is worth telling in full, because it is not a failure - it is a cost.

Partway through, the owner decided the old write endpoint would be removed entirely: one door, one write path, no dual entry. Correct decision, and it simplified everything downstream. It also erased, in one line:

nine iterations of parity work from a single day;
the cumulative field-by-field contract diff between the two answers;
every timing measured on the old path;
and the whole defect class "the two implementations disagree", which stopped being a defect class at all.

What was missing was not the decision. It was the date it would take effect, agreed in advance. A comparison is worth its cost only while both sides are going to keep existing; the moment one is scheduled for removal, the parity work has a known expiry and can be scoped to it.

The real bill: finding out what was wanted

Add it up and the shape of the period is not "we wrote buggy code":

23 requirements that had two defensible readings;
7 reversals of decisions that were already implemented;
~13 of 27 iterations spent working out what was wanted rather than fixing code (an estimate over intervals, not a measurement);
15 iterations of briefs written and discarded within a day;
and in 5 reversals out of 6, the code was correct against the specification it was written against.

That last number is the one I keep. The implementation was rarely the problem. The requirement moved, and nothing moved with it.

What keeps them in step

None of the remedy is exotic. It is a loop, run per iteration rather than per release:

Freeze the requirement in words, with the owner of each value named.
Write the check before the code, in two halves: what proves the value, and which reads do not prove it.
Run it against the live system, on everything rather than on a sample.
Re-sync deliberately: when the requirement moves, the check and the specification move in the same change, or the requirement hasn't actually moved.

The one addition the multi-system case forces: when a comparison is part of the method, write down when it expires. Which side is going away, and on what date.

What automation settled, and what it handed back

Nine iterations in a single day are only possible because the run itself is cheap - generated, executed and folded by an automated executor. Worth splitting precisely what that bought.

Mechanical, and automated end to end: building the request matrix per object, running all 392 rather than a sample, folding both answers into a field-by-field diff, repeating the whole pass on the next build, attaching the trace contents to every failure so the report outlives the retention window.

A decision, and still ours: which of the two sides is entitled to be right; who owns a value the two disagree about; when the comparison expires because one side is being removed; the observation that a switch was set differently in the test environment and in production; and the distinction that mattered most - did the build break, or did the requirement move?

The last one is the whole reason this period cost what it did. To an executor, a requirement that changed overnight and a regression look identical: yesterday's expectation, today's mismatch. It will report both with the same confidence and the same formatting. Deciding between them is not a gap in the tooling - it is the job the tooling exists to make room for.

So the honest framing of the multiplier: fast iteration does not settle what "correct" means, it multiplies whatever answer is currently written down. Nine cheap runs against a definition that changed by evening produce nine artefacts and no verdict. Freeze the definition, name the owner of each value, give the comparison an expiry date - and the same automation stops producing noise and starts producing evidence.

Anatomy of a service split — Part 4. Next: the cascade that does not fit inside a response - a constant deadline against a cost that grows with the number of children, and a write that succeeds after the caller has already been told it timed out.

If you do this better, tell me how you keep a reference frozen while it is still being changed. If you have been through this, what did your green suite turn out not to be testing? If you see it differently, say where parity against an old implementation earns its cost. How is it solved on your side, and what broke there?

The Client Decides How Much Work You Do

Anton Brilliantov — Thu, 22 Oct 2026 14:50:57 +0000

A synchronous answer over work the caller sizes is not a contract. It is a promise the caller gets to break for you.

👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services. This series is about one write path that left the monolith and what it took to accept it as working. This part is about the thing we never wrote down before shipping it: how much work one caller is allowed to ask for. Running notes live on my GitHub: github.com/brilliant-almazov.

This is how I do it right now, with the price attached - maybe you already do it better, maybe you'd do it another way.

The path in one paragraph

The shape that matters here is not the topology, it is one sentence about the work: one call from the user, and then a pass over every object below the one they addressed. Save rules on a leaf and it is a small write. Save the same rules at the top of an account and it is the same call with a different amount of work behind it - decided entirely by how big that account happens to be.

Three objects was the whole test bench

For the first weeks the checks ran against a playground of three objects. Everything passed, and the passes were honest: real requests, real assertions, real data written and read back.

What three objects cannot show you is a slope. Every run cost about the same, so cost looked like a constant - a property of the endpoint. It is not a property of the endpoint. It is a property of the caller.

The slope only appeared when a run went across the whole environment: 76 top-level objects, 392 configurations, roughly 2 300 map rows - and, at the account level, one entity with 366 children.

Three against 366. The sample was 1/122 of the largest legitimate case in the same environment, and nothing in the sample said so.

What one write actually costs

Two writes from the same set of traced runs, drawn to the same scale:

  one object addressed directly
    121 spans · 4 services · 53 database round trips (11 of them repeats)

  one account addressed at the top, 366 children below it
    10 822 spans

10 822 ÷ 366 ≈ 30 spans per child. That number is derived, not measured directly - I divided one measured total by one measured count - and it is enough to answer the only question that mattered: is the cost of this call flat, or does it track the caller's data?

It tracks the caller's data. Which means the operation has a slope, and the response has a ceiling, and nobody had ever written either of them down.

Round trips are the unit that scales

The other measured number on that single-object write is the one I now put in the capacity budget instead of milliseconds: 53 database round trips, 11 of them repeats of a query already made in the same call.

Milliseconds are a property of the day - of the machine, the cache, the load next door. Round trips are a property of the code, and they are the thing that gets multiplied when the caller is bigger. Fifty-three round trips for one object is a number you can reason about before anything is deployed; "about a second" is not.

  per one object            53 round trips · 11 of them repeats
  per account of 366        the same shape, multiplied
  what a repeat costs       nothing on the first call, everything on the thousandth

So the two useful sentences about this endpoint, both available without a stopwatch:

its cost per unit of caller data is a small number of round trips and about 30 spans;
its budget is a fixed 20 seconds that does not grow with anything.

One of those scales with the customer. The other does not. That is the entire finding, and every measurement after it is decoration.

A constant against a line

The deadline on that call is a constant: 20 seconds, the same 20 seconds for a leaf with two children and for an account with 366. The work behind it is a line through the origin.

One observed run at the top of that account: the caller was cut off at 20.26 s, and the work carried on for another 8.7 s after the answer had already gone out - about 28.96 s of work behind a 20-second promise.

Divide again: 28.96 s over 366 children is roughly 79 ms per child. Same caveat - derived, one measured duration over one measured count, and it is a per-account average rather than a per-child guarantee. It still puts a number on the crossing point:

   children     derived work         budget
        3   ▏            0.24 s      1.2 % of it
      100   ████▏         7.9 s      40 %
      253   ██████████▏  20.0 s      the last one that fits
      366   ██████████████▏ 28.96 s  145 % - over by 8.7 s
                         ▲ 20 s deadline

Around 253 children is where this endpoint stops working, and the number of accounts above that line is a product fact, not an engineering one. Nobody had asked for it. Part 1 of this series recorded what that looks like in aggregate: deadline failures on 13 of 76 objects, and then, after a change meant to help, on 27 of 76.

The failure is not "the service is slow". A service that is uniformly 3x faster moves the crossing point from 253 to 759 children and fails on the next account that has 800. Optimisation moves the line. It does not change the fact that there is one.

The answer that arrived before the work stopped

The second half of that run is the part I would want to know before designing anything similar.

At 20.26 s the caller got its failure. At 28.96 s the work finished - and it finished successfully, in the database, exactly as if nothing had gone wrong. Which is what happened: nothing did go wrong. A deadline is a decision by the caller to stop waiting. It is not a rollback.

  0 s                                  20.26 s               28.96 s
  ├───────────────────────────────────────┤───────────────────┤
  │  work in progress                     │  8.7 s more work  │
                                          │
                          caller is told: the call failed
                                                              │
                                          database holds: the write landed

So the response code and the state of the database disagree, legitimately, and any check that treats the code as the verdict will be wrong in a way that repeats. Everything downstream inherits this: a retry on timeout is a second write, not a repair; an alert on 5xx counts calls that did their job; a parity comparison against a reference system reads a failure on one side and a row on the other and calls it a defect.

Three notes on reading any of these figures, because I would rather hand over the caveats than the confidence:

The measurements come from three different observation windows with different instrumentation. Comparing across them compares methods of observation, not states of the system.
Round-trip time to the environment sits at a steady 220-260 ms regardless of endpoint. It is not part of the work above, and it is not ours.
No share is quoted without its absolute value. 28.96 s over a 20 s budget, never "145 %" on its own - a percentage of a moving denominator is the easiest honest-looking way to say the wrong thing.

The rule I now write into the acceptance notes before the endpoint exists:

A deadline bounds the wait, not the work. Response code and stored state are two separate measurements and must be asserted separately.

The questions that belong in the design, not the postmortem

None of the above needed a trace to predict. It needed five answers, and none of them is an engineering answer:

Question	What we had before the sweep	What the sweep gave
How many children does a caller have?	"a few" - three, in the test bench	366 at the largest account, 76 top-level objects, 392 configurations
What does one child cost?	unknown, never asked	~ 30 spans, ~ 79 ms - derived
Where does the cost cross the deadline?	no deadline written down	~ 253 children, at a 20 s ceiling
What happens when the data grows?	not considered	the crossing point moves into the middle of the population
What do we do for callers past the line?	nothing defined	still open - the reason for the next section

This is what "know your traffic" means concretely, and it is cheap: four of those five rows are one query against your own data plus one division. We paid for them in iterations instead, which is the expensive currency.

What changed in how this gets accepted

The practices below cost nothing to adopt and are the reason the same class of surprise does not repeat:

Every run includes the largest caller in the environment, not a convenient one. A sample that is 1/122 of the largest case tells you about the sample.
Response code and stored state are asserted separately. Two checks, never one; the timeout case exists precisely because they can disagree.
Round trips per unit of caller data are recorded per build, next to the durations. A shape regression shows up there first and shows up nowhere else - the response stays 200 the whole time.
The population is part of the specification. How many callers sit above the crossing point is a line in the requirements, alongside the deadline, before anyone writes a handler.

None of that is tooling. It is four sentences written down before the code, which is the cheapest artefact in this entire story and the one we produced last.

Where this goes: change the shape, not the step

The exit is not a faster cascade. It is a different shape of operation.

  compute on request                     compute ahead, select on request
  ──────────────────────                 ────────────────────────────────
  caller sets the size                   size is known before the request
  cost grows with the caller             cost is paid where it can be scheduled
  deadline is a constant                 the request is a selection over ready rows
  failure mode: cut off mid-write        failure mode: stale, and visibly so

If the result is computed ahead of the request, a change of rules stops being a write that fans out and becomes a different selection over data that already exists. The question "will we fit inside the deadline" is then answered by the form of the operation rather than by optimising a step inside it. The cost does not vanish - it moves to a place where it can be scheduled, measured and bounded, instead of a place where a user's account size decides it.

I am not going to pretend that is free, or that it is done, or put a date on it. What I will say is that we arrived at it from the numbers above rather than from taste: an operation whose size is set by the caller and whose answer is synchronous has a population of callers it cannot serve, and you find out which ones by measuring your own data, not by tuning.

The multiplier line

The full sweep that produced these numbers was cheap because it was automated: run every object, pull the trace, fold it into counts, run it again on the next build and print the difference. What automation could not do is say how large a legitimate caller is allowed to be - that is a product answer, and it has to exist before the endpoint is designed, not after a run finds the ceiling. Fast measurement amplifies whoever wrote down the limits; where no limit was written, it produces a very precise description of a design that was never bounded.

Anatomy of a service split — Part 5. Next: the contract itself - a value that exists, is stored and is correct, and still cannot be distinguished from "not set" once it is on the wire; and the ordering rule nobody ever specified, which held as a norm for 17 months because it was a property of an implementation.

If you do this better, tell me where you write down the largest caller your synchronous endpoints are allowed to serve. If you have been through this, what did your timeout turn out to have written anyway? If you see it differently, say where precomputing costs more than it returns. How is it solved on your side, and what broke there?

Zero Is Not Absent, and Nobody Wrote the Order Down

Anton Brilliantov — Wed, 21 Oct 2026 15:34:05 +0000

A field that cannot separate "absent" from "zero" is not one field. It is two facts sharing a slot.

👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services. This series is about one write path that left the monolith and what it took to accept it as working. This part is about the contract: a value that exists, is stored and is correct and still cannot be shown - and a rule the system obeyed for a year and a half that nobody had ever written down. Running notes live on my GitHub: github.com/brilliant-almazov.

This is how I do it right now, with the price attached - maybe you already do it better, maybe you'd do it another way.

The path in one paragraph

Two of those rules can match the same domain. Something has to decide which one wins. That decision is a number on the rule - its priority - and everything else in this article follows from the fact that this number was never fully described anywhere.

The value was right the whole time

The symptom was narrow and boring, which is why it survived so long. A rule is saved with a priority. The write is accepted. The row is in the database with the right number in it. Read the rule back and the number is there. Read the map row the engine produced - the thing that says which rule marked this domain - and the priority is gone.

Nothing was lost, nothing was corrupted, nothing was recomputed wrongly. The value simply could not be presented, because the highest priority in this system is written as 0, and on the wire 0 is exactly what a field looks like when nobody set it at all.

  the field has three states                 the wire carries two
  ┌────────────────────────────┐             ┌────────────────────────────┐
  │ 0     the strongest rule   │  ────────►  │ nothing on the wire        │
  │ 7     a weaker rule        │  ────────►  │ 7                          │
  │ —     not set at all       │  ────────►  │ nothing on the wire        │
  └────────────────────────────┘             └────────────────────────────┘

  reader's rule: "no value means not set"  →  the strongest rule reads as unset

This is not exotic. It is the default behaviour of a wire format that omits zero values to save bytes, meeting a domain where zero is the most meaningful value there is. Both halves are reasonable on their own. Together they mean the strongest rule in the system is indistinguishable from a rule with no strength at all.

Both sides were locally correct

The part worth sitting with is that neither end of this is a mistake anyone would flag in review.

The producer follows the serialisation rules it was given: a numeric field holding its zero value carries no bytes, and that is the documented, intended behaviour of the format - it is how the wire stays small. The consumer follows the only rule available to it: a field that arrived with nothing in it was not provided, so fall back to the default. Read either repository on its own and you find a service behaving exactly as specified.

The defect exists only in the sentence neither side owns: what does absence of this field mean for this particular value? No reviewer is asked that question by a diff. It belongs to the contract, and the contract had never said.

The registry carried two entries for it: one where the map row carried no priority, which took five iterations to close and came back once after being fixed, and one where priority 0 arrived as absent on the wire, which took three. In a period of 32 days and 26 numbered defects, that is a lot of attention for a value that was never wrong.

Eight of 26, and 91 days of the contract changing its mind

That pair was not an accident of one field. Counting the whole registry, eight of the 26 numbered defects were defects of contract shape - a field missing, a field ambiguous, a value that could not be told apart from its own absence, an identifier that was a string in one place and a number in another. It is the largest single class in the set. Nothing in it was a computation error.

The same field's own history says why the class stays alive:

  day 0                                day ~45                        day 91
    │                                     │                              │
    ●  priority removed from the contract ●  put back                    ●  meaning pinned down
       "the engine derives it"               "callers need it after all"    "0 is a value, and
                                                                             absent is a third state"
       ─────────────────────── 91 days of three different answers ───────────────────────

Each of those three moves was defensible when it was made. What nobody did until the third one was write down what the field means when it is absent - and until that sentence exists, every consumer invents its own answer, and the answers disagree quietly.

A field that leaves a contract and comes back three months later is worth treating as a signal in itself: it means the question "who needs this and why" was answered by removing it rather than by writing it down. The cheap version of that conversation is one sentence in the contract. The expensive version is two services, several runs and a class of defects that keeps producing new numbers.

The count that made it visible

The reason this ran for weeks is that nothing in the responses said anything was wrong. Status codes were fine, payloads validated, the data was correct in storage. So the check that finally made it observable was not an assertion about a response - it was a count over state: how many map rows exist that carry no priority at all.

  map rows with no priority

  first measurement   ████████████████████████                27
  after the next run  ██████████████████████████████████████  41   ← growing
  after the fix       ▏                                        0

Two honest notes about those three numbers, because they are easy to over-read.

First, 27 → 41 is not a measurement of a system getting worse at a steady rate; it is two counts taken while data kept being written. Second - and this holds for every number in this series - a share is never published without its absolute value. Three windows of observation exist here with different instruments, and comparing across them compares the instruments, not the system.

What matters is the zero at the end, and the fact that a single SELECT produced the whole story. The class of defect that response codes cannot express is exactly the class that a count over stored state can.

The other half: how did this ever work in the monolith?

While that was going on, a plainer question came up and stopped everyone: two rules match the same domain and have the same priority. Which one wins?

Nobody knew. Not "nobody remembered" - nobody had ever decided. Reading the original specification produced nothing: it described what a rule is and what marking does, and said not one word about ties, order or precedence.

The behaviour existed all the same, and it was completely deterministic: the first matching rule in the order the array happened to be in wins. Not a rule anyone chose. A property of how a loop was written, promoted to a norm by nothing except the fact that it never broke.

  where the tie-break rule lived

  the written specification        ─  silent on order, silent on ties
  the interface contract           ─  no field expressing it
  the documentation                ─  not mentioned
  the code                         ●  "first match in array order wins"
                                      ▲
                                      the only copy, held as norm for 17 months

Seventeen months of production behaviour resting on a line of code nobody had ever been asked to agree with.

This is what makes a migration expensive in a way that no estimate contains. Moving a feature out of a monolith is not "reimplement this specification elsewhere". It is, first, archaeology: find out what the system actually did. Then, second, an argument nobody can settle from documents: was that intended, or is it a bug we have been shipping since it was written?

Those are two separate bills, and only the first one gets estimated. Recovering the behaviour is finite work: read the loop, reproduce it, write down what it does. Deciding whether the recovered behaviour is a requirement is not finite, because there is no document that can settle it - only people, and only by making a decision that should have been made a year and a half earlier, now, under the schedule pressure of a migration.

The follow-up defect is still open at the end of the period, and its wording is the giveaway: rule order is not provable. Not "wrong" - unprovable. You cannot demonstrate that a system obeys a rule that was never stated. Until the rule is written, the check can only assert that today's output equals yesterday's output, which is a description, not a requirement.

Requirements read out of code are a liability, not a spec

Both halves of this article are the same failure wearing different clothes.

In the first, the requirement existed in someone's head - "priority 0 means strongest" - and never reached the contract, so the contract encoded a different, weaker requirement: "priority is a number that may be omitted".

In the second, the requirement existed in the code - "first match in array order wins" - and never reached a document, so for 17 months there was no way to tell an implementation detail from a promise.

Neither absence is visible from inside a running system. Code with an unstated rule behaves identically to code with a stated one, right up until somebody has to reimplement it somewhere else - and then the difference is the entire cost of the move.

Across the whole period, 23 places in the specification admitted two readings. Each of those is a small coin toss between two implementations, and the toss is resolved late, by a run, by an argument, or by a defect - never for free.

What changed in how this gets accepted

The practice that came out of it is not "write more documentation". It is narrower and it is checkable: a new task starts by freezing the tests for its requirements, and only then the code gets written. For a field crossing a service boundary, freezing it means answering four questions in writing before anyone implements a reader:

  1  presence     what does absent mean, and is it different from the zero value?
  2  ordering     when two candidates tie, what decides - and where is that written?
  3  authority    which service is allowed to write it, and which only reads it?
  4  proof        which stored count goes to zero when this is correct?

Question 4 is the one that would have caught this article's first half in an afternoon. "Priority is carried through correctly" is not testable as written; "zero map rows without a priority" is a SELECT, it runs forever, and it was the thing that eventually made the defect visible anyway - just several weeks later than it needed to be.

Question 1 is what turns a field description into a contract. "Priority: integer, optional" describes bytes. "Priority: integer; 0 is the strongest; absent means the rule carries no priority and loses every tie" describes a decision, and a decision can be checked.

Question 2 is the one that would have caught the second half, and it costs even less: someone types the sentence "ties are broken by X", the sentence turns out not to exist, and the archaeology happens on day one of the design instead of eighteen months later during a migration.

What this does not prove

The material behind this article covers 32 days of acceptance runs, and the instruments changed inside that window - the earlier stretch has verdicts without numbers, the later one has traces and counts. So none of this supports "the monolith was better" or "the split made it worse". It supports something narrower and, I think, more useful: the split did not create these defects; it made them expressible. A single process can hold an unstated tie-break rule indefinitely, because there is no boundary at which the omission has to be declared. Two services cannot: the boundary forces every implicit fact to be either written down or lost.

I also cannot put a money figure on any of it. Time and iteration counts were recorded, cost was not, and inventing it afterwards would be worse than leaving the column empty.

The multiplier line

Both defects were found and closed faster than they would have been by hand: an executor can re-run a full sweep on every build, pull the state counts, and print the difference without getting bored on the fortieth object. What it could not do is decide that 0 means "strongest" rather than "unspecified", or that ties should be broken by explicit order rather than by whatever the array happened to hold. Those are product decisions, and they are exactly the sentences that were missing. Fast iteration multiplies the requirements that exist in writing; where a requirement exists only in code, it multiplies the archaeology instead.

Anatomy of a service split — Part 6. Next: the last part of this set - what happened when one automated executor was pointed at the work of another, told to add the missing traces, run the measurement and report back on what it found.

If you do this better, tell me how you write down the difference between "zero" and "not set" in your contracts. If you have been through this, which of your production rules turned out to live only in a loop? If you see it differently, say where freezing requirement tests before code costs more than it returns. How is it solved on your side, and what broke there?