DEV Community: Anton Brilliantov

What an Executor Needs Before It Can Test Production

Anton Brilliantov — Sat, 17 Oct 2026 14:53:30 +0000

The quality of a check is set by the inputs, not by the executor.

👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services.

A lot of that work is checking production: does the extracted service behave there the way the piece it replaced did. Most of those checks are handed to an executor - person or program, the rules below hold either way.

Running notes live on my GitHub: github.com/brilliant-almazov. This is the assignment I write today, price attached - maybe you write yours better, and I'd like to hear how.

The thesis

The quality of a production check is decided by the completeness of the assignment, not by the intelligence of whoever executes it.

Everything below is a consequence of that one line.

What one line of the assignment was worth

I ran a sweep across a vertical of 276 configurations: does each one have a result, and if not, where does the path stop. 184 came back empty - against the assignment I had written, that was 184 problems.

Then one condition entered the assignment: a configuration's schedule can be of a type that produces no tasks of this kind at all. With that line the same 184 became normal - the setup was doing what it was configured to do.

One line changed 67% of the picture, and no care from the executor would have found it. The fact is not a field, not a status, not a counter - it lives in how the domain works, and it enters a check only if someone writes it into the assignment first.

One line of the assignment, 276 configurations
the same sweep, read against the assignment before and after that line

without the line
┌──────────────────────────────────────────────┬───────────────────────┐
│ 184  empty result - filed as a defect        │ 92  has a result      │
└──────────────────────────────────────────────┴───────────────────────┘

with the line
┌──────────────────────────────────────────────┬───────────────────────┐
│ 184  empty - the schedule makes no such task │ 92  has a result      │
└──────────────────────────────────────────────┴───────────────────────┘

67% of the picture changed, and no care from the executor would have
found it. The fact is not a field, not a status, not a counter.

That is the whole argument. What follows is the list I now write before any check starts.

The ten inputs

1. The surface, and a ban on every other one

Name the one entry point the check goes through, and state that every other one is off limits. A system in migration has more than one front door, and results measured through two of them are not comparable.

Without this line the executor picks whichever door answers, and the report silently mixes two systems.

2. How to authenticate, with secrets living only in the environment

State the authentication scheme and where credentials come from: the environment, never the assignment, an artefact or a commit.

This is not only hygiene - an executor that has to improvise credentials improvises something that works once and cannot be repeated.

3. The exact shape of every request

Path, method, every parameter with its range, every body field - the literal shape, not "call the listing endpoint".

Anything left open gets interpreted plausibly, and a plausible-but-different request produces numbers that look comparable and are not.

4. How each link is measured, and what counts as the fact

For every link in the chain, say which single value is the fact: this field, this header, this counter. One number per link, read cheaply.

Two people measuring "does this have data" in two different ways will disagree forever, and neither will be wrong.

5. The domain conditions that no read returns

Provider coverage. Schedule types. What each task status actually means. No endpoint exposes any of it.

This is the line that was worth 67% of my picture, and the one most likely to be missing - it is the only one that cannot be discovered by reading the system.

6. What counts as a normal empty

Say which empties are expected outcomes and which are defects, and how to tell them apart.

An executor with no such rule counts every empty as a finding.

7. Where the artefacts go, and in what form

One folder per run, one file per probe, written as the run goes, not assembled at the end.

Left unspecified, the report becomes the only record - and a report is a summary of evidence, not the evidence.

8. The rule for writing in production

Snapshot, act, restore, a unique probe marker, and a confirmation by reading again.

This one gets its own section below.

9. The shape of the verdict, and the duty to separate "not checked" from "passed"

State the vocabulary of the verdict and require that unchecked cases be reported as unchecked.

Left unstated, "not checked" quietly becomes "nothing found", which reads as "passed" to everyone downstream.

10. An explicit ban on concluding from one field without a control case

Every conclusion needs a control beside the suspect: an object where the thing is known to work, read the same way, in the same run.

I put this in writing because I broke it myself, twice.

Ten inputs, written before the first request
the executor fills every one of these itself if the assignment leaves it out

    01  one surface, every other one banned
    02  auth scheme, secrets only from the environment
    03  the exact shape of every request
    04  one measured fact per link
 ▶  05  the domain conditions no read returns
        not exposed by any read - this one line moved 67% of the picture
    06  what counts as a normal empty
    07  artefacts: one folder per run, one file per probe
    08  snapshot, act, restore, verify
    09  verdict format: not checked ≠ passed
    10  no conclusion without a control case

Nine of them can be checked against the system. The fifth cannot - it is
a hand-written record of the world outside.

Why the normal empty gets its own line

If a system answers "no data" with an error, every mass run looks like a catastrophe. Two thirds of the objects come back as not found, the genuine failures sit inside that pile, and every run starts by clearing the noise away.

A normal empty is a success with zeros in it. Note where that requirement lands: on the system, not on whoever is checking it. Writing "treat two thirds of the answers as noise" into an assignment is a workaround for a contract that should have said so itself.

Why the write rule is not up for discussion

Any write executed against production follows one shape:

snapshot  →  act  →  restore  →  verify by reading again

Every object touched carries a unique probe marker, so anything left behind is identifiable as mine and removable without guessing. Restore is not done when the restore call returns - it is done when a fresh read shows the original state again.

A check that leaves the polygon different from how it found it is not a check. It is a change, and whoever runs anything there next inherits it. Without the marker, cleanup depends on memory - the one part of a run nobody can audit.

Every write executed against production

┌──────────┐   ┌─────┐   ┌─────────┐   ┌─────────────────────────┐
│ snapshot │──▶│ act │──▶│ restore │──▶│ verify by reading again │
└──────────┘   └─────┘   └─────────┘   └─────────────────────────┘
──────────────────────────────────────────────────────────────────
                              │
                              ▼
        every touched object carries a unique probe marker
   so anything left behind is identifiable as mine and removable
                        without guessing

What happens when these are missing

The executor does not stop. It fills the gaps itself, plausibly. A missing input does not produce an obviously broken report - it produces a coherent, confidently worded, wrong one, with nothing marking which parts came from the system and which from a guess.

My own example, and not a flattering one: my first conclusion from a sweep was that a set of objects would not rebuild at all. They had failed on time, back to back. The objects were fine - the failure belonged to the pace I was driving the endpoint at, and one at a time with a pause they came back almost entirely clean. That distinction is the next part.

My assignment did not contain rule 10. Without it, one run and no control was enough to file a defect against a service that had lost nothing.

What it costs

The inputs are assembled by hand, up front, before any checking starts. On a vertical of this size that is several hours whose only output is a document.

On a one-off check that is pure overhead: if the question is asked once and never again, writing the ten inputs takes longer than answering it badly and answering it again. The trade pays when the check gets repeated - by the second run the assignment exists, while the guesswork would be re-invented from scratch.

The other part of the price: the fifth input has no owner inside the system. It is a hand-written record of how the world outside works, and it goes stale silently.

The part that generalises

A fast executor amplifies whatever is in the assignment, holes included. Complete inputs get a complete answer sooner; a gap gets a confident wrong answer sooner, and the confidence reads identically in both.

The advantage goes to whoever can make correctness visible from the assignment itself, rather than from a second run and a comparison.

This is my list right now, on one vertical, in a system halfway through a migration, price included.

So: maybe you do this better, and your assignments already carry the domain conditions in a form that stays current. Maybe you have been through this and found which of the ten you can drop without paying for it. Maybe you see it differently and think an executor should be trusted to work the conditions out for itself.

How is this solved on your side, and what broke there?

Testing production with an executor - Part 1.

Next: a failure on time, checked twice - once in a stream and once one at a time with a pause - and how to tell a property of the object from a property of the pace.

Processing and Completeness Are Different Problems

Anton Brilliantov — Thu, 15 Oct 2026 11:16:55 +0000

One verdict for two aspects keeps a closed task open forever.

👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services.

This series is about the complaint that keeps landing on that migration: a user sees no result, and one service gets blamed by default. This part is about the last step of such a breakdown - writing down what was actually solved.

Running notes live on my GitHub: github.com/brilliant-almazov. It is how I write a verdict today, not a methodology - maybe you already do it better, maybe you'd do it another way entirely.

The thesis

Mixing two aspects into one verdict turns a closed task into a permanently open one.

Everything below is a consequence of that.

The two aspects are processing and completeness. They come out of the same breakdown and get written into the same report, and they are not the same problem.

Two definitions

Processing - how the system computes and answers on the data it already has: rules applied, the result rebuilt, the response deadline, concurrency, response codes.

Completeness - whether the data reaches the system at all: the schedule, the scheduler, the provider's queue, the region coverage.

┌──────────────────────────────────┐   ┌──────────────────────────────────┐
│ PROCESSING                       │   │ COMPLETENESS                     │
│ how it computes on the data      │   │ whether the data arrives at all  │
│ it already has                   │   │                                  │
├──────────────────────────────────┤   ├──────────────────────────────────┤
│ rules applied                    │   │ schedule                         │
│ rebuild of the result            │   │ scheduler                        │
│ response deadline                │   │ provider queue                   │
│ concurrency                      │   │ region coverage                  │
│ response codes                   │   │                                  │
├──────────────────────────────────┤   ├──────────────────────────────────┤
│ owner: the processing side       │   │ owner: the collection side       │
└──────────────────────────────────┘   └──────────────────────────────────┘

no arrows: neither side moves the other column by working harder

No qualifiers on either definition.

A fact belongs to one column or the other, and one question decides which: does the system have the data yet?

Why one number can't hold both

The columns have different owners and different repairs.

The team that owns processing cannot speed up somebody else's collection queue, and cannot create a schedule on a user's behalf.

The team that owns collection cannot fix a response deadline.

Neither of them moves the other column by working harder on its own.

So a single "percent solved" over both averages two scales. Work on either side barely moves it. It stops being a signal and becomes a thing people argue about in the standup.

What it looked like in one breakdown

Processing. Every entity that had source data had a result: 76 of 76.

A reference computation run beside production agreed with it on 1 752 entities, with 0 discrepancies.

The two states where the accused service would actually be losing data - raw rows in with no snapshot, snapshot in with no result - came back 0 and 0.

That is a verdict, and the verdict is solved. Where there was input, there was output, and the output matched an independent computation of it.

Completeness. 21 configurations were waiting for collection. 184 are scheduled for a different type of collection entirely. 14 have no keyword queries to collect for.

Those are facts, and they stay facts. They are not a percentage of anything, and none of them says the computation is wrong.

The shape of the verdict I write now

At the bottom of every breakdown, one line per aspect:

processing:   solved - 76/76 with data have a result; 1 752 entities, 0 discrepancies
completeness: 21 awaiting collection, 184 other schedule type, 14 without queries
              owner: the collection side

The owner sits on the completeness line on purpose. A number with no address is a number somebody has to chase.

The reader of a report should not have to derive the verdict. They should be able to read it.

Everything above those two lines is the evidence. The two lines are the answer.

What one verdict would have done

The task would have been "partially solved" - and it would have stayed that way.

Follow it through. Under a single verdict, the task closes only once the missing data arrives.

That data arrives when a schedule fires, a queue drains, a region gets covered. None of those sit with the team that owns processing.

So the task closes on somebody else's timeline, which in practice means it does not close.

ONE VERDICT                            TWO VERDICTS
one number over two aspects            two markers, no line between them

┌──────────────────────────────┐       ┌──────────────────────────────┐
│ partially solved             │       │ processing - closed          │
└──────────────────────────────┘       └──────────────────────────────┘
               │
               ▼
┌──────────────────────────────┐       ┌──────────────────────────────┐
│ closes only when the missing │       │ completeness - waiting,      │
│ data arrives                 │       │ owner named                  │
└──────────────────────────────┘       └──────────────────────────────┘
               │
               ▼
┌──────────────────────────────┐       each closes on its own owner's
│ it arrives on somebody       │       timeline
│ else's timeline              │
└──────────────────────────────┘
               │
               ▼
┌──────────────────────────────┐
│ so it comes back at every    │
│ review                       │
└──────────────────────────────┘

This isn't status bookkeeping. An open task has a running cost: it comes back at every review, it gets discussed again, and the breakdown gets requested a second time by someone who wasn't in the first conversation.

That is paid weekly, forever, for a computation that was already correct.

Where the split stops being honest

The split is honest only while processing genuinely does not affect completeness.

If the system loses data on its own leg - rows arrive and no snapshot is taken, a snapshot exists and no result is produced - that is processing, however much nicer it would be to hand it outward.

So the split comes with a check, and the check gets named out loud: the two loss states must both be zero.

Here they were 0 and 0, and that is the only reason the completeness column was allowed to belong to someone else. If either one is non-zero, there is no conversation about splitting aspects - there is a defect in processing, and it is mine.

What it costs

Two verdicts mean two addressees and two discussion threads, where one line used to do.

The reader has to be walked through the difference once, and for a while afterwards the two still get merged in conversation - usually by whoever quotes the number without the line above it.

I pay that, because the alternative is a task that never closes.

The part that generalises

An executor of any speed returns a verdict in the shape it was asked for.

Ask for one number over two aspects and you get one number over two aspects, delivered faster. The averaging happens either way; it just happens sooner.

The shape of the verdict is part of the assignment, not a matter of taste for whoever is reporting. It decides whether the work can ever be called done.

This is how I split a verdict right now, and the price of it is written above. I'd like to hear the other side: if you do this better, tell me how you separate what your system computes from what it receives; if you have been through this, tell me which aspect ended up owning the never-closing task; if you see it differently, say where drawing this line does more harm than the merged verdict. How is it solved on your side, and what broke there?

Why there is no result - Part 4.

Next series: what an executor has to be given before a production check is repeatable - the surface, the exact shape of every request, the domain conditions no read exposes, and the difference between "not checked" and "passed". It starts in what an agent needs to test production.

Two Cut-Off Factors Your Analysis Is Lying Without

Anton Brilliantov — Wed, 14 Oct 2026 16:57:43 +0000

Coverage and backlog close a third of the cases before any debugging starts.

👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services.

This series follows one question that keeps coming back from that work: why does this entity have no result? Part 1 counted the empty screens, Part 2 split the path into links with owners. This part is about the two conditions that live outside the system entirely, and what happened when I ignored them. Running notes live on my GitHub: github.com/brilliant-almazov.

This is how I do it right now, with the price attached - maybe you handle it differently, and I'd like to hear how.

The thesis

Part of the empty cases should not be taken apart at all. They are closed by two external conditions, and neither of them is returned by any handle in the system.

Everything below is a consequence of that one sentence.

Factor one: coverage

The provider of the result data does not operate in some regions at all. Not slowly, not partially - it has no coverage there.

That changes what the decision is made from. If the configuration sits in a region the provider does not cover, the answer is settled by the region alone.

There is no point opening the task, reading its counters, or checking whether anything was ever queued. The chain of links has nothing to say here, because the chain never had an input.

In the vertical I took apart, that was 4 configurations - removed from the investigation by one attribute of the configuration, before any measurement.

The uncomfortable part: coverage is not a fact about my system. It is a fact about somebody else's product, and it can change without anyone telling me.

Factor two: the queue

The second condition is the provider's own backlog. A task can be created, accepted, and simply waiting - queued, not lost. From inside, both look identical: no rows, empty result.

What separates them is the status of the task, and there are three of them:

Task status	What an empty result means here
waiting to be queued	the task exists; nothing has been collected yet
receiving data	collection is in progress - empty is a point in time, not an outcome
completed	collection is over - this is the only status where empty is a question

An empty result on a task that is receiving data is a wait. There is nothing in it to debug, nothing to file, and nobody to hand it to.

Read as a failure, it produces a defect report against a service that is doing exactly what it should.

┌────────────────────────────────────────────────────────────┐
│ all empty cases                                            │
│ every entity a user reported as having no result           │
└────────────────────────────────────────────────────────────┘
                              │
                              ▼
    ┌────────────────────────────────────────────────────┐
    │ region not covered                       4 removed │
    │ decided by the configuration's region alone        │
    └────────────────────────────────────────────────────┘
                              │
                              ▼
        ┌────────────────────────────────────────────┐
        │ queued at the provider          21 removed │
        │ a wait, not a loss - collection            │
        │ is in progress                             │
        └────────────────────────────────────────────┘
                              │
                              ▼
            ┌────────────────────────────────────┐
            │ worth debugging                    │
            │ the only ones opened link by link  │
            └────────────────────────────────────┘

neither cut-off is returned by any handle in the system

The one-field trap

Here is the part I got wrong, and it is the reason this article exists.

I reported seven configurations as "the task never reached the provider".

The evidence was a single field: on the row of the keyword query, the field naming the provider was empty. Empty field, no provider, task never sent. It reads like a fact.

Then I ran the same read against a control object - one where the data had already arrived, where everything demonstrably worked. The same field was empty there too. And the task behind it was in status receiving data.

┌──────────────────────────────────┐   ┌──────────────────────────────────┐
│ suspect                          │   │ control - data already arrived   │
│ reported as broken               │   │ demonstrably working             │
├──────────────────────────────────┤   ├──────────────────────────────────┤
│ key row                          │   │ key row                          │
│   provider: -                    │   │   provider: -                    │
├──────────────────────────────────┤   ├──────────────────────────────────┤
│ task, one level down             │   │ task, one level down             │
│   status: receiving data         │   │   status: receiving data         │
└──────────────────────────────────┘   └──────────────────────────────────┘

7 configurations reported as "the task never reached the provider"
on the evidence of a field that is empty in the working case too

So the field was never the fact. It says nothing about whether a task was sent, because it is empty in the working case as well. The fact was one level away, in the status of the task itself.

The rule that comes out of it is short: read the task status, not the key row.

The cost was not the debugging time. It was a rewritten report and a separate correction in the tracker: the conclusion had already been handed to people who would have gone looking for a defect that did not exist.

A wrong conclusion travels. You have to go and catch it.

The shape of the mistake is worth naming exactly, because it is not "I misread a field". It is: a conclusion drawn from one field, with no control case next to it.

One field can only tell you what it contains. Whether that content means anything is answered by looking at the same field where things work.

Why these conditions can't be derived from the system

Neither factor can be read out of the system at all. Not the provider's coverage, not the depth of its queue, not which type of schedule a configuration is on.

No handle returns any of it, and no amount of careful measurement will produce it, because it isn't there.

That knowledge lives with people. And while it lives only with people, every new breakdown rediscovers it - usually late, usually after a wrong conclusion has been written down.

Hence the practical consequence: such conditions belong in the brief handed to whoever runs the check, as plain text, on the same footing as the shape of the request.

Not as background, not as "ask me if something looks odd" - as a stated condition, next to the endpoint and its parameters. That is the subject of a later part of this series, on what an executor needs before it can test production.

What it does to the numbers

Two cut-offs, applied before any link is read:

4 configurations removed by coverage - decided by the region alone.
21 configurations moved out of "broken" and into "waiting for collection" - tasks queued at the provider, being collected right now.

What is left after both cut-offs is a different question. Not why is this empty, but when does it arrive. That question has an answer, an owner, and no defect attached to it.

What it costs

Both conditions are external, and that is not a detail - it is the whole price.

I cannot verify either of them with my own code. There is no assertion I can write that fails when the provider adds a region or drains its backlog.

And I cannot keep them current automatically: a note about coverage is only true on the day it was written.

So the honest statement is this: a stale record of coverage is exactly as damaging as no record at all - worse, in one respect, because it looks like knowledge.

It will close cases that should have been opened, and it will do it quietly, in the same voice as the correct record.

I do not have a solution to that. The best I have is keeping the record small enough that its age is obvious, and writing down when it was last confirmed.

The part that generalises

An executor of any speed - a teammate, a new hire, an automated one - will fill in a missing condition on its own.

Not out of carelessness: an analysis needs the condition to reach a conclusion, so if it isn't supplied, it gets inferred from whatever field is closest to hand. That is precisely how "the task never reached the provider" was produced.

So the arithmetic is simple. Writing the condition down once costs a sentence. Unpicking a conclusion built on its absence costs a report, a correction, and other people's time spent looking for a defect that was never there.

Speed doesn't change that balance. It just makes the second outcome arrive sooner.

This is my experience and my price, on one vertical, in one system halfway through a migration.

If you do this better, tell me where external conditions like these live in your setup and how you keep them from going stale. If you have been through this, tell me which single field turned out to be lying. If you see it differently, say where reading one field without a control case is actually good enough.

How is this solved on your side, and what broke there?

Why there is no result - Part 3.

Next: processing and completeness are two different aspects and deserve two different verdicts - what a service computes on the data it has, versus whether the data arrives at all.

A Chain of Links, Not a Guess

Anton Brilliantov — Tue, 13 Oct 2026 12:54:19 +0000

Measure the path the data takes, one number per link - not the endpoint that answered last.

👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services. This series is about a complaint that keeps landing on that migration: a user sees an empty screen, the system thinks everything is fine, and one service gets blamed by default. This part is the method I use now to take those apart. Running notes live on my GitHub: github.com/brilliant-almazov.

It is how I work today, not a methodology - maybe you already do it better, maybe you'd do it another way entirely.

Two different questions

Checking a handle means asking: did the call answer correctly? Status code, body shape, latency, error text. Clean question, clean test.

Checking data means asking a different one: should this entity have had anything here at all?

The second question closes most of the complaints. It also can't be asked until the path the data takes is split into links - "should there be data here" is meaningless for a whole path at once. It only has an answer per link.

Before that split, the first question was all I had, and it produced a specific kind of noise: two thirds of the entities came back with nothing, that read as a contract defect, and the few real failures drowned in it.

The move from the first question to the second isn't a better assertion library. It's a different object under test: not the call, the path.

The chain, and who owns each link

A link is a place where data is obliged to appear, and it has exactly one owner. Top to bottom, from the schedule to the finished markup:

schedule of the configuration        → the scheduler
set of keyword queries               → the dictionary/catalog service
collection task                      → the tool-task service
task run + its counters              → the tool-task service
raw result rows                      → the provider + the raw-data service
snapshot of entities                 → the domain-rule-map service
applied rules → markup               → the rule-set-markup service
rebuild of the result on request     → the rule-set-markup service

Two things about that list before any measurement.

The owner column is the point of the exercise, not decoration. A link without a single named owner isn't a link - it's a region of the system, and a region can't be handed to anyone.

And some links are governed by conditions no read exposes: which types of schedule exist and what each means, which regions the provider covers at all. Not defects, not gaps in the code - domain facts that live outside every handle. The largest cohort below is decided by one of them.

Note what the last two lines mean. The service blamed for the empty screen owns the bottom of the chain. It can't produce markup from a snapshot that was never taken, or snapshot raw rows that never arrived. Everything above it is somebody else's obligation - and until the chain is written down, saying that out loud sounds like deflection.

One number per link

The measurement rule is what makes this cheap enough to repeat.

Per link, take exactly one number, with a cheap request at page size 1: read the total count off the response header, don't download the selection.

That's the difference between a method and an intention. Pulling rows for eight links across a vertical takes hours, so nobody runs it twice. Reading eight counts takes tens of seconds, so it gets re-run after every fix - and "before" and "after" become the same measurement rather than two investigations.

The number isn't evidence of correctness. It's evidence of presence. That's all a link has to answer: is there anything here. Whether what's there is right is a later question, asked only of the links that came back full.

Choosing that number is the one genuinely hard step, and it's a decision rather than a lookup: one thing per link is declared to be the fact, and every later reading uses it - including the ones that contradict what I expect.

A control subject next to the suspect

One number on its own is a guess. A link gets checked as a pair: a control subject where everything works, and the suspect, same measurement, side by side.

                  control subject          suspect subject
                ┌─────────────────┐      ┌─────────────────┐
schedule        │      n > 0      │      │      n > 0      │
                ├─────────────────┤      ├─────────────────┤
queries         │      n > 0      │      │      n > 0      │
                ├─────────────────┤      ├─────────────────┤
task            │      n > 0      │      │      n > 0      │
                ├─────────────────┤      ├─────────────────┤
runs            │      n > 0      │      │        0        │
                ├─────────────────┤      ├─────────────────┤
raw rows        │      n > 0      │      │        0        │
                ├─────────────────┤      ├─────────────────┤
snapshot        │      n > 0      │      │        0        │
                ├─────────────────┤      ├─────────────────┤
markup          │      n > 0      │      │        0        │
                ├─────────────────┤      ├─────────────────┤
rebuild         │      n > 0      │      │        0        │
                └─────────────────┘      └─────────────────┘

a conclusion from one field without the pair is a guess

In this breakdown I got that wrong twice, the same shape both times: a conclusion drawn from one field, without the pair.

The clearer of the two: I concluded a task had never reached the provider, because the field naming the provider on the keyword-query row was empty. Then I ran the same read against a control subject that already had data - same field, also empty, while the task's own status said it was accepting data. The field was never the fact; the status was. The full version of that trap, and the coverage factor next to it, is in Part 3.

The pair isn't a nicety or "running it twice". It's the only thing separating this entity has no data from this reading has no data - an empty number means both, and the response won't say which.

Both mistakes cost the same: a conclusion about a service that was doing its job. Worse, they're wrong in the expensive direction - they send work to a team with nothing to fix.

Eight cohorts

Once the chain is measured, every case lands in exactly one cohort:

result present - nothing to explain
schedule of a different type - never scheduled for this kind of collection · the scheduler
no keyword queries - nothing to collect for · the dictionary/catalog service
schedule present, no runs - it should have fired and didn't · the tool-task service
runs present, no raw rows - collected, nothing back yet · the provider + the raw-data service
raw rows present, no snapshot · the domain-rule-map service
snapshot present, no result · the rule-set-markup service
region the provider does not cover - decided from the configuration's region alone · the provider

COHORT                              OWNER OF THE EMPTY LINK      COUNT
─────────────────────────────────   ──────────────────────────   ─────
result present                      nothing to explain              76
schedule of a different type        scheduler                      184
no keyword queries                  dictionary/catalog service      14
schedule present, no runs           tool-task service                —
runs present, no raw rows           provider + raw-data service     21
raw rows present, no snapshot       domain-rule-map service          0
snapshot present, no result         rule-set-markup service          0
region the provider does not cover  the provider                     4

The rule that makes this useful: "unclear" is not a cohort. It's an unfinished measurement. A case that doesn't land means the next link hasn't been read - a task, not a verdict.

How a conclusion is phrased

Not "the service is broken". The form is:

link X is empty; the owner of link X is role Y.

That's the only phrasing I can hand to another team without a discussion attached. It names a fact and an address, it alleges no defect, and the receiving team can check it with the same number I read. "The markup service is losing data" starts an argument. "The snapshot link is empty, and that link belongs to the domain-rule-map service" starts a look.

It also survives being wrong. If the owner reads the same link and finds it full, the disagreement is about one measurement, not about whose service is at fault.

What the numbers did

The vertical had 276 configurations. Sorted by cohort:

Cohort	Configurations
result present	76
schedule of a different type	184
no keyword queries	14
region the provider does not cover	4
waiting for collection	21
raw rows present, no snapshot	0
snapshot present, no result	0

The last two rows carry the accusation. Those are the two states where the blamed service would actually be losing data - raw rows in, no snapshot; snapshot in, no result - and both are zero. Where there was input, there was output.

The 184 is the row to read twice. Those configurations are scheduled for a different type of collection: they were never going to have this result, and the system was right about every one of them.

After the sorting, two configurations were left as genuine questions: one whose schedule is active but has never fired once, and one whose tasks are created daily and sit in the provider's queue. Two hundred-odd "problems" became two - and neither sat in the service the complaint pointed at.

What it costs

Two costs, and I'd rather name them than pretend the method is free.

The chain has to be written down once, by someone who knows the whole path: eight links, eight owners, and per link a decision about which single number counts as its fact. Nobody hands you that - it comes out of reading the system, before any measurement starts.

And every link has to be measurable that cheaply, which isn't always true. Where a total-count header lies, or a schedule isn't exposed by any read at all, the link gets measured by hand, subject by subject. There the breakdown stops being a sweep and becomes enumeration, and the cost argument above collapses for that link. It's worth knowing which links those are before promising anyone a fast answer.

The part that generalises

A chain of links with named owners is a form, not a document. It's the shape in which work can be handed to any executor - a teammate, a new hire, a fast automated one - and checked afterwards without re-running it, because the instruction and the result are both single numbers with addresses.

Without that form, a faster executor just brings a wrong conclusion sooner. Speed multiplies whatever discipline is already there; it supplies none.

This is my current answer to "there's no result", and the price of it is written above. I'd like to hear the other side: if you do this better, tell me what your links are and how you measure them; if you have been through this, tell me which link turned out to be the liar; if you see it differently, say where the pair-based reading breaks down. How is it solved on your side, and what broke there?

Why there is no result - Part 2.

Next: the two factors that cut cases out of the breakdown before any link is read - a provider that does not cover a region at all, and a provider queue where a task is waiting rather than lost.

An Empty Screen Is Not Always a Failure

Anton Brilliantov — Mon, 12 Oct 2026 12:09:05 +0000

One visible complaint, six different causes - five of them outside the service being blamed.

👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go.

Most of what I do right now is carving a live PHP monolith into Go microservices, and much of that job is checking whether the extracted services behave the same way in production.

This series is about one question that keeps coming back from that work: why does this entity have no result? Running notes live on my GitHub: github.com/brilliant-almazov.

This is how I answer that question today, with the price attached. Maybe some of it transfers; maybe you look at it differently, and I'd like to hear how.

The complaint

Users open a screen and see nothing where they expect markup. The rule is configured, the result is empty.

The complaint arrives already addressed: the rule-set-markup service. One sentence, one accused service, no numbers.

It is a reasonable accusation: that service is the last link before the screen, and the last link is what the user sees.

The first way I counted, and why it was wrong

My first move was a sweep: run every object in the vertical through the endpoint and count the answers.

Roughly two-thirds of the objects answered not found. A further group failed on time. Read as a report card, that says: unusable.

It was wrong, and the reason is worth naming precisely. The sweep measured the endpoint, not the data.

Nobody had asked whether those objects were supposed to have a result at all. Not found is a truthful answer about an object that never had input. Counted as a defect, it becomes noise - and the handful of real failures drown inside it.

What the recount from the data showed

The second pass asked a different question of each object: does it have input, and if so, does it have output? That turns one complaint into arithmetic.

What	Value
Configurations in the vertical	276
With a source snapshot	76
Of those, with a result	76
Snapshot rows / result rows	433,348 / 2,239
"Raw rows exist, no snapshot"	0
"Snapshot exists, no result"	0

Those two zeros are the whole verdict on the accused service. Every object that had input had output.

Separately, a reference calculation of the same markup agreed with production on 1,752 entities with 0 discrepancies - so it is not producing merely something.

The empty screens are real, and the service blamed for them lost nothing. That is the actual shape of the problem.

Six causes behind one empty screen

The rest of the vertical does not collapse into a single explanation. It splits into named cohorts, and each has a different owner:

184 - the configuration's schedule collects a different type of data, one that produces no position tasks at all. Owner: the scheduler.
14 - the configuration has no keyword queries at all. Nothing to plan. Owner: the dictionary/catalog service.
1 - the schedule exists and is active, but has never fired once. Owner: the scheduler.
1 - tasks are created daily and stand in the provider's queue, waiting for collection. Queued, not lost. Owner: the provider-request service.
4 - the configuration's region is not covered by the provider at all. Decided by region alone. Owner: outside our system entirely.
and the sixth, the only one belonging to the accused service - failures on time, which turned out to be a property of the pace of my own sweep rather than of the objects.

Five of the six sit before the service that received the complaint.

The chain, and where it breaks

The cohorts are addressable because the path from a schedule to a rendered result is a chain of links, and every link has an owner:

LINK                                 OWNER                          EMPTY HERE
─────────────────────────────────    ──────────────────────────     ──────────────
schedule of the configuration        scheduler                      184 · 1
set of keyword queries               catalog service                14
collection task                      tool-task service              ·
task run + its counters              tool-task service              ·
raw result rows                      provider + raw-data service    1 queued · 4 region
snapshot of entities                 domain-rule-map service        0
applied rules → markup               rule-set-markup service        0
rebuild of the result on request     rule-set-markup service        timeouts

Read the right-hand column and the complaint stops being a complaint. It becomes a distribution: most of the emptiness is decided at the first link, and the two links owned by the accused service carry a zero.

The honest mistake, and what it cost

Here is the part I got wrong. Leaving it out would make this a sales pitch.

The first conclusion I published from the sweep was: "23 objects don't rebuild." That was a defect report against the service, and it was false.

The failures on time came in batches - 20 to 23 of them, back to back rather than scattered. I took the same objects and ran them one at a time with a pause: 21 succeeded immediately, 2 after a pause.

The failure was a property of the pace I was driving the endpoint at, not of the objects.

The cost was a whole extra round of measurement.

The cause is not mysterious: a conclusion drawn from one run without a control case. One run tells you what happened; it does not tell you what the run itself contributed.

What the breakdown changes in the work

While "no result" is a single complaint, it cannot be closed and it cannot be handed to anyone. It sits on whichever service is nearest the screen, and every conversation about it restarts from zero.

Split into cohorts with an owner each, every part has an addressee: the scheduler, the keyword catalog, the provider, the processing service.

Some parts turn out not to be work at all. 184 configurations that were never meant to produce this type of result are a fact about the setup, not a backlog item.

What remains is small, specific, and belongs to someone by name.

That is the difference. Not "the service is fine", but "here is where each empty screen stops, and here is who owns that link."

What it costs

The breakdown took several runs and a lot of manual work, because the system does not answer "why is this object in this state" through its own interfaces.

There is no place to ask whether a configuration even has a schedule that produces this type of task, so the answer is assembled by hand, one link at a time.

Until the system returns those answers itself, every new breakdown starts by clearing the noise from the previous one - a cost paid again on every investigation.

The part that generalises

A mass sweep is cheap now. Any executor can push a whole vertical through an endpoint before lunch.

The conclusion drawn from it is not cheap. "23 objects don't rebuild" was produced quickly and was wrong, and being wrong quickly cost more than being right slowly would have.

The discipline that matters is not speed. It is making the conclusion checkable without re-running the thing: a control case next to the suspect, one number per link, and a stated rule for what counts as a normal empty. Speed multiplies whatever discipline is already there - including its absence.

This is my experience and my price, on one vertical, in one system halfway through a migration.

So: maybe you do this better - maybe your services already answer "why is this empty" themselves. Maybe you have been through this and found where it stops being worth the effort. Maybe you see it differently and think the sweep was the right instrument all along.

How is this solved on your side, and what broke there?

Why there is no result - Part 1.

Next: the chain of links as an instrument - one cheap number per link instead of guessing which service to accuse, and the eight cohorts any case reduces to.

What Checks A Release When There Is No Canary

Anton Brilliantov — Sun, 11 Oct 2026 15:39:50 +0000

A canary and shadow traffic both ask for something the chain has to be able to do - the second binary is the cheap half of either one.

👋 Hi, I am Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services. This block has been about the things a fleet has to do identically or every service reinvents them; this last part is about the moment after all of them: a change is built, and the only question left is what stands between it and everybody. Running notes live on my GitHub: github.com/brilliant-almazov.

This is how I do it right now, with the price attached - maybe you already do it better, maybe you have been through it, maybe you see it differently.

Where this sits

A short primer, so this reads on its own.

A PHP monolith is being taken apart into Go services. The Go API gateway is the only door into the new services from outside; behind it the extracted services talk to each other over gRPC, and every one of them is raised from the same shared platform library - which is where health endpoints, metrics, tracing and the messaging layer come from, rather than from each service writing its own. Some work is answered inside the call; the rest is published as a domain event and picked up later by a background handler in a separate process.

Two things follow from that shape and both matter below. The platform supplies the observation, so the question is never "do we have metrics" but "do the metrics separate the thing we are comparing". And the publish boundary is a real boundary: everything past it is a different process, a different tree in the trace, and - as it turns out - a different answer to whether a copy of a request is harmless.

What stands between a commit and everybody

Before any opinion about canaries, the inventory - counted rather than described.

Three pipelines. A build fires on a pushed tag vX.Y.Z and stamps the version and the commit into the binaries. Tests run on every pull request, on every push to the main branch, and nightly on a schedule. Deployment is a manual pipeline run with two parameters: which version, and which services to move - server, worker, all.
67 records in the machine-checked metrics snapshot: 59 of them come from the platform library, 6 from the service, plus one <dynamic> entry. 13 domain metrics sit on top of that. Drift between the snapshot and reality fails CI.
Nine alert rules, described in text and created on the fleet monitoring side. The dashboard is assembled by a generator next to the rest of the fleet dashboards; neither dashboards nor alerts live as files in the service repository.
86.7 % line coverage, checked on every run against a ratchet whose threshold currently sits at 0.
Zero requests routed into a canary.

That last line is not an omission in the list. There is no routing of a marked request into a new version in production on my side, and the previous part is the reason: the mark has to reach the last hop, and mine would not have.

What "checking a release before everyone" actually means

Two techniques, and it is worth writing them down in one line each, because they are routinely discussed as if they were the same thing.

A canary. Part of the live traffic is sent to the new version while everybody else stays on the old one, and the two are compared on the metrics that matter.

Shadow traffic. A copy of the live traffic is sent to the new version, and the answer the user receives still comes from the old one. Nothing the new version produces is shown to anybody.

Both answer exactly one question: does the new version behave the same way on real input. Neither of them answers it by being deployed. They answer it by being observable, which is a different requirement and a more expensive one.

The tempting reading is that the hard part is running two versions at once. It is not - on a deploy that already takes a version as a parameter, standing the second one up costs one more pipeline run. The hard part is everything the two techniques assume about the chain around them, and that is what the rest of this is about.

How this is usually done

Four shapes come up in public write-ups and standards, and all four are reasonable: a gradually increasing share of traffic with the failure rate watched as the dial moves (CanaryRelease); mirroring requests into a second, non-answering path; two colours side by side, where a whole second environment is raised and traffic is switched to it at once (BlueGreenDeployment); and a sandbox on a copy of the environment, exercised with no live traffic at all.

The previous part compared those four by blast radius - how much of the world meets one mistake. This part compares them along a different axis, and it is the axis that decides whether you can have them at all: what each one demands from the chain and from the data. I am naming no gateways, no service meshes and no tracing products; the choice of product is not what makes any of this work or fail.

What each way needs

Three rows. The last one is not a technique, it is what is actually there.

Way	What has to work	What it costs	Have it
canary	mark reaches the last hop; two generations, one set of rows	a share of traffic; a version cut in metrics	no
shadow traffic	a copy with no side effects; writes split from reads	reads served twice; a separate write path	no
what I have	executable checks before, metrics and alerts after	a defect found after everyone met it	yes

Read the first row from the right: the two things a canary needs before it is worth anything are a marked request that survives every hop and metrics that can tell the generations apart. Neither of them is the second binary.

There are no numbers in that table on purpose. The share of canary traffic, the observation window, the rollback threshold, how long a rollout takes - none of those has been measured on my side, and a plausible figure would be worse than a visible hole.

The canary runs into the wall from the previous part

Short, because that part already did the work. A canary needs the mark on the request to survive every hop; if it stops after the first one, the first service runs the new code and everything it calls afterwards runs the old one - one user action executed by two generations, with each half looking correct from where it stands. It needs the contract to be compatible in both directions, and data written by the new version to be readable by the old one. It needs whoever is testing to know which branch they are in, or the debugging is done blind.

None of that is technical difficulty. It is two generations of a system being true at the same time, and people holding both in their heads while they work.

There is a second wall, and it is the quieter one: there is nothing to compare the generations by. The tenant identifier is deliberately kept out of metric labels for cardinality reasons, and there is no version dimension in the snapshot today either. A canary without metrics that separate the two versions is not a controlled experiment - it is a rollout on live people with someone watching a graph that cannot tell them apart.

Where a shadow contour stops being free

Shadow traffic looks like the safe one of the two, because nothing it produces reaches a user. That holds exactly as long as the copied request does not write anything.

The facts on my side that decide it:

delivery is at-least-once, and the broker does not deduplicate;
a redelivery carrying the same message_id is normal behaviour, not an error - the incident is a spike in redeliveries, not their existence;
the normal state of the outgoing queue is empty, which is what makes a spike legible at all.

So a mirrored request that gets as far as publishing does not produce a shadow message. It produces a real message, on the real stream, for the real consumers, indistinguishable from one a user asked for.

Which gives the rule in one line: a shadow contour is meaningful exactly up to the point where writing starts. Past that point it needs a separate path for every write, and building that path costs about what building the new version cost. That is the whole reason shadow traffic is rare in systems that publish events, and common in systems that mostly read.

What actually checks a release here

With both techniques unavailable, the honest question is what is left - and the answer is a list of things that fail the build rather than a list of intentions.

Before the deploy, the test pipeline runs, in this order:

  build
  linter
  bring up the shared test containers        one script, one container per type, per run
  run the tests with coverage
  check the time budget                      a separate step, not a note in the log
  tear the containers down                   if: always()
  coverage report
  threshold check
  ratchet                                    raising it is allowed, lowering fails the PR

There is one container of each infrastructure type for the whole run - isolation between tests is done with schemas and data, never by raising another container - and the teardown is explicit rather than left to a reaper. The time budget is a step of its own for the same reason everything else here is: a run that quietly got three times slower is a defect, and a defect that fails nothing is a defect nobody will look at.

Nightly, a separate job runs the race detector and the volume tests with -p 1, so that a comparison of ns/op between runs means something.

Two catalogues fail the build on drift. The environment variable catalogue - 59 variables, 45 from the platform and 14 from the service - and the metrics snapshot above. Both are machine-checked artefacts, not documents somebody is supposed to update.

One check exists purely because of a silent failure mode. The version is stamped into the binary with a linker flag, and the linker silently ignores a flag aimed at a symbol that does not exist: the build stays green and the version stays dev - in the audit trail, in the trace attribute, everywhere. So the build target compiles both binaries with a test tag and greps the binary for it. Empty output means the symbol is wrong.

After the deploy, /health and /ready are served by the platform, and there is no "wait for health" step in the pipeline - that check belongs to the environment. Then the metrics and the nine alert rules.

The case: the ratchet is wired, the bar is not set

This is the one I would fix first, and it is small enough to state in three lines.

  coverage    0 %                              86.7 %              100 %
              |------------------------------------|-----------------|
  threshold   ^ 0                                  ^ actual
              the ratchet can only move right - and it starts here

What is there. A coverage ratchet: the threshold can be raised, never lowered, and a pull request that tries to lower it fails.

What it costs today. The threshold sits at 0 while actual coverage is 86.7 %. So the mechanism is wired and works in one direction only - it cannot be walked back - but the bar it is holding is on the floor. Lowering coverage is impossible; shipping a release with coverage far below 86.7 % is not, and nothing in the pipeline would notice.

What is holding form next to it. The two drift checks and the container discipline above are the same idea done properly: an artefact that fails the build, not a rule somebody is meant to remember.

The reading that generalises: a release here is checked by executable checks before the deploy and metrics after it. A canary is not in that row, and neither is anything else that depends on a marked request travelling further than the first hop.

Why it is arranged this way

Because a check you can miss is not a check. The things listed above all share one property: they fail something. The linter fails, the threshold fails, the snapshot drift fails, the ratchet fails a pull request. That is worth more per hour spent than any technique whose failure mode is "nobody looked at the dashboard". So the order is: ratchet and prohibiting tests first, canary second.

Because the observation is not there yet. With no way to separate the two generations in the metrics, a canary would produce a feeling rather than a comparison. Building the version cut is the prerequisite, not the follow-up.

What it costs

Stated plainly, because the arrangement above is a trade, not a victory.

A defect that tests did not catch is found after everyone has met it. The blast radius of a mistake is the entire traffic. That is the price of the whole section above, and it is not a small one.
The ratchet holds nothing at the moment. 0 against 86.7 % - wired, one-directional, and set on the floor.
The nightly signal arrives a day late. Races and volume are checked once a night, not per pull request, so a race introduced in the morning is a tomorrow problem.
One check exists only because a tool fails silently, and silent failures of that shape are almost certainly not down to one.
A human is part of the deploy procedure. A manual run with two parameters means somebody picks the version and the service set, and that choice is checked by nothing.

When not to do this

When traffic is small, a canary gives you no statistics. A share of a small number is a smaller number, and a comparison on it is noise wearing a lab coat.
When writes are not separated from reads, a shadow contour is dangerous, not cautious. The copy stops being a copy at the first publish.
When the mark does not reach the last hop, a canary checks the first step and lies about the rest - and a half-migrated request is worse than no rollout mechanism at all.

And the boundary of this write-up, said outright: there is no canary and no shadow contour here, there is no routing of a marked request in production, there are no declared SLAs and no formal SLO, and the coverage bar is not set. Everything above is a breakdown of what these techniques require and what they cost, plus an inventory of what is holding the line instead - not a report on a working scheme.

The multiplier line

An automated executor makes the mechanical half of this cheap: wiring the threshold check, writing the teardown step, generating the snapshot, adding the version cut to a metric. What it does not do is decide that a check which cannot fail the build is not a check, or notice that a mirrored request stops being a copy at the first publish. Those are judgements, and they are the whole content of this article. Speed amplifies whoever set the definitions; it does not supply them.

Access, tracing, rollout - Part 6. That closes this block: who arrived, what they may do, where their request has been, what decides a branch, how one request reaches a new version, and - here - what checks that version when nothing routes traffic to it yet.

If you do this better, tell me what your release has to pass before it reaches everybody, and which of those checks can actually fail a build. If you have been through this, what did your shadow copy turn out to write that nobody expected? If you see it differently, say where shipping to everyone at once and watching the graphs is the honest answer. How is it solved on your side, and what broke there?

The Mark That Stops After The First Hop

Anton Brilliantov — Sat, 10 Oct 2026 13:31:44 +0000

A new version is not ready when it is up. It is ready when the mark reaches the last hop - and mine would not have.

👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services. This block is about the things a fleet of services has to do identically or every service reinvents them; this last part is about putting a new version in front of one request instead of all of them. Running notes live on my GitHub: github.com/brilliant-almazov.

This is how I do it right now, with the price attached - maybe you already do it better, maybe you have been through it, maybe you see it differently.

The answer I already had

This starts with something that was already broken, before the idea below was on the table.

An audit of one of my own services asked a single question: where in this service is there code that the shared platform library already provides. One finding reads like a footnote and is not one:

The publish was going out with empty headers.

The first consequence was a decoding one. With no header to select a codec by, the platform typed receive was physically inapplicable, so the consumer had to call JSON decoding by hand, in the handler, on raw bytes.

The second consequence is the one this article is about. Anything that travels in headers did not cross the queue boundary. Not the trace context, not the claim about who arrived, and not a mark on the request.

A mark on the request is exactly the mechanism the rollout technique below stands on. So the question "will the mark reach the last hop" was never open on my side. It already had an answer, and the answer was no.

Where this sits

A short primer, so this reads on its own.

A PHP monolith is being taken apart into Go services. The Go API gateway is the only door into the new services from outside; behind it the extracted services talk to each other over gRPC, and every one of them is raised from the same shared platform library. Two things are already settled at that door: the edge parses the input once and passes on a claim about who arrived, carried in the call metadata; and, as the previous part argued, a flag is decided there too - the service reads a finished yes or no rather than the address of a flag store.

Some work answers inside the call. The rest is published as a domain event and picked up later by a background handler in a separate process. That second half is where every mechanism in this article either survives or quietly stops.

The technique, in one paragraph

The new version is raised beside the old one. The standard path stays exactly as it was for everybody. Only a request carrying a mark is routed into the new version.

The mark is not a new invention. It is the same kind of statement as the identity claim and the flag: a decision taken before the service runs, travelling in the call metadata, with the lifetime of one request. Nothing new appears in the service; a field that already exists gets one more neighbour.

That paragraph is the entire appeal of the technique, and it is also why it is easy to underestimate. Standing a second copy of a binary next to the first one is a deployment parameter. Getting a mark to the end of a chain is not.

The condition that makes or breaks it

Written as a condition, because that is the only form of it that can be checked:

The technique is ready when the mark arrives at the last hop - not when the second version is up.

Stated that way it becomes something a test can hold: send one marked request, walk the whole chain, and assert that the last participant saw the mark. Stated as an intention - "we will pass the flag through" - it becomes a rule, and a rule that produces no symptom when broken is a rule that is already broken somewhere you have not looked yet.

If the mark stops after the first hop, the first service runs the new code, and every call it makes afterwards - synchronous or not - goes to the old one. That is not "the rollout is partially on". It is one user action executed by two generations of code, with nobody in a position to notice, because each half looks correct from where it stands.

  edge          hop 1            hop 2         hop 3         hop 4
  decides  -->  new version  |   old        -->  old      -->  old
  mark: yes     mark: yes    |   mark: -        mark: -       mark: -
                             |
                       the mark stops here

The two places the thread breaks

Both of them are already documented on my side, which is the reason I trust the condition above rather than my optimism.

A dropped call context. The context is passed as the first parameter everywhere - handlers, repositories, managers, background handlers, the publish call itself - and it is checked by cancellation tests. Breaking that rule produces no symptom: nothing fails to build, nothing throws, no response code changes. It shows up as a thread that quietly stops being connected.

A publish with empty headers. The finding above. Everything the messaging layer needs in order to know what it is holding travels in the same place as everything the rollout needs in order to know which generation this request belongs to. When those headers are empty, both are broken at once - and fixing either one fixes the other.

The useful way to read that pair: the mark is not asking for a new transport. It is asking for the transport that was supposed to be working already.

The message that repeated forever

Here is the concrete episode, from the same audit, and it is the reason I treat contract compatibility and consumer behaviour as one requirement rather than two.

What we had. A subscription that set no redelivery limit, no backoff and no dead-letter queue. A decode error was returned as an ordinary error. The driver did what a driver does with an ordinary error: it redelivered.

What that cost. One message the consumer could not parse occupied the stream and was discarded by nothing. Not slowed down - repeated, indefinitely, with nothing in the arrangement able to give up on it.

Why it belongs in an article about rollout. Two generations of code on one stream is precisely the situation where a message the consumer cannot parse arrives as a matter of routine rather than as an accident. The new version writes an event the old consumer does not understand, or the reverse, and the poison loop above is no longer a latent defect - it is the expected outcome of the technique working as designed.

So the requirement is one requirement, not two: the contract is compatible in both directions, and the consumer has a defined behaviour for a message it cannot parse. Satisfying only the first is how you find out about the second.

How this is usually done

Four shapes show up in public write-ups and standards, and all four are reasonable:

Routing on a header at the entry layer. The decision is taken once at the door, and the entry layer sends the request to one of two upstreams. This is the shape the technique below is a version of.
Two colours side by side. A whole second environment is raised, traffic is switched to it at once, and the old one is kept ready to switch back - see BlueGreenDeployment.
A gradually increasing share of traffic, with the share moved up while the failure rate is watched - see CanaryRelease.
A separate sandbox on a copy of the environment, where the new version is exercised with no live traffic at all.

The four side by side, in the only terms that decide between them:

Shape	What it needs from the chain	Blast radius of one mistake	What it costs
routing on a header	the mark reaching every hop	one marked caller	a mark that must survive the whole chain
two colours	nothing - the whole environment moves	everything, briefly	a second environment, kept warm
a growing share	nothing at the chain level	the current share	a way to watch failure rates and move the dial
a separate sandbox	nothing - no live traffic at all	none	a copy of the environment, and no real load in it

I am not arguing any of them is wrong, and I am deliberately naming no gateways, no service meshes and no entry providers - the choice of product is not what makes this work or fail. The row that matters for this article is the first one, and its middle column is the reason to want it: the blast radius is one caller who agreed to be that caller. Its right-hand column is the reason it is hard.

How it lands on the deploy I already have

Worth separating what is already there from what is not, because they land very differently.

What is already there. Deployment is a manual pipeline run with two parameters: the version, and which services to move (server, worker, all). Tags are set by one person, pseudo-versions are forbidden, and a service carries exactly the tag it was given. The version and the commit are stamped into the binary with linker flags, and there is a build target that checks the symbol is alive - because the linker will silently ignore a flag aimed at a symbol that does not exist, leaving the version at dev and sending dev into the audit trail and into the trace attribute.

So "stand the new version beside the old one" costs nothing new: it is the same pipeline, run twice, with two version arguments. There is no new code, no new component and no new runtime.

"Get the mark to the last hop" lands on nothing at all. It is the part that has to be built, and the two breaks above are its actual scope.

One honest boundary while I am on the subject of the pipeline: there is no "wait for health" step in it. Health and readiness are served by the platform, and the check after a rollout is the environment's job, not the pipeline's.

What it costs

The four items below are the ones I would put in front of anybody who wants to try this, and none of them is a token entry.

Two generations of code live at the same time. Not two branches in a repository - two running versions, both correct, both reachable, both maintained. Every fix in the period applies to a question nobody enjoys answering: which of the two, or both.

The contract has to be compatible in both directions. Forward compatibility is the one people plan for: the old side must tolerate what the new side sends. Backward compatibility is the one that gets discovered: the new side must tolerate what the old side is still sending, for as long as the old side exists.

Data written by the new version is read by the old one. On my side that is survivable by construction, because there is no destructive UPDATE in the domain tables - a change is a new version of a row, and the old generation keeps seeing its own rows. That holds right up until the new version starts writing something the old version cannot read at all, at which point the storage layer, not the contract, is what ends the experiment.

Whoever is testing has to know they are in the new branch. Otherwise debugging is done blind: a behaviour is observed, attributed to the wrong generation, and the conclusion is confidently wrong. This is the item that looks smallest on the list and produces the most wasted hours.

  old version  <-- contract compatible both ways -->  new version
       |                                                   |
       +-------------------> data <-----------------------+
                  written by new, read by old

Put together, the shape of the price is not what it looks like from the deployment side:

This is organisational complexity, not technical complexity. The technical part is a routing rule and a header. The expensive part is two generations of a system being true at the same time, and the people around it having to hold both in their heads.

Why do it at all, then

Two reasons. I do not have a third.

The alternative prices every mistake at the whole traffic. Rolling out to everybody at once is not a cheaper technique; it is the same technique with the blast radius set to maximum. The marked request exists so that the first thing a mistake meets is one caller who agreed to meet it.

The mechanism already exists. The mark is the same decision, taken at the same place, travelling in the same call metadata as the identity claim and the flag. No flag store appears, no client appears in each service, no cache and no refresh loop appear - which matters, because a piece of state with a refresh loop is runtime, and runtime is what the platform provides. The technique earns its keep partly by not adding anything.

When not to do this

When the chain is longer than it can carry the mark. Then the honest order is: stitch the chain first, argue about routing second. Routing a mark into a chain that drops it produces a half-migrated request, which is worse than no rollout mechanism at all.
When the change touches storage in a way that makes reading back impossible. Compatibility in both directions is not achievable there, and the technique simply does not apply - no amount of routing discipline fixes a row the other generation cannot read.
When there are few enough consumers to move all of them at once. Then the whole apparatus is ceremony, and the cheaper answer is a coordinated switch.

And the boundary of this write-up, stated plainly because it matters: there is no routing of a marked request into a new version in production on my side. This is a breakdown of a technique and its price, not a report on a working scheme. There are also no numbers here - not the share of marked traffic, not how long two generations coexist, not the number of hops in the chain, not how long a rollout takes - because none of those have been measured on my side. A hole you can see is worth more than a plausible figure.

The multiplier line

An automated executor makes the mechanical half of this cheap: threading one more field through the call metadata, setting headers at every publish site, writing the test that fails when the mark does not reach the background handler. What it does not do is decide that the readiness criterion is the last hop rather than the second binary, or notice that a poison loop and a contract change are the same requirement seen from two sides. That judgement is the work. Speed amplifies whoever set the definitions; it does not supply them.

Access, tracing, rollout - Part 5. Next, and last in this block: checking a release before everybody starts using it - the coverage ratchet sitting at zero, one infrastructure container for the whole run, and what drifts between what is declared and what is actually there.

If you do this better, tell me how you prove a request mark reaches the last hop rather than the first. If you have been through this, what did your two coexisting generations turn out to disagree about? If you see it differently, say where rolling out to everyone at once is the cheaper answer. How is it solved on your side, and what broke there?

A Flag Is A Decision Made Before The Service

Anton Brilliantov — Thu, 08 Oct 2026 20:40:50 +0000

The service receives a finished yes or no the same way it receives who arrived - not the address of a flag store.

👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services. This block is about the things a fleet has to do identically or every service reinvents them; this part is the shortest of the four, and it is one thought with one price attached. Running notes live on my GitHub: github.com/brilliant-almazov.

This is how I do it right now, with the price attached - maybe you already do it better, maybe you see it differently.

Where this sits

A short primer, so this reads on its own.

A PHP monolith is being taken apart into Go services. The Go API gateway is the only door into the new services from outside; behind it the extracted services talk to each other over gRPC, and every one of them is raised from the same shared platform library. Identity is already settled at that door: the edge parses the input once and passes on a claim about who arrived, carried in the call metadata next to the call itself. By the time a service runs, who is calling is a finished statement, not a question.

This part asks whether a second statement can travel the same way.

The thesis, in one line

A flag is not a variable inside a service. It is a decision made before the service runs.

The edge decides whether a branch is on for this caller, and the result travels the same way the identity claim does. The service reads a finished yes or no. It never learns where the answer came from, and it is never handed the address of a flag store.

a flag client in every service        the decision at the edge
------------------------------        ------------------------

  [service A] --+                       +-- [service A]  yes
   config       |                       |
   cache        +--> [ flag store ]     |
   refresh      |                     [edge] -- [service B]  no
  [service B] --+     each service     decides  |
   config       |     keeps its own    once     |
   cache        |     copy of the               +-- [service C]  yes
   refresh      |     machinery
  [service C] --+                       services hold no config,
                                        no cache, no refresh

The case that made me write this down

Not a production incident, and I will not dress it as one. It is an audit finding.

What we had. An audit of one of my own services asked a single question: where in this service is code that the platform library already provides? Among the findings, two matter here.

Three caches written by hand - a map plus a read-write mutex, no TTL, no capacity limit, no metrics. And a retention schedule nailed down as a constant in the code instead of arriving as configuration.

What that cost. The platform library already ships a cache with a TTL, a capacity bound and metrics, and configuration arrives as an import rather than as code somebody writes. So each of the three hand-written copies was a piece of runtime that nobody could observe: no eviction anyone could reason about, no size anyone had bounded, no number on a dashboard when it misbehaved. Nothing was on fire. That is exactly the problem - a piece of runtime with no metric under it does not announce itself, it just sits there being load-bearing.

Why it belongs in an article about flags. A flag client living inside a service reproduces exactly these shapes - a piece of configuration, a cache, a refresh loop - and adds one more on top. It is the same mistake with a nicer name.

How this is usually done

Three shapes show up in public write-ups, and all three are defensible depending on what you are building:

An in-process client of a flag store in every service, with a local cache and a background refresh. This is the default in most write-ups on the subject, and its appeal is real: the decision sits next to the code it affects, so the branch reads naturally.
Flags as configuration that ships with the deploy. No runtime lookup at all - a value arrives with the release. Simple and observable; the cost is that changing it means a deploy.
The decision made on the entry layer, with the result attached to the request as it travels inward. Less common, and the one this article is about.

The three shapes, side by side, in the only terms that decide between them:

Shape	What the service holds	Who changes the value	What a change costs
client in every service	configuration, cache, refresh	a human, in an admin page	nothing at request time, a runtime layer forever
configuration with the deploy	one value, read at start	whoever cuts the release	a deploy per change
decided at the edge	nothing	a human, at one participant	one more field in the call metadata

Martin Fowler's Feature Toggles is still the clearest public breakdown of the categories and their lifetimes, and the vendor-neutral OpenFeature specification is worth reading for what a flag evaluation is actually made of - a key, an evaluation context, a default. Both describe the first shape, because that is what most systems need. I am not arguing they are wrong. I am saying the first shape has a price that a fleet of small services pays repeatedly.

What a flag client in a service drags in

Four things, and each one is a separate piece of runtime:

Its own configuration - where the store is, how to authenticate to it, what the defaults are.
Its own cache - because you are not making a network call per branch.
Its own refresh - because a cache that never updates is a constant with extra steps.
Its own admin page - somewhere a human changes the value and sees what it currently is.

There is a standing rule on my side that decides this before taste gets a vote: the platform owns the runtime, and writing your own version of a platform layer is forbidden. The service writes its input, its processing and its output. Everything else - the cache, the configuration, the scheduling, the metrics registry - is an import.

So "a flag client in every service" is not a preference I dislike. It is a request to hand-write a runtime layer in N services, in a fleet where that is already prohibited, and where an audit has already found the exact same shapes written by hand three times over in a single service.

One line, and it is the only thing here I would defend without qualification:

A "small" feature that has state and a refresh is runtime.

How I do it - one example

One behaviour branch, enabled for some callers and not others. Neutral domains: an entity belongs to an account, and an order is written against it.

The edge decides, once per request, whether the new branch is on for this caller. It is the only participant that has the caller resolved and the only one that has to be told anything.
The result travels in the call metadata, next to the claim about who arrived - the same channel, the same lifetime, the same rules.
The handler reads a finished yes or no. No lookup, no client, no cache, no default value to get wrong.

And the part that is easy to forget until it bites: the mark has to survive the queue boundary the same way the call context does. If the synchronous half of the work runs the new branch and the published message goes out without the mark, the background handler picks up the old branch - and now one user action ran two different versions of the same logic.

  edge          handler         publish      background      next call
  decides  -->  new branch -->   ...     |   old branch  -->  old branch
  mark: yes     mark: yes                |   mark: -         mark: -
                                         |
                                   the mark stops here

That failure is worse than the branch being off everywhere, because it is not visible from either side. The synchronous path looks correct. The background path looks correct. Only the pair is wrong.

What actually travels

Worth being precise, because "the edge decides" is easy to over-read.

What travels is a mark on this request: this call, for this caller, runs the new branch. It is a fact about one request, with the lifetime of one request, sitting in the same place as the claim about who arrived.

What deliberately does not travel is the rule that produced it - the percentage, the account list, the date after which everyone gets it. None of that is a service's business, and none of it belongs in a message that crosses four processes. A service that received the rule instead of the answer would be right back to evaluating flags, only with a worse input.

That distinction is also what keeps the compatibility story sane: a mark is a boolean with a name, and a name plus a boolean is a thing you can add, deprecate and remove. A rule is not.

Why the edge and not the service

Two reasons, and I do not have a third.

The decision needs the caller, and the caller is resolved at the edge. The service receives an input that has already been parsed; whether it also receives the resulting yes or no is a matter of putting one more field next to one that is already there. Deciding in the service means re-deriving something that was already known one hop earlier.

Nothing new appears in the service. No flag configuration, no cache, no background refresh - which is to say, no runtime that then has to be maintained separately in every service that has a flag. The count matters here: one decision point does not scale with the number of services; a client in each of them does.

What this costs

An honest list, not one token item:

Product logic moves to the edge. "Is this branch on for this account" is a product decision now living in the door, which is not where anyone looks for it. That is a real navigational cost for whoever debugs it next.
The set of marks in the call metadata grows, and it becomes part of the compatibility surface between the edge and everything behind it. Removing one is a fleet-wide change.
Local development needs a way to inject a mark by hand. Otherwise a service is only runnable with the edge in front of it, and the developer experience quietly degrades into "you can only test this on a stand".
A mark that does not survive the whole chain gives you a half-enabled branch - the failure described above, and the most unpleasant of the lot.
What I have just described is a direction, not a running system. There is no flag store on my side and no page to manage flags from a browser. The cost of "its own admin page" is therefore named here, not measured - I am counting a cost I have not paid yet, and it belongs in the list marked as such.

When not to do this

When the switch is purely technical and lives inside one service. A timeout, a batch size, a worker count - that is configuration, not a flag, and it arrives as an environment variable derived from the resource declaration. Routing it through the edge would be ceremony.
When the decision depends on data the edge does not have. This is the same dividing line as with permissions: a question that needs rows is answered where the rows are, which is the service. The edge decides what it can decide from the caller alone.
When there is exactly one service. Then there is no fleet to protect from N copies of a runtime layer, and the whole argument is premature.

And the honest boundary of this write-up, stated once more because it matters: there is no flag store here, no admin page for one, and no marked request routed into a new version in production. There are also no numbers - not how many flags, not the TTL of a flag cache, not a refresh delay, not a share of enabled callers - because none of those have been measured on my side. A hole you can see is worth more than a plausible figure.

One line

A flag is an answer, not a source. The service reads the answer; it does not go and fetch one.

The multiplier line

The mechanical half of this is cheap now: threading one more field through the call metadata, adding a parameter to a handler signature, writing the test that fails when the mark does not reach the background handler. What did not get faster is the judgement that a flag is a decision rather than a lookup, and that a piece of state with a refresh loop is runtime no matter how small it looks in the diff. Get that wrong and you get a perfectly written flag client, in every service, passing review because it looks like every other flag client. Speed amplifies whoever set the definitions; it does not supply them.

Access, tracing, rollout - Part 4. Next: what follows from this - a marked request routed into a new version running beside the old one, two generations of code alive at once, and the price that is organisational rather than technical.

If you do this better, tell me where your flag decision is evaluated and how the answer reaches a background handler. If you have been through this, what did your half-enabled branch turn out to break? If you see it differently, say where a client in every service is worth the runtime it brings. How is it solved on your side, and what broke there?

The Trace Ends Where The Message Starts

Anton Brilliantov — Wed, 07 Oct 2026 21:19:04 +0000

The synchronous path is one tree; everything after the publish is another one - and the audit found out why: the headers were empty.

👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services. This block is about the three things a fleet of services has to do identically or each service reinvents them: who arrived, what they may do, where their request has been. This part is the third one - where the request has been, and the exact point where the answer stops being available. Running notes live on my GitHub: github.com/brilliant-almazov.

This is how I do it right now, with the price attached - maybe you already do it better, maybe you see it differently.

The shape of the system, briefly

A PHP monolith is being taken apart into Go services. The Go API gateway is the only door into the new services from outside; behind it the extracted services talk to each other over gRPC, and each of them is raised from the same shared platform library. Some work is synchronous and answers inside the call. The rest is published as a domain event and picked up later by a background handler in a separate process.

That last sentence is the whole subject of this article. Everything before the publish is one call. Everything after it is a different process, at a different time, with a different life expectancy - and by default, a different trace.

What arrives without a line of service code

Before any argument about instrumentation, the inventory. A snapshot taken from a freshly generated service - name, type, help text, labels, histogram buckets, where each one is declared - reads 67 records: 59 of them from the platform, 6 from the service, plus one <dynamic> entry for the dynamic-metric factory. On top of that the service declares 13 domain metrics of its own: queue depth, age of the oldest queued row, relay cycle duration, publish failures by reason, resolve duration, audit records written.

Collection is pull. An agent scrapes the system port with a bearer token and remote-writes into a store; the application never pushes anything. Traces leave over OTLP.

The line I would keep from that paragraph: observability here is an import, not a body of code inside the service. Health, readiness, the metrics endpoint, gRPC metrics, broker driver metrics, connection pool instrumentation, trace export - all of it comes with the runtime the service is built on. Nobody writes it per service, so nobody writes it slightly differently per service.

The one thing the service still has to do

Exactly one requirement is left on the service side, and it is small enough to state in a sentence:

Do not lose the call context.

It is passed as the first parameter everywhere - handlers, repositories, managers, background handlers, and the publish call itself - and it is checked by cancellation tests. In tests the context comes from the test runtime; an empty one is forbidden.

That last clause is the part that makes the requirement real, and it took a while to arrive at. A rule about context has a nasty property: breaking it produces no symptom. A dropped context does not fail a build, does not throw, does not change a response code. It shows up as a trace that quietly stops being connected, which nobody notices until they need it.

So the requirement is not written as a rule. It is written as a failure: a handler that drops the context fails its cancellation test, because the operation no longer ends when the caller goes away.

The case: a requirement that was only a rule

Here is what it looked like while it was still an agreement, and the numbers are the point.

What we had. "Pass the context everywhere" as a written rule, honestly believed by everyone. Meanwhile the test code had accumulated 421 calls building an empty context, across 184 test files.

What that cost. A test built on an empty context does not cancel with itself, so it hangs until the package timeout instead of failing. Worse, it asserts nothing at all about the context: if production code dropped the incoming context and substituted its own, that entire suite would stay green. A test suite of that size, running on every push, was structurally incapable of noticing the one failure mode the rule existed to prevent.

What changed. Four things, and only the last one is enforcement:

the context comes from the test runtime, t.Context() and b.Context();
a port stub keeps the context it was handed, as a field, and gives it back to the test - instead of accepting it as an ignored parameter;
every transport, repository, manager and consumer has a cancellation test: cancelled context in, an error that is true under errors.Is(err, context.Canceled) out, and no row written;
writing a test with an empty context is blocked at the tool level, so the count cannot climb back.

The ladder underneath this is the same one from the first part of this block: a reminder lives one session, a rule works while it is being read, a check works. 421 is what the middle rung looks like after a year.

How this is usually done

Four shapes show up in public write-ups and standards, and all four are reasonable:

A correlation identifier passed by hand. One header, generated at the door, logged by everyone. Cheap, works everywhere, and gives you grep rather than a tree - you can join records, but you cannot see nesting or duration.
Standardised context propagation in the transport. W3C Trace Context defines the headers; client and server libraries carry them, and application code never touches them.
Instrumentation through interceptors rather than call sites - one place per protocol wraps every call, so no handler contains tracing code.
A separate collection layer next to the application, receiving spans and forwarding them, so the application knows one export protocol and nothing about the backend.

We sit on the second and third: propagation is a property of the transport, instrumentation is a property of the platform library, and the service contributes neither. Which leaves the asynchronous path, where the transport is a queue and no library does it for you unless you decide what travels. The messaging conventions say the same thing in more words: the producer puts the context into the message, or there is no link.

How we do it: what comes from where

The whole arrangement fits in one table, and the last column is the useful one:

What you see in the trace	Where it comes from	What the service does	How it breaks
service version attribute	the value the build stamps in	nothing	a dead `-X` target leaves `dev`
spans of the synchronous path	platform instrumentation	passes the context first, everywhere	a dropped context
durations	platform and service histograms	declares its domain metrics	names drift apart
the link to background work	the message headers	sets them at publish	empty headers

Three of those four failures are invisible in a green build. That is the shared property worth naming: every way this breaks produces correct-looking output. The service answers, the data is right, the tests pass, and the trace is quietly wrong.

Where the thread breaks

Now the specific loss, the one this article exists for.

The synchronous path traces beautifully. Door, service, validation, transaction, publish - one root, nested children, durations that behave the way nesting says they should. Then the message lands in a queue, a worker in another process picks it up some time later, and that work has its own root. Two trees, no edge between them.

one trace                              a second tree, its own root
---------                              ---------------------------

gateway root                           consumer handler
└─ service                             ├─ decode, called by hand
   ├─ validate                         └─ domain work
   ├─ transaction
   └─ publish   ← the last span            no parent, no link back
                  with a parent

The thread across that gap is stitched with message headers - the same mechanism the serialisation already uses. And here is the finding from our own audit of the service, which is why this is written as a direction and not as a report:

The publish was going out with empty headers.

The consequence was not primarily a tracing one. With no header to select a codec by, the platform's typed receive was physically inapplicable - the consumer had to call JSON decoding by hand, in the handler, on raw bytes. One defect, two problems: the receiving side could not be typed, and the trace could not be continued.

That is the part I find genuinely useful to notice. The headers you need for propagation are not an extra tax invented by observability. They are the same headers the messaging layer already needs in order to know what it is holding. If they are empty, both things are broken at once, and fixing either one fixes the other.

What the message carries, and what it deliberately does not

Since the whole argument rests on headers, it is worth being exact about the division:

The body is protojson of the domain event: event id, tenant, type, subject, timestamps, revision, acting principal.
The row identifier in the outgoing queue table is the same value as the message id.
Beyond the message id, the service was setting no headers of its own. That is the defect above, stated as an inventory line.
Delivery status and attempt counters are not in the message at all: queue depth, age of the oldest row and relay cycle duration live in the metrics endpoint and never travel to the broker.

Which settles a design question that otherwise gets argued in circles: the thread is stitched with headers, not with a field in the event body. The body belongs to the event contract - consumers depend on its shape, and it outlives any particular observability arrangement. Metadata about the delivery of a message is not part of what happened in the domain.

The version attribute that lies quietly

One more failure from that table, kept separate because it is about trust rather than connectivity.

The build stamps the service version into a symbol the runtime reads, and the runtime puts it into the trace as an attribute. The linker will silently ignore an -X flag aimed at a symbol that does not exist. No warning, no error, green build - and the version stays dev. Then dev travels into the audit trail and into the trace attribute, and every span claims to come from a build nobody can identify.

The check is embarrassingly cheap: a build target that compiles both binaries with a test version tag and greps the binary for that string. Empty output means the symbol is wrong. And the flags have to be right in two files, the Makefile and the Dockerfile - a template carrying a dead target in one of them hands the same silent defect to every service generated from it afterwards.

A trace attribute nobody checks lies quietly.

Why it is arranged this way

Two reasons, and neither is about elegance.

Observability written by hand in a service is runtime written by hand in a service. Runtime is what the platform provides - registries, instrumentation, export, pooling. The moment a service writes its own version of it, its trace stops lining up with its neighbour's: different span names, different attribute spellings, different histogram buckets. A fleet where each service is individually well instrumented and collectively incomparable is worse than it sounds, because every cross-service question turns into a translation exercise first.

The single requirement left to the service is expressed as a check. Not because rules are worthless, but because 421 is what a rule looks like when it is only a rule. A requirement with no executable form is a shared intention, and the failure it permits is silent by construction.

What it costs

An honest list:

Context as the first parameter is a discipline in every signature. It shows up in every review, forever, and it is the kind of thing that feels like ceremony right up until the day it isn't.
Cancellation tests are written per transport and per handler. That is real test code, and it grows with the surface, not with the feature list.
Message headers become part of compatibility. Once a consumer reads a header, changing it is a coordinated change, with the same care as changing a field in the contract.
Cardinality is kept low on purpose. The tenant identifier does not go into labels - there are too many tenants, and the label would blow up the series count. So a per-tenant cut is simply not available in traces or metrics; it lives in queries against the data.
A thread stitched across a queue makes the tree long. A trace spanning a synchronous call, a queue and a background handler is more expensive to read than two short ones, and a reader who does not know the shape will get lost in it.

When not to do this

When there is no asynchronous path. Nothing to stitch; propagation in the transport already covers the whole story.
When the message leaves the perimeter and its headers are controlled by someone else. Then the thread ends at your boundary by agreement, and the honest thing is to say so rather than to hope.
When the volume makes recording everything cost more than it returns. That is a conversation about sampling, which is a different decision from whether the thread exists at all.

And the honest caveat about the boundaries of this write-up: there are no names of specific tracing systems here, and no numbers on trace volume, sampling share, span durations or tree depth - the first because naming them adds nothing, the second because those figures were never measured on my side. A hole you can see is worth more than a plausible figure.

The multiplier line

An automated executor made the mechanical half of this cheap: threading a parameter through hundreds of signatures, writing a cancellation test per transport, converting a rule into a blocking check. What it did not do is decide that the requirement should exist as a failure rather than a sentence, or notice that empty headers were simultaneously a decoding defect and a broken trace. That judgement is the work. Speed amplifies whoever set the definitions; it does not supply them.

Access, tracing, rollout - Part 3. Next: feature flags, and why the decision belongs at the boundary - the service receives a finished yes or no, not the address of a flag store.

If you do this better, tell me what you put in the message headers and how you prove it is still there. If you have been through this, what did your broken thread turn out to be hiding besides the trace? If you see it differently, say where a correlation identifier and good logs beat a stitched tree. How is it solved on your side, and what broke there?

A Decision That Needs Data Belongs In The Service

Anton Brilliantov — Tue, 06 Oct 2026 17:59:41 +0000

Three places can decide what a caller may do - only one of them can answer whether this tenant owns this row.

👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services. The previous part was about who arrived. This one is about the second question - what that caller may do - and the only thing I am confident about is where the answer has to be computed. It is the shortest piece in this block, because it is one rule with one test under it. Running notes are on my GitHub: github.com/brilliant-almazov.

This is how I do it right now, with the price attached - maybe you already do it better, maybe you see it differently.

Where this sits

A short primer, so this reads on its own.

A PHP monolith is being taken apart into Go services. The Go API gateway is the only door into the new services from outside; behind it the extracted services talk to each other over gRPC, and each of them owns its own data. Identity is settled at that door: the edge parses the input once and passes on a claim about who arrived, so by the time a service is running, who is calling is a finished statement rather than a question it has to work out.

Which leaves the second question, and it does not resolve itself the same way.

The thesis, in one line

"Who arrived" is answered before the service runs. "What they may do" is not - and it can be answered in three different places, at three different prices.

Picking one of them is usually discussed as a matter of taste, or of which layer feels tidier. It is neither. Two of the three places physically cannot answer some of the questions.

The three places

At the edge. The general check - may this caller reach this at all. It is cheap, it happens before any service is involved, and it knows nothing about the data behind the call.
In the service. The check that needs a row to answer - does this tenant own this row. This is the only place where such a decision is possible at all, because it is the only place the row exists.
In a shared resolver. A separate service that answers "allowed or not" on request. It removes duplicated rules and gives you one place to look. It also adds a network call on the hot path and a new dependency between the service and something that must be up for it to serve.

Nothing on that list is wrong. They are priced differently, and the price is paid at different times: the edge charges you at design time, the shared resolver charges you on every request.

What the edge can see

Worth being precise, because the edge is the place people overestimate. It holds the claim about who arrived, the method being called and whatever the caller sent. That is enough to answer a large class of questions, and it is genuinely the cheapest place to answer them: nothing has been started yet, no connection has been taken from a pool, no transaction is open.

What it does not hold is a single row of anything. It can be told which tenant is calling; it cannot be told which tenant owns the row that is about to change, because that fact exists only in the storage of the service that owns it. Handing the edge that fact means handing it the data, and we are back to the copy.

One question decides the place

The dividing rule is short enough to fit in a review comment:

Does answering this check need rows from this service?

Two answers, and each names the place:

No - decide it as early as possible on the request path.
Yes - decide it in the service, next to the row.

The reason I like this phrasing is that it is a checkable condition, not a preference. It has exactly two answers, anyone can arrive at the same one, and it does not require agreeing on what "business logic" means - a discussion I have never seen finish.

Applying it in a review

The rule is only worth having if two people apply it to the same check and land in the same place. Four checks, run through the question:

Check	Needs rows from this service?	Place
may this caller reach this method at all	no	as early as possible
is the payload shaped correctly	no	as early as possible
does this tenant own this row	yes	in the service
is this row in a state that allows the change	yes	in the service

The last one is the interesting entry. It is not usually filed under permission at all - it reads as ordinary validation - and it lands in exactly the same place for exactly the same reason. That is a good sign about the rule: it stops producing a separate category for checks that were never separate.

One request, two checks

The concrete case is not an incident, and I am not going to dress it as one. It is the ordinary shape of a write.

What we had. A request to change one row needs two different answers before anything is written: this caller is allowed to use this method, and this tenant owns this row. Both are permission questions. Both are asked on the same request. They are not the same kind of question at all.

What that cost when they were treated as one thing. The second check cannot be moved outward unless the data moves with it. To decide ownership somewhere else, that somewhere else needs to know which tenant owns which row - which means a copy of the ownership data, kept somewhere it is not authoritative. A copy starts to drift the day it is made, and a permission copy that has drifted fails in the worst available direction: it answers confidently.

What follows. The first check is settled earlier, without touching data. The second lives where the row lives - and once you say that out loud, it stops looking like security and starts looking like what it is: an invariant of the domain, standing in the same row as every other rule about what may be written. In our services those are expressed as ordinary typed checks - a Rule over an input, a Guard from an input to an output - and the ownership check is one of them, not a separate security layer bolted to the side.

There are no numbers in this episode, so I am not attaching any. Not how many rules exist, not how many checks the edge runs, not what a hop to a resolver would cost us. Those go in when they are taken from the repo, not before. A visible hole is worth more than a plausible figure.

The same rule after the queue

Work does not always finish on the request that started it. Part of it is published as a message and picked up later by a background handler, and at that point the neat picture of "the edge decided the general part" needs re-reading.

The claim about who arrived travels with the message, which is what keeps the later record naming the person rather than the machine. What does not travel is the edge: there is no door in front of a background handler. So the general check is not re-run there, and the data-dependent one still is - by the service that owns the row, exactly as before.

That asymmetry is worth stating rather than discovering: the earlier a decision is made, the more places it has to survive to still be true. A general check made once at the edge is a decision that has to hold for the whole life of the work it authorised, queue hops included.

How this is usually done

Three shapes show up in public write-ups and standards, and all three are defensible:

Policy as a separate decision service. The vocabulary for this is standardised - the split between the point that enforces a decision and the point that decides it is described in NIST SP 800-162, along with the attribute-based model most such systems use. Rules live in one place and are auditable by querying it.
A policy library embedded in every service. The same rules, evaluated locally, no hop. The cost moves from latency to distribution: every service carries a version of the rules, and "which version is live where" becomes a question you need an answer to.
Role checks at the entry layer. The oldest and cheapest arrangement, and the one that quietly stops being sufficient the moment a check needs a row.

The interesting part is that none of the three removes the constraint this article is about. A decision service still has to be given the data to decide on; an embedded library still has to read the row. The question of where the data is survives every one of these designs.

How we do it - one example

One write, spelled out:

request: update one row of entity

  check 1   may this caller use this method     rows needed: none      decided at the edge
  check 2   does this tenant own this row       rows needed: one       decided in the service

  check 2 stands next to the other invariants of entity, not in a layer of its own

Check 1 never sees the database. Check 2 cannot exist anywhere else: the row that answers it is in this service's storage, and the service is the only participant that can read it without a copy.

One thing that falls out of writing it this way: the two checks stop competing for the same slot in the code. Check 1 is a property of the route and lives with the route. Check 2 is a property of the write and lives with the write - which also means it runs inside the same transaction as the change it guards, rather than as a separate lookup that was true a moment ago.

Why this way

Two reasons, and I would not add a third.

Moving a data-dependent check outward means moving the data outward. There is no version of that trade where the data stays put. Either the deciding party reads this service's storage - which makes the storage shared, and the service's ownership of it a fiction - or it keeps a copy, and the copy drifts.

A shared resolver is another runtime on the hot path. The rule I work by is that the platform supplies the runtime and a service supplies its input, its processing and its output. Introducing a component that every request must consult, that must be available for any of them to answer, and whose failure mode is "everything is denied" or "everything is allowed" - that is a runtime decision, and it is a much bigger commitment than it looks like when it is drawn as one box.

Neither reason is an argument that a shared resolver is a bad design. Both are arguments that it is a larger decision than the one this article is about, and that the data-dependent check would still be in the service afterwards.

What it costs

An honest list, not one token item:

The rules are split between the edge and the services. There is no single place where "who may do what" can be read off. Answering that question means reading two things, and knowing that you had to.
Identical general checks in different services have to be kept identical without shared code. Anything that is generic enough to share belongs in the platform layer; anything that is not drifts quietly.
With no resolver, auditing the rules is reading code. You cannot ask a service "list everything this caller may do" - nothing is holding that list. It is a real capability we do not have.
The tenant is not a dimension in the metrics. A tenant identifier does not go into metric labels - the cardinality is not affordable - so "who was denied and why" is a question for queries against data, not a panel someone can open.
A denial at the edge and a denial in the service look different to the caller. One arrives before any work started, one arrives after part of the request has been read and validated. Making them look the same to a client is extra work that nobody schedules.
New services can get this wrong silently. The dividing rule is easy to state and easy to skip, and skipping it produces code that works in the happy path and is wrong exactly once.

When not to do this

When the rules are many and change faster than the code. Then a shared resolver pays for itself, and it is not a debate - rules that change on a different clock than deployments want a different home.
When a consumer needs "am I allowed to" without performing the action. A decision endpoint is genuinely useful there, and inventing one out of the checks you already run is worse than declaring one.
When there are two services. The whole discussion is premature; the split costs more than it returns.
When the rules are owned by someone who does not write code. If the people who decide what is allowed are not the people who deploy, a place they can edit is worth a hop, and no amount of tidy layering substitutes for it.

And the honest boundary of everything above: there is no single permission-decision service on my side. The general check lives at the edge, the data-dependent check lives in the service, and there is no third participant deciding both. This is an account of three places and their prices, not a report on a resolver that runs in production.

The one line

The place of a check is decided by whether it needs data, not by which architectural layer it looks like it belongs to.

The multiplier line

Writing the checks themselves is fast now - a typed rule over an input is a few lines, and generating the twentieth one is not where the time goes. What did not get faster is deciding which of the two questions a given check actually is. Get that wrong and you produce a perfectly written ownership check running in a place that has to be handed a copy of the data to answer - correct code, wrong location, and it will pass review because it looks like every other check. Speed amplifies whoever set the definitions; it does not supply them.

Access, tracing, rollout - Part 2. Next: one request identifier that has to survive from the handler all the way into the queue and back - and the two places the thread reliably snaps.

If you do this better, tell me where you draw the line between the general check and the data-dependent one. If you have been through this, what did your copy of the permission data drift into before anyone noticed? If you see it differently, say where a shared resolver on the hot path was worth its hop. How is it solved on your side, and what broke there?

The Token Stops At The Edge

Anton Brilliantov — Mon, 05 Oct 2026 20:21:41 +0000

The service never sees the caller's token - it receives a claim about who arrived, and a red test stand guards that.

👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services. This block is about the three things a fleet of services has to do identically or each service reinvents them: who arrived, what they may do, where their request has been. This part is the first and the smallest of the three - what actually travels between services when a person is behind the call. Running notes are on my GitHub: github.com/brilliant-almazov.

This is how I do it right now, with the price attached - maybe you already do it better, maybe you see it differently.

The shape of the system, briefly

Short primer, so this reads on its own.

A PHP monolith is being taken apart into Go services. The Go API gateway is the only door into the new services from outside; behind it sit the extracted services, talking to each other over gRPC. The PHP monolith still runs behind its own front door - the two doors are peers and do not route into each other. Every one of the Go services is raised from the same shared platform library, which is what makes a fleet-wide rule like this one enforceable in the first place: there is one place where "how a service reads who is calling" is decided.

So when I say "the edge" below, I mean the door in front of the new services, and when I say "a service", I mean anything behind it - including the background worker that picks up a message an hour later.

The check that fails

The rule in this article exists in exactly one executable form, and it is a failure:

The test stand fails if a token header arrived in the incoming call metadata.

Not a warning. Not a lint note somebody reads on a good day. The stand goes red, and the run stops.

That is worth stating before the design it protects, because the design is the boring part - most teams end up somewhere near it - and the enforcement is the part that decides whether the design is still true six months later.

What is forbidden, and what travels instead

The forbidden thing: the caller's token is not forwarded into the system. It stops at the outer boundary.

What travels instead: the edge parses the input once and passes on a claim about who arrived, carried in the call metadata alongside the call itself. A service downstream receives a finished claim. It does not verify tokens - it has neither the keys for it nor a reason to.

Drawn as two paths through the same fleet:

token forwarded                      claim propagated
---------------                      ----------------

  request (token)                      request (token)
        |                                    |
        v                                    v
     [ edge ] -- token -->             [  edge  ]  <-- the only place with keys
        |                                    |
        |                                    |  claim: who arrived
   +----+----+------+                   +----+----+------+
   v         v      v                   v         v      v
[svc A]  [svc B]  [svc C]            [svc A]  [svc B]  [svc C]
 key      key      key                no key   no key   no key
 verifies verifies verifies           reads the claim, and that is all

Say it as a property rather than a policy and it gets sharper: a service in this fleet is not capable of validating a caller's token. Not "is asked not to". Not capable. There is no key material in it, no library wired for it, no configuration pointing at an issuer. If a token showed up, the service would have nothing to do with it - which is precisely why a token showing up has to be an error and not a shrug.

The episode: a rule with nothing under it

Here is the actual sequence, and it is not a production incident - I am not going to dress it as one.

What we had. The rule "no tokens travel inward" existed as an agreement. Everyone knew it. Everyone would have repeated it correctly if asked in a review.

What that cost. There was nothing to check it with. An agreement holds exactly until the first edit made by someone in a hurry, and the violation it produces is invisible: forwarding a header is not a compile error, not a failing assertion, not a 500. It shows up in production, and only if somebody goes looking for it. Nobody goes looking for a rule everybody agrees with.

What changed. The rule was turned into a prohibiting check: the test stand fails if a token header arrived in the incoming call metadata. That is now the only form in which this rule exists in an executable state on my side.

There are no numbers attached to this episode, so I am not attaching any. Not how many services accept a claim, not how large the metadata is, not how long the edge takes to parse the input - those numbers go in when they are taken from the repo, not before. A hole you can see is worth more than a plausible figure.

How this is usually done

Four approaches show up in public write-ups and standards, and all four are defensible depending on what you are building:

Forwarding the caller's token all the way down. Every service validates it independently. Simple to reason about, and the reason it is popular is that it needs no new format - the thing the client sent is the thing the service reads.
Exchanging the token at the boundary for a narrower internal one. This is a standardised operation - RFC 8693, OAuth 2.0 Token Exchange - and it keeps the external credential outside while still giving services something verifiable.
A signed assertion about the caller placed in the call metadata, usually as a JWT issued by the boundary rather than by the external issuer.
Mutual authentication between services at the transport layer, which answers a different question - which process is calling - and is a separate layer from which person is behind the call. The two are often confused; they compose, they do not replace each other.

None of these is wrong. The first is the cheapest to start with and the hardest to reverse: once three services have a verification path, removing it is a change to three teams' code. The second and third both move the parsing to one place; they differ in whether what comes out is still a credential. What we do is the third shape with the verification removed on the receiving side - the boundary issues the claim, and nothing behind it checks the issuing. That is a real trade-off and it gets a price list further down, rather than a shrug.

How we do it

Three statements, one line each:

One place parses the input. The outer boundary, once, per request.
Identity travels in the call metadata, next to the call itself - not in the body, not in a lookup the service performs.
A service is a consumer of the claim, not a verifier of it.

And a fourth that is easy to forget until the logs go quiet: the same requirement applies on the asynchronous path. The claim has to survive the queue boundary the same way the call context does - passed as the first parameter everywhere, including background handlers and the publish call itself. If it does not survive, the background handler is acting as the machine, and every record it writes says so. Our domain events carry a reference to the acting principal in the message itself (principalRef in the event payload), which is what makes "who did this" answerable after the fact rather than at request time only.

Why a check and not a rule

There is a ladder I keep coming back to, and it has three rungs:

A reminder - said in a thread. Lives one session.
A rule - written in an instruction file. Works while it is being read.
A check - a linter, a prohibiting test, a structure test, a blocking hook. Works.

The stand that fails on a token header sits on the third rung, in the same row as the other prohibiting checks we run: the forbidden test asserting that the row-scan loop appears in exactly one package; the block on a test that uses an empty context instead of the test runtime's; the drift check that fails the build when the environment catalogue and the metrics snapshot no longer match the code.

One line, and it is the only thing in this article I would defend without qualification:

An agreement about identity with no executable check under it is not a rule.

It is a shared intention. Shared intentions are useful. They are not enforcement, and confusing the two is how a fleet ends up with three services that quietly do it the old way.

What the check actually asserts

It is worth being precise about what a prohibiting check like this does and does not prove, because it is easy to oversell.

It asserts one thing: on the path the stand exercises, no token header reached a service. That is narrow. It does not prove the claim is well-formed, it does not prove the edge parsed correctly, it does not prove some other path exists where a header does get through.

What makes it worth having anyway is the failure mode it removes. The violation it catches is the one nobody would otherwise notice - the header that gets forwarded because forwarding all headers was the convenient way to write the client, and that then works perfectly for months. A check that catches a class of silent, correct-looking mistakes is worth more than a broad check that catches loud ones, because the loud ones were going to be caught anyway.

Three consequences

One place parses the input. Parsing, signature checking, clock skew, key rotation, malformed-input handling - all of it exists once. Not once per service, and not once per service plus the four services that copied it from the first one before it was fixed.

Changing how people log in does not touch the services. The shape of the input is a property of the boundary. Behind it, services consume a claim whose meaning has not changed, so a change in the way identity is established is a change in one component instead of a fleet-wide migration.

Logs and traces name the person who acted, not the machine that ran. This is the one I underestimated. Because the claim rides with the call and continues into the event stream, an audit record can name the acting principal - and that record is written in the same transaction as the mutation it describes, not appended by a best-effort listener afterwards. "Who changed this" stops being an investigation.

Where identity ends and permission begins

A short divider, because these two get mixed constantly:

"Who arrived" and "what they may do" are different questions.

A decision that needs the service's own data - does this tenant own this row - is made in the service, because that is the only place it is answerable at all. Everything else is made as early on the request path as it can be. That is the whole boundary for the purposes of this article; the full breakdown of where permission decisions live is the next part.

What it costs

An honest list, not one token item:

The boundary becomes a single point of trust. The service takes the claim as given and cannot check it. If the boundary is wrong, everything behind it is wrong, confidently and silently.
A claim format exists now, and every participant must understand it. Changing it is a breaking change across the fleet, with all the coordination that implies.
A locally raised service needs a way to inject a claim. Otherwise it is unusable without the boundary in front of it, and the developer experience quietly degrades into "you can only test this on a stand".
Debugging "why is the wrong principal here" moves to the boundary - that is, into code the service's author does not own and may not have read.
The prohibiting stand has to be maintained. A check that gets switched off "just for now" is not a check any more; it is a comment with a longer commit history.

When not to do this

When there is exactly one service and no boundary as a separate participant. Parsing the input in place is cheaper, and a claim format between one sender and one receiver is ceremony.
When the service genuinely needs the original input - for instance, when it is the boundary, or when it re-issues credentials.
When an external consumer is contractually owed a response derived from the original token. Then the token has a reason to travel, and the reason is written down in a contract rather than in habit.

And one honest caveat about the boundary of this whole description: there is no single permission-decision service on my side. The general "may this caller reach this at all" check lives at the edge; the data-dependent check lives in the service; there is no third participant deciding both. Everything above is about identity - who arrived - and not about permission.

The multiplier line

Wiring the check, moving the claim through the metadata, threading the context into the background handlers - that part is fast now, fast in a way that is genuinely different from a few years ago. What did not get faster is the decision that this rule deserves a prohibiting check at all, and that the check belongs in the stand rather than in a document. That call sets what "correct" means, and everything generated afterwards is correct or wrong relative to it. Speed amplifies whoever set the constraints; it does not supply them.

Access, tracing, rollout - Part 1. Next: the same question one step further in - who decides what is allowed, and the three places that decision can be made, each with a different price.

If you do this better, tell me what your boundary passes inward and how a service proves it did not get a token. If you have been through this, what did your identity agreement turn out to have drifted into before anyone noticed? If you see it differently, say where forwarding the caller's token all the way down is the cheaper answer. How is it solved on your side, and what broke there?

The Same Decision at Three Depths

Anton Brilliantov — Sat, 03 Oct 2026 15:18:10 +0000

Layers, daemons, services - one boundary drawn at three depths, and each depth has its own bill.

👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services. This block is about what a service is to everyone else: whose it is, who calls it, what it promises. Part 5 listed the four lines of a storage requirement; this last part picks one of them - the ratio of reads to writes - and follows it as far down as it goes. Running notes are on my GitHub: github.com/brilliant-almazov.

This is how I do it right now, with the price attached - maybe you already do it better, maybe you see it differently.

Where the boundary gets drawn

Short primer, so this reads without the earlier parts of the block.

Each of our Go services raises itself from one declaration - a manifest listing its daemons, its databases, its queues, its schedules. The runtime and the deploy pipeline read that file; nothing about a service is true because a page somewhere says so. One line of the requirement feeding that file is the ratio of reads to writes, and it is written before the first table exists.

What that line buys is not only a schema decision. It decides which executor runs a read and which one runs a write - and "executor" turns out to mean three different things, depending on how far down you take it.

The thesis

Splitting reads from writes is not a decision about whether. It is a decision about how deep.

The same boundary can sit between two packages inside one process, between two daemons built from one image, or between two services with a stream of events between them. All three are the same decision taken at a different depth. Each step down buys something specific, and each step down is paid for in consistency, in latency and in the number of moving parts.

The case: 44 copies of four read shapes

Before any of that is worth discussing, the first depth has to actually be one thing rather than forty-four. Here is the audit that started this, run on the service in question in August:

  multi-row loop   ..............  18 copies
  single-row read  ..............   6
  COUNT(*)         ..............   8
  existence probe  ..............  12
  ────────────────────────────────────
  total            ..............  44   of which 12 differ only
                                        in the text of the error wrapper

Four shapes of reading rows out of a database, written by hand in every domain that needed them. Nothing in that list was wrong on its own. Every copy compiled, every copy was tested, every copy did what its caller wanted.

The problem is what the four shapes differed by: the result type and the dependency, and nothing else. That is the definition of a generic - written out with copy-paste instead of a type parameter. And the correct form already existed in the tree: a reader carrying an executor, a statement and a scanner, with one Query(ctx, in) method. It was sitting inside a domain package, where no other domain could reach it, so everyone else wrote their own.

How it got to 44 is the least interesting part and the most familiar one. Each copy was written by somebody who needed a list of rows that afternoon, could see three similar loops nearby, and had no reason to believe the fourth one was a problem. Nobody made a bad call. The count is what makes it visible: one loop is a loop, eighteen loops are a missing abstraction, and nothing in between produces a moment where anyone notices.

The fix was not clever. The reader moved into one cross-cutting package, the 44 copies became calls into it, and the question was closed by a forbidding test: for rows.Next(), rows.Err(), rows.Close() and QueryRow( occur in exactly one package of the tree. Not "should occur" - the test fails the build if they occur anywhere else. A convention that is only written down decays; a convention with a failing build behind it does not.

That is the connection to everything below. The first depth is only worth having when the read side is one primitive. A read layer that is forty-four hand-written variants is not a layer; it is a naming convention. Splitting it from the write side buys nothing, because there is nothing coherent on either side of the line.

How this is usually done

Fairly, because all three of these are reasonable and I have used the first two.

The common framing is command-query separation inside one application: the write path and the read path get their own models and their own objects, in one process and one deployment - the shape described in Fowler's note on CQRS. One level up, the read side becomes its own model, kept current by events emitted from the write side; that is the version in the Azure architecture centre's CQRS pattern, usually paired with a materialised view. And one level up again, the read model becomes a separate deployable, with its own scaling and its own store.

Those are three points on the same axis, not three competing schools. What follows is where I place each one and what it costs me.

Depth 1 - layers inside one process

Two layers touch the database, and only two. repository reads and only reads. manager writes and only writes, find-or-create included, because that is a write with a read in front of it.

There is no store, no writer, no dao, no service in between. Those names exist to avoid deciding which of the two things a type does, and every one of them ends up doing both. When a manager needs to read, it calls a repository; it does not carry its own copy of the SQL.

The package layout follows the component type, with the domain nested inside it:

  internal/repository/order/methods/     one method = one file = one type with Execute
  internal/manager/order/methods/
  internal/repository/account/methods/
  internal/manager/account/methods/

Two details make the split hold rather than erode:

One method is one file and one type with an Execute method. A package that reads eleven things has eleven files, not one file with eleven functions on a struct that slowly grows a write path.
The transaction is a decorator on the outside, and only over mutations. tx pgx.Tx never appears in a signature; the executor comes from the context. A domain method does not know whether it is running inside a transaction, which is exactly why the read side can never quietly acquire one.

Worth being precise about what this is not. Two layers is not two models. There is one representation of the data, one schema, one transaction boundary; what is split is which code is allowed to write and which code is allowed to read, not what the data looks like on each side. The distinction matters because the word "CQRS" gets attached to both, and only the deeper version carries the consistency bill.

The price of this depth is close to zero. It is a rule about where files live, not a piece of infrastructure. Nothing new runs, nothing new can fail, and there is no consistency question because it is all one process and one connection pool.

Depth 2 - two daemons of one image

One image, two binaries: server answers synchronous calls, worker drains the outgoing-event queue, runs background computation, and executes partition retention on the audit tables.

Both are declared in the manifest and deployed by the same pipeline with a parameter saying which of them to bring up. That declaration is the whole mechanism - there is no second repository, no second image, no separate release.

One rule about the composition of that list, and it is not a technical one: the set of daemons is decided by the owner of the service, and by nobody else. New background work goes into an existing daemon. A new cmd/* is not something an implementer introduces on the way past, because a binary that nobody declared is a binary that nothing deploys.

What this depth buys is that a long job stops sharing a request budget with a synchronous handler. What it costs:

A second process, with its own resource configuration, its own restarts and its own place in every dashboard. Two processes is where "the service is up" stops being a single fact.
One publisher per gauge. Queue depth and partition age are published by worker only, never by server. Two processes publishing one gauge produce a graph that flips between their two versions of the truth, which is worse than having no graph, because it looks like data.

Depth 3 - separate services

The third step is a write-side service and a stream of events, with the read side living somewhere else entirely, on its own store and its own schedule.

We do not have this in production. What we have is depth 1 and depth 2. I am describing the third because the mechanism is worth writing down and because its price is what keeps the decision honest - not because there is an installation behind it.

Everything that was free at the first two depths becomes an explicit obligation here:

Delivery is at-least-once. The broker does not deduplicate. Idempotency by event identifier is the consumer's job, and it is not optional - it is the thing that makes the stream usable at all.
The relay is the single drainer. Events leave in strict order by id, and FOR UPDATE SKIP LOCKED is deliberately not used. Parallel draining is faster and it breaks order, and order is precisely what lets a consumer rebuild a correct replica.
Redelivery is normal, a spike of it is the incident. If the process dies between publishing a message and deleting its row, the same message_id goes out again on the next pass. That is designed behaviour, not a fault. The alert is on the rate, not on the event.

Read that list again as a cost sheet rather than a design. Three obligations, all of them permanent, all of them on code that did not previously need to exist. And note where they land: not on the service that made the split, but on every consumer downstream of it. That is the part that makes this depth expensive in a way the first two are not - you are spending somebody else's budget as well as your own.

The depth is chosen by a line of the requirement

This is the part I would defend hardest, and it is not about reads and writes specifically.

An argument about "CQRS or not" is settled by a line of a non-functional requirement, not by preference. If no such line exists, there is nothing to argue about - the requirement has not been gathered yet, and two people are comparing instincts.

For this particular decision the mapping is short enough to fit in four lines:

  read/write ratio near 1, small load        →  depth 1, and stop
  different load, different shape            →  depth 1, from the first table
  heavy background work beside handlers      →  depth 2
  different owner, different response target →  depth 3, with the bill below

The read-to-write ratio is one of the four lines of the storage requirement from Part 5. That is where this decision is made - months before anybody opens an editor to make it.

The one conclusion

Every step deeper is paid for in three currencies: consistency, latency and the number of moving parts. Until a line of the requirement shows there is something to pay with, the decision stays at the depth above.

That is the whole rule. It is deliberately not "start at depth 1 and grow into depth 3" - growth is not the trigger, a named requirement is.

What each depth costs

depth	what it buys	what it costs
layers in one process	a read is written once, not 44 times	moving between two package trees to read one feature
daemons of one image	background work stops sharing a request budget	a second process, a second configuration, one gauge owner
separate services	read and write scale and fail separately	eventual consistency, idempotent consumers, ordering by design

The third row is the one worth staring at, and it has one more entry than the cell holds: debugging a problem across two systems instead of one. Nothing in that row is a one-off cost. Every line of it is a permanent obligation on everyone who touches that code afterwards.

What a skipped depth looks like later

The failure mode never announces itself as "we picked the wrong depth". It arrives wearing something else.

A background recalculation that shares a process with request handlers is not reported as a topology problem. It is reported as latency, intermittently, on endpoints that have nothing to do with it - and it gets three separate investigations before anyone looks at what else is running in that process. That is a missing depth 2.

The opposite direction is quieter and more expensive. A team goes to depth 3 for a domain whose read and write have the same owner and the same response-time target. Nothing breaks. There is simply a stream, a relay, an idempotency key and a second store to reason about forever, in exchange for a separation that nobody ever needed. Nothing about that shows up as an incident - it shows up as every subsequent change taking longer than it should.

When not to go deeper

When the ratio is close to one and the load is small. Depth 1 is the whole answer. Going further adds parts and removes no problem, and "we might need it later" is not a requirement line.

When there is no separate response-time target for reads. Without that, depth 3 buys nothing that depth 2 did not already buy, and it charges you eventual consistency for the privilege.

And the honest boundary, stated once more because it is easy to read this piece as a report rather than an analysis: one domain split across separate read and write services does not exist in my production. Layers exist. Daemons exist. The third depth is written up here as a mechanism and a price, and that is all it is.

The multiplier line

Collapsing 44 copies into one reader is exactly the kind of work an automated executor does well - mechanical, well-specified, verifiable by a test that either passes or does not. What it did not do was notice that four shapes differing only by result type and dependency were a generic written by hand, or decide that the right answer was one cross-cutting primitive rather than a tidier copy in each domain. That judgement is the whole exercise, and speed applied before it is made produces forty-four beautifully refactored copies. Speed amplifies whoever set the boundary; it does not draw it.

Services and commitments - Part 6. That closes this block: a service exists when it is declared, its owner is a field, its call map is observed rather than claimed, its promises are metrics with targets on them, its volume is a requirement line, and that same line decides how deep the read/write boundary is cut.

If you do this better, tell me what made you go one depth further than you had to. If you have been through this, what did the extra depth cost you that nobody had costed up front? If you see it differently, say where separating reads and writes at the service level pays off earlier than I claim. How is it solved on your side, and what broke there?