A second pass earns its cost only if it can add a measurement the first pass never had.
👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services. This series is about one write path that left the monolith and what it took to accept it as working. This last part is about the arrangement that produced most of the numbers in the other six: one automated executor pointed at the output of another, told to instrument what was not instrumented, run the measurement and report what it saw. Running notes live on my GitHub: github.com/brilliant-almazov.
This is how I do it right now, with the price attached - maybe you already do it better, maybe you'd do it another way.
The path in one paragraph
A user saves a set of matching rules; every domain under that account gets marked by them. Four participants: the Go API gateway as the only door from outside, the rule-set-markup service which takes the write and folds the cascade, the domain-rule-map service which owns rule-set identity and recalculates the map, and PostgreSQL behind them - the monolith's database for the rule data, the identity service's own for identity.
Work on that path ran in iterations: a brief describing what to change, an executor producing the change, a run against a live environment, a verdict. This part is about the second executor in that sentence - the one whose whole job is to look at what came out of the first, and whose findings are the only reason several of the numbers in this series exist.
What the second pass actually is
Not a review of the diff. A review of the diff is cheap and mostly useless here: every change in this period was locally correct in its own repository, and I have a whole part of this series about two individually correct commits producing one defect.
The second pass is five steps, in this order:
1 read what the other pass produced the change, and the claim it makes
2 find the zone that has no numbers which span would have to exist to check that claim
3 carry the instrumentation there usually into a neighbouring service, not this one
4 run the measurement on a live environment same input, before and after, same day
5 report two kinds of finding "this is what it takes to run it at all"
"this looks wrong - here is the number, take a look"
Step 3 is the one that makes the arrangement worth anything, and it is the one that gets skipped. A pass that only measures where measurement already exists will confirm whatever the first pass believed. The finding below existed because the executor was told, in the brief, that adding spans in a service it was not otherwise touching was in scope.
The zone with no spans
The claim under check was ordinary: a read endpoint had been reworked and was, by the response, fine. Status 200, correct data, content assertions passing on every object in the sample.
What did not exist was any number underneath that verdict. The service's own spans said "the call took this long"; nothing said what the call had been doing. So the second pass added, in the service that answers that read, the one field that turns a duration into a finding: the statement text on every database span, with a repeat count.
Then it ran the same request - two domains, one account - against three consecutive builds on the same day:
build A B C
─────────────────────────────────────────────────────────
database round trips 92 7 7 ◄ the shape
database time, ms 440.0 37.2 31.8 ◄ 14× less
server root, ms 474.7 76.2 59.1 ◄ ours, 8× less
client wall, ms 715.9 324.0 433.1
network, ms 242.1 250.1 381.3 ░ not ours, never moved
response code 200 200 200 ← unchanged throughout
Ninety-two round trips on a call that needs seven. Nothing about that is visible in a response code, a status page, an error rate or a duration alone. It is visible in exactly one place - the count and the text of the statements - and that place did not exist until a pass was sent to create it.
Two things I take from that run, and keep:
- The unit of the finding is round trips, not milliseconds. 92 → 7 is a fact about the shape of the work. 440.0 → 31.8 ms is what the shape happened to cost on that day, on that data volume, on that machine.
-
A share without its absolute value is not publishable. The network's share of the call went from 33.8 % to 88.0 % across those three builds and reads like a catastrophe; in milliseconds it never moved, and everything we own fell. I write
440.0 ms · 61 %, never61 %.
The two kinds of finding, and why the difference matters
Everything the second pass reported fell into one of two boxes, and I ask for them separately now:
- "This is what it takes to run at all." Missing configuration, a wrong assumption about a field, an endpoint that needs a list where a single value was sent. Mechanical, verifiable, and safe to act on immediately.
- "This looks wrong - here is the number, take a look." 92 statements where 7 would do. A resolver called once per rule. A wait with no span around it. These are suspicions with evidence attached, not verdicts.
The second box is the valuable one and it is the one that must not be auto-applied. A suspicion is a claim about intent - somebody meant this call to cost one round trip per request, not one per rule - and intent is not in the trace. What the pass supplies is the number that makes the conversation short. What it cannot supply is the decision that the number is wrong.
I have seen the failure mode of skipping that distinction, in this same period: a probe read a value from a listing endpoint that quietly ignored its filter, printed a confident finding, and produced two phantom defects - one of them filed as a comment on another team's ticket before being withdrawn. The executor was not sloppy. It reported what it measured, with the same formatting and the same confidence it uses when it is right.
The instruction surface is where the cost sits
The cheap part of this arrangement is the running. The expensive part is the writing.
brief files in the repository 289
of them, in 33 numbered sets 137
─────────────────────────────────────────
what a brief has to contain the exact surface, the exact request shape,
what counts as a normal empty, where artefacts go,
and an explicit ban on concluding from one field
That is the honest bill for "the executor did it". Two hundred and eighty-nine files of instruction accumulated behind a period whose code changes would fit in a much smaller number. None of it is boilerplate: a brief that does not say what a normal empty result looks like will get a normal empty result reported as a failure, and I have the reports to prove it.
What it buys is repeatability. A brief that is complete enough to hand to an executor is, by construction, complete enough to hand to a person, to a pipeline, or to yourself in three weeks. The instruction surface is the durable artefact here; the run is disposable.
Defects do not die on the first pass
The other thing a second pass changes is the arithmetic of a defect's life. Twenty-six numbered defects over the period, and 39 defect-iterations between them - a defect on average outlived more than one pass, and the worst of them outlived six:
iterations 0 1 2 3 4 5 6
├────┼────┼────┼────┼────┼────┤
D-01 ████████████████████████████▌ 6 a rule written, the engine kept the old set
D-04 ███████████████████████▌ 5 map row carried no priority (+ a regression)
D-11 ██████████████▌ 3 priority 0 arrived as null on the wire
false ✗ ██████████████▌ 3 no defect at all - measured the wrong source
D-14 █████████▌ 2 timeout on a third of the objects
D-16 █████████▌ 2 map row did not name its set - came back later
eight more ████▌ 1 each
──────────── still open at the end of the period ────────────
D-23 █████████▶ ≥2 rule order not provable
D-25 ████▶ ≥1 cursor paging says "more", returns no token
D-26 ▶ ≥0 async rebuild lands in tens of seconds - or not
The line worth staring at is the one labelled false ✗: three iterations spent on something that was never a defect, because a probe read the wrong source and reported it with full confidence. Three iterations is the same cost as the third-worst real defect in the set.
So the second pass is not a heroic single sweep that closes things. It is a loop that has to survive being wrong about its own findings, which is why every finding it reports carries the measurement that produced it rather than the conclusion.
A rule written only into prose comes back
I keep a write-up for every mistake I make in acceptance work. Over this period that is 28 of them, which is a number I am fine publishing because the interesting part is not the count.
The interesting part is that 5 of those mistakes repeated, and the gap between the first occurrence and the repeat was one to six days:
- reading a configuration's rule set from an endpoint that does not authoritatively answer it - came back after six days, four iterations burnt;
- writing a request from memory instead of from the contract - three separate times;
- writing to a production environment from a state that had been read wrongly - four days later, with 17 rules wiped and a four-minute gap in the data;
- misreading an auth
401as a defect - five days later, one ticket filed and withdrawn; - overwriting a shared artefact with a blind write - the same day.
Every one of those already had a write-up. The write-up did nothing. A rule that lives in prose is re-derived by whoever reads it next, which is to say never, and an executor never reads it at all.
What actually stopped the repeats was moving the rule into the two places that are read on every run: the brief (as a numbered constraint, with the ban stated as a ban) and the check (as an assertion that fails, not a note that warns). That is the same movement as freezing requirement tests before code, applied to my own process rather than to the product.
Closing the loop, and the steps that stay manual
Right now the handovers between passes are mine. I take the analysis, I decide it is worth a run, I hand it to the run, I read the report, I decide what it means. The obvious next step is to close that: one orchestrating pass that fans out analysis, instrumentation and measurement, and returns a report instead of returning to me three times on the way.
Three steps I am deliberately not closing:
- what "correct" means - the decision that 92 round trips is wrong and 7 is right is a product decision, and the trace does not contain it;
- writing to a live environment - a write from a wrongly-read state has already cost me 17 rules and four minutes of data; it stays behind a person on purpose;
- accepting a defect as a defect - because the cheapest failure mode in the whole set is a confident, well-formatted, entirely fictional finding.
Calling those three "ritual" is fair, and I keep them anyway. Not because automation cannot perform them - it can perform all three - but because the cost of being wrong on them is asymmetric, and the cost of doing them by hand is a few minutes per iteration.
What I never measured
The honest hole in this part, in one line: there is not a single day in this period with a measurement of what the automated work cost. No token counts, no money, no per-iteration accounting. I have 985 minutes of run durations and I have zero currency.
That means everything above is a statement about what the arrangement found, and nothing above is a statement about whether it was worth it. Those are different claims and I can only make the first one. The second needs a measurement I did not take, and taking it costs nothing next to the thing being measured - which is exactly why not having it is embarrassing rather than defensible.
The multiplier line
The second pass produced findings the first pass could not have produced, and it did it in an afternoon: the missing spans, the run, the comparison, the report. What it did not produce was the instruction that sent it into a service it was not otherwise touching, or the judgement that a correct 200 response can still be a defect. Those came out of a brief - one of 289 - written by a person who had already been wrong enough times to know which zone had no numbers in it. Fast execution multiplies the definition of "done" that it is given; where that definition is a response code, it multiplies green.
Anatomy of a service split — Part 7. That closes this set. The remaining piece is the whole ledger in one place: 32 days, 34 runs, 26 numbered defects and every figure from these seven parts, read end to end as one timeline rather than seven angles on it.
If you do this better, tell me how you decide which zone gets instrumented before a run. If you have been through this, what did your second pass find that the first one had reported as green? If you see it differently, say where a second automated pass costs more than it returns. How is it solved on your side, and what broke there?
Top comments (0)