The quality of a check is set by the inputs, not by the executor.
👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services.
A lot of that work is checking production: does the extracted service behave there the way the piece it replaced did. Most of those checks are handed to an executor - person or program, the rules below hold either way.
Running notes live on my GitHub: github.com/brilliant-almazov. This is the assignment I write today, price attached - maybe you write yours better, and I'd like to hear how.
The thesis
The quality of a production check is decided by the completeness of the assignment, not by the intelligence of whoever executes it.
Everything below is a consequence of that one line.
What one line of the assignment was worth
I ran a sweep across a vertical of 276 configurations: does each one have a result, and if not, where does the path stop. 184 came back empty - against the assignment I had written, that was 184 problems.
Then one condition entered the assignment: a configuration's schedule can be of a type that produces no tasks of this kind at all. With that line the same 184 became normal - the setup was doing what it was configured to do.
One line changed 67% of the picture, and no care from the executor would have found it. The fact is not a field, not a status, not a counter - it lives in how the domain works, and it enters a check only if someone writes it into the assignment first.
One line of the assignment, 276 configurations
the same sweep, read against the assignment before and after that line
without the line
┌──────────────────────────────────────────────┬───────────────────────┐
│ 184 empty result - filed as a defect │ 92 has a result │
└──────────────────────────────────────────────┴───────────────────────┘
with the line
┌──────────────────────────────────────────────┬───────────────────────┐
│ 184 empty - the schedule makes no such task │ 92 has a result │
└──────────────────────────────────────────────┴───────────────────────┘
67% of the picture changed, and no care from the executor would have
found it. The fact is not a field, not a status, not a counter.
That is the whole argument. What follows is the list I now write before any check starts.
The ten inputs
1. The surface, and a ban on every other one
Name the one entry point the check goes through, and state that every other one is off limits. A system in migration has more than one front door, and results measured through two of them are not comparable.
Without this line the executor picks whichever door answers, and the report silently mixes two systems.
2. How to authenticate, with secrets living only in the environment
State the authentication scheme and where credentials come from: the environment, never the assignment, an artefact or a commit.
This is not only hygiene - an executor that has to improvise credentials improvises something that works once and cannot be repeated.
3. The exact shape of every request
Path, method, every parameter with its range, every body field - the literal shape, not "call the listing endpoint".
Anything left open gets interpreted plausibly, and a plausible-but-different request produces numbers that look comparable and are not.
4. How each link is measured, and what counts as the fact
For every link in the chain, say which single value is the fact: this field, this header, this counter. One number per link, read cheaply.
Two people measuring "does this have data" in two different ways will disagree forever, and neither will be wrong.
5. The domain conditions that no read returns
Provider coverage. Schedule types. What each task status actually means. No endpoint exposes any of it.
This is the line that was worth 67% of my picture, and the one most likely to be missing - it is the only one that cannot be discovered by reading the system.
6. What counts as a normal empty
Say which empties are expected outcomes and which are defects, and how to tell them apart.
An executor with no such rule counts every empty as a finding.
7. Where the artefacts go, and in what form
One folder per run, one file per probe, written as the run goes, not assembled at the end.
Left unspecified, the report becomes the only record - and a report is a summary of evidence, not the evidence.
8. The rule for writing in production
Snapshot, act, restore, a unique probe marker, and a confirmation by reading again.
This one gets its own section below.
9. The shape of the verdict, and the duty to separate "not checked" from "passed"
State the vocabulary of the verdict and require that unchecked cases be reported as unchecked.
Left unstated, "not checked" quietly becomes "nothing found", which reads as "passed" to everyone downstream.
10. An explicit ban on concluding from one field without a control case
Every conclusion needs a control beside the suspect: an object where the thing is known to work, read the same way, in the same run.
I put this in writing because I broke it myself, twice.
Ten inputs, written before the first request
the executor fills every one of these itself if the assignment leaves it out
01 one surface, every other one banned
02 auth scheme, secrets only from the environment
03 the exact shape of every request
04 one measured fact per link
▶ 05 the domain conditions no read returns
not exposed by any read - this one line moved 67% of the picture
06 what counts as a normal empty
07 artefacts: one folder per run, one file per probe
08 snapshot, act, restore, verify
09 verdict format: not checked ≠ passed
10 no conclusion without a control case
Nine of them can be checked against the system. The fifth cannot - it is
a hand-written record of the world outside.
Why the normal empty gets its own line
If a system answers "no data" with an error, every mass run looks like a catastrophe. Two thirds of the objects come back as not found, the genuine failures sit inside that pile, and every run starts by clearing the noise away.
A normal empty is a success with zeros in it. Note where that requirement lands: on the system, not on whoever is checking it. Writing "treat two thirds of the answers as noise" into an assignment is a workaround for a contract that should have said so itself.
Why the write rule is not up for discussion
Any write executed against production follows one shape:
snapshot → act → restore → verify by reading again
Every object touched carries a unique probe marker, so anything left behind is identifiable as mine and removable without guessing. Restore is not done when the restore call returns - it is done when a fresh read shows the original state again.
A check that leaves the polygon different from how it found it is not a check. It is a change, and whoever runs anything there next inherits it. Without the marker, cleanup depends on memory - the one part of a run nobody can audit.
Every write executed against production
┌──────────┐ ┌─────┐ ┌─────────┐ ┌─────────────────────────┐
│ snapshot │──▶│ act │──▶│ restore │──▶│ verify by reading again │
└──────────┘ └─────┘ └─────────┘ └─────────────────────────┘
──────────────────────────────────────────────────────────────────
│
▼
every touched object carries a unique probe marker
so anything left behind is identifiable as mine and removable
without guessing
What happens when these are missing
The executor does not stop. It fills the gaps itself, plausibly. A missing input does not produce an obviously broken report - it produces a coherent, confidently worded, wrong one, with nothing marking which parts came from the system and which from a guess.
My own example, and not a flattering one: my first conclusion from a sweep was that a set of objects would not rebuild at all. They had failed on time, back to back. The objects were fine - the failure belonged to the pace I was driving the endpoint at, and one at a time with a pause they came back almost entirely clean. That distinction is the next part.
My assignment did not contain rule 10. Without it, one run and no control was enough to file a defect against a service that had lost nothing.
What it costs
The inputs are assembled by hand, up front, before any checking starts. On a vertical of this size that is several hours whose only output is a document.
On a one-off check that is pure overhead: if the question is asked once and never again, writing the ten inputs takes longer than answering it badly and answering it again. The trade pays when the check gets repeated - by the second run the assignment exists, while the guesswork would be re-invented from scratch.
The other part of the price: the fifth input has no owner inside the system. It is a hand-written record of how the world outside works, and it goes stale silently.
The part that generalises
A fast executor amplifies whatever is in the assignment, holes included. Complete inputs get a complete answer sooner; a gap gets a confident wrong answer sooner, and the confidence reads identically in both.
The advantage goes to whoever can make correctness visible from the assignment itself, rather than from a second run and a comparison.
This is my list right now, on one vertical, in a system halfway through a migration, price included.
So: maybe you do this better, and your assignments already carry the domain conditions in a form that stays current. Maybe you have been through this and found which of the ten you can drop without paying for it. Maybe you see it differently and think an executor should be trusted to work the conditions out for itself.
How is this solved on your side, and what broke there?
Testing production with an executor - Part 1.
Next: a failure on time, checked twice - once in a stream and once one at a time with a pause - and how to tell a property of the object from a property of the pace.
Top comments (0)