Entry four of this diary laid out five instruments that keep Sentinel's code and documents honest with each other. Every one of them was added because the one before had an edge, and something had just fallen off it. This entry is about the sixth, added the same way, which reads the one thing none of the other five does: the resolutions. It is also about the register checks moving out of the repository into a tool of their own.
The column nobody reads
A deferral is a row in the roadmap register, a DW-*, for work the build owes and has not done. When the work is done the row moves to Resolved deferrals, and its last column says what was built: which function does it, which test proves it.
That column is written by whoever made the commit, usually an agent, in the same commit that made it true or not. Then nothing reads it again. The per-step audits map exit criteria onto tests, and the reviews of the development guide read chapters against the code. By the time anybody looked on purpose, 135 rows had been resolved and none had been checked by anyone but its author.
Three had already been found wrong, each by accident and each in the same shape: a mechanism built, tested by calling it directly, and called by nothing in the running core. One of the three was in the last entry, where derived points were computed and never evaluated or written.
So about five weeks ago the first deferral audit read all 135 against the code on main, rather than against the commit that made each claim. For every mechanism a row says was built, it looked for a production caller. Every cited test was read, and for the newest fixes, reasoned against the code before the fix to confirm the test could have failed. Then the open rows were read the other way round, looking for one that was done and still open.
| Rows | Verified | Wording stale | Discrepancy | Unverifiable | |
|---|---|---|---|---|---|
| Resolved | 135 | 121 | 6 | 7 | 1 |
Most of it held, but the worst of what did not was live.
A reply that beats its own command
The command plane follows every write to a device through a state machine. The Command Plane, published here a few weeks ago, defines the failure state like this:
timeout- no driver response within the deadline
The row that built the issuing path said the command "moves to sent after the publish returns". It did not. The publish happened inside the reconcile pass's transaction, which commits only after every point in the pass has been driven.
The half nobody had recorded was the other direction. A driver's reply is handled in a transaction of its own, which cannot see a command the pass has not committed yet. A reply that arrived quickly enough was looked up, not found, logged as a result that "does not apply", and dropped. The command sat in sent until its deadline and closed as timeout. A driver had answered well within the deadline, and the record says it never did.
The audit's note on which replies lose that race:
The replies most likely to win that race are the instant ones -
rejected,stale_epoch, a host error - which are exactly the ones whose reason matters.
No test had a driver reply before the commit. The audit filed it and left the fix to a change of its own, because closing the window was a design choice.
The week a resolution is written
The smaller discrepancies had a family resemblance. A test cited as proof of a database query fed the code a hand-built dictionary. A default that every test materialised by calling the function directly, and never by starting a core.
The useful finding was where they were. Of the 24 oldest rows, 21 held exactly. The unproven claims clustered in the rows resolved in the two days before the audit; two had been resolved that same morning, and one of those was wrong. The audit's closing note:
A resolution is least reliable in the week it is written, which is also when nobody rereads it.
That set how the audit runs. The first one reads everything; each later one starts from the commits the last one read and takes every row resolved since, plus all the open rows. The second ran two days after the first, on twenty rows, and found one wrong. The row said reprocess, the maintenance operation that rewrites stored history, refuses a second run while one is going. The code checked whether a run was in progress, then awaited a dry-run plan, and only then recorded that a run had started. Two requests overlapping by that much both started. The test for the refusal marked a run as in progress by hand, so it never saw the gap.
The same audit found a second race in the command plane's reads, in a place a decision had explicitly set aside as "not examined". Its note for the next one:
A check that precedes an
awaitis a claim about one request, and a sentence saying something "was not examined" is a row nobody filed.
It also began running both repositories' suites against real services and counting every skip, "because a skip is a test that silently does not run". That found 38 tests in the driver host that CI had never run, because the variables that enable them were never set there.
The column the code still named
The third audit, four days later, changed how it read. Six readers took the rows in parallel, and every finding they reported was checked against the code a second time before it was written down. What nobody had checked again went under its own heading, Read and not checked again, and none of it was routed. The next day a different session, one that had not reported them, read those eight again. All eight held.
The finding that mattered most was about something the code had stopped doing. The published copy of Ingest Path and Log Schema declares a column on the per-point state table:
watermark_seq bigint -- last control_event applied
The restart step that read it was later struck, and the column with it. The resolution said "the checkpoint stops naming the column", and the plan for dropping it rested on that: once staging and production both ran a release carrying the change, a migration could remove it.
The change had taken the key out of the dictionary of values the checkpoint writes. It left the statement those values are bound to, which is built from the table's model, and the model still declared the column. So the SQL on every checkpoint flush still ended watermark_seq = excluded.watermark_seq. The model's own comment said "Nothing reads or writes it".
Nothing was broken, because the column was still there. But a migration dropping it on that condition would have run while the outgoing core was still serving, and its next checkpoint would have failed. The test asserted that the key was absent from the dictionary, which was true, and never looked at the statement the dictionary was handed to. The audit's lesson is the most general thing any of the three has said:
A claim that code stopped doing something is checked on what reaches the database or the wire, and the audit that reads only the lines a pull request changed will agree with it every time.
The audit was not above its own method either. Its test run started throwaway message brokers on the only Docker daemon it could reach, which was the deploy host. It removed them, and said so: "The last audit withheld it on purpose, and this one should have."
Six instruments
Entry four's table gains a row:
| Instrument | Tense | Its normative home | What keeps it honest |
|---|---|---|---|
| the specification | what the system will be | itself | nothing internal; everything else comes at it from outside |
| the four registers | what is open now | the specification | the register checks, for the registers' shape only |
| the audits | what happened | the build step | a person, once per step, looking back |
| the seam reviews | what nobody said | the joins between documents | a rule that every step ends with one |
| the development guide | what exists today | the code |
test_dev_docs.py, and a marker on every claim |
| the deferral audit | what was said to be done | the resolved row | the last audit's commits, and a second reading of every finding |
A resolved row found wrong keeps its text and gains one line pointing at what was found. The wrong sentence is the evidence, and editing it would make the register look as if it had never been wrong.
The checks leave home
The register checks started as one test file in the Sentinel repository: ids numbered contiguously from one, every relative link resolving, every link's label naming an id that exists. Entry three was about where that file's reach ends.
A second project adopted the same registers and copied the test, changing the constants. Within weeks the copies disagreed. One wrote component ids C-12 and the other C12, they checked different documents, and the copy had two fixes the original lacked. That is the drift the registers exist to prevent, in the tests that check the registers.
So the checks now live once, in a repository called register-tools, as a GitHub Action each consumer runs inside its required test job. A small config file describes that repository's registers, and everything that had differed between the two copies became a field in it. One check improved in the move. The old copy counted a row's cells by splitting on " | ", so a pipe inside a code span broke a row's rendering on GitHub and the test did not see it. The action counts cells the way GitHub does. Each check's docstring records the incident that caused it, so the rule and the thing it once caught sit side by side.
Two branches, one number
The tool also carries the renumberer. Register ids are sequential. Two branches written in parallel, which with several agents working at once happens often, both add a decision and both call it D-184. The first to merge keeps it.
Detecting that was never the problem. The contiguity check refuses a duplicate, and it is a strict required check, so the second branch cannot merge. The cost was in the repair. Register ids are cited from nearly three hundred files, prose and code and YAML and migrations, and renumbering by hand had gone wrong twice, each time
leaving citations that resolved, named a real row, and meant a different one.
That is the worst kind of broken link, because every link check passes it.
Ids that cannot collide were weighed and turned down. The pull request's number in the id, or a random or date-based one, would each lose contiguity, the check that no row was ever deleted. A merge queue
Catches the collision earlier and removes none of the renumbering.
So the rule is the one now in the tool's README:
An id is fixed when it reaches
main, and not before.
The branch that merges second moves its own rows to the numbers after main's highest and rewrites every citation of them. It is one command, with a dry run that prints the plan.
The hard part is telling the branch's citations of D-184 from main's. The first version chose lines by where the diff placed them. With rows in a different order on each side, git can report one of main's own lines as removed and re-added, and the tool renumbered main's row. Lines are now judged by their content, and the merge base's lines are excluded too, after a smoke run on a real clone caught a branch still holding an older main's version of a line.
The first deferral audit's own rows had to be renumbered. Another branch had taken DW-161 while it was being written.
A finding becomes a check
The third audit also corrected component statuses by hand. One component was marked 🟢, complete, with an open deferral against it, written two days after that deferral was filed. Before that, the opposite shape had turned up: a component marked 🟡, partly met, whose citations of what it lacked were all to rows that had since resolved. Three repositories were each checking some version of that locally, each in its own way.
At the end of last week the second shape became a check in register-tools: a row marked partly met must cite an open row for what it lacks. Its docstring names the incidents. The first shape, a 🟢 that should not be, still needs someone to read the code and notice.
That is entry four's pattern, turned once more. A reader, person or agent, finds a shape. Once the shape can be stated, it becomes a check and nobody has to read for it again. The reading moves on to what no check can see yet, starting from the commits the last audit read.
This Wednesday - The Rules Engine. The specification's seventh part: an engine kept small because almost every trigger a control system needs is already a derived point or a quality transition. What is left is a predicate over transitions, with a claim as its output.
Next Monday - the Dev Diary. Build step five, read against The Command Plane.
Start of the diary: Four Registers and a Drift Check. Why the registers exist at all, and the one rule about where a sentence is allowed to live.
Top comments (0)