Sentinel Dev Diary: Read First, Run Second

Build step five of Sentinel is the command plane: the claim set and the arbitration over it, the command record and the path that issues it, verification, fencing, and the API a person uses to ask for a write. It is the step where the system stops only watching devices and starts changing them. The document that owns it, The Command Plane, was published here on 7 October.

The last two entries said this one would cover where the specification and the build disagree, and which of the two had to change. For step four the answer was usually both. For step five it splits cleanly by when each gap was found. Everything found by reading, before there was any code, changed the document. Everything found by running the code changed the code, and in every case the mechanism had not been run by anybody before.

Step five is also the first step whose seam review came before the build rather than after it. Entry five held back what that review found until the document it was about had been published. That has happened now, so this entry starts there.

Found before any code

A seam review reads each document against the ones it touches and asks what each assumes of the other, and where neither says. The one for The Command Plane ran on 7 September, the day before step five's first commit. Its method section explains why that timing mattered:

Findings are OI-* without exception - none gets a D-*, because no code has been written and so no reading has been taken.

An OI-* is an open question against the specification; a D-* is a decision recorded because the code had to pick something. The review found four. One of them is about a document that publishes on Wednesday and waits for it. These are the other three.

Two vocabularies for one key

The published copy, on what an interlock does when it cannot evaluate:

- id: heatsink-overtemp
  band: interlock
  interlock_class: thermal
  on_input_loss: fail_safe
  fail_value: false

The Rules Engine, published last Wednesday, introduces its own input-loss policy as "the same fork as interlocks" and lists release, hold and assert(value). That is three names against two, and none of them is shared. Both documents put the key on the same object, a rule in the registry, and an interlock is a rule. So one loader would have had to parse one key against two grammars.

The two vocabularies line up in pairs, fail_safe with assert and fail_clear with release. The third value has no partner, and the review called it the part with teeth:

hold exists only in Rules Engine, and an interlock is a rule ... If it is legal, hold_max: 30m means a cutout keeps asserting for half an hour on data it knows is stale

It survived because The Rules Engine uses both vocabularies itself: the list in one section, and fail_safe with fail_value in its warmup prose and its exit criteria. Read alone, each document is consistent.

The review proposed keeping both spellings as accepted values. That was rejected when the question was settled three days later, because two spellings a loader must both accept are two grammars on one key, which is the problem the question was filed about. There is now one set, release, hold and assert, defined in The Rules Engine. The Command Plane keeps fail-safe and fail-clear as the names of the two answers to its own question, which way protective cuts. fail_value became assert_value, because fail is the interlock's word and an ordinary rule taking a defensive position has no failure in it. hold is refused at load for band: interlock.

None of this could be tested at the time, and the resolution says so: the rule loader is step six, so nothing in the code parsed either vocabulary yet.

A limit read from a field that does not exist

The published copy lists what the core rejects before anything is queued. The last item:

rate limit per point, from the descriptor - a physical relay has a duty cycle, and this is the last defence against a rule loop wearing out hardware

The Semantic Model owns the descriptor and prints its writable fields in full. There are five, and none of them is a rate, a duty cycle, a minimum interval or a cooldown. Every other descriptor field The Command Plane relies on is there, which made this one an omission rather than a different design. And it could not be the driver's, because a duty-cycle limit is a timer and The Driver Contract forbids a module to hold timers.

Adding the field was the easy half. The descriptor gained min_write_interval, optional, a minimum interval rather than a count per window, "because one number is what a datasheet gives".

The review had asked a second question as well: what a breach does. Answering it moved the check out of the list it was printed in. The other three checks in that list refuse requests that can never succeed. A sensor does not become writable. A rate limit refuses a request that will succeed a minute later. The paragraph just below the list in the published copy already made that distinction for deliverability, "Deliverability is checked here too, but it is not a rejection", so the rate limit moved beside it. A held-back write is reconsidered on the next pass like any other owed write.

Two more things came with it. An interlock is exempt, because a duty cycle is about wear across many cycles and a late cutout is the worst failure available. And a point with no min_write_interval has no limit, which the document now states as a weakness: a fleet-wide default would be a number nobody owns.

Command state with nowhere to live

The Command Plane needs state that outlives a single command. It asks for one in-flight command per point with the rest queued, coalescing of the queue, and this on restart:

Anything in sent or verifying at shutdown is marked timeout on recovery

That requires knowing what was in sent or verifying. The specification declared nine tables across two documents and none of them was a command. Command events had a home in the control log, which is append-only and kept forever, but that log had no command_id column. Working out "what is in flight now" would have meant folding a lifetime of events through a JSON key to answer a question about the last few seconds.

The review's first draft got this wrong. It said no document says where commands are stored, and had to be corrected: Ingest Path and Log Schema says plainly where command events go. What has no home is their current state.

Both answers needed a migration, so cost did not decide it. The serialiser did. One in-flight command per point, as a partial unique index on a command table, is a constraint the database keeps across a restart. As a fold over the log, it is something the core rebuilds in memory at the moment it is least able to, coming back with a queue half sent. Step five built the table the day after the review, and Ingest Path and Log Schema printed it a week later, copied from the migration rather than written for the document.

What reading first bought

Each of these became a paragraph or two of specification, written by someone weighing two readings with no code depending on either. The review's last line:

Nothing here is code that is owed, and nothing has been routed around - which is the entire value of having looked before step 5 rather than during it.

Found by running it

The rest of step five's findings came from the code, and two of them came from probing a criterion rather than reading the line that said it passed.

A note that explained the amber away

The published copy on fencing:

a stale epoch means two cores existed and one of them was still issuing commands. It escalates rather than retrying.

The exit criterion for it was marked amber, partly met, with a note explaining why that was fine. The core "does not retry it, which is trivially true of a core with no retry".

Both halves were false. On 10 September the criterion was probed directly. After a driver refused a command as stale_epoch, the next reconcile pass issued another command, and the pass after that a third, until the retry budget ran out. Then the core reported the point diverged, which says the device refused. A fenced core went on writing to a bus it was no longer entitled to, and its own record blamed the hardware.

The fix says what escalating means. The point is fenced and this core never drives it again, because nothing a running core can do raises its own epoch. The core asks the database at once who holds the lock, since a driver refusing its epoch is the strongest evidence available that somebody else does. And a new signal, core.{n}.fenced-points, is published, because any value above zero means two cores existed and alerting lives outside the core.

It does not exit from the fence itself, and the reason is written down. The handler runs inside a NATS subscription callback, the client library catches whatever a callback raises, and there was no error callback. An exit raised there would have been swallowed. The exit stays with the periodic lock check, which the process actually watches.

The step's audit drew the lesson from the note:

The marker was not overstating a gap; it was describing a mechanism nobody had run.

The double-click

The published copy's whole treatment of idempotency is two sentences:

idempotency_key on the intent, deduplicated over a short window. A UI double-click or a bus redelivery must not produce two commands.

The double-click was written as a test against the code as it stood, and it produced two commands. So the sentence was false rather than unbuilt. Writing the fix found five things the two sentences did not say: how short the window is, what a repeat returns, what the key is scoped to, what a repeat with a different body does, and whether a withdrawal carries a key.

All five are answered, and the section is nine paragraphs now. The window is 60 seconds. A repeat returns the original answer, the same command_id included, because a caller that could tell its retry from its original would be back to guessing whether the first one landed. The key is scoped to the principal. A key reused for a different request is refused, and the comparison includes the point named in the path as well as the body. Without that, one principal reusing a key on a second point would be told the first point's outcome: a write they believe happened, which never did.

Code nothing called

Entry one quoted a decision titled A guard with no caller is not a guard. Entry five found the next one, derived points that were computed and never evaluated. Entry six said three resolutions had been found wrong in that shape before the first deferral audit. The other two were in this step.

The first was verification. Every confirmation mode was correct and tested, by calling the verification module directly. Nothing in the running core called it. A command a driver accepted stopped at acked, and because acked counts as in flight, the database's one-per-point constraint refused every later write to that point for the life of the process. A restart closed the command, which bounded the damage, and is also why nobody noticed. It was found on 17 September by a session writing the developer guide from the code, without the specification open. The register row:

every mode was proved correct in isolation ... which calls enter and on_observation directly

The second was recovery. The restart behaviour quoted above, close what the last core left, was a function that did exactly that, with tests that called it by hand. Nothing called it at startup. A migration's docstring asserted that "a core starting on this schema runs recovery before anything else". A queued command survived the restart, and could later be sent carrying the previous core's epoch, which a driver refuses as stale_epoch and the core then treats as a fencing incident. The criterion it undermined had read ✅ for over two weeks.

There was a third on the same day, of a slightly different kind. A rule changing a positional group, the three channels of a colour, wrote it one member at a time. Moving a light from (A, A, A) to (B, B, B) published (B, A, A) and queued two more commands behind it. The criterion that a write group is written as one command had been proved for the API a person uses, and for nothing else. A rule is a writer like any other, and nothing proved it for a rule. It was latent, since no rule claims a write group yet, and fixed in the commit that filed it.

The step audit's paragraph on verification has the line I keep coming back to:

the lesson did not generalise the first time

A test of the wiring had been written after derived points, and it covered the issue path. The verification path went on having exactly the tests that finding had been about.

What ✅ said

Entry five rested on one audit of the step, written once the step was done. Step five never had one. Its notes were written into the roadmap's Build steps section, a paragraph at a time, by the changes they describe. On 27 September that prose was moved into the directory where the other steps' audits live, and the file opens by saying so:

Moved, not re-audited. ... Nobody re-read the code against Command Plane to produce it.

Two passages were audits, on 10 and 14 September. The rest of what this entry describes was found by other readers: the seam review before the build, a probe of one criterion, the developer guide, and the deferral audit from entry six. The file marks three statements that were false by the time it was moved, and leaves their text standing.

The status marker itself was wrong in both directions. It read not started for four merges after the claim set and resolver had landed, while the component register marked each piece as it arrived. It read amber over a fencing note that explained the gap away, and ✅ over verification and recovery while both were broken. And for a while it read ✅ while seven open questions still stood against the step's own documents; the last of them were settled on 14 September. The audit's note on that says plainly what the tick means:

✅ says the exit criteria pass, and an OI-* is a gap in the specification rather than in the build.

A finished step does not mean a settled document. And for this step, ✅ meant the criteria passed when somebody tried them, not that somebody had read the code against the document from start to finish.

What each kind cost

The three seam findings cost a reading and an amendment each. Nothing had been built around them, so the decisions were made on the merits, with the next document open.

The findings from running the code cost a defect each, and a defect has already done something by the time it is found. A fenced core wrote to a bus it no longer owned, a point refused writes until a restart, and a queued command outlived the core that queued it. The last two had tests that passed the whole time the bug stood, because those tests called the mechanism directly and nothing in the running core did.

Reading could not have found those three. The rate limit could be specified properly before any code existed, but whether anything called verification could only be found by starting a core and watching what it did.

---

This Wednesday - The Core Runtime. The specification's eighth part: why only one core may run at a time, what enforces it, and what the system does while no core is running. Its startup order is what the seam finding this entry held back is about, and that finding comes the Monday after.

Next Monday - the Dev Diary. Build step six, read against The Rules Engine, and the seam finding The Core Runtime leaves open.

Start of the diary: Four Registers and a Drift Check. Why the registers exist at all, and the one rule about where a sentence is allowed to live.