DEV Community: Philip Shaw

Sentinel: The Command Plane

Philip Shaw — Wed, 07 Oct 2026 09:59:39 +0000

Everything in this series so far has been about reading from devices. This is the part where Sentinel starts writing to them, and a mistake here can leave a heater running or unlock a door twice.

A writable point has two values. reported is what the device says and desired is what the system wants. Nothing writes to a device directly: a rule or a person sets a claim, arbitration resolves the claims on a point into desired, and a reconciler drives reported toward it. A device that reboots into the wrong state is corrected on its next report, and nobody has to send the command again.

Every claim carries an owner and one of six bands, and the band decides who wins. A person's manual claim has to expire, because otherwise one touch of a switch disables the automation on that point until someone notices in November that the heating never came back on. Frost protection can be suspended for half an hour by someone who names it and gives a reason. An interlock can't be suspended at all.

A write on the bus is fire-and-forget, so each command is followed through a state machine to an outcome. For a relay whose state can't be read, the outcome is unverified and the point is marked assumed, which a rule can choose not to trust. A core that restarts in the middle of a write marks the command timeout and works out what to do next from desired and reported. It never sends the same write a second time. And no command is created for a device that can't be reached: the claim stands and waits, so an interlock asserted during cold start is issued the moment its driver is ready, with no deadline it could have missed in the meantime.

Rules never send commands. They set claims and release them, and that is what lets the rules engine, the next part, be tested without a device in the loop.

What this document owns: desired and reported as separate values on a writable point, the durable claim set and the projection resolved from it, the convergence states and the bounded retry, the command state machine, per-point serialisation and queue coalescing, the four confirmation modes, deliverability and re-drive, the six bands and how arbitration resolves them, manual expiry and withdrawal, suspension of a named owner, how a rule's band is declared and checked at registry load, fail-safe versus fail-clear on input loss, the limit a confirmation mode puts on what an interlock can protect, the intent API and the direct-command escape hatch, validation before the wire, and idempotency of an intent. It does not own core_epoch and the fencing behind it, which belong to Core Runtime; how a principal is established, which belongs to Security Model; the command subject, the driver's acknowledgement, the instance states that deliverability reads and the idempotent flag on a driver assignment, which belong to Driver Contract; the timer that claim expiries share with freshness deadlines, which belongs to Watchdog & Derived Points; or the shape of a rule and when it sets or releases a claim, which belong to Rules Engine.

The command plane is where the system stops being an observer and becomes able to be wrong in ways that matter. Two things carry the weight: the state machine that turns a fire-and-forget write into a verified outcome, and arbitration, which decides whose intent wins.

Desired and reported are separate points

A writable point has two values, not one. reported is what the device says. desired is what the system wants. They are different fields on the same point, with independent quality.

Nothing writes to a device directly. Something sets a claim, arbitration resolves the claim set into desired, and a reconciler drives reported toward it. This is the difference between "send an off command" and "this light should be off", and it is what makes the system self-healing: a device that reboots into the wrong state gets corrected on its next report without anyone re-issuing anything.

The claim set is durable; desired is a projection of it. Arbitration has to fall through to the next-highest claim when one is withdrawn, and to restore a suspended claim when its suspension expires. Both require every active claim to be stored, not just the winner.

CREATE TYPE band AS ENUM (        -- declaration order is precedence order
  'default','automation','schedule','manual','safety','interlock');

CREATE TYPE convergence AS ENUM (
  'no_intent','undeliverable','converging','converged','diverged');

CREATE TABLE claim (
  point_id    text        NOT NULL,
  owner       text        NOT NULL,   -- 'rule:frost-protect', 'user:alice', 'system:default'
  band        band        NOT NULL,
  value_num   double precision,
  value_bool  boolean,
  value_text  text,
  set_at      timestamptz NOT NULL,
  expires_at  timestamptz,            -- required for band 'manual'
  reason      text,
  PRIMARY KEY (point_id, owner)
);

CREATE TABLE point_desired (
  point_id     text PRIMARY KEY,
  value_num    double precision,
  value_bool   boolean,
  value_text   text,
  owner        text,                  -- null when no claim is active
  band         band,
  resolved_at  timestamptz NOT NULL,
  convergence  convergence NOT NULL,
  last_command uuid
);

Postgres compares enum values in declaration order, so the band is the comparison. There is no separate priority number to keep in step with it.

One claim per owner per point. A rule asserting a different value updates its own row rather than accumulating claims.

Convergence is evaluated on every reported transition. desired set with no matching reported after the confirmation deadline is a divergence, and divergence is a first-class alertable condition, surfaced as a self point per writable point so that ordinary rules and dashboards can act on it.

Divergence gets a bounded retry: reconcile, wait, reconcile once more, then stop and mark diverged. Unbounded reconciliation against a device that physically cannot reach the desired state means a relay clicking every thirty seconds until someone notices.

What the reconciler does when the device is not reachable at all is a different question, dealt with under deliverability below.

Command state machine

                  ┌──────────→ rejected
                  │
created → queued → sent → acked → verifying → verified
                    │       │         │
                    │       │         └────→ unverified
                    ├───────┴─────────────→ failed
                    └─────────────────────→ timeout

created — intent accepted, arbitration passed, command_id assigned
queued — waiting on the per-point serialiser
sent — published to cmd.{host}.{instance}.{point_id}
acked — driver reports transport accepted
rejected — driver refused before attempting (unknown point, bad value, instance not ready, stale epoch)
verifying — awaiting confirmation per the point's mode
verified — reported now matches desired
unverified — optimistic mode, or verification window elapsed with no contradiction; point quality becomes assumed
failed — driver reported an error
timeout — no driver response within the deadline

Every transition is a control_event, carrying the principal that caused it and the core epoch that issued it. Terminal states carry the elapsed time, which gives per-device command latency as a self point without any extra instrumentation.

Every command carries core_epoch. It is the fencing token from Core Runtime, and a driver rejects any command whose epoch is below the highest it has seen, with reason stale_epoch. That rejection is treated differently from every other rejection: an unknown point or a bad value is a configuration error, but a stale epoch means two cores existed and one of them was still issuing commands. It escalates rather than retrying.

Serialise per point. One in-flight command per point, others queued. Concurrent writes to the same relay are never what anyone wanted, and a queue depth of more than one or two is itself a signal worth exposing.

Coalesce the queue. If two commands for the same point are queued and neither has been sent, the older is dropped. A rule flapping should produce one write, not a backlog that plays out after the condition has passed. Coalescing never crosses a sent boundary — the ordering guarantee has to hold for anything already on the wire.

Confirmation modes

inline — the driver's device_ack carries the new value. Compare against desired, go to verified or failed. Latency is one round trip.

async_event — wait for a subsequent observation on the point. Correlate by value match within the verification window, because most protocols do not echo command IDs. Ambiguity is real here: if something else changed the point to the same value in that window, it will be attributed wrongly. That is accepted, and the window kept tight, typically a couple of seconds.

poll_verify(delay) — the core schedules a read after the delay. This is the Modbus path, and it is the core issuing a read, not the driver remembering to. Deliberately a re-read rather than waiting for the next scheduled poll, because a 30 s poll interval is too long to hold a command open.

optimistic — no verification exists. The command goes to unverified, reported becomes the desired value with quality assumed, and it stays assumed until something real contradicts or confirms it. This is honest about the write-only relay whose state cannot be read, and the assumed quality means a rule can choose not to trust it.

Verification timers are durable, in control_event, rehydrated at restart. A core that restarts mid-verification resumes the wait rather than losing the command.

On restart, in-flight commands do not resume being sent. Anything in sent or verifying at shutdown is marked timeout on recovery, and the reconciler re-derives from desired versus reported. Replaying a write whose outcome is unknown is how a door gets unlocked twice.

Deliverability and re-drive

Reconciliation only means anything when the device is reachable. Two conditions decide that, and the driver plane already publishes both.

A point is deliverable when its driver instance is ready or degraded, and the point's own quality is not unavailable. An instance still connecting and a device the driver has declared off the network are the same thing from here: a command would go into the dark.

Undeliverable is a state of the intent, not the outcome of a command. No command is created while the target is undeliverable. The claim stands, desired holds its value, convergence sits at undeliverable, and nothing is queued against a wall-clock deadline. A deadline that expires for reasons unconnected to the device says nothing about the device.

This is why convergence is an enum rather than the converged boolean that would otherwise be the obvious choice. A boolean has to mean three different things — no intent, not yet delivered, and genuinely failing — and that is the same null-means-several-things trap the semantic model exists to escape.

The retry budget counts only attempts made while deliverable. The bounded retry above exists to stop a relay clicking at a device that cannot reach the desired state. A device that is not there has not refused anything. Burning both attempts against an unreachable instance produces a diverged that means nothing, and consumes the budget the real divergence would have needed later.

Re-drive on the transition to deliverable. When an instance reaches ready, or an observation clears a point's unavailable, every non-converged claim on the affected points is re-driven and its retry budget resets. It is the same reconciliation a reported transition triggers, on a different edge.

Spread the re-drive. A driver host with two hundred points reaching ready would otherwise issue two hundred commands in the same instant, at a device and a transport that have just finished coming up. Modbus in particular does not tolerate that. Concurrency is bounded per instance and the issue jittered. The per-point serialiser and the per-point rate limit do not help here, because the burst is across points rather than within one.

Cold start is the ordinary case of this, not a special one. At core start every instance is connecting, so every claim is undeliverable and nothing is sent. Claims made during warmup — in practice the fail-safe interlocks, since every other rule is gated — are recorded immediately and issued the moment their instance becomes deliverable. The protective claim is never racing a timer it can lose; it is waiting on a condition it can only win.

Undeliverable is not silent for protective claims. An ordinary claim can wait indefinitely and nobody needs to know. An interlock or safety claim sitting undeliverable is unenforced protection, which is exactly as serious as an interlock that has diverged. Convergence is exposed as a self point per writable point; for interlock_protected points, undeliverable past a short bound escalates rather than waiting quietly.

Arbitration

The part that is cheap now and expensive later.

Every claim carries an owner and a band. The band is the whole of the precedence model.

band	typical owner	notes
`interlock`	protective rules that may not be overridden	requires `interlock_class`
`safety`	frost protection, over-temperature cutouts	suspendable, never outrankable
`manual`	a person	`expires_at` required
`schedule`	time-based rules
`automation`	ordinary rules
`default`	`system:default`	optional resting state from the descriptor

Resolution. The active claim set for a point is every claim whose expires_at is unset or in the future and whose owner is not currently suspended. The winner is the highest band; ties within a band go to the most recent set_at. The resolver is a pure function from the active set to a winner, and it runs on every claim insert, update, withdrawal, expiry and suspension change.

default is an ordinary claim, and it is optional. A point descriptor may declare default_value; when it does, the registry materialises a non-expiring claim at band default owned by system:default. When it does not, no such claim exists and there is no special case in the resolver.

An empty active set means no intent. point_desired.owner is null, convergence is no_intent, and the reconciler does nothing. Releasing the last claim on a point leaves the device where it is rather than driving it somewhere. Asserting a resting state is something a point opts into by declaring default_value.

That is right for a light and wrong for anything protective, so the descriptor carries require_intent. A point with require_intent: true must declare default_value, checked at registry load, which makes an empty active set unreachable and desired never null. Any point marked interlock_protected gets require_intent implicitly, and the load fails if it has no default.

Manual claims expire. Band manual requires expires_at, defaulting to an hour or two and configurable per point. On expiry the claim is deleted, arbitration re-resolves, and the point falls through to whatever automation wants. Without expiry, every manual touch permanently disables automation for that point, and the discovery comes in November when the heating never came back on.

Claim expiries run on the same timer mechanism as freshness deadlines, and like them they are derived rather than stored: recomputed from expires_at when the claim set loads, so a restart cannot lose one.

Withdrawal, not just setting. An owner can release its claim, and the point falls through to the next-highest active claim, or to nothing. Rules that have stopped applying must release rather than continue asserting, otherwise arbitration resolves against stale claims forever.

Every resolution is logged to control_event with the winner, the full losing set, and the principal responsible. "Why is this light on" is then a query against the claim set, not an exercise in reasoning about rule ordering.

Two load-time checks:

Two rules claiming the same point at the same band is a lint warning. It is almost always unintentional, and last-writer-wins between them is rarely what anyone meant.
Two interlock rules claiming the same point with different values is an error, not a warning. Resolving that at runtime is worse than failing the build.

Safety bands

interlock and safety are separate bands, and the difference between them is not precedence. manual outranks neither. The difference is that a safety claim can be suspended and an interlock claim cannot.

Suspension is a distinct mechanism from precedence. It is not that manual wins the comparison; it is that the suspended claim is temporarily withdrawn from arbitration on explicit request.

Suspension is scoped to the owner, not to the point. A frost-protection rule holding three pumps is suspended once, for all three. Per-point suspension would let someone servicing the system suspend one pump, believe protection was off, and be wrong about the other two.

CREATE TABLE claim_suspension (
  owner        text PRIMARY KEY,
  suspended_by text        NOT NULL,
  suspended_at timestamptz NOT NULL,
  expires_at   timestamptz NOT NULL,   -- never open-ended
  reason       text        NOT NULL
);

POST /claims/rule:frost-protect/suspend
  { for: 30m, reason: "servicing the pump" }

The request is rejected if the named owner holds any claim at band interlock. Otherwise the suspension is recorded, every point that owner claims is re-resolved, and the response returns those points with their new winners — so the caller sees the actual blast radius of the decision. The suspension and its expiry are both control_event entries, and on expiry the claims return automatically.

expires_at is not nullable and there is no unsuspend-forever. A suspension that outlives the attention of the person who set it is the failure this mechanism exists to prevent.

Modelling it as suspension of a named owner rather than as a precedence comparison forces the person to name what they are overriding. "Turn this on" is ambiguous. "Suspend frost protection for thirty minutes" is a decision someone can be held to, and it is legible in the log afterwards.

Which band a claim gets

The band belongs to the rule, declared in the registry, and validated at load rather than trusted at runtime:

- id: pump-dry-run-cutout
  band: interlock
  interlock_class: thermal
  ...

band: interlock requires interlock_class (thermal | electrical), and the registry rejects it otherwise. That forces the classification to be explicit and reviewable in a diff rather than emerging from whoever wrote the rule.

Two load-time checks:

No point may have interlock claims from more than one rule unless they assert the same value. Two interlocks disagreeing is a design error, and resolving it by precedence at runtime is worse than failing the build.
Any point that a rule targets with band: interlock gets interlock_protected: true on its descriptor. That flag is what the API and UI key off, and it means protection is a property of the point that is visible without walking the rule set.

Fail-safe versus fail-clear

The harder question, and the one the band split does not answer on its own: what does an interlock do when it cannot evaluate?

An over-temperature cutout depends on a temperature point. If that point goes stale, bad, or unavailable, the rule has no basis for a decision. Two possible behaviours, and they are opposite:

fail_safe — assert the protective value. Cut the heater. Correct for anything where the failure mode is thermal runaway or overcurrent.
fail_clear — release the claim. Correct where asserting the protective value is itself harmful — a circulation pump forced off during a heat cycle can cause the boiler overheat the interlock was meant to prevent.

There is no sensible default, so it is required:

- id: heatsink-overtemp
  band: interlock
  interlock_class: thermal
  on_input_loss: fail_safe
  fail_value: false

fail_safe requires fail_value. The registry rejects an interlock rule without on_input_loss — this is the one place where inference is refused outright, because getting it wrong is silent until the day it is not.

For fail_safe interlocks, the asserted claim persists until inputs return to live and the rule evaluates cleanly, not merely until data reappears. A single good reading after a fault does not clear a cutout.

Confirmation mode constrains what an interlock can protect

An interlock on a point with confirmation_mode: optimistic cannot guarantee anything — it asserts a value with no way to verify it, and the point sits at quality assumed. That is acceptable for a light and not for a thermal cutout.

This is a load-time rejection: interlock_class: thermal or electrical requires the target point to have a verifiable confirmation mode (inline, async_event, or poll_verify). Protecting a write-only relay requires a second point that reads the actual state — a current sensor on the circuit, an auxiliary contact — rather than an interlock that cannot see what it is doing.

Interlock non-convergence is treated differently from ordinary non-convergence in both directions it can fail. diverged gets a shorter deadline, no bounded-retry give-up, and escalation rather than a settled state. undeliverable escalates too, on the bound described under deliverability. An interlock that has failed to take effect is the loudest thing this system can say, and it does not matter whether it failed because the device refused or because the device was not there.

The intent API

POST   /points/{id}/claim
  { value, expires_in?, reason?, idempotency_key? }

DELETE /points/{id}/claim        withdraws the caller's own claim

Owner and band are derived from the authenticated principal, never supplied in the body. A rule's band comes from its registry declaration; a person's is manual. An API that takes owner and a priority number as parameters lets any caller assert whatever authority it likes, which is not an authorisation model. The mechanism for establishing the principal is in Security Model.

Every mutating call names a principal, and there is no default. A request that arrives without one is rejected rather than attributed to a service identity, and the principal is written to every control_event the call produces. That requirement is what makes the claim log an audit trail rather than an activity feed: nothing can change state without saying who asked.

The response is the resolved arbitration outcome and the resulting convergence state, plus a command_id if this claim won and a write was issued. A claim that won on an undeliverable point returns undeliverable and no command ID, which is a normal answer and not an error. A losing claim is still recorded, and the response returns the current winner. That is not an error either, and the caller may well want to know it was overruled.

Withdrawal removes only the caller's own claim. Removing someone else's is suspension, which is the mechanism above and carries its rules.

A direct-command escape hatch exists for commissioning and debugging, gated behind its own capability and loudly logged, bypassing arbitration. Two limits on it, both non-negotiable: it does not override an asserted interlock claim, because an escape hatch that can defeat a thermal cutout is the absence of one; and enablement is bounded and expiring exactly like a suspension, with both edges in control_event, rather than a mode someone leaves on. It is needed when a device is misbehaving late at night, and building it deliberately is better than the alternative of someone reaching for a raw publish.

Validation before the wire

Rejection at the core, before anything is queued:

point is direction: out or inout
value type and enum membership check
range check against the descriptor
rate limit per point, from the descriptor — a physical relay has a duty cycle, and this is the last defence against a rule loop wearing out hardware

Deliverability is checked here too, but it is not a rejection. An undeliverable target means the claim is accepted and the command deferred until re-drive, never queued against a deadline waiting for an instance to come up.

Command values are supplied canonical and converted to native on the way out, using the same table as ingest, inverted. Same registry version, same affine handling. An asymmetric conversion path is a bug waiting to happen, so it is one function used in both directions.

Idempotency

idempotency_key on the intent, deduplicated over a short window. A UI double-click or a bus redelivery must not produce two commands.

The idempotent flag on the driver assignment is separate and narrower: it tells the driver whether a single transport-level retry is safe. A coil write is idempotent. A pulse or a momentary is not, and is marked so — the driver never retries it, and a lost command becomes a timeout that the reconciler handles rather than a double-fire.

What this buys the rules engine

Rules never send commands. They set claims with a value and a band, and release them when they stop applying. Everything about delivery, retry, verification, convergence and conflict happens below them.

That makes rules testable in isolation: a rule's output is a claim, and a claim is data that can be asserted against without a device in the loop.

Exit criteria

Arbitration is a pure function, so it is a fixture table. Claim set in, winner out, with no device and no bus. The cases that need covering are the ones the model exists for rather than the obvious ones:

withdrawal falls through to the next-highest active claim, and to nothing when there is none
an expired manual claim falls through without anyone withdrawing it
a suspension removes an owner's claims across every point it holds, and they return automatically on expiry
suspending an owner that holds an interlock claim is refused
an empty active set resolves to no_intent and the reconciler does nothing
a point with default_value falls through to the band-default claim instead

Interlock cold start delivers rather than times out. Assert a fail-safe interlock claim before any driver instance is ready; assert that no command is created, that convergence is undeliverable, and that the command is issued the moment the instance reaches ready. A timeout here is a failure, not a pass.

The retry budget is not consumed while undeliverable. Take a point unavailable, leave a claim standing, bring it back, and assert the bounded retry still has its full allowance. The bug this catches is a diverged that means nothing, which then hides the real one later.

Fencing. A command carrying a core_epoch below the driver's highest is rejected with stale_epoch, and the core treats that outcome as an escalation rather than a retry.

Restart does not replay writes. Commands in sent or verifying at shutdown become timeout on recovery, and the reconciler re-derives from desired versus reported. Assert that no command is re-published — this is the door-unlocked-twice case.

Every mutating path names a principal. A request without one is rejected, and every control_event written during the fixture run has principal and core_epoch populated. The NOT NULL constraints make this hard to get wrong, which is the point of putting them in the schema.

A break until November. This is the last new article here for October. The specification and the Dev Diary both pause after this part and pick up again at the start of November.

The next part - The Rules Engine. An engine kept small because almost every trigger a control system needs is already a derived point or a quality transition. What is left is a predicate over transitions, with a claim as its output.

The latest Dev Diary - Auditing the Watchdog. Build step four read against the previous part: a sentence with two halves and a test asserting the one that worked, and a stale point whose checkpoint never moved.

Start of the series: An Introduction. The map, the two decisions every later document is downstream of, and why a specification at this scale is being published in public while the system it describes gets built.

Sentinel Dev Diary: Auditing the Watchdog

Philip Shaw — Mon, 05 Oct 2026 09:23:13 +0000

Build step four of Sentinel is the freshness watchdog and derived points. It was audited about four weeks ago against the document that owns it, The Watchdog and Derived Points, which went out last Wednesday.

That post promised this entry would say where the specification and the build disagree, and which of the two had to change. For this step the answer is usually both. In one case the code changed, then the decision that explained the code, then the document, in that order and each because of the one before.

Half a sentence

The published copy says this about a stale point recovering:

An observation with observed_at older than the current deadline - a buffered replay landing late - does not clear staleness, though it is still logged.

There are two claims in it. A late observation must not push the point's deadline out, and it must not change the point's quality.

The first was built. Ingest hands the watchdog each observation's observed_at, so a replay of hours-old data schedules a deadline that is already in the past.

The second was not, and could not have been where the code stood. The projection applies an observation's quality without ever seeing a deadline; the information is not available to it. A three-hour-old replay carrying live moved a stale point to live, and the watchdog put it back on the next tick. That is two transitions and two control-log rows where the document asks for none.

It survived because of the test. test_a_late_replay_does_not_push_the_deadline_out is named for the rule, starts from a live point, and asserts that it goes stale. The sentence is about a point that is already stale, and no test started there. The audit's line on it, at the time:

How it survived: the test that quotes the sentence asserts the half that works.

The fix was wider than the sentence

Holding the quality and applying everything else is what the sentence appears to ask for. The decision row records why that does not work.

A host reconnecting after an outage replays many rows, oldest first. The projection wrote observed_at unconditionally, so an old row moved it backwards, and every deadline is derived from observed_at. Holding only the quality

would let the first row lower the bar the second is judged against, and a few rows in a replayed hour would read as current.

The transition published on the bus also carries the observation's own quality. A held row that was still published would tell every subscriber the point read live while the projection said stale.

So the reading taken was the stronger one: a late replay is written to the log and advances nothing.

It is a row in the history and no part of the present.

The same row names the trap in the obvious implementation. "Older than the current deadline" is true of every healthy observation; a point reporting every 30 seconds under a 45-second window is always inside its deadline. A rule keyed on lateness alone

would freeze the entire projection on the first observation after startup.

Two more rows came out of building it. One out-of-order observation could mark a healthy point stale, for the same reason: observed_at went backwards and took the deadline with it. The other was filed as a cosmetic note about three fields nobody reads. It turned out that a withheld row was still being treated as the predecessor of the row after it, which suppressed a real edge on the next reading.

Then the document moved

The decision gated its rule on the point being stale, because that is the state the sentence is about. Which other states a late replay may promote was left as an open question against the document.

The document's answer was to stop asking what state the point is in. The current text says a reading that was already past its own deadline when it was ingested "is expired on arrival: it is written to the log and advances nothing". It is judged on its own age, measured from ingested_at, instead of against whatever deadline the point happens to hold. That closed the question and superseded part of the decision that had prompted it.

One sentence in the copy published here is now six paragraphs.

Criteria that passed without a core

The first exit criterion in the published copy is specific about how to test it:

Load a checkpointed projection, start the core, and assert zero restored → stale transitions inside each point's grace, and the correct ones after it.

What existed was six unit tests over a watchdog constructed inside a test. The integration test that does load a checkpoint and start a core asserted nothing about staleness. The second criterion was covered in three pieces, with core.{n}.staleness-suppressed asserted against a stub that returns what it is given. The audit: "Nothing asserts the chain".

Both got the tests the document asked for, against a started core swept by its own timer. The register row for that work:

The second test found a live bug on its first run, which is what the deferral was about

staleness-suppressed is the only boolean among the core's signals about itself. The code that turns a sample into a row narrowed the value to a number and excluded booleans without putting them anywhere. So the signal the published document describes as saying "plainly that this is happening" was written on every pass as a row with no value in any column. The stub had asserted one layer above the row.

The third criterion is the quality propagation table, one fixture per row. It passed. The row that records what was behind it:

Derived points were validated, computed, and neither evaluated nor written. Exit criterion 3 passed on pure functions with no loop around them

Entry one quoted a decision titled A guard with no caller is not a guard, and I wrote that I had no idea how many more of those there were, presumably not zero. This was the next one, and it covered every derived point in the system.

Wiring it found a collision

A derived point's output goes through the ingest writer into the observation table, because rehydration reads that table and a chain of derived points has to be rebuildable from the log.

The dedup key is the one from Ingest Path and Log Schema: instance, epoch, sequence and ingested_at. Every row in one derived pass shared three of the four, and the pass wrote one evaluation at a time with the sequence restarting at zero. Every point got the same key. The insert kept the first and dropped the others silently, and the symptom was a chain whose second point stayed unknown for ever.

No unit test of a single derived point could have seen that. It needed two points written in the same pass.

The stale point whose checkpoint never moved

This one was found in a running core.

Ingest Path and Log Schema gives the checkpoint's write policy as "immediately on quality change". That write was made by the ingest pipeline, from the observation that caused the change.

A point going stale is a point that has stopped sending observations. So the path never ran for it. The checkpoint row sat at the last quality the point had reported, while the core's in-memory count of stale points moved. The two disagreed for as long as the point stayed quiet.

Patch coverage had flagged it. Seventeen changed lines were uncovered on the pull request, and they were the watchdog's entire wiring into the running core. Nothing acted on the flag, because the test job was the only required check. The repository's own note:

A coverage gate that reports and does not block is a gate that tells you afterwards.

The gate now sits inside the test job, and every changed line has to be covered for it to pass. I have written before about a rule being only as strong as what enforces it, and this is the same lesson costing a bug.

Where the document changed

Three things in this step ended with the document amended. A fourth did not.

The fan-out

The Driver Contract promises that a driver dying silently "is caught by its own heartbeat going stale - the core marks the instance unavailable after a grace period". The published watchdog document gives one route to unavailable: an availability event from a driver. A dead driver sends no event, and a missed deadline produces stale. No rule anywhere could perform the promise.

The watchdog document took it, before the step started. A stale instance heartbeat now marks every point that instance owns unavailable. The heartbeat itself stays stale, because "it is the evidence rather than a casualty". And "there is no second grace period": the heartbeat's own interval and grace are the wait. Because the fan-out is triggered by a staleness evaluation, it inherits both suppressions, so "neither a restart nor a slow core can take a fleet unavailable".

The propagation table

The published copy gives six rows as a list. Two of its answers are transitions The Semantic Model forbids, unavailable → stale and unknown → stale, and both are reachable. Row one asks whether an input "has ever been live", and nothing stores history. restored appears in no row, though every input is restored after a restart. And the order decides the answer for an input that is bad having never been live, in a list that does not say it is ordered.

The table is longer now, it is ordered, and it says first match wins. Where a row's answer is an illegal move, the point holds what it has and the control event carries the reason held. That is the two-exit rule on unavailable from entry two, applied one level up.

Flap damping

The published text says transitions "are counted in a rolling window; past a threshold" further ones coalesce "until it settles". It gives no window, no threshold and no definition of settled, and underneath that it does not say whether the window is per point or global. The audit left it unbuilt on purpose:

Building to a guess would put the only absolute duration in the mechanism.

The document now says the window is per point and four times the point's own tolerated silence, the threshold is six, and settled means a whole window with no crossing. That last one is deliberately not "the point is live for a window", because a dead device settles into stale and would never clear.

It has been built since. Building it found that the register row describing the work had named the wrong place to put it. The row said damping was a filter over the watchdog's output, which would have counted entries into stale. A point can legally go stale → bad → stale, so that is a different number from the live/stale crossings the document counts.

The wheel

The published copy asks for a hierarchical timer wheel. The code has a flat one, and the document still says hierarchical. The document names the structure and then says what it is for: one deadline per point, one-second resolution, O(1) reschedule. A flat array of buckets has all three at four thousand points. The difference is recorded as a decision, along with the condition under which hierarchy becomes the answer, and the document has been left as it is.

A seam review, pointed back

Introduction now requires every step to end with a seam review, and step four was the first to end under that rule. The first run of the procedure pointed forward, at documents that had not been built yet. What it found is still reserved for the Mondays after those documents publish.

The second run pointed back at this step, reading the watchdog document against its neighbours. Because the step was already built, its findings arrived with a reading already taken in code. Two of them sit between documents that are both public.

An escape hatch that exists in one document

The Semantic Model, on logarithmic quantities:

Derived points refuse to compute mean on a non-agg-safe quantity unless the definition explicitly opts into log_mean

The watchdog document owns the operator set. Its table lists thirteen operators and log_mean is not one of them. Its mean row reads "rejected when the input quantity is not agg_safe", and its exit criteria ask for exactly that refusal. The name appeared in no vocabulary and nowhere in the code.

The code refused unconditionally, which is what the owning document says. The review gave it no decision row, and said why:

That is a choice the code should not have had to make silently.

The fix went the generous way. The operator set gained log_mean, computed as 10 * log10(mean(10^(x/10))), and refused on an agg_safe quantity as the mirror of mean's refusal.

Two columns the vocabulary never had

The published watchdog document says a derived point declares its output quantity, and that a mismatch with "what the operator produces from the input quantity" is a validation failure at load. Checking that needs something that says what a rate over energy is, or a difference of two temperatures.

The amended operator table reads two fields off the quantity vocabulary for it, delta_quantity and rate_quantity. The Semantic Model owns that vocabulary and printed it with four columns. Neither field was one of them.

The registry file declared both, with a rule no document stated: leaving one out means the operator is refused. The decision row has the reason for refusing instead of inheriting:

20 °C minus 21 °C is −1 degree of interval, not −1 °C of anything

The Semantic Model has the two columns now. The amendment got one sentence wrong on the way in, about when an omission is refused, and it was corrected against the code before it was committed. An amendment written to close a finding about drift had drifted from the code itself, by one sentence, before it landed.

A third finding from the same review waits for its document.

What the step has in common

Entry three's findings were all about placement. These are about joins.

The late-replay sentence had two halves and its test held one of them. The exit criteria were met by parts that each worked, with no core running them together, and the dedup key was fine for one derived point and collided as soon as there were two. A stale point's checkpoint depended on the one write path a stale point never takes. The promise about a dead driver sat in one document while the machinery to keep it belonged to another.

Each piece was right when looked at alone, which is why the unit tests passed throughout. What caught these was running the pieces together, and for the first two exit criteria that was the test the document had asked for in the first place.

This Wednesday - The Command Plane. The specification's sixth part: where Sentinel starts writing to devices, with desired and reported held as separate values and a write followed through to a verified outcome.

At the start of November - the Dev Diary. The diary pauses for the rest of October, along with the specification. The next entry picks up where this one leaves off: a sixth check has joined the five from last time: an audit that reads every resolved deferral against the code it says was built. Plus the register checks moving out into a tool of their own, including the one that renumbers rows when two branches claim the same id.

Start of the diary: Four Registers and a Drift Check. Why the registers exist at all, and the one rule about where a sentence is allowed to live.

Sentinel: The Watchdog and Derived Points

Philip Shaw — Wed, 30 Sep 2026 06:24:29 +0000

A door sensor that only reports when the door opens or closes can go months without sending anything. If nothing has happened, that silence is correct. If the sensor has died, the silence looks identical, and no freshness window can tell the two apart. So the registry won't accept an on_change point unless it also declares a heartbeat or explicitly sets never. Whoever adds the point has to say how anyone would know it had died, and the answer is recorded in a commit.

A derived point can mislead in the same way. Computed naively, a 15-minute mean of a flow temperature keeps producing numbers after its sensor goes quiet, and a mean built from a single reading looks entirely reasonable on a dashboard. So the output carries a quality worked out from its inputs. A window with fewer than min_samples readings comes out bad, with the reason insufficient_samples. A stale input makes the output stale unless the point has opted into live_or_stale. And a derived point that stops emitting misses its own deadline and is marked stale by the same watchdog that watches the devices.

Restarts are where silence is easiest to misread. Each point's deadline is recomputed from its last observed_at when the core starts, so nothing has to be persisted. Without a startup grace, though, a fleet that reports every five minutes would go stale within seconds of the restart. A core running behind on ingest is worse: every point would pass its deadline while its data sat queued on the bus, and fail-safe interlocks would assert across the whole estate. Staleness is paused in both cases, and the lag pause is announced on its own signal so it can't hide a real fault. The slow part of a planned restart is rebuilding derived-point windows, and running those queries on a small pool, with one query shared by every derived point that reads the same input, keeps it bounded.

Staleness and derived values both reach the rules engine as ordinary transitions on ordinary points, so the engine, two parts from now, only has to understand one kind of input.

What this document owns: freshness policies and the single timer wheel that enforces them, the heartbeat requirement on on_change points, liveness as a freshness policy, how staleness fires, how it recovers and how it ranks against unavailable, startup grace, suppression during ingest lag and the three limits on it, flap damping, the derived point descriptor and its operator set, quality propagation, evaluation order and the cap on chain depth, windows and the retention guard that bounds them, the cold-start rehydration budget, emit policy, and the restricted expression grammar with its two binding contexts. It does not own the lag mechanism, its arithmetic or the scale targets, which belong to Core Runtime; the retention classes, their policy values and how each rolls up, which belong to Retention & Compaction; the raw-table query a window rehydrates from, which is defined in Ingest Path & Log Schema; or how a rule triggers on a transition, claims an output and uses the grammar in its when clause, which belong to Rules Engine.

These two land together because the watchdog's output is just another observation, and a derived point with no fresh inputs is exactly the staleness problem again one level up.

Watchdog

A single timer wheel in the core, not a timer per point. Every point with a freshness policy has one pending deadline; when an observation lands, the deadline is cancelled and rescheduled. A hierarchical wheel at 1 s resolution handles tens of thousands of points on one thread and stays O(1) on reschedule, which is the operation that runs constantly. At the four thousand points targeted in Core Runtime this is not close to a constraint.

Deadlines survive restart because they are derived, not stored: observed_at + window + grace, recomputed for every point during projection load. Nothing to persist.

Policies, from the descriptor:

periodic(interval, grace) — deadline at observed_at + interval + grace. Grace defaults to half the interval, capped at 60 s.
on_change(max_silence) — deadline at observed_at + max_silence. Requires the device to heartbeat.
ttl(duration) — hard expiry, no grace.
never — no deadline. Config values, static setpoints.

on_change without a heartbeat is the trap. A door sensor that reports only on transition is silent for months when nothing happens, and there is no window that distinguishes "quiet" from "dead". The registry rejects on_change unless the point also declares heartbeat: <interval> or explicitly sets never. Forcing that choice at registration is the whole point: the question "how would I know if this died?" gets answered once, in a commit, rather than never.

For BLE advertisement points, periodic against the advertising interval is usually more honest than on_change, since advertisements repeat regardless of change.

Liveness is a freshness policy, not a value. A point that reports "still running" every second is not a measurement; the information is in the gap, not in the sample. It is modelled as an ordinary point with a tight periodic policy, with a rule triggering on quality rather than on value. The system already treats absence as a first-class signal, so nothing extra is needed — and a rule watching for true to become false would be watching the wrong thing, since a dead process does not report false.

Firing writes a quality_changed entry to control_event and publishes a transition on state.{point_id}. Same envelope shape as a value transition, changed: false, value untouched. To the rules engine it is indistinguishable in structure from any other transition, which is the design goal.

stale and unavailable interact by precedence: an availability event from a driver sets unavailable and cancels the point's deadline. When availability returns, the point stays unavailable until real data arrives — no synthetic promotion. This stops a flapping device from generating alternating stale/unavailable noise.

Recovery is not automatic. A point goes stale → live only when an observation arrives that satisfies its policy. An observation with observed_at older than the current deadline — a buffered replay landing late — does not clear staleness, though it is still logged. This is the second place observed_at versus ingested_at earns its keep.

When staleness is suppressed

Two conditions pause staleness evaluation entirely. Both exist because the absence of data is sometimes a fact about the core rather than about the device, and reporting it as device staleness is a lie that causes the system to act.

Startup grace. For the first N seconds after core start, restored → stale transitions are suppressed, where N is the point's window plus grace. Otherwise every point in a fleet that reports every 5 minutes goes stale within seconds of a restart and floods the log. The suppression is per point, so a genuinely dead device still surfaces at the right time.

Ingest lag. While the core is behind, all staleness evaluation pauses. Freshness is computed from observed_at, so a core ten minutes behind would take every point in the system past its deadline while the data sits queued and healthy on the bus — and rules with on_input_loss: release would drop their claims, and fail-safe interlocks would assert, across the whole estate, because ingest was slow. That is close to the worst available response and it is entirely self-inflicted.

Three constraints on the lag suppression, because withholding a signal must not become a way to hide problems:

The threshold is computed, not configured. Suppression engages when lag exceeds the smallest window + grace in the registry. Past that point staleness is unreliable for at least one point, and there is no principled reason to trust it for the others.
It is global, not per point. Unlike startup grace, the condition is a property of the core, so all staleness evaluation pauses rather than pausing selectively.
The condition is loud. Lag is a first-class signal, sustained lag is alertable, and core.{n}.staleness-suppressed says plainly that this is happening. The system is not going quiet; it is replacing a false signal with a true one.

When lag clears, deadlines are recomputed from each point's observed_at and genuinely dead devices surface within one window. The lag mechanism and its arithmetic live in Core Runtime.

Flap damping. A point crossing live/stale repeatedly is marked flapping rather than emitting every transition. Transitions are counted in a rolling window; past a threshold, a flapping control event is emitted and further transitions coalesce until it settles. This is a rules-engine concern as much as a watchdog one — a rule triggering on staleness of a flapping point will fire endlessly.

Derived points

A derived point is a point. Same ID scheme, same descriptor fields, same quality states, same freshness, same bus subject. Consumers cannot tell the difference and should not be able to.

id            boiler.ch1.flow-temp-15m
kind          measurement
quantity      temperature
unit_native   Cel
subject       water
subject_ref   zone:upstairs-ch
derivation:
  op          mean
  inputs      [ boiler.ch1.flow-temp ]
  window      15m
  emit        on_input | interval(1m)
  min_samples 3
  input_quality  live_only | live_or_stale
  freshness   periodic(1m, grace=30s)

The descriptor declares quantity and unit explicitly rather than inheriting them, because some operators change both. A rate over energy yields power in W; a delta over temperature yields temperature_delta, which is a different quantity precisely because it is not affine-convertible. If the engine cannot verify that the declared output quantity matches what the operator produces from the input quantity, that is a registry validation failure at load time.

Operator set, deliberately small:

op	notes
`mean`, `min`, `max`, `sum`, `count`	`mean` rejected when the input quantity is not `agg_safe`
`median`, `p95`	for noisy analogue sources
`stddev`	a staleness-adjacent signal — a stuck sensor has zero variance
`delta`	last minus first in window; output quantity differs from input
`rate`	delta over time; output quantity differs
`debounce(duration)`	boolean stable for a duration
`hysteresis(on, off)`	boolean from a numeric input
`expression`	restricted arithmetic over named inputs; grammar below

hysteresis as a derived point is the most consequential of these. Thresholds belong here, not in rules. A rule that says "pump on when flow-temp-low is true" is readable and testable; a rule carrying its own threshold pair is neither, and the same threshold used by three rules is three chances to get it inconsistent.

Quality propagation is where most implementations of this go wrong. The rule:

No input has ever been live → output unknown
Any required input unavailable → output unavailable
Any required input bad → output bad
Window has fewer than min_samples live samples → output bad, reason insufficient_samples
Any required input stale and policy is live_only → output stale
Otherwise → live

The min_samples case matters more than it looks. A 15-minute mean computed from one sample is not a 15-minute mean, and it will look completely reasonable on a dashboard. min_samples defaults to a figure derived from window and expected input interval — around 50% of expected — rather than to 1.

input_quality: live_or_stale exists for windows long enough that a missed poll should not invalidate them. It is opt-in, and the output carries lower confidence, but it stops a 24-hour average collapsing because one reading was late.

Evaluation. Derived points form a DAG. Acyclicity is validated at registry load and cycles rejected there rather than detected at runtime. Evaluation is in topological order within a single transition batch, so a chain of three derived points settles in one pass rather than three bus round trips.

Chain depth is capped at three or four levels. Deeper than that is a rules problem wearing a derived-points costume.

Windows are queries, not buffers. As established in the ingest design, rehydration is a SELECT against the raw table for the input's retention class. The in-memory window is a cache of that query, and it can be rebuilt at any time. This is what makes derived points restart-transparent, and it is worth protecting: no derived-point state that cannot be recomputed from the log.

Windows are bounded by both duration and sample count, so a misconfigured 5 Hz sensor with a 24-hour window does not consume the core.

Windows are bounded by the retention of their inputs

A window longer than the retention behind it rehydrates from a truncated range: it produces a value computed over less time than it claims, which min_samples will usually but not always catch.

Retention differs by class, so the check is per input, against that input's class, not against a single global figure. A derived point reading a short-retention diagnostic must satisfy the diagnostic horizon even where it would comfortably pass a measurement one. That is a strictly better check than the global version and it catches the case most likely to be wrong.

Two validations, and the second is the one that will actually bite:

At registry load, reject any derived point whose window exceeds its input's class retention less one chunk interval, and warn above half of it. Zero slack is not enough, because a chunk dropping partway through a rehydration truncates the window silently. The check reads the retention policy from the database rather than a constant in the code — retention is a deployment parameter, and hardcoding a figure here means changing it later breaks derived points with no warning at all.
At retention change, refuse a shortening that would orphan an existing window. Retention gets shortened when a disk fills, which is exactly the moment nobody is thinking about derived points. This is the same class of guard as refusing to remove a vocabulary member that points still reference.

A window that cannot fit takes the rollup as its input instead. A thirty-day mean cannot rehydrate from raw samples dropped at fourteen days, but it can read the hourly aggregate, which is kept far longer and is exactly the right granularity for a window that long. The choice is explicit in the descriptor and enforced at load, rather than discovered at the next restart when the window comes back short.

Rehydration is the cold-start budget

At the scale targets in Core Runtime — a thousand or more derived points — this step is essentially all of the planned-restart time, and it is the only part of cold start that grows with configuration rather than with fleet size. Three things keep it bounded:

Bounded concurrency, not sequential. A thousand round trips executed one at a time is minutes of latency spent waiting rather than working. Run on a small pool, the wall time collapses toward the slowest few queries.
Share queries across derived points reading the same input. With a thousand derived points over two thousand inputs, several derived points will read the same source over overlapping windows. Group by input, issue one query covering the widest window in the group, and slice in memory. This is where the largest saving is, and it costs nothing at registry load to compute the grouping.
Watch the compression boundary. A window reaching past the compression age decompresses chunks to serve, which is far slower than a hot-chunk read. Windows that cross it are worth knowing about; a long window is not wrong, but it should be a deliberate choice rather than a surprise at the next restart.

Rehydration time is measured as a self point from the first restart. It is the number that degrades quietly as the rule set grows, and the one that makes an outage worse than it needs to be, since recovery is cold start plus drain.

Emit policy. on_input recomputes and publishes on every input transition, which is right for debounce and hysteresis where latency matters. interval(period) recomputes on a schedule, which is right for aggregates over slow-moving inputs where per-sample recomputation is wasted work. Aggregates generally use interval, and the interval becomes the basis for the derived point's own freshness policy.

Derived points respect their own change_policy deadband exactly as device points do, so a moving average that moves by 0.001 °C does not emit a transition.

The restricted expression grammar

Defined here and referenced by the rules engine. There is one grammar with two binding contexts, not two grammars, and it is written down before either consumer is built because an informal grammar is how eval gets in.

Legal:

arithmetic + - * / % and unary minus
comparison == != < <= > >=
boolean and or not
min, max, abs, clamp, round
numeric, boolean, quality-member and enum-member literals
names bound by the enclosing declaration, carrying exactly two accessors: .value and .quality

Rejected at load, not at runtime: function definitions, lambdas, loops, comprehensions, indexing, general attribute access, imports, string formatting, any call not in the list above, and any name not declared in the enclosing inputs.

.value and .quality are an enumerated accessor pair, not general attribute access. The distinction has to be explicit, because "no attribute access" and flow.quality == live otherwise contradict each other. The accessor set is closed, it has two members, and adding a third is a grammar change rather than a convenience.

Quality members are legal only as operands of == and !=. Ordering comparison on quality is meaningless — stale > live is not a question — and the parser rejects it rather than coercing something.

Binding contexts. In a derived expression point, names bind to input values and a bare x means x.value; .quality is available but rarely wanted, because quality propagation is handled by the rules above rather than inside the expression. In a rule's when, names bind to the full (value, quality) pair and there is no bare-name sugar: a rule that reads a value without considering its quality is usually a bug, and it should have to say so in writing.

Both contexts parse to an AST at registry load, using one parser. Two parsers is how the contexts diverge.

The property that keeps the rules engine small

Derived points feed derived points, and staleness of a derived point is computed by the same watchdog with the same policies. A derived point whose inputs went quiet stops emitting, misses its own deadline, and goes stale — and a rule can trigger on that without knowing or caring that the point was derived.

That uniformity is what lets the rules engine stay small. Rising edge, falling edge, moving average, staleness, fresh data and bad data are each either a transition predicate or a derived point, and the engine only needs to understand transitions over points.

Exit criteria

No spurious edges at cold start. Load a checkpointed projection, start the core, and assert zero restored → stale transitions inside each point's grace, and the correct ones after it. This is the whole reason startup grace exists, and it is easy to get subtly wrong in a way that only shows up as a burst of log entries nobody reads.

Staleness suppression engages and clears. Simulate lag past the computed threshold and assert that no staleness transitions are emitted, that core.{n}.staleness-suppressed is true, and that when lag clears the deadlines are recomputed from each point's observed_at so a genuinely dead device surfaces within one window. The failure mode to catch is suppression that never lifts.

Quality propagation, one fixture per row of the table. Including the two that produce plausible wrong answers rather than errors: a window below min_samples yields bad with insufficient_samples rather than a mean of two readings, and a non-agg_safe input rejects mean at load rather than computing one.

The window guard fails the build. A derived point whose window exceeds its input's class retention is rejected at registry load, and separately, a retention shortening that would orphan an existing window is refused. Both directions, because the second is the one that happens under pressure.

Rehydration is measured, and the grouping demonstrably works. Record the wall time, and assert that the number of queries issued is materially below the number of derived points — if it equals it, the shared-query grouping is not doing anything and cold start will degrade linearly as the rule set grows.

On Monday - the Dev Diary. The latest on where the specification and the build disagree as the system takes shape, and which of the two had to change.

The next part - The Command Plane. Where Sentinel starts writing to devices: desired and reported held as separate values, a durable claim set with banded arbitration, and a state machine that follows a fire-and-forget write through to a verified outcome.

Sentinel Dev Diary: Checks & Balances

Philip Shaw — Mon, 28 Sep 2026 14:03:16 +0000

This entry is about the checks and balances. Nine build steps into Sentinel, five processes have grown up around the specification, and each one catches a different way the code and the documents describing it come apart. Most of the work is done by AI coding agents, but nothing in these processes depends on that. They would work the same way on a team of engineers.

They all start from one assumption: things will diverge. A specification written before the code will be wrong in places. The code will move after someone has described it. A note that was true on Tuesday will be stale by Thursday. On a project that runs for months you cannot stop that, and trying to is how a team ends up trusting documents that stopped being true a while ago. The realistic goal is to find divergence soon after it happens and to correct it somewhere the next person, or the next agent, will see it.

The rest of this entry lays out the five: what each one watches, what keeps it honest, and where it stops. First, an example of the kind of divergence they exist to catch, found eleven days ago by the newest of them.

A setting nothing read

The part of the specification that went out last Wednesday, Ingest Path and Log Schema, says this about writing to the database:

Inserts are batched - multi-row INSERT flushed on 500 rows or 100 ms, whichever comes first.

The code had both numbers. An accumulator took batch_max_rows and batch_max_delay_ms from configuration, 500 and 100, and offered a method, due(now), that answered exactly the question the sentence asks: is this batch full, or old enough, to flush?

Nothing in the daemon ever called it.

The ingest loop passed now=0.0 into every row it added, so the batch could never age, and then took the whole batch unconditionally. Meanwhile the two settings travelled faithfully through the system: from the settings class into the dataclass, into .env.example, into the deploy workflow, into the GitHub Environment the deploy reads from. Every layer carried them, and none of them did anything. The register row that eventually recorded it put it better than I can:

The two fields travelled from settings.py into the dataclass and affected nothing, which is worse than absent: a knob that appears to work.

The part I did not enjoy: due did have a caller, just not in the application. The throughput benchmark had built its own loop on top of it.

Entry one held up a row as the register earning its keep: CP-1, ingest throughput, 4,369 observations a second, with a field reading Do not pre-decide: that the margin holds. I was quite pleased with that row. The benchmark behind it was measuring a batching strategy this system has never run.

It was honest by coincidence. The unused knob said 500 rows, and the setting that actually bounds a batch, how many messages one pull from the bus may return, also says 500. When the benchmark was repointed at the real one the figure did not move, so the reading stands. But nobody knew that until somebody looked, and nothing was looking.

What found it

The tests missed it. test_batch.py passed throughout, because it tested the accumulator's flush policy directly; in the words of the row that closed it, it

asserted a flush policy nothing consulted, which is how a suite comes to certify behaviour the system does not have.

The audits missed it, because no exit criterion asks whether a configuration field is read. So did the drift check, which reads the code against the specification, and would have found a class implementing the sentence perfectly well.

It turned up when I sat down to write, from the code, a description of what the ingest path does. There is no way to write "the batch flushes when due returns true" with a citation next to it and not go and look for the caller.

That was on the day the fifth process, the development guide, was created. Writing the guide found five things on its first pass, and this was one of the gentler ones.

I have been describing these processes one at a time for three entries, and from here on I call them instruments. Here are all five together.

Five instruments, five tenses

Instrument	Tense	Its normative home	What keeps it honest
the specification	what the system will be	itself	nothing internal; everything else comes at it from outside
the four registers	what is open now	the specification	`test_register_integrity.py`, for the registers' shape only
the audits	what happened	the build step	a person, once per step, looking back
the seam reviews	what nobody said	the joins between documents	a rule in Introduction that every step ends with one
the development guide	what exists today	the code	`test_dev_docs.py`, and a marker on every claim

The specification is where it starts, and it is the only one of the five that defines anything. It is also the only one with nothing inside it checking it. A document cannot audit itself; the best it can do is be written so that the others can.

The registers number what the specification says, so that it can be cited, and hold what the build keeps producing that the specification has no home for. Entry one was about why they exist. They are under test, and entry three drew the edge of that: a register can be machine-checked for everything that is a statement about itself, and nothing that is a statement about the world outside.

The audits are the retrospective. One per build step, in two halves: what changed during implementation, and what the step did not deliver. They are kept honest by a person, and only as far as the exit criteria reach. Entry two was the first of them finding silence; entry three was one of them failing to mention a fork in the road, because there was no exit criterion for it to fail.

The seam reviews are the most recent thing the specification was amended to require, and the oldest complaint. Entry one ended on an open question, Nothing reads the documents against each other, and three weeks ago it was answered by making it a step's own work: Introduction now says that "every step ends with a review whose unit is the seam rather than the document". It is the only instrument whose unit is a pair of documents. The one-owning-document rule optimises for consistency inside a document and provides nothing at the joins, and every cross-document gap found before it existed sat between two documents that were each perfectly consistent on their own.

It has a joke in its history that I am still enjoying. The paragraph describing the drift check went on calling the seam review "standing, and currently unowned" for two days after the amendment that gave it an owner. In the document that defines the drift check. The repository's own note on it at the time:

a pointer is only as current as the last person to follow it.

What the first run actually found waits for the documents it found things in; that one is coming.

The development guide is the fifth, and it is the one that runs the other way round.

The one that points at the code

Everything else in that table is about the specification. The registers point at it. The audits measure the code against it. The seam reviews read it against itself.

Nothing said how the thing that exists behaves. Nine build steps in, the daemon was about 36,000 lines across two repositories, and every piece of work, in every session, was paying to reconstruct it. Since nearly all of that work is done by coding agents, "every piece of work" really does mean every one: an agent opening a session has read the repository and none of my recollection, so it rebuilt its understanding of the ingest loop from scratch each time and, presumably, slightly differently each time.

(It did not help that the project's own CLAUDE.md, the first file an agent reads, described two application directories as populated. Both hold a .gitkeep. They always have. They are the layering from a template for a different kind of app, and this process is a daemon. Writing the guide found that too.)

So docs/dev/ was written as fifteen chapters, around 3,300 lines, and it defines nothing, exactly as the registers define nothing. The difference is only what it points at. A register row points at the specification and is cited by id. A chapter points at the code and cites path.py::symbol. The constraint that makes that legitimate rather than a second definition was already in the specification:

Prose in these notes explains them; it does not define them once the code exists.

And where the code and the specification disagree, the guide describes the code and links the register row recording the disagreement. Where there is no row, the disagreement is a finding and gets routed like any other, instead of being written around.

Proved by, or unverified

This is the part I would steal for any project.

Every mechanism section in the guide ends in one of two ways. Either with Proved by: and the test that would fail if the claim stopped being true, or with the bare word unverified. There is no third option, and in particular there is no option for explaining why checking was not necessary. The reasoning, from the guide's own introduction:

a marker reading "not checked" invites the check; one reading "trivially true" ends it.

Two days in, sixty-five claims carried a Proved by:. Three were marked unverified.

And every chapter names the commit it was derived from,

because four pull requests means four different truths about main.

A chapter is as current as that line says, and no more.

Then there is a test, test_dev_docs.py, which asserts that every cited path exists and defines the symbol it names, that a list marked as an enum's members holds exactly that enum's members, and that a table marked as a mapping parses to exactly that mapping.

That makes it the fifth place in the repository where a document is coupled to the code by a test, and the first that runs this way round. The other four assert that the specification still describes the code. This one asserts that the documentation still describes the code, which sounds like a small distinction until you notice that nothing checked that before.

It rotted in two days

Of course it did.

Two days after the guide was written, it was audited against the code, the way the audits check a build step. Eleven commits had touched it in the meantime, and not with the same care.

Eleven chapters claimed a currency they did not have. Every edited chapter still carried its original derived-from commit, in the layer whose own README makes that line the definition of how current a chapter is. Three described code that had changed under them. One citation credited a function with publishing messages when all it does is format a subject string; the claim underneath was true, the citation resolved, so the test passed. A citation hung on a symbol that made the sentence look checked.

That last one is the docs test's edge, in exactly the way entry three found the register test's. It can prove the symbol exists. It cannot prove the symbol is the one doing the thing.

My favourite goes straight back to the batching. The ingest chapter had to say what really decides a batch, now that the knob was gone: one pull from the bus is one batch is one transaction. It cited test_ingest_pipeline.py for that. Nothing in test_ingest_pipeline.py asserts it. So the audit demoted it, and the section now ends:

For the cadence itself - that run_once takes once per fetch, unconditionally - unverified.

followed by the test that would close it.

I like that more than any Proved by: in the guide. It is the marker working in reverse: a claim that had a check pinned to it losing the check, and saying so, where the reader will trip over it. The specification described a batching rule that never ran. The guide describes the one that does, and is honest that nothing yet proves it. That is the right way round for those two documents to be wrong.

Different ways to rot

Every one of the five goes stale in its own way, including the checks themselves. The tenses are the useful way to hold all this, because each one decays differently, and so each needs something different watching it.

The specification rots by being overtaken. It is amended as the build discharges its open items, which is why the copies published here are a frozen snapshot, and why the gap between the snapshot and the current text is what this diary is for.
The registers rot by saying things about the world they cannot see, like entry three's sentence placing a test somewhere it had never been, which stood for three pull requests.
The audits do not rot so much as squint. They are history, but they only see as far as the exit criteria reach.
The seam reviews rot like any pointer: the rule was two days old before the paragraph describing it caught up.
The guide rots by the code moving underneath it, and the derived-from commit is the only reason anyone can tell.

None of them can do another's job. The register tests cannot look outside the registers. The audits cannot see a fork that no criterion asks about. The drift check reads the code against the prose, but only from one side. The seam review reads document against document and never touches code. The guide reads code into prose, and the one thing it cannot do is say what the code should do; that stays with the specification, which is where this started.

Every one of the five was added because the one before it had an edge, and something had just fallen off it. I had assumed the newest would spend its first week finding small things in the newest code. Instead, on its first day, it found a hole under entry one's proudest number.

That is the pattern I would take to any long-running project, with or without agents. Assume the documents and the code will drift. Give each kind of drift something that looks for it, and when one of those checks finds its own edge, add the next one.

This Wednesday - The Watchdog and Derived Points. The specification's fifth part: one timer wheel for every freshness deadline, and what a derived point is allowed to say when its inputs go stale.

The next entry - Auditing the Watchdog. What the build sent back to the specification from step four: a sentence with two halves and a test asserting the one that worked, and a stale point whose checkpoint never moved.

Start of the diary: Four Registers and a Drift Check. Why the registers exist at all, and the one rule about where a sentence is allowed to live.

Sentinel: Ingest Path and Log Schema

Philip Shaw — Wed, 23 Sep 2026 10:33:04 +0000

Last week's sensor is still reporting 71.6 in Fahrenheit, and its driver has published that number untouched, as the contract requires. The conversion to Celsius happens here instead, at the eighth of thirteen stages between the bus and the database, and the row it writes keeps both numbers: 22.0 in the canonical column, and 71.6 in the native one beside the unit the device actually used.

Keeping both is what makes a conversion reversible. Suppose that weeks later the descriptor's scale factor turns out to have been wrong. Nothing collected in the meantime is lost, because the number the device produced was never overwritten; a reprocess recomputes every canonical value from native under a corrected registry version, rebuilds the current state of the affected points, and refreshes the aggregates built over them. It can only reach rows that still exist, and it refuses outright rather than correcting the part of a request it can reach. What it will not do is re-run the rules that fired on the wrong numbers, because those decisions were made on the data as it stood.

The other load-bearing decision is smaller and much easier to get wrong: where ingested_at comes from. Timescale requires the partitioning column in every unique index, so the dedup key cannot be just the three columns that identify a message; it has to carry the timestamp as well. Stamp that from the core's clock at processing time, and a message redelivered after a crash arrives with a new timestamp, slips past the index and inserts a duplicate, while the index goes on looking as though it enforces dedup. Take it from the bus instead, where it is fixed once and identical on every redelivery, and the same index turns at-least-once delivery into exactly-once effects without a watermark table of its own.

Those two decisions are the spine of this document. The two logs, the per-class hypertables, the thirteen stages and their ordering rules, the restart sequence and the reprocess command are what the rest of the schema has to look like once both are held fixed.

What this document owns: the split into an observation log and a control-event log, the observation schema and its physical split into one hypertable per retention class, where ingested_at comes from and why, the versioned and content-addressed registry, the thirteen-stage ingest pipeline and its three ordering rules, the in-memory projection and its checkpoint, the restart sequence, reprocess and its refusal to apply partially, and the properties of the continuous aggregates that belong with the schema rather than the policy. What it deliberately does not own is how long anything is kept, how each class rolls up and how state-class compaction works, which belong to Retention & Compaction; keeping cold-start rehydration bounded, which belongs to Watchdog & Derived Points; the core epoch, restarting against a backlog and the utilisation arithmetic the ingest ceiling feeds into, which belong to Core Runtime; and who a principal is and why a wrong_instance count is an alert rather than a dashboard figure, which belong to Security Model. Everything downstream that asks what a point's value is, or was, is reading a projection of the two logs defined here.

Two things drive the whole design here: the bus message's own identity doubles as the dedup key and the ingest watermark, and value_native in the log is what makes conversion reversible. Get those right and restart and reprocess both fall out.

Two logs, not one

The pure event-sourcing instinct is one ordered log. It is the wrong shape here. Observations and control events have different volumes, retention, and query shapes, and Timescale hypertables want homogeneous rows.

observation — high volume, append-only, compressed, aged out. Ordering matters only within a point. Physically several hypertables, one per retention class; see below.
control_event — commands, availability, quality transitions, assignment changes, reprocess operations. Low volume, globally sequenced, kept forever, not classed.

The projection is built from both. Quality transitions from the freshness watchdog go in control_event because they carry no new value; in a healthy system they are rare, and when they are not rare that is the thing most worth a permanent record.

Schema

CREATE TYPE quality AS ENUM (
  'unknown','restored','live','stale','unavailable','bad','assumed');

-- identical definition for each retention class:
--   observation_measurement, observation_state, observation_diagnostic
CREATE TABLE observation_<class> (
  ingested_at       timestamptz  NOT NULL,   -- bus-accept time; see below
  point_id          text         NOT NULL,
  observed_at       timestamptz  NOT NULL,
  device_time       timestamptz,

  value_num         double precision,   -- canonical
  value_bool        boolean,
  value_text        text,               -- canonical enum member / string
  native_num        double precision,
  native_text       text,               -- raw device representation

  unit_native       text,
  quality           quality      NOT NULL,
  reason            text,
  detail            jsonb,

  changed           boolean      NOT NULL,
  registry_version  integer      NOT NULL,
  driver_instance   text         NOT NULL,
  source_epoch      integer      NOT NULL,
  source_seq        bigint       NOT NULL,
  skew_flagged      boolean      NOT NULL DEFAULT false
);

SELECT create_hypertable('observation_<class>','ingested_at',
                         chunk_time_interval => INTERVAL '1 day');

CREATE UNIQUE INDEX ON observation_<class>
  (driver_instance, source_epoch, source_seq, ingested_at);
CREATE INDEX ON observation_<class> (point_id, observed_at DESC);

Typed columns rather than JSONB for values. JSONB costs space on the highest-volume table and blocks index-only scans; five nullable narrow columns cost less, with only one or two populated per row.

One hypertable per retention class. Timescale attaches retention, compression and continuous-aggregate policies to a table rather than to a row, so per-class policy means a physical split. The class comes from the resolved descriptor, which ingest already has by the time it writes. The policies themselves are in Retention & Compaction; what belongs here is that the split exists, that the schema is identical across classes, and that it cannot be deferred — adding the class field later is easy, splitting a populated table is a rewrite of everything ever collected.

An observation view over the three, as UNION ALL, exists for ad-hoc querying. The hot paths do not use it: rehydration and reprocess resolve the class from the descriptor and query the specific table.

Chunk on ingested_at, not observed_at. A driver host replaying a buffered partition writes rows with old observed_at values, which would land in chunks that are already compressed. ingested_at is monotonic by construction, so chunks close cleanly and never reopen. The cost is that analytical queries filtering on observed_at lose chunk exclusion, which the secondary index mostly covers at these volumes.

Every observation writes a row, including one that changed nothing. Suppressing unchanged rows at write time is a tempting optimisation and it breaks two mechanisms: derived-point windows rehydrate by querying these tables, so mean, count and min_samples need the samples that were suppressed; and reprocess can only reconvert rows that exist. Volume is managed afterwards, by retention and compaction, where the trade is explicit and bounded.

Where `ingested_at` comes from

It is the JetStream message timestamp, not now() at processing time. This is load-bearing and easy to get wrong.

Timescale requires the partitioning column in every unique index, so the dedup index is necessarily (driver_instance, source_epoch, source_seq, ingested_at) rather than the three-column key that would otherwise be correct. If ingested_at were assigned when the core happened to process the message, a redelivery after a crash would carry a different value, land outside the index, and insert a duplicate row. The index would look like it was enforcing dedup and would not be.

The bus timestamp is fixed when the message enters the stream and is identical on every redelivery, which makes the four-column index behave exactly like the three-column one. It also keeps the monotonicity that chunking depends on, because a single obs stream assigns timestamps in stream order. If observations ever arrive from more than one stream, that property weakens and this decision needs revisiting.

The cost is a change of meaning: ingested_at is when the bus accepted the observation, not when the core recorded it. That is the more useful of the two anyway — it is the moment the observation became durable, it comes from one clock rather than from whichever core process handled it, and it is stable across a core restart. Core processing lag belongs on a self point, not on a column of the highest-volume table in the system.

CREATE TABLE control_event (
  seq          bigserial PRIMARY KEY,
  occurred_at  timestamptz NOT NULL,
  kind         text        NOT NULL,   -- quality_changed, command_*, availability, assignment, reprocess
  point_id     text,
  subject_key  text,                   -- driver_instance / device / component
  principal    text        NOT NULL,   -- who caused it; see Security Model
  core_epoch   integer     NOT NULL,   -- which core instance; see Core Runtime
  payload      jsonb       NOT NULL,
  registry_version integer  NOT NULL
);

principal and core_epoch are both NOT NULL deliberately. A code path that cannot name who asked, or which core acted, cannot write a control event — which means it cannot change state.

The registry is versioned data

Vocabularies and point descriptors load from git into Postgres. Every registry load creates a version row, and descriptors are stored per version by content hash:

CREATE TABLE registry_version (
  version    integer PRIMARY KEY,
  loaded_at  timestamptz NOT NULL,
  git_sha    text        NOT NULL,
  content_hash text      NOT NULL
);

CREATE TABLE descriptor_blob (
  hash       text PRIMARY KEY,          -- of the canonicalised descriptor
  descriptor jsonb NOT NULL
);

CREATE TABLE point_descriptor (
  version   integer NOT NULL REFERENCES registry_version,
  point_id  text    NOT NULL,
  hash      text    NOT NULL REFERENCES descriptor_blob,
  PRIMARY KEY (version, point_id)
);

The indirection exists because of scale. At a few hundred points a full descriptor copy per version is a rounding error; at the four thousand points targeted in Core Runtime it is around 1.6 MB per commit, growing without bound, when a typical commit changes a handful of points. Content-addressing collapses that to one small row per point per version plus one blob per genuinely new descriptor.

It preserves the property that matters: any historical observation can be reinterpreted under the exact descriptor that produced it, or under any other version, because every version still names a complete descriptor set.

Ingest pipeline

Stages, in order, all core-side:

Consume from the JetStream durable consumer on obs.>, taking ingested_at from the message
Validate envelope — required fields, parseable timestamps, observed_at not absurdly in the future
Dedup on (driver_instance, source_epoch, source_seq) against an in-memory set of recently seen keys; this is a fast path, and the unique index is the authority
Resolve point — registry lookup; miss goes to quarantine and the message is acked
Check publisher — the host and instance assigned to this point must match the ones the message was published from; a mismatch goes to quarantine with reason wrong_instance and the message is acked
Unit check — driver-supplied unit_native disagreeing with the descriptor gives quality bad, reason unit_mismatch, no conversion attempted
Validate value — range check, enum codebook lookup; failures give bad with the offending value preserved in native_text
Convert — value_native × factor + offset, affine-aware
Skew check — device_time against observed_at, sets skew_flagged
Evaluate change — deadband against the in-memory projection, sets changed
Append to the hypertable for the descriptor's retention class, batched, as INSERT ... ON CONFLICT DO NOTHING ... RETURNING
Commit, then ack the JetStream messages
Update projection and publish transitions to state.{point_id}, for returned rows only

Stage 5 is what makes bus permissions mean anything. Subject scoping confines a host to publishing under its own prefix, but says nothing about which point IDs it puts inside those subjects. Without this check, one compromised driver host could publish readings for every point in the system, including the inputs to thermal interlocks, entirely within its permissions. It is one comparison against the assignment mapping the registry already holds.

The result is a security signal rather than a data-quality one. A unit_mismatch is a configuration error; a wrong_instance is a host claiming points that were never assigned to it, and the count belongs in the alerting path rather than on a dashboard. See Security Model.

Stage 11 batches per class, since a batch has to target one table. That is a grouping in the flush, not three pipelines — a batch is partitioned by class immediately before the insert and committed in one transaction.

Stages 2 to 10 are the throughput ceiling, not the insert. Per-observation validation, registry lookup, codebook resolution and conversion all happen once per row in application code, and that is where sustainable ingest rate is actually determined. It is the unmeasured term in the utilisation arithmetic in Core Runtime, and it is worth benchmarking against the design rate before sizing anything that depends on it.

Three ordering rules that are easy to get wrong and expensive to debug:

Ack after commit. The consumer's ack position is the ingest watermark. Crash between commit and ack, and the redelivered messages carry the same (instance, epoch, seq, ingested_at) and are swallowed by the unique index. Crash before commit, and they are redelivered and processed properly. This produces exactly-once effects from at-least-once delivery without a separate watermark table — but only because ingested_at is deterministic. If that ever changes, the whole property goes with it.

Publish after commit. Rules must never fire on data that could roll back. A relay that clicks because of an aborted transaction is a bad afternoon.

Project only what actually inserted. ON CONFLICT DO NOTHING makes a redelivered batch a no-op in the table, but the projection update and the transition publish are not idempotent in the way that matters: re-publishing a transition re-fires the rules engine on an edge that already happened. Stage 13 is driven from the RETURNING set rather than from the batch submitted. This is the redelivery case doing its job silently, and it is invisible in testing unless a core is crashed mid-batch on purpose.

Inserts are batched — multi-row INSERT flushed on 500 rows or 100 ms, whichever comes first. Per-row inserts will not keep up at this scale, and the batch window is invisible next to the polling intervals.

Failure at stages 6–8 still writes a row. A bad observation is data — it refreshes freshness, it feeds bad-data rule triggers, and it is the evidence when working out why a sensor went strange at 3am. Silence is the one outcome that helps nobody.

Failures at stages 4 and 5 do not write a row, because there is no descriptor to write one against, and therefore no class to route to. Both go to quarantine with the full native payload.

Projection

The authoritative copy is in memory in the core. Postgres holds a checkpoint:

CREATE TABLE point_state (
  point_id        text PRIMARY KEY,
  value_num       double precision,
  value_bool      boolean,
  value_text      text,
  quality         quality     NOT NULL,
  reason          text,
  observed_at     timestamptz,
  ingested_at     timestamptz,
  changed_at      timestamptz,          -- last value change past deadband
  quality_since   timestamptz NOT NULL,
  watermark_seq   bigint                -- last control_event applied
);

Write policy: immediately on quality change, at most every 5 s per point on value change, plus a full flush every 30 s. Per-observation UPDATE on a hot table generates MVCC bloat for no benefit — the value churn does not need durability because the log already has it. Quality changes are rare and operationally important, so those go through synchronously.

The value in point_state is never nulled when quality degrades. Consumers get (value, quality, observed_at) and decide.

Restart

Load vocabularies and current registry version
Load point_state, force every quality to restored (except rows already unknown)
Resume the JetStream consumer from its ack position; replay anything the checkpoint missed
Replay control_event above each point's watermark_seq
Rehydrate derived-point windows by querying the appropriate class table for the required lookback
Rehydrate durable timers from control_event
Start drivers; instances go connecting → connected → ready
Rules leave warmup as their inputs reach live, or when their warmup deadline expires

Step 3 can be blunt precisely because of the dedup key: anything already committed is re-consumed and discarded, so the consumer resumes from a conservative position without the core having to reason about how far it got.

Step 5 is the quiet payoff of log-first. A 15-minute moving average needs no separately persisted ring buffer — it is a query. Derived points become stateless to restart, which removes an entire category of subtle post-restart wrongness. It is also the step that dominates cold start, and Watchdog & Derived Points covers how to keep it bounded.

Step 2's blanket restored is deliberately pessimistic. Something that was bad before the restart might be fine now, and the driver will say so within one poll interval.

Where the restart follows an outage long enough to have built a backlog, steps 3 onward run against a draining queue rather than a current one. What that changes is in Core Runtime.

Reprocess

value_native is never rewritten. That is what makes this possible:

reprocess --point-glob 'boiler.*' --from 2026-06-01 --registry-version current

Recompute canonical values from native under a chosen registry version, update the affected rows, rebuild the projection for those points, refresh every dependent continuous aggregate over the affected range, and write a reprocess entry to control_event recording the scope, the old version, the new one, and the aggregate ranges refreshed.

ingested_at is never rewritten either. Reprocess changes what a row means, not when it arrived, and the dedup key has to keep pointing at the same message.

Refreshing the aggregates is part of the operation, not a side effect of it. Timescale records an invalidation when the underlying rows change, but the background refresh policy only revisits its own recent window — typically the last day or two. A correction to June's data leaves an invalidation nobody will ever process, and the aggregates keep serving the old numbers indefinitely while the raw table is right. That divergence is worse than the original error, because the raw table and the dashboard now disagree and nothing says so.

So the reprocess command calls refresh_continuous_aggregate explicitly for each aggregate over the affected range, after the row updates commit. Two details:

Round the range outward to bucket boundaries. A range that starts mid-bucket produces a partially recomputed bucket, which is a worse artefact than either the old or the new value.
The refresh is point-blind. Aggregates bucket by time, not by point glob, so refreshing recomputes every point in those buckets, not just the ones reprocessed. That is more work than the reprocess itself and it is the right trade: filtering by point would mean maintaining a parallel invalidation map, and the whole operation is rare.

The horizon is per class, and partial reprocess is refused

Reprocess can only reach rows that still exist, and raw retention differs by retention class. A point glob can therefore span classes with different horizons, and a range that is valid for a measurement point may be long gone for a diagnostic one.

The command validates the whole scope first and refuses outright if any point in it cannot be covered, reporting which points and which classes made it impossible. It does not correct the part it can reach.

That refusal is the important behaviour. Silently correcting two thirds of a glob leaves the system in a state where some points are right, some are wrong, and nothing distinguishes them — which is precisely the divide-and-be-wrong failure the operation exists to prevent. Narrowing the glob or the range is then a deliberate second command, taken with the horizon in view.

The practical consequence: a unit error found within a class's raw retention is fully correctable, and one found after that is permanent in the aggregates. If a horizon is wrong, the retention policy is the thing to change, not this.

Compressed chunks need decompress-update-recompress, which is slow, but this is a maintenance operation run rarely. It is built as a real command with a dry-run mode from the start, while the surface is small. Dry run reports the row count, the classes in scope with their horizons, the aggregate ranges it would refresh, and a time estimate, because the refresh is usually the expensive half.

Reprocess does not re-run rules. Historical automation decisions were made on the data as it stood, and rewriting them would make the control event log a lie.

Retention and rollups

The continuous aggregates are defined here as schema; how long anything is kept, which functions each class rolls up with, and how state-class compaction works are all in Retention & Compaction.

Three properties of the aggregate definitions belong with the schema rather than the policy:

Store sum and count, not avg. An average is a view expression over the two. The moment anything rolls an hourly bucket up to a day, or a minutely bucket up to an hour, an average of averages is wrong wherever the bucket counts differ — and it is wrong quietly, by a few percent, in exactly the direction nobody checks. Sum and count compose; averages do not.

Aggregates are defined directly on the raw class tables, never hierarchically. Building the hourly on the minutely would be cheaper to maintain and would introduce a refresh ordering dependency that reprocess has to get right every time. Defining both on raw costs some materialisation work and removes the ordering question entirely.

agg_safe is enforced in the aggregate definition, not only in derived points. An hourly mean of RSSI in a continuous aggregate is exactly as wrong as one computed live, and much harder to notice once it is sitting in a dashboard.

Only rows where quality = 'live' are aggregated, and live_count is carried alongside count so a gappy hour is visibly gappy rather than silently averaged over three samples.

Deleting data by age happens in Postgres, under the policies in Retention & Compaction. The observation stream on the bus is bounded by size instead, for the reasons in Driver Contract.

Exit criteria

This is the step whose claims are most expensive to discover are false, and most of them are otherwise only prose. Prose cannot fail loudly. Each of the following is a test that runs in CI with a fault injector, not a procedure performed by hand once.

Exactly-once effects from at-least-once delivery. Kill the core mid-batch, restart, and assert two things: no duplicate rows, and no transition re-published for a row that was already committed. The second half is the one that gets missed, because the table looks correct either way — it is the rules engine that would have fired twice.

Both crash windows, separately. Crash before commit: the messages are redelivered and processed normally. Crash after commit but before ack: the redelivered messages are swallowed by the unique index and produce no projection update. If the second case ever starts inserting duplicates, ingested_at has stopped being deterministic and the whole property has gone with it.

Publisher check. An observation for a point assigned to a different host or instance is quarantined with reason wrong_instance, no row is written, and the message is acked.

Class routing. One point per retention class, each landing in its own hypertable; a count over the UNION ALL view equals the sum of the three. A point whose class changes is not silently split across two tables.

Reprocess refuses rather than partially applies. A dry run over a glob spanning classes, with one point out of horizon, refuses and names the offending points and classes. Assert that no rows were changed and no aggregate was refreshed.

Reinterpretation under an older registry version reproduces the original canonical value from native, which is the property that makes the whole log-first argument true.

Two measurements to record rather than assert, because their thresholds are open:

Sustainable ingest throughput against the design rate, measured with synthetic load through stages 2 to 10 rather than against the insert alone.
Cold-start duration, broken down by startup step.

On Monday - the Dev Diary. Not one finding this time but the whole apparatus: the five instruments that have grown up around the specification, and the different thing that keeps each of them honest.

The next part - The Watchdog and Derived Points. One timer wheel for every freshness deadline, why a sensor that only reports on change cannot be told apart from a dead one unless it is made to heartbeat, and how the restart step that dominates cold start is kept bounded.

Sentinel Dev Diary: Where the Check Lives

Philip Shaw — Mon, 21 Sep 2026 08:10:58 +0000

Build step three is the driver contract: a shared conformance harness, one push module and one poll module, and the host process that supervises them. It was audited a fortnight ago against the documents that own it, and the first of those, The Driver Contract, went out last Wednesday.

Entry two was about silence. A specification that described things carefully, at length, and then never said whether they were required.

This one is the opposite failure, and I did not see it coming. The document is not silent. It says the thing. It says the thing twice, in two places, and the two do not agree.

Everything below is that same shape. Not a false statement about the system; a true statement in the wrong place, or two true statements that cannot both be the contract.

The contract said both, and the code believed the one with a code block

From Push and poll are the same thing:

The driver module implements one method: emit observations. Whether that is an MQTT callback or a poll loop is internal.

From Conformance, eighty lines further down, where the interface is printed:

read(points) → observations

Those are two different contracts. The first says a module emits, on its own initiative, whenever it has something. The second says a host asks and a module answers.

The driver host built the second. I would build the second again from the same document, and the reason is not subtle: it is the one with a code block beside it. A signature beats a sentence, every time, and it beats it silently.

So a push module latches the latest frame per point, and the host samples that latch on the assignment's interval; thirty seconds where a point declares nothing.

For a level reading that is fine. A temperature that arrives at an awkward moment gets read at the next tick, and it is still the temperature.

For a transient it is not fine at all. A button press. An alarm that raises and clears itself. A notification with no status counterpart. Those complete inside one interval and are never published. Not delayed; never published.

And the document has a sentence about exactly that, which it can no longer support. Buffering says of the bounded disk queue that

this is the single place in the whole path where data is dropped by design.

The latch is a second place. It has no drop count, and it has no self point. So the problem is not that a sentence went out of date; it is that the guarantee the sentence makes, that data is never lost quietly, is defeated by the other reading of the same document.

The audit did not mention it

This is the part that stings.

build-step-3.md is the audit of the step that built that interface. It runs to twenty thousand characters, it is unsparing elsewhere, and it does not mention the choice at all.

Not because anything was hidden. Because an audit asks whether the exit criteria pass, and "which of the document's two readings did you implement" was never an exit criterion. There was nothing there to fail.

Which means the pull shape has been load-bearing since step three with nothing behind it. No decision row. No open item. No line anywhere saying a fork had been reached and a side taken.

It is three rows now, because it is three different kinds of statement: the gap in the document, the reading the code took while it stood, and the build that closes it. Splitting them is not bureaucracy. The gap is discharged by the document saying which reading is the contract and by nothing else; the reading is history the moment the build lands; the build is work somebody has to do. One row would have to be wrong about two of those.

The step went amber for a day

The amendment landed over the weekend, and it brought four new exit criteria with it. None of them passed.

So step three, complete since a fortnight ago, went back to amber.

Nothing regressed. No code changed. Every test that passed before the amendment passed after it. The document simply started asking for more than had been built, and a step is finished when its owning document's exit criteria pass.

It was green again a day later, with the delivery path built and both hosts in the field running it.

I want to be precise about why that yo-yo is the machine working rather than the machine embarrassing itself, because a status marker that moves backwards looks like a mistake and this one is the opposite.

The marker follows the definition. The alternative is a marker that follows the work, and that quietly changes what the marker means: it stops recording whether the thing the document asks for exists, and starts recording how much effort has been spent. Those come apart precisely when a document is amended, which is the moment you most need the marker to be honest.

A step going amber because its document grew is not a regression. It is the only truthful thing the marker can do.

Three checks, and one place the third can go

Same shape, smaller blast radius, and it is sitting in the post that went out last Wednesday where anyone can check it. Conformance names

Three additions to the harness that come from later documents and belong in the same suite

and lists epoch rejection, publish scope, buffer bounds.

Only publish scope is in the harness. The other two are the host's, and the suite says so as a matter of design:

The suite is deliberately silent on anything the host owns. Backoff, buffering, epoch fencing, self-points, sequence numbers and subjects never reach a module, so a module cannot get them wrong and the harness does not ask.

That is the right call, and the argument for it comes from the document itself, in the section immediately before. What drivers must never do forbids a module to hold verification state or timers. A module that tracked the highest epoch would be holding a fencing token; a module that owned the durable queue would be holding buffering state. The queue is one per host, behind every instance, so there is not even a coherent thing a single module could be asked to demonstrate.

A check cannot be placed in a suite that runs against an interface which never sees the thing being checked. The code is right. The document put three checks in a place that can hold one of them.

And that is a gap in the document rather than in the code, which decides what happens next: shipping the code that routed around it does not close it. So it was filed against The Driver Contract, with the reading recorded separately so nobody would have to re-derive it meanwhile, and the document was amended a week later to say whose each check is. One of the three is the harness's; two are the host's.

The order that repair happened in is the part worth keeping. The requirements register carried a paraphrase of the original sentence, and the paraphrase was wrong in an extra way of its own: it said the check happens in the harness "as well as core-side", and buffer bounds has no core half at all. It was still left alone while the finding was open, and moved in the same commit as the document rather than ahead of it. Correcting a paraphrase before the thing it paraphrases does not fix anything; it just produces a second answer, and now you have two.

The register said otherwise, for three pull requests

This is the one I keep coming back to.

ROADMAP.md carried a sentence claiming that publish scope and buffer bounds "were module checks in the harness from the start". A docstring in the driver host repository said the same two live under tests/conformance/.

Buffer bounds never did. Not moved, not relocated: never there.

Epoch rejection's move out of the harness was noticed and recorded when it happened. Buffer bounds' move was not, and so a sentence asserting the opposite survived three pull requests, in two repositories, saying something false about the codebase it shipped with.

Entry one ended on the registers being under test. test_register_integrity.py asserts that ids are contiguous and unduplicated, that a requirement is never both a bare row and a set of sub-ids. Fifty-seven of those tests at the time of this audit. I wrote then that pointing the specification's own convention back at the process documents felt faintly ridiculous, and that it had caught things.

It had, and it does. It could not catch this, and the reason is worth saying plainly:

Not one of the fifty-seven can check a claim about where a test lives.

They check the shape of the registers, which is the thing a register can be wrong about on its own. A sentence asserting that a particular test sits in a particular file is a claim about the world outside the register, and the register has no way to reach out and look.

The same audit turned up two stale cross-reference anchors the same way: by a person reading, while all fifty-seven passed.

And those did have a fix available, which is what makes the boundary visible rather than merely assertable. An anchor points at a section inside the register, so a test can follow it and see whether the row is there. One was written, straight after the audit, and it found four more the moment it ran.

A sentence about which file a test lives in points outward, at a repository the register cannot see. There is nothing there for a test to follow.

So the idea holds, and its edge is now drawn, which is more useful than the idea was on its own. A register can be machine-checked for everything that is a statement about itself. Everything it says about the world outside is still something somebody has to go and look at, and the cadence at which that happens is the audit, not the test suite.

Two smaller ones, and they rhyme

The command dedup window is a fixed five minutes, deliberately not derived from the command's own deadline. Sizing it from the deadline is the obvious move and it is actively harmful: it puts the core's clock inside a driver-side decision, so a host running fast forgets a command id early and pulses the relay on exactly the redelivery the window exists to suppress. Wrong in the direction that costs memory, rather than wrong in the direction that moves a physical thing twice. The published copy still leaves the figure open as "a short window"; it is a number now, and fixed on purpose.

Two subjects were built and never called. Availability and quarantine each had a subject builder, an envelope type, and a publish path asserted against a real broker, and nothing in the driver host ever constructed a message for either. A test that proves the machinery works is not a test that proves anything runs it.

Availability has both ends now. The core subscribes at startup and narrows each envelope to what the declaring instance actually owns, and a driver declares on exactly two edges: losing the transport while it is serving, and stopping deliberately. Quarantine was lifted out of that row rather than carried along inside it, because its producer end turns out not to be buildable yet at all, and a resolved row still quietly carrying half its subject is a row nobody ever comes back for. That is entry two's distinction between deferred and forgotten, met again from the other direction: the danger this time was not an absence with no row, but a row marked done with an absence still in it.

What the shape is

Entry two found one root cause behind three symptoms. This step has a different unity, and it took me a while to see it.

Every finding is about placement. Which of two sentences is the contract. Which side of a process boundary a check belongs on. Which file a test is in, as asserted by a document that cannot see the file. Whether a built path has anything calling it.

None of that is the kind of error a test catches, because a test runs the code, and these are all statements about where things are.

And the drift check, which reads the code against the document and is the thing I would have expected to catch all of it, only works from one side. It finds where the code departs from the prose. When the prose departs from itself, both halves are documented, both are defensible, and the code picks one without recording that it picked.

The honest summary of the step: the specification was wrong in three places, none of them about the system, all of them about itself.

This Wednesday - Ingest Path & Log Schema. The specification's fourth part: what an observation does between arriving on the bus and being answerable, and the log that the current state is only ever a projection of.

The next entry - Checks & Balances. The five processes that keep code and documents aligned laid out whole, including the seam review that filled the hole entry one ended on, and the newest of them finding a batching setting nothing read on the day it was written. What the seam review's first run found still waits for the documents it found things in to be published.

Start of the diary: Four Registers and a Drift Check. Why the registers exist at all, and the one rule about where a sentence is allowed to live.

Sentinel: The Driver Contract

Philip Shaw — Wed, 16 Sep 2026 14:31:10 +0000

A temperature sensor reports 71.6, and it means Fahrenheit. Meanwhile, the canonical unit for temperature in the system is Celsius. It is a simple conversion: a subtraction and a multiplication, and the driver bringing the sensor's reading into the system has everything it needs to perform it.

But it must not convert. What it publishes is 71.6, tagged with the unit the device actually used: the conversion happens on the far side of a wire boundary.

Two things break if the conversion happens at the driver. The number the sensor actually produced stops existing anywhere, so the reading cannot be reproduced once the unit tag or the scale factor turns out to have been wrong - and on a long enough timeline one of them is bound to be. The other is that the driver becomes a second place where the meaning of a point gets decided, when the registry is supposed to be the only one. A protocol adapter running on a Raspberry Pi in a cupboard is a poor place to keep half a semantic model.

Converting a unit is the first item on a list of eight things a driver is forbidden to do, and that list is the spine of this document. Everything else here - the process model, the assignment export, the subject layout, the buffering rules, the deliberately narrow command contract - is what remains once protocol ugliness has been confined to one side of that line and meaning to the other.

What this document owns: the driver host and instance process model, the assignment export and why it is pushed rather than queried, the instance lifecycle, the bus subject layout and the limits on the observation stream, the driver-supplied half of the observation envelope, local buffering and the one place in the path where data is dropped by design, availability declarations, driver-side command handling, discovery and quarantine, the prohibitions list, and the conformance harness every module is tested against. What it deliberately does not own is anything about what a value means, which belongs to The Semantic Model; command verification and the epoch whose token the driver checks, which belong to Core Runtime; and the permissions that turn the subject layout into an authorisation surface, which belong to Security Model. The boundary drawn here is what lets all three assume a device looks the same whatever protocol it speaks.

The driver contract has one job: make every device look identical to the core while keeping all protocol ugliness on the far side of a wire boundary. Everything below follows from drivers being dumb by design.

Process model

Driver host is the deployment unit — one container or systemd unit per network segment. It loads driver modules and supervises driver instances, one per configured device or device group.

driver host "rpi-c1"
├── modbus-tcp    → instance "boiler-relay"
├── mqtt          → instance "esp32-fleet"
└── ble           → instance "ble-local"

Supervision is per instance. A wedged Modbus poller must not take down MQTT ingest in the same host.

One northbound transport, for every protocol without exception. Every driver host reaches the core the same way: over the bus, in the observation envelope below. A second northbound path for any one protocol buys two failure models, two buffering implementations, and two places to fix backpressure, in exchange for convenience in a single module.

The temptation is strongest for BLE, because the radio work genuinely is bespoke — supervision, retry-on-connect, per-device connection state — and it is easy to let that bespokeness leak upward into how the host talks to the core. It stays below the line. Whatever a BLE host has to do to hold a connection is its own business; what it emits is an observation like any other.

Assignment, not configuration

Drivers must not read the registry, but they need to know what to poll. The registry exports assignments rather than having drivers query it.

On startup a driver host requests its assignment from the core and caches it to local disk. If the core is unreachable, it starts from cache. An assignment carries addressing only:

driver_instance: boiler-relay
module: modbus-tcp
transport: { host: 10.0.4.12, port: 502, unit_id: 3 }
points:
  - point_id: boiler.ch1.flow-temp
    read:  { fn: input_reg, addr: 30001, type: int16, scale: 0.1 }
    poll:  { interval: 30s }
  - point_id: boiler.ch1.pump
    read:  { fn: coil, addr: 1 }
    write: { fn: coil, addr: 1 }
    idempotent: true

No quantity, no unit, no subject, no freshness policy. The driver sees a point ID, an address, and a scale factor. scale is a raw-register-to-native adjustment, not a unit conversion — a Modbus register holding tenths of a degree is a wire encoding, not a unit.

Credentials are referenced, never carried. Where a transport needs one — an MQTT broker password, a vendor API key — the assignment names it:

transport: { host: mqtt.lan, port: 8883, credential: mqtt/fleet }

The driver host resolves that reference against its own local store, provisioned out of band. The bus never carries a secret and neither does the on-disk assignment cache, which matters because the cache sits on every host and the assignment subject is readable by anything that can subscribe to it. See Security Model.

Assignment changes arrive as a push on the bus with a version number. The driver applies them without restarting: new points start polling, removed points stop, changed addresses rebind. This is also the load-shedding lever — raising poll intervals across a fleet is an assignment change that takes effect in seconds, which is the manual response to sustained ingest lag described in Core Runtime.

Lifecycle

init → connecting → connected → ready → degraded → connecting
                                  ↓
                              stopping → stopped

connecting — transport not established, exponential backoff with jitter, capped around 60s
connected — transport up, initial reads in flight
ready — initial read pass complete, or explicitly skipped for points with no readable state
degraded — transport up, some points failing; instance stays up and keeps serving the rest

The connected → ready transition is what makes cold start work. Until an instance is ready, the core keeps its points in restored or unknown rather than promoting them, rule warmup gates on it, and the command plane treats every claim on its points as undeliverable rather than sending into the dark.

Instances publish their own lifecycle as points under subject: self — driver.boiler-relay.state, .reconnects, .poll-latency, and the queue and rejection counters below. Driver health is not a separate observability channel; it is the same pipeline, with the same freshness and alerting. Driver monitoring comes free, and the ingest path is exercised constantly as a side effect. The full catalogue of what a driver must publish is in Observability.

Bus subjects

obs.{host}.{instance}.{point_id}         observations
avail.{host}.{instance}                  device/component availability
quarantine.{host}.{instance}             unregistered points
cmd.{host}.{instance}.{point_id}         commands, core → driver
cmdres.{host}.{instance}.{command_id}    command outcome, driver → core
assign.{host}                            assignment push, core → host

Every driver-plane subject leads with the host, and that is what makes bus permissions workable: one credential grant per host covers every instance it will ever run, so credentials stop needing to change whenever an assignment does. Two places to keep in step is one too many, and the one that gets forgotten is the credential.

cmdres carries host and instance for the same reason. A subject of cmdres.{command_id} alone would force every driver to hold cmdres.>, and any driver could then forge an outcome for another driver's command. See Security Model.

Commands route by instance rather than by point, because the core knows the binding and the driver should not have to filter a firehose. Each instance subscribes to cmd.{its host}.{its own id}.>.

Stream limits

obs is a single JetStream stream, and it is the system's outage buffer. Three properties, all load-bearing:

Reject new publishes when full; never discard old. A full stream pushes back onto the driver's local queue, which is bounded and reports its drops. Discarding old messages would delete observations that had already been accepted, between two components that both believe the handoff succeeded.
Bounded by bytes and messages, with no max_age. An age limit deletes unconsumed messages silently, which is the same failure by a different route. Age-based deletion belongs only where dropping old data is the actual intent, which here is Postgres retention.
Sized from tolerable outage and utilisation, not from a duration that feels sufficient for a restart. The arithmetic is in Core Runtime, and the size cannot be computed until capacity targets exist.

Durability lives in Postgres. The stream exists so that a core outage costs nothing, not as an archive.

Northbound: the observation

Exactly the envelope from the semantic model:

point_id, value_native, unit_native?,
quality: live | bad | unavailable,
reason?, detail?,
device_time?, observed_at, source_seq,
driver_id, driver_instance, assignment_version

unit_native is populated only when the source self-describes — Shelly and UniFi JSON do, Modbus does not. When present it drives the mismatch check; when absent the registry is trusted.

source_seq is monotonic per instance and resets on restart, with the restart marked by an epoch counter. (epoch, seq) is what the core dedups on.

A driver may publish only for points its own assignment names. The core enforces this rather than trusting it — see the publisher check in Ingest Path & Log Schema — because subject permissions confine a host to its own subjects but say nothing about which point IDs it puts inside them.

Push and poll are the same thing

The driver module implements one method: emit observations. Whether that is an MQTT callback or a poll loop is internal.

Poll scheduling lives in the driver, never the core, for two reasons: the scheduler must survive a core outage, and it needs to be adjacent to the transport to do jitter, coalescing, and batching. A Modbus driver reading twenty registers should issue one multi-register read, not twenty round trips the core scheduled independently.

Every poll produces an observation. A poll that returns the same value still publishes, because the deadband decision is core-side and freshness needs the refresh. A poll that fails produces quality bad with the reason — not silence.

Buffering

When the bus is unreachable or full, the driver buffers to a local disk queue and replays with original observed_at values intact.

This is exactly why ingested_at exists as a separate field. A driver host that buffers through a twenty-minute partition and then floods must not have those readings evaluated as fresh. The core orders by ingested_at, computes freshness from observed_at, and a rules engine seeing a burst of old readings treats them correctly.

The queue is bounded by size and age. On overflow it drops oldest and publishes a self point recording the drop count — silent data loss is the failure mode nobody notices, and this is the single place in the whole path where data is dropped by design.

Availability

The driver declares unavailability; the core fans it out.

avail.{host}.{instance}: {
  scope: instance | device | component,
  target: "boiler" | "boiler.ch1",
  available: false,
  reason: transport_timeout,
  observed_at
}

The driver names the scope and the core marks every affected point unavailable. This is the difference between "the cause of the silence is unknown" (stale) and "the device is off the network" (unavailable), and only the driver can tell which.

A driver that dies without saying anything is caught by its own heartbeat going stale — the core marks the instance unavailable after a grace period. Both paths converge on the same state.

Southbound: commands

command_id, point_id, value, issued_at, deadline, idempotent, core_epoch

The driver's contract is narrow. It attempts the write and replies once:

command_id,
outcome: accepted | rejected | failed | timeout,
device_ack?: { value, observed_at },
reason?, detail?

accepted means the transport accepted it. device_ack is populated only when the protocol returns confirmation inline. Everything else the core owns.

core_epoch is a fencing token. The driver tracks the highest epoch it has seen and rejects any command carrying a lower one, with outcome rejected and reason stale_epoch. It needs to understand nothing else about the value, only that it never goes backwards. The reasoning is in Core Runtime; the effect is that a core which has already been replaced cannot still be driving relays through a partition it has not noticed yet.

Verification is core-side, without exception. poll_verify is the core issuing a read command after a delay. async_event is the core correlating a later observation. optimistic is the core marking the point assumed. The driver has no concept of confirmation modes, no timers, and no memory of what it was asked to do.

That boundary matters because verification needs durable timers and registry knowledge, and because a driver that retries on its own behalf will eventually double-fire a non-idempotent command. Drivers retry transport establishment, never command delivery — unless idempotent: true, which permits one retry within the deadline.

Commands are deduplicated on command_id within a short window, so a bus redelivery does not pulse a relay twice.

Discovery and quarantine

An observation for an unknown point_id goes to quarantine.{host}.{instance} with its full native payload, never to obs.*. The core records it, deduplicates by point ID, and surfaces it for triage. Nothing auto-registers.

This gives discovery as a side effect of normal operation. Connect a new device, let the driver run, and watch what accumulates in quarantine. Promoting a point is a registry commit supplying quantity, subject, unit, and freshness — the four things no driver can guess correctly.

For protocols with real discovery (mDNS, vendor announce messages, device lists), the driver publishes candidates to the same quarantine subject in the same shape. One triage path regardless of how the point was found.

What drivers must never do

Worth writing down and enforcing in review, because each of these is a plausible-looking shortcut that dissolves the boundary:

Convert units, or emit anything but value_native
Assert quality other than live, bad, or unavailable
Invent, rename, or auto-register points
Publish for a point its assignment does not name
Implement debounce, deadband, moving averages, or hysteresis
Retry a non-idempotent command
Hold verification state or timers
Read the registry directly

Conformance

The module interface is small enough to test properly, and with six or more modules to write, a shared harness pays for itself immediately.

connect() → transport
read(points) → observations
write(point, value) → outcome
close()

A conformance suite is built alongside the first driver: a mock transport that produces malformed payloads, unmapped enum values, out-of-range readings, mid-poll disconnects, slow responses that exceed a deadline, and clock skew. Every module runs the same suite. The failure modes are protocol-independent even though the protocols are not.

mqtt and modbus-tcp come first — push and poll, self-describing and not, verifiable and not. Between them they exercise every branch of this contract. Other protocols are then variations rather than new problems.

Exit criteria

The conformance suite above is this step's exit criterion, not a nice-to-have. A driver module is done when it passes the shared harness, and mqtt and modbus-tcp both pass it before anything else is written.

Three additions to the harness that come from later documents and belong in the same suite:

Epoch rejection. A command carrying a core_epoch below the highest seen is rejected with stale_epoch, and the driver does not attempt the write. Tested by replaying an old command after a higher epoch has been observed.
Publish scope. The module publishes only under its own host and instance prefix, and only for point IDs its assignment names. This is checked here as well as core-side, because a driver that gets it wrong should fail its own tests rather than fill quarantine.
Buffer bounds. Fill the local queue past its limit and assert that the oldest entries are dropped, the drop count is published, and nothing is lost silently.

The step as a whole is done when both modules pass, and when an instance taken from connecting to ready to degraded and back publishes the lifecycle transitions that cold start and deliverability depend on.

On Monday - the Dev Diary. The third entry of what the build has had to send back to the specification.

The next part - Ingest Path and Log Schema. Two logs rather than one, and the column that makes the unit conversion a driver is forbidden to do reversible once it happens in the right place.

Sentinel Dev Diary: What the Specification Left Unsaid

Philip Shaw — Mon, 14 Sep 2026 16:52:49 +0000

Build step one is the semantic model: identity, the closed vocabularies, units, quality, retention class. It was marked complete a fortnight ago. Then it was audited against The Semantic Model, which is the document that owns it and which went out last Wednesday, and the audit is the part worth writing about.

Here is the thing I did not expect. The specification was not wrong anywhere. Not once. Every delta between what it says and what got built came from the same place: it described something carefully, at length, with the reasoning attached... and then never said whether it was optional.

Silence is not neutral. Silence resolves, and it resolves to whatever the code does by default.

Six rejections became twenty-eight

The exit criteria name six ways the registry must refuse bad input. app/registry/errors.py defines twenty-eight.

Most of that is not interesting. malformed_point_id, duplicate_point_id, canonical_unit_missing, kind_mapped_twice: these are finer-grained versions of what the criteria already asked for, and they need no justification beyond a load reporting what is wrong rather than that something is. If you are going to refuse, refuse specifically.

Two of the twenty-eight are different. They are new rules, in the sense that the specification does not state them.

A numeric point must declare a deadband (change_policy_required). The descriptor carries change_policy, and the document argues hard for why the deadband exists; without one "a noisy analogue sensor generates an edge per sample and the rules engine spends its life on 0.01 °C". Having made that argument, it never says the field is required.

So what happens if it is absent? Every sample becomes an edge. The exact failure the document spent a paragraph warning about, arrived at by leaving a field out. Booleans, enums and strings still default to any, because a boolean that changes is an edge by construction; but a float with no deadband is a mistake wearing the costume of a default.

A writable point must declare a confirmation mode (confirmation_mode_required). Same shape, worse consequence. There is no safe default here, and that is the whole argument: the mode an unset field would most naturally mean is optimistic, which is the mode that verifies nothing. A default that quietly means "assume the write worked" is not a default; it is a trap with good manners.

Both could have been solved with a default value. I want to be clear about why they were not, because "make it mandatory" is the more annoying choice and it needs defending.

A default is a decision made once, by whoever wrote the schema, on behalf of every point that will ever exist. It is invisible at the point of use. Nobody authoring a descriptor sees the deadband they did not write. Whereas a load-time refusal is a decision made by the person who knows what the point actually is, at the moment they are thinking about it, with a named error telling them precisely which of the twenty-eight things went wrong.

It costs a line of YAML per point. It buys the guarantee that no point in the system has a policy nobody chose.

Turning prose into types found a missing number

The Semantic Model writes its policies as strings: periodic(30s, grace=15s), absolute(0.2), poll_verify(500ms), assume(v). They are readable, they are compact, and they are exactly what Introduction warns against: "anything encoded as a string suffix will have to be parsed later."

So they became discriminated unions in app/schemas/descriptor.py. That is a straightforward application of a rule the specification already made, and it would not be worth a paragraph except for what fell out of it.

Writing the type forced a number the document never gives. periodic has a grace shown as optional, with no statement of what it defaults to. In prose that reads fine; you skim past it. In a typed model you cannot skim past it, because the field either has a default or it does not, and you have to type one or the other.

PeriodicFreshness.effective_grace now resolves it to half the interval, capped at 60 s, at load. Not at every read; the distinction matters, because a value computed at read time is a value that can differ between two readers of the same descriptor.

That is the second-order benefit of turning prose into types, and I had not really appreciated it before this step. The type system does not just enforce the document. It interrogates it, and it asks about precisely the things a careful reader glides over.

Three smaller decisions, one pattern

Units are nested under quantity rather than becoming a sixth vocabulary. The specification names five closed vocabularies and gives the unit table no home of its own, so it could have gone either way. It went inside quantity because the quantity owns its canonical unit and there is no useful reading in which the conversions into that unit live somewhere else. Nesting keeps the count at five as specified, makes unit_native convertibility a membership test rather than a lookup across files, and lets affine: false reject an offset at load; which, as the document itself points out, is where a 5 °C rise mistaken for 41 °F gets caught.

Derived fields are declared anyway, so authoring one is a named refusal. unit_canonical and interlock_protected are both derived. The document says so and says nothing about how a hand-authored one should fail. They are declared on PointDescriptor regardless, so writing one yourself fails as derived_field_authored rather than being swallowed as an unknown key. A typo and a misunderstanding deserve different errors; the first wants a spell-checker, the second wants an explanation.

dBm was kept rather than "corrected" to strict UCUM dB[mW]. The quantity table spells it dBm, which is not a valid UCUM code. It is, however, what every datasheet, vendor API and engineer writes for RSSI. The owning document was followed rather than quietly corrected in code, and the divergence was recorded in the register instead. This one is a small rule with a big blast radius: canonical values are frozen in the log at ingest, so changing that spelling stops being a YAML edit and becomes a migration over the highest-volume table in the system, the moment the first driver publishes.

The pattern in all three: where the document had made a choice, follow it and record the disagreement. Where the document had made no choice, make one and record the reasoning. Never quietly improve a document from inside the code.

What the step did not deliver

This is the half of an audit that earns it. Three findings, and the first two share a root cause: nothing compared registry version N against version N−1 at load.

A guard that existed as a function with a fixture

vocabulary_member_removed_while_referenced is one of the six exit-criteria rejections, and its fixture passed. Green. Ticked.

The check needs a baseline, and only the tests supplied one. Both real call sites passed none: the runtime at startup, and the CI check that runs against the actual registry. So removing heatsink from the subject vocabulary would have failed as unknown_vocabulary_member; a different error, from a different check, that happens to look exactly like the guard working.

That is the bit that took a moment to sit with. The guard was not missing. It was not broken. It was unarmed, and the symptom of it being unarmed was a passing test and a plausible-looking error message.

The fix was not "call it with a baseline". The fix was to make the resolver refuse when it cannot find a previous revision, rather than returning "no baseline" and carrying on. A fallback would have reproduced the original failure exactly where it matters most, which is a shallow CI checkout: no previous revision available, so no comparison, so no guard, so green. The cost is that CI now needs fetch-depth: 0, and the failure message says so in as many words.

A guard that degrades gracefully is not a guard. It is a guard-shaped hole that reports success.

The register had no row for `addresses`

The Semantic Model is the document that places addresses in the registry:

The registry holds addresses separately: MQTT topic, Modbus unit ID and register, BLE MAC, UniFi site path. These change freely without touching identity.

There is no field on PointDescriptor, no file under registry/, and no table. Nothing was built.

And that is defensible, because the document that consumes them is later in the series and the work is genuinely blocked behind it. What is not defensible is that the step's ledger had no row saying so. The gap between "deliberately deferred" and "forgotten" is invisible from outside; both look like an absence. The only thing that distinguishes them is a row somebody wrote at the time.

The transition criterion was delivered in half

The criterion asks that the quality transition table "exists as a checked enum, with a test asserting that no code path can produce a transition outside it".

The first half landed: the full 7×7 matrix is asserted, including the two an implementation gets wrong, which are that unavailable has only two exits and that nothing returns to a startup state. The second half could not land, because at that point nothing in the system wrote a quality at all, and the deferral was written into a docstring honestly.

The interesting part is what happened next. A later step added the first real writer, and a test that it cannot write an illegal transition. What was never added is the structural assertion that no writer bypasses the function; and the docstring still reads as though nothing writes a quality, which is no longer true. So the deferral note itself has rotted, in the specific way that deferral notes rot: not by being wrong when written, but by being right about a world that moved on.

The shape of the whole thing

One root cause produced three symptoms across two build steps, and none of the three looked related while I was in them. That is what an audit buys that a code review does not: not a closer look at a change, but a look at the step from the outside, against the document rather than against the commits.

The step was already marked complete when the audit ran. It stayed complete; nothing found here unmade it. What changed is that "complete" now means something checkable, because the enumeration of what is missing exists and is written down where it can be argued with.

I would rather ship a step with three named gaps than one with none I have looked for.

This Wednesday - The Driver Contract. The specification's third part: making every device look identical to the core while keeping the protocol ugliness on the far side of a wire boundary.

The next entry - Where the Check Lives. The driver contract turned out to say a module emits and to say a host reads, and only one of those was built; plus three conformance checks placed where one of them can go.

Start of the diary: Four Registers and a Drift Check. Why the registers exist at all, and the one rule about where a sentence is allowed to live.

Sentinel: The Semantic Model

Philip Shaw — Wed, 09 Sep 2026 09:08:10 +0000

A UUID is the obvious primary key. It is opaque, globally unique, immune to renames, and every tutorial reaches for it. It is also the wrong choice here, and the reason has nothing to do with database design.

boiler.ch1.flow-temp shows up in a NATS subject, a Postgres row, a rule definition, a log line and a stack trace. Every one of those is a place a person reads under time pressure. f47ac10b-58cc-4372-a567-0e02b2c3d479 is correct in all five and useful in none of them: it moves the question "which point is this?" one join away, and one join away is far enough that nobody makes the trip at eleven at night.

So the identifier is a dotted slug, and it is the primary key rather than a label hanging off one. That choice has a price, and it is paid at rename time and nowhere else. The rest of this document is the same kind of trade made about twenty more times - what a point is, what it is allowed to mean, and what the system is required to admit it does not know.

What this document owns: point identity and hierarchy, the descriptor fields, the five closed vocabularies, units and conversion, the seven quality states and their legal transitions, the three timestamps, and the observation envelope. Everything downstream reads the registry defined here - the log schema, the watchdog, the rules engine and the command plane all resolve against it. It is the first build step for the same reason it is the first document: nothing later can correct a semantic model that was wrong, only migrate away from it.

Identity

point_id is a dotted path: {device}.{component}.{point}, each segment [a-z0-9-]+. This maps 1:1 onto NATS subjects, survives being pasted into logs and grep, and reads correctly in a stack trace.

The slug is the primary key in Postgres, not a UUID with the slug as a label. Stability comes from the ID being assigned at registration and never derived from anything the network can change. A rename is a deliberate migration event, and it should feel like one.

The registry holds addresses separately: MQTT topic, Modbus unit ID and register, BLE MAC, UniFi site path. These change freely without touching identity.

Hierarchy

Device → Component → Point, where Component may be trivial (main) so a one-sensor device does not pay ceremony for structure it does not have.

Component earns its place because it is the unit of independent failure and hot-swap: the four channels of a multi-channel relay, a sensor behind one leg of an I²C multiplexer. When the multiplexer leg goes dead, one component is marked unavailable, not the device.

Points are scalar. If a device emits a struct, the driver explodes it into points. No exceptions — the moment a point can hold a composite, every downstream consumer has to understand payload shapes, which is the coupling this model exists to prevent.

Point descriptor

The declarative half, living in the registry:

id                 boiler.ch1.flow-temp
device_id          boiler
component_id       ch1
direction          in | out | inout
kind               measurement | state | setpoint | command | diagnostic
value_type         float | int | bool | enum | string

quantity           temperature
unit_native        Cel                    # UCUM
unit_canonical     Cel                    # derived from quantity
subject            water
subject_ref        zone:upstairs-ch       # optional, nullable
modifiers          { aggregation: instant }
tags               [ heating, primary-loop ]

valid_range        { min: -10, max: 120 }
precision          1
change_policy      absolute(0.2)
freshness          periodic(30s, grace=15s)
retention_class    measurement            # defaults from kind

# out / inout only
init_strategy      read_on_connect | restore_from_log | assume(v) | assert(v)
confirmation_mode  inline | async_event | poll_verify(500ms) | optimistic
default_value      false                  # optional
require_intent     false
interlock_protected false                 # derived at rule load, not authored

kind and direction are not redundant. kind describes what the value means; direction describes flow. A setpoint is inout; a command is out with no retained value; a diagnostic is in but must never appear in a measurement dashboard or share a freshness policy with real sensors.

default_value and require_intent are the point's half of the arbitration model in the command plane. default_value is optional; when present, the registry materialises a non-expiring claim at the lowest band, which is the resting state the point falls back to when every other claim is released. When absent, releasing the last claim leaves the point alone rather than driving it anywhere.

require_intent: true says the point must never be without a resolved intent, and registry load fails if it has no default_value. interlock_protected is set at rule load rather than authored, and implies require_intent. Between them, a protective point is one that cannot accidentally be left with nothing asserting a state.

Retention class

retention_class is a closed vocabulary — measurement, state, diagnostic — and it decides how the point's data is stored, aged and summarised. The policy behind each class is in Retention & Compaction.

It is rarely authored, because it defaults from kind:

`kind`	default class
`measurement`	measurement
`state`, `setpoint`, `command`	state
`diagnostic`	diagnostic

Two properties belong here rather than in the retention document, because they are facts about the descriptor:

The default is resolved at registry load and stored explicitly. The class determines physical placement, and placement must not change silently because kind was edited for an unrelated reason. What the registry stores is the resolved class, not the rule that produced it.

Changing it afterwards is a migration, in the same category as a point rename: existing rows live in a table chosen by the old class and have to be moved. A deliberate operation, not a config edit.

The semantic axes

quantity is a closed vocabulary, and it owns the canonical unit. One table, one entry per quantity, and the canonical unit is a property of the quantity rather than a per-point decision.

Decision: pragmatic canonical units (°C, Wh) rather than strict SI ones (K, J). Strict would buy clean dimensional algebra — divide energy by time, get power, units fall out. Pragmatic buys rules and dashboards that read without conversion. This system does not do dimensional algebra, so pragmatic is the right choice.

unit uses UCUM codes (Cel, mA, V, W.h, %) rather than invented strings, with a separate display_symbol. UCUM is a real standard with libraries, and it permanently kills the °C / degC / C / celsius ambiguity. The point stores unit_native; the observation stores both native and canonical values.

subject is the class of thing measured: air, water, heatsink, core_body, skin, battery-cell, mains. Also a closed vocabulary.

Subject alone is underspecified. Two air-temperature sensors in different rooms have the same subject, and that difference is location rather than subject. So the axis splits:

subject — the class of thing (closed vocabulary)
subject_ref — an optional typed reference to a specific modelled entity: zone:loft, device:rpi-t2, person:x

subject_ref is nullable and unresolved for now. It is the hook that allows spatial or logical modelling to be added later without reworking every point. Declaring the field costs nothing; retrofitting it costs everything.

modifiers is a small typed map for things that qualify a reading without changing quantity or subject:

current_type: ac | dc
phase: l1 | l2 | l3 | n
aggregation: instant | mean | min | max | sum | delta — this is what distinguishes a daily energy total from an instantaneous power reading, and it is a real semantic difference that must not hide in the point name

aggregation earns its place twice over: it is also what tells the rollup layer that a cumulative counter must not be averaged. See Retention & Compaction.

tags are free-form and exist to hold things not yet formalised. One governance rule, enforced strictly: tags are never load-bearing. No rule, no unit conversion, no aggregation may branch on a tag. The moment one needs to, the tag is promoted to a field. Without that rule, tags become a shadow schema within six months.

Vocabularies are closed

quantity, subject, enum_type, retention_class, and the modifier value sets live in the registry as versioned tables, not free strings. Adding heatsink is a deliberate registry change. This is the only thing that prevents accumulating heatsink, heat_sink, and heatSink alongside a rule that silently matches nothing.

Enums and codebooks

A point with value_type: enum references an enum_type:

enum_type: switch_state
members:   [ on, off, unknown-position ]

Drivers map native representations (Modbus 0/1, Shelly "on"/"off", UniFi status strings) to canonical members. A native value that maps to nothing produces quality bad with the unmapped value in the detail field. It does not invent a member and does not throw.

Quality

Seven states, plus a machine-readable reason and free-text detail:

State	Meaning
`unknown`	never observed
`restored`	loaded from log at startup, not yet reconfirmed
`live`	observed within its freshness window
`stale`	observed, now past its window, cause unknown
`unavailable`	driver or device known-disconnected
`bad`	last attempt gave an invalid or device-reported-error value
`assumed`	written, and unverifiable

stale and unavailable are separate because the difference is actionable: stale means the cause of the silence is unknown, unavailable means it is known. Alerting and rule tolerance differ.

Two states are not entered by any runtime transition:

unknown is the state a point is in from registration until its first observation.
restored is entered only at core start, when the checkpointed projection loads. A point checkpointed as unknown stays unknown; everything else becomes restored regardless of what it was.

Legal runtime transitions:

unknown     → live | bad | unavailable | assumed
restored    → live | stale | bad | unavailable | assumed
live        → stale | bad | unavailable | assumed
stale       → live | bad | unavailable | assumed
bad         → live | stale | unavailable | assumed
assumed     → live | stale | bad | unavailable
unavailable → live | bad
*           → unavailable          (driver reports disconnect)

Three of those need stating explicitly, because they are the ones an implementation gets wrong:

Anything readable can become assumed. A write with confirmation_mode: optimistic asserts a value irrespective of what the point held before, so unknown → assumed (a cold-boot relay with init_strategy: assert(v)), restored → assumed, stale → assumed and bad → assumed are all reachable. The prior state has no bearing on it; that is exactly what assumed means.

unavailable has only two exits, and both require real data. A point known to be off the network is never promoted synthetically, so there is no unavailable → stale and no unavailable → live by timer expiry. It leaves only when an observation arrives. There is deliberately no unavailable → assumed either: writing to a device known to be disconnected must not manufacture confidence in the result.

No transition returns to unknown or restored. Both are startup states. A point that has been observed once has been observed forever, and restored is a claim about a particular process start.

The rule that matters most: when quality is not live, the value field still holds the last known value. It is never nulled. Consumers receive (value, quality, observed_at) and decide for themselves. Clearing the value on staleness destroys information and reintroduces the null-means-three-things problem this model exists to escape.

Drivers may only assert live, bad, or unavailable. Everything else is the core's to derive. This keeps the driver contract small and prevents a badly-behaved driver from declaring things it cannot know.

Time

Three timestamps. Collapsing them to two destroys information that later mechanisms depend on:

device_time — nullable, untrusted. Devices without an RTC will lie. Used for analysis only, and flagged when skew against observed_at exceeds a threshold.
observed_at — when the driver saw it. This is what freshness is computed against.
ingested_at — when the bus accepted the observation. This is what the log orders by.

ingested_at is taken from the bus message itself, not from the core's clock at the moment of handling. It is part of the dedup key, so it must be identical on every redelivery of the same message; the reasoning is in Ingest Path & Log Schema and it is not safe to change here in isolation. The practical reading is that it means "when this observation became durable in the system", which is also the more useful of the two candidate meanings.

The second and third diverge exactly when a driver host buffers through a network partition and then floods. Any host on a wireless link or a segment that drops makes this routine rather than exceptional, and BLE and radio hosts are the obvious candidates. Collapsed together, a burst of buffered readings all look fresh at the moment they land.

source_seq, monotonic per driver instance, exists for dedup and out-of-order detection.

Change vs refresh

change_policy (any | absolute(delta) | relative(pct) | none) belongs in the semantic model because it determines what the rules engine sees as an edge.

Every observation refreshes freshness. Only observations that clear the change policy emit a value_changed event. Without the deadband, a noisy analogue sensor generates an edge per sample and the rules engine spends its life on 0.01 °C.

The changed flag this produces has a second life in storage: it is what state-class compaction keeps and what it drops. See Retention & Compaction.

The observation envelope

What a driver publishes:

point_id, value_native, unit_native,
quality (live|bad|unavailable), reason, detail,
device_time?, observed_at, source_seq,
driver_id, driver_instance

What the core adds on ingest:

event_id, ingested_at,
value_canonical, unit_canonical,
prev_value, prev_quality,
changed (bool, per change_policy)

That second block is the full transition tuple, which is exactly what the rules engine consumes. Edge detection, staleness triggers, and bad-data triggers all read the same record.

Normalisation is a write-time decision

Core-side normalisation plus a log-first core means canonical values are computed once, at ingest, and then frozen in the log. If a wrong unit_native is corrected in the registry six months later, every historical event stays wrong.

The fix is cheap now and impossible later: the log stores native and canonical side by side, plus the registry version that did the conversion.

value_native, unit_native, value_canonical, unit_canonical, registry_version

Reprocessing then becomes a real operation — replay the log, reconvert from native under the current registry, rebuild the projection. Without value_native in the log, a unit error is permanent data loss. This is codified as a reprocess command from day one rather than a script written in anger later.

Registry authority and the two failure cases

With drivers dumb, the core needs explicit answers for two things that happen in week one:

Unregistered point. A driver publishes boiler.ch1.return-temp and the registry has never heard of it. It is neither dropped nor auto-registered. It goes to a quarantine stream with full native payload, raised for triage. Auto-registration guesses at semantics, and a guessed quantity is worse than no point at all. Quarantine also provides discovery as a side effect: connect a new device, let the driver run, and watch what accumulates.

Unit mismatch. Some sources self-describe (Shelly and UniFi JSON carry units), most do not. Where a driver supplies unit_native and it disagrees with the registry, the observation gets quality bad with reason unit_mismatch. Neither claim is converted from — either could be the wrong one, and silently picking is how a heatsink reading of 90 °F becomes a value a rule treats as 90 °C.

The quantity table

Pragmatic canonicals. Two columns that are easy to omit and expensive to add later:

quantity	canonical (UCUM)	affine	agg_safe
temperature	`Cel`	yes	yes
temperature_delta	`Cel`	no	yes
relative_humidity	`%`	no	yes
pressure	`kPa`	no	yes
voltage	`V`	no	yes
current	`A`	no	yes
power_active	`W`	no	yes
power_apparent	`V.A`	no	yes
power_factor	`1`	no	yes
frequency	`Hz`	no	yes
energy	`W.h`	no	yes
charge	`A.h`	no	yes
illuminance	`lx`	no	yes
flow_volumetric	`L/min`	no	yes
volume	`L`	no	yes
signal_strength	`dBm`	no	no
sound_level	`dB`	no	no
duration	`s`	no	yes
ratio	`%`	no	yes
count	`1`	no	yes

affine exists because °F → °C needs an offset, not just a scale factor. A multiplicative-only conversion layer is correct for every quantity except temperature, which is the one most likely to be converted. The conversion is value × factor + offset from the start.

agg_safe: false marks logarithmic quantities. The mean of two dBm values is not the dBm of the mean, and a naive moving average over RSSI produces a number that looks plausible and means nothing. Derived points refuse to compute mean on a non-agg-safe quantity unless the definition explicitly opts into log_mean, and the same flag governs which functions a rollup may use.

temperature_delta as a distinct quantity is the same class of trap. A 5 °C rise is not 41 °F.

Two notes on the table. pressure in kPa is awkward for both atmospheric and hydraulic readings, since neither is the natural unit; one quantity handled through display_symbol is preferred to splitting, and the decision is worth revisiting once real pressure points exist. And quantity is required for kind: measurement and null for boolean and enum state points — a state pseudo-quantity must not be invented to fill the column.

Subject and subject_ref

Starter vocabulary, flat, to be edited rather than accepted as given:

air, water, surface, heatsink, enclosure, soil, mains, circuit, battery, core_body, skin, ambient, self

self covers diagnostics — a device reporting its own uptime, free heap, or signal strength. Those are real points with real freshness needs and must not be forced into a physical subject.

subject_ref format is {kind}:{slug}, with kinds zone, device, circuit, asset, person. The format is validated, the string stored, and nothing resolved. No entity tables until a second consumer needs them.

The signal that flat subjects have stopped working is a point declared as subject: water with tags: [flow] and another with tags: [return], and then a rule branching on the tag. That is the tag governance rule firing as designed. When it happens the distinction gets promoted — either to a circuit_position modifier or to a nested subject — and by then there is real evidence for which.

Vocabularies as code

All five closed vocabularies (quantity, subject, enum_type, retention_class, modifier value sets) live as versioned files in the repository, loaded into Postgres by a migration. The registry version stamped on every observation refers to this.

Three checks belong in CI rather than at runtime:

every point's unit_native is convertible to its quantity's canonical unit
no point references a vocabulary member that does not exist
no vocabulary member is removed while points still reference it

The last is the migration guard. Renaming heatsink to heat-sink fails the build rather than silently orphaning forty points.

Exit criteria

This step is done when a test suite demonstrates that the registry rejects what it is supposed to reject. Not that it accepts good input — that will be obvious — but that bad input fails at load rather than in production.

A fixture set of deliberately invalid descriptors, each expected to fail with a named reason:

unit_native not convertible to its quantity's canonical unit
a reference to a vocabulary member that does not exist
on_change freshness with neither heartbeat nor an explicit never
a retention_class whose rollup functions cannot apply to the point's value_type or quantity
require_intent: true with no default_value
removing a vocabulary member while points still reference it

Plus two positive checks that are cheap and catch real errors:

Conversion round-trip against known pairs, affine included: °F → °C at 32 and 212, mA → A, W.h at scale. A multiplicative-only conversion layer passes every case except the one that matters, so the affine cases must be in the table.
The quality transition table exists as a checked enum, with a test asserting that no code path can produce a transition outside it. The prose table above is for readers; this is what keeps it true.

On Monday - the Dev Diary will look at what needed to change when the model spec'ed above met reality.

The next part - The Driver Contract. Why every driver speaks one northbound transport no matter what protocol it talks to the device in, and why converting a unit is the first item on a list of eight things a driver is forbidden to do.

Sentinel Dev Diary: Four Registers and a Drift Check

Philip Shaw — Tue, 08 Sep 2026 10:21:49 +0000

The Sentinel specification is ten documents, and it was finished before a line of the system was written. That was the whole idea: decide what the thing is while deciding is cheap, then build it.

Changes to the spec were anticipated. I have a series of Dev Diary look-backs planned that address exactly this set of deltas starting next week. But barely had the first lines of code been written before a wholly different gap surfaced: registers!

Correcting specifications, recording decisions, creating issues to come back to things later, hand-written notes...these are all fine in their own right, but they rot so fast... This is where registers earn their keep from day one.

Two false starts

The registers were the third arrangement, not the first. Let me be honest about that up front. First the story of the short-shelf-life solutions I mentioned.

Stage one: PR bodies. Anything getting deferred, assumed or decided-in-passing was written in the pull request. This is a natural default and it is not nothing; it is at least written down, next to the diff that motivated it. Where it fails is on retrieval and discovery. Nobody greps forty merged PRs to find out whether a thing was deliberate.

Stage two: one file. So it went into docs/deferred.md: blocked work, overdue work, measurements, and findings about the specification, all in one place. Wonderfully arranged by category with notes and even some commit hashes. Better. It lasted... hmm... a few days if I am generous. Why?

It was carrying four different kinds of statement in one file. Because its content was sourced from PRs merged, it organised along similar lines and mixed findings and deferrals into one because it lacked a set of organising principles to keep it disciplined.

A file that holds four kinds of thing has no rule for what belongs in it, so it also has no rule for what has fallen out of it. Nothing was lost dramatically. Things were lost quietly, which is worse, because a register you have stopped trusting is one you stop reading.

And it had a twin. The driver-host repository had grown its own docs/deferred.md, on the same convention and for the same reasons, so the same mistake was now being made twice, in two repositories, about one specification. It was time to do things properly. Enter, the Registers...

Any deferral, every decision or a freshly discovered risk goes in the register that covers the specification, naming the repository it belongs to. Numbered, immutably and durably, and referenced. Importantly, never repeated elsewhere - when the temptation to do so arises, squash it, and link back to the register that owns it instead.

One set of registers spans both repositories, deliberately. Two registers on one specification is two places to look and one to forget, and the seam between two repositories is exactly where a row goes missing; build step three is split across both, so its deferrals were too. Code does not move; documentation does.

The four registers

docs/
  spec/              ← normative. Ten documents. Nothing outside this defines anything.
  REQUIREMENTS.md    ←  SR-*  what the system must do, grouped by owning document
                        CN-*  the constraints that may never be violated
  COMPONENTS.md      ←  C-*   components, each with a build status and its package
                        I-*   integration contracts
  DECISIONS.md       ←  D-*   every choice taken where the spec was silent, superseded ones kept
  ROADMAP.md         ←  CP-*  checkpoints: evidence that does not exist yet
                        A-*   assumptions
                        R-*   risks
                        OI-*  open questions - gaps in the specification
                        DW-*  deferred work - code that is owed
  spec-deltas/       ← one audit per completed build step

Everything except spec/ is derivative. That word matters rather more than it looks:

Nothing in a register defines anything. They point at docs/spec/, which is normative; a register that restates a mechanism has created a second home for it, which is the failure the specification exists to prevent.

The registers number what the specification already says, so that a component, a commit, an Issue or a review can cite it. SR-14 is not a requirement written in REQUIREMENTS.md; it is a pointer with a stable handle, and the requirement itself lives in the document that owns the mechanism. Where the two disagree, the row is the bug. Always the row.

Then a precedence order, because two documents will disagree eventually and I would rather settle it now than in the moment:

On what a mechanism is, or what a term means - the owning specification document wins, always.
On why a decision was taken where the specification is silent - DECISIONS.md wins.
On what is built right now - COMPONENTS.md and ROADMAP.md win over any document's description of intent.
On implementable work - the GitHub Issue wins over all of the above.

Without that written down, every disagreement gets settled by whoever happened to notice it, on the day, in whatever mood they were in. And half of those settlements go the other way next month.

The distinction that is actually load-bearing

If you take one thing from any of this: an open question is not a task, and shipping code does not close one.

An OI-* is a gap in the specification - something that could have been decided and was not. A DW-* is code that is owed. In the moment they feel identical. They are not, and getting them confused is filed as a risk against my own process:

R-2 - A specification gap is "closed" by the commit that routes around it. The work that reveals a gap is almost always the work that works around it, so a finding filed as a task is struck by the commit that proves it real.

Which is exactly how it goes. You are building, you find the document never said which way a case falls, you pick a reading, you ship. If that finding was filed as a task, the task is now done. The specification is still silent - and nobody will ever notice, because the code works. That last part is the one that gets you.

So an OI-* is discharged by amending the owning document and by nothing else. Where a gap was routed around in code, the reading taken gets recorded as a D-* and the open question stays open, pointing at it. Two rows, two lifecycles. One says what the code does; the other says the document never said.

Snippets from the first three steps

Some of what actually landed in the registers as the build got going, quoted as written. The shape of the thing rather than the content of it.

The register vocabulary managed to break the specification's own citation rule, more or less immediately:

OI-1 - Introduction, Retention & Compaction. There is no build step that owns retention policy values. In the absence of one, this repo invented "step 10" - the document's reading-order number wearing a step's clothes - and it has spread into the register, the self-point catalogue test's deferral map and several docstrings. It also directly contradicts Introduction's rule that nothing cites a document by number.

Which is why every specification document here is cited by title, in italics, and never by number or path. The rule pre-dates the incident. The incident is what made it enforceable, because there is now a row explaining precisely what happens when it lapses.

The constraints had the opposite problem. Real, load-bearing, and entirely uncitable:

OI-17 - Introduction. The constraints have no numbered home in the document that states them. Introduction carries twenty-two Conventions - the properties everything else is measured against, each extracted from a specific error - in a document that says of itself "this note is the map and the rationale. It defines nothing." A definition found there is stated to be a bug in the note, and yet these are the closest thing the system has to invariants. They are also uncitable: every reference to one quotes its prose.

They are indexed as CN-1 to CN-22 now, which is an index and not a copy. The argument for each one stays exactly where it was.

An assumption is a different animal again, because work is already resting on it before anyone thinks to check:

A-1 - The test environment has TimescaleDB, not plain Postgres, and NATS started with JetStream. If it is wrong: ~95 tests skip rather than fail and the suite reports green; the tell is the skip count and coverage near 80% rather than ~97%. Tested by: every run - a correct one has 0 skipped.

A green suite that is green because a hundred tests did not run is the most expensive kind of wrong. And it is invisible unless somebody wrote down what the number is supposed to be.

A good many decisions turn out not to be about the system at all, but about the ground it is standing on:

D-07 - Enum members are quoted in YAML. YAML 1.1 resolves bare on, off, yes and no to booleans, which would have turned the switch_state codebook into [true, false] silently. Not a specification matter - a trap in the serialisation format the specification chose.

No specification was ever going to catch that one. It is the format's trap rather than the design's, and without the codebook check it would have been found eventually by something behaving very strangely at runtime.

And occasionally a decision is really a finding about the tests:

D-14 - A guard with no caller is not a guard. The check passed its fixture and neither real call site supplied a baseline, so the operation it was meant to refuse failed with a different error, from a different check, that happens to look like the guard working.

That one is worth considerably more than its two lines. A test that exercises a function directly, plus a codebase where nothing calls that function in anger, gives you a green build and an unarmed guard - and the failure presents as the guard working correctly. I have no idea how many more of those there are. Presumably not zero.

Then the category for numbers that do not exist yet. A checkpoint says what produces it, what it must yield, what it unblocks, and one more field that is the whole point of the exercise:

CP-1 - Ingest throughput against a populated database. Current value: 4,369 obs/s, taken 2026-08-31 against freshly-truncated hypertables with the three aggregates defined and no refresh policy. Steady state is 220/s, so utilisation ≈ 5%. Do not pre-decide: that the margin holds. Re-taking can only move the figure down.

Do not pre-decide is the field that earns the register its keep. A number measured under favourable conditions reads as headroom, and the caveat only survives if it travels in the same row as the figure. That is R-3 - and R-3 is retired by re-taking the measurement, not by anyone deciding it is probably fine.

The status marker that lies

COMPONENTS.md and ROADMAP.md mark every entry built, partial, or not started. Partial is the interesting one:

A 🟡 is a claim about what is missing, not only about what is there. It is the marker most likely to be wrong, because it is the one that requires someone to have looked.

So partial is only legal when what is missing is enumerated, with a deferred-work row behind each gap. A 🟡 with one named gap and three real ones is worse than an honest "not started"; it has spent the credibility of a status marker to buy an inaccuracy.

Which is where the list of things reviewers are asked to attack begins: any 🟢 that is really a 🟡, and any 🟡 whose account of what is missing looks suspiciously short.

The drift check

None of this stays true on its own. Registers rot exactly like the notes they replaced, unless something makes them false loudly. So there is a ritual, and it does not vary.

At the end of each build step: enumerate the owning document's exit criteria and map each one to a test. A criterion with no test is a finding, not something to quietly fix - all three audits so far have turned one up. Then audit the code against the documents rather than against the commit history, because the commits record what I intended and only the code records what I did. Write it up as spec-deltas/build-step-N.md, in two halves: what changed during implementation, and what the step did not deliver. Then route every finding into exactly one register, which is the part that actually matters.

On each pull request: if it defers something, the DW-* row lands in the same commit. If it adds a mechanism, that mechanism's signals go into the observability document in the same commit. That rule exists in writing because I broke it once.

The audits keep earning their place. The third one found a paragraph in a register asserting the opposite of what the code did - not drift between the code and the specification, but drift between the register and reality, in a document whose entire job is to describe reality. Humbling.

Why this shape, specifically

Sentinel is built almost entirely with coding agents, and that has shaped the registers more than anything else here has.

An agent opening a session has read the repository and none of my recollection. It cannot know that a field is mandatory because of an argument three weeks ago, or that a number is provisional, or that a gap in a document was spotted and deliberately left open. Every one of those is something I would have supplied from memory without noticing - and, if I am honest, supplied differently on different days.

So the project's CLAUDE.md does not explain the registers. It points at them, and says to read the index before adding to any of them. What that buys is that a decision can be cited rather than re-derived. A decision that gets re-derived every time gets decided differently at least once.

Superseded decisions stay for the same reason:

A decision that was overridden is more useful than one that was erased, because the override is itself evidence about what the design gets wrong.

And the registers are themselves under test. test_register_integrity.py asserts that ids are contiguous and unduplicated, and that a requirement is never both a bare row and a set of sub-ids. That is the specification's own convention - every claim a document makes has a test that would fail if it stopped being true - turned around and pointed back at the process documents. It felt faintly ridiculous to write. It has caught things.

What is still unowned

One gap is worth ending on, because it is open and I do not have an answer for it:

OI-7 - Introduction. Nothing reads the documents against each other. All three cross-document gaps found so far were found by building rather than by reading, and each sat between two documents that were each internally consistent.

The single-owning-document discipline optimises for consistency within a document, and it works. The price is that the seams between documents are where the gaps actually are, and nothing currently looks there. The drift check runs per build step, so every seam does get inspected eventually - by the build, at the point where it costs the most to find.

A real hole in the process, then. Filed as an open question, in the register whose job is to hold the questions the process cannot yet answer. Which is at least the machine working as designed.

Next - The Semantic Model. Part two of the specification proper: why the primary key for every point in this system is a dotted string you can paste into a grep, and never a UUID.

The next entry - What the Specification Left Unsaid. What the first audit found in a step that had already been called complete: six named rejections that had become twenty-eight, and a guard that existed as a function with a fixture and no caller.

Sentinel: An Introduction

Philip Shaw — Wed, 02 Sep 2026 09:22:19 +0000

import PlaneArchitecture from '../../assets/sentinel-plane-architecture.svg';

Does it matter if a soil moisture sensor goes quiet for three days?

On its own, probably not. But if an irrigation decision is derived from that reading, three days of silence is three days of watering against a number that stopped being true, and nothing in the system said so.

Move up the scale. Indoor air quality is degrading because the air conditioning has stopped working — not thrown an error, stopped, which from the outside looks identical to an air conditioner with nothing to do. Move up again. A battery bank charging off solar, in an enclosed space, begins to climb in temperature. Do you want to know about that now, or the next time you happen to look at a dashboard?

Those three questions have the same shape but completely different answers and priorities. What separates them is not technical, it's a judgement about what matters, how much, and how quickly. That call gets made somewhere regardless. Written down, it is a specification. Left unwritten, it is whatever the code or user happened to do.

Problem Statement: Underneath all three questions is the same problem: a system that cannot tell a reading from a memory of one. A value that arrived three days ago still presents as a value, a device that has stopped reporting looks like a device with nothing to report, and a command that was sent looks like a command that worked — none of it announcing itself, all of it producing plausible numbers and unbroken graphs right up until a decision gets made against something that stopped being true. Sentinel exists to make that difference explicit at every hop, so that acting on a number the system cannot vouch for is a choice rather than an accident.

There is an obvious objection, which is that this is a solved problem and you can buy it. However, I have spent a long time cajoling systems and their processes to fit around real world situations they were not designed to address — reframing the problem until it fitted the tool. That works until the reshaping costs more than the tool saves, and at that point it is worth exploring something with the flexibility those use-cases actually need.

So this design is written down, and in public.

A specification at this scale is not trivially derived from a set of working assumptions; the assumptions are the easy part, and they usually create a false sense of simplicity that unravels the moment you think the work is finished.

Publishing holds me to the choices I made during the design. Not as choices I must live with, but ones that I must litigate clearly when I adopt or change them. What I decided, what I claimed they would achieve, and what had to change when the design met the thing it describes.

That pressure is the same in a one-maintainer repository as it is in a corporate monorepo, and so are the failure modes. Arguing the changes in public makes each one a worked example rather than a commit message.

The spec is here to be read and argued with.

What is Sentinel?

A platform for monitoring and controlling devices across several network segments: collecting readings over mixed protocols, normalising them into a single semantic model, deriving values from them, evaluating automation rules, and driving physical outputs with verification and safety interlocks.

The name is the job. A sentinel stands watch, knows the difference between quiet and absent, and raises an alarm rather than acting on a guess. Two properties of this design follow that description more closely than they follow any conventional monitoring platform: data quality is a first-class field, so the system is explicit about what it does not know; and absence of data is a signal in its own right rather than a gap to be interpolated over.

It is written for a system built and operated by one person, and several decisions follow from that constraint rather than from first principles. Where they do, the constraint is stated alongside the decision.

Components take the system name as a prefix: sentinel-core, sentinel-driver-modbus, sentinel-registry.

How this series is organised

Eleven parts, published weekly. This one is the map and the rationale for Sentinel's design; the ten that follow are the documents that define it.

Every mechanism in this system has exactly one owning document. Where this part mentions a state set, a policy name, or a lifecycle, it names it only to say where it is defined.

Each part is published as it was drafted, rather than as a tidied-up retrospective. The system is being built while the series runs, so where the plan meets reality and loses, a follow-up says what changed and why. Three figures in the design are unmeasured today — sustainable ingest throughput, cold-start time, and the retention policy values — and all three get measured during the build, inside the window this series runs in. You will see the numbers when I do.

Reading order

The Semantic Model
Driver Contract
Ingest Path & Log Schema
Watchdog & Derived Points
Command Plane
Rules Engine
Core Runtime
Security Model
Observability
Retention & Compaction

Core Runtime, Security Model and Observability come after the mechanisms they constrain: what enforces single-instance operation and what the system is sized for, which boundary carries the authorisation weight, and what signals every mechanism must expose.

Retention & Compaction reads last because it uses descriptor fields defined across four earlier documents, but it is a mechanism rather than a policy appendix — it constrains the log schema, the derived-point layer and the reprocess operation, and part of it cannot be deferred without a data migration.

Scope

These ten documents cover the core and the data pipeline: collection, ingest, state, derivation, automation, control, runtime, security, instrumentation and storage lifecycle.

Two things have been deliberately left out, because neither changes anything above: alert routing and notification delivery, and the read API and UI. Observability says what must be watchable and what should raise an alarm; but how an alarm reaches a person is not specified here. Rules Engine names actions with a channel; the channel model is not specified here either.

The one constraint those inherit: alerting runs outside the core, because alerting inside the core cannot report the core's absence.

The two decisions that shape everything else

1. Log-first core. Current state is a projection of an append-only event log, never the primary artefact. Every observation, quality transition, command and command outcome is an appended event. Restart is snapshot plus replay of the tail. This is the whole answer to restarting without losing state, and retrofitting it later is painful, so it goes in first.

2. Hard driver/core boundary. The core has zero protocol knowledge. Drivers have zero knowledge of rules, units or storage. The boundary is exactly two contracts: a normalised observation flowing north, and a command envelope flowing south. The failure this prevents is an observability or collection concern leaking outward into the layers around it, so that every part of the system ends up coupled to whichever part was built first. Making the boundary a wire protocol between processes means it cannot be casually violated.

Everything else in these documents is downstream of those two choices, and most of it follows fairly directly.

Non-goals

This is a system built from scratch, with no migration path in or out. No document should acquire a migration section, a dual-run plan or a compatibility shim.

Multiple concurrent cores are not a goal. Exactly one core instance runs at a time, enforced rather than assumed. The reasoning, the enforcement mechanism, and the two conditions that would reopen the question are in Core Runtime.

Keeping raw data indefinitely is not a goal. Nothing is retained in original sample detail forever; measurements are rolled up and their samples dropped. State transitions are a different thing and are kept — the distinction is samples versus history, and it is in Retention & Compaction.

A second storage tier is not a goal. No object-store archive, no cold tier. Data not worth keeping in Postgres is not worth keeping.

Nothing here describes existing infrastructure. These documents specify what to build. Where they name a protocol, a device or a deployment shape, that is a design target rather than an inventory, and no document should assert that any component currently exists.

The planes

The diagram is illustrative. This section is the normative description of the decomposition; where the two disagree, this text wins and the diagram should be corrected.

Devices are anything that produces readings or accepts writes, over any protocol, push or poll.

Driver plane. Protocol adapters, deployed as driver hosts, one per network segment, each supervising per-device driver instances. Drivers translate a protocol into observations and attempt writes. They hold no semantics, no verification state and no automation. Defined in Driver Contract.

Bus. Durable subject streams carrying observations, availability, quarantine, commands and command results. Every driver-plane subject leads with the host, which is what makes per-host credentials workable. The observation stream is the system's outage buffer, bounded by size rather than age and sized from tolerable outage and utilisation — see Core Runtime. Subject layout is defined in Driver Contract; it is also the primary authorisation surface, so the two are designed together — see Security Model.

Core. A single process — see Core Runtime — containing four stages that are logically distinct and physically together:

Normalise — validate, deduplicate, resolve against the registry, check the publisher, convert to canonical units, evaluate the deadband. Defined in Ingest Path & Log Schema.
Event log — the durable append-only record, split by volume, retention class and query shape. Schema in Ingest Path & Log Schema; the class split in Retention & Compaction.
State and registry — the current projection, plus the versioned registry that says what every point means. Projection defined in Ingest Path & Log Schema; registry content defined in The Semantic Model.
Derivation and watchdog — freshness deadlines and derived points, both of which emit into the same pipeline as device data. Defined in Watchdog & Derived Points.

The diagram places derived points and timers inside the rules engine box. That is an over-simplification, and it is the one worth flagging, because it survived several drafts before the distinction it blurs turned out to matter. Derived points are ordinary points, computed before rules evaluate; the watchdog that ages them is not part of the engine at all. That separation is exactly what keeps the rules engine small, and it is argued properly in Watchdog & Derived Points.

Rules engine. Evaluates predicates over transitions and produces claims. It sends no commands, holds no state between evaluations, and has no visibility into whether a claim reached hardware. Defined in Rules Engine.

Command plane. Owns the durable claim set and its resolution into a desired value, deliverability and reconciliation of reported against desired, confirmation and verification, and the safety and interlock model. Returns to the bus as commands to driver instances. Defined in Command Plane.

The single closed loop is: device → driver → bus → core → rules → command plane → bus → driver → device, with the log recording every hop.

Scale

The design targets, stated once here and derived from in Core Runtime: 200+ devices across 16+ network segments, 2,000+ device timeseries points, 1,000+ derived points, and roughly 700 self-points that fall out of instrumenting the rest — about 4,000 points in total, at ~220 observations per second, designed for 500.

Two figures matter more than their size suggests. Self-points are a fifth of the registry and are easy to omit when sizing anything. And sustainable ingest throughput is the one term still unmeasured, which makes it the term everything else waits on.

Where things are defined

Concern	Owning document
Point identity, hierarchy, descriptor fields	The Semantic Model
Closed vocabularies: quantity, subject, enum types, retention class, modifiers	The Semantic Model
Units, canonicalisation, affine conversion, aggregation safety	The Semantic Model
Quality states and legal transitions	The Semantic Model
Timestamps and sequence numbers	The Semantic Model
Observation envelope	The Semantic Model (shape), Driver Contract (driver-supplied half)
Registry authority, quarantine and unit-mismatch handling	The Semantic Model
Driver host and instance model, lifecycle, assignments	Driver Contract
Bus subjects, stream limits, buffering, availability	Driver Contract
Driver-side command handling and the prohibitions list	Driver Contract
Log schema, hypertable split, ingest pipeline ordering, dedup, publisher check, projection, restart, reprocess	Ingest Path & Log Schema
Registry versioning and descriptor storage	Ingest Path & Log Schema
Freshness policies, watchdog behaviour, staleness suppression, flap damping	Watchdog & Derived Points
Derived point descriptors, operators, quality propagation, evaluation, rehydration	Watchdog & Derived Points
Restricted expression grammar, and its two binding contexts	Watchdog & Derived Points
Claim model, bands, resolution, suspension	Command Plane
Desired/reported, deliverability, convergence, command state machine, confirmation modes	Command Plane
Fail-safe versus fail-clear, interlock constraints	Command Plane
Intent API, validation, rate limits, idempotency, escape hatch	Command Plane
Rule shape, triggers, evaluation, claims, warmup, loop protection	Rules Engine
Rule and registry load-time validation, rule testing	Rules Engine
Single-writer enforcement, core epoch, startup and shutdown ordering	Core Runtime
Scale targets, outage figures, buffer sizing, drain behaviour	Core Runtime
Backpressure and ingest lag	Core Runtime
Availability while the core is down, restart time, geographic split	Core Runtime
Threat model, principals, credentials, bus permissions	Security Model
Human authentication, capabilities, what each principal may do	Security Model
Secret handling, audit requirements, exposure assumptions	Security Model
Self-point catalogue, naming, what raises an alarm	Observability
Retention classes, policy values, rollup derivation, compaction	Retention & Compaction
Exit criteria for each build step	the document owning that step's mechanism

Conventions

Most of these were extracted from a specific error rather than asserted as principle, and each names the failure it prevents.

One definition per concept. Cross-reference rather than restate, including in this note. A concept that appears in two documents has one normative home and one pointer.

Vocabularies and schemas are code. The closed vocabularies, point descriptors and rule definitions live as versioned files in the repository and load into Postgres by migration. Prose in these notes explains them; it does not define them once the code exists.

State machines are enums with tests. Quality transitions, driver lifecycle, command lifecycle, convergence and arbitration bands appear as tables in prose here and must exist as checked enums in code, with a test that no code path produces an illegal transition. Prose tables are for readers and will rot.

Every claim a document makes has a test that would fail if it stopped being true, and that test lives in the same document. Prose cannot fail loudly. The exit criteria at the end of each document are that test set, and they are what says a build step is finished — not a feature checklist, but a demonstration that the step's central property actually holds.

A boolean that has to carry three meanings is an enum. Quality was the first instance and convergence the second. When a field needs to say "not yet", "no", and "not applicable", write the enum the first time.

Store what composes, not what is convenient to read. Native alongside canonical, sum and count rather than average. Anything derived can be recomputed from a composable base; a base that has already been reduced cannot be recovered, and the reduction is usually wrong at the next level up.

Storage optimisations that discard raw rows are not optimisations. Derived-point windows and reprocess both read the raw tables, so a row not written is a mechanism quietly broken. Every observation writes a row; volume is managed by retention and compaction, which are deliberate and bounded, rather than by suppression at write time.

Behaviour that varies by data type is derived from the descriptor, not authored per point. Which rollup functions are valid follows from value_type, agg_safe and the aggregation modifier; which retention class applies follows from kind. Restating what the registry already knows is an invitation to state it wrongly.

One number per concept. Where an ordering exists, it is expressed once. Bands are an ordered enum and there is no parallel priority integer; the same discipline applies anywhere else a rank appears.

Two numbers that differ by an order of magnitude are two numbers. Planned restart and tolerable unplanned outage were briefly treated as one quantity, which would have sized the outage buffer against the restart case. Before reusing a figure in a second place, check that the two cases actually coincide.

Identity comes from the message, not from the moment of handling. Anything a component assigns to an event at processing time is different on a retry, and therefore cannot be part of an identity or dedup key. This is why ingested_at is taken from the bus rather than from the core's clock.

Authority comes from the connection, never from the payload. Who a request or a message is from is established by the credential that carried it or the binding it arrived on, never by a field inside it. Owner and band are derived from the authenticated principal; a driver's right to publish for a point is checked against the registry rather than taken from the subject it chose.

Assumptions that several documents rely on are enforced somewhere. Single-writer was assumed by five mechanisms before anything checked it. An assumption held by convention is one that gets broken on a quiet afternoon, and the failure modes that follow are usually silent.

Nothing expires data by age except a retention policy chosen for it. Age limits on a buffer delete messages that were never consumed, silently, between components that both believe the handoff succeeded. Buffers are bounded by size and refuse new work when full; age-based deletion is confined to the places where dropping old data is the actual intent.

Physical layout that varies by policy is decided before there is data. Retention attaches to a table rather than to a row, so a per-class policy is a table split, and a table split after the fact rewrites everything ever collected. The values can wait; the shape cannot.

An operation that cannot cover its whole scope refuses, rather than covering part of it. Reprocess spans retention classes with different horizons, and correcting the reachable two thirds would leave right and wrong data indistinguishable. The scope is validated first, the refusal carries a reason, and narrowing it is a deliberate second command.

A signal about a component must not depend on that component. Core self-points are injected at the projection rather than published through the queue they describe, and alerting runs outside the core so it survives the core being down. A monitor that fails with the thing it monitors is not a monitor.

Never report a fact about the core as a fact about a device. Ingest lag would otherwise surface as fleet-wide staleness, and the system would take protective action across the estate because it was slow. Where the cause is internal, the system says so and suppresses the derived signal — staleness and warmup deadlines both pause on the same condition.

Wait on conditions, not on clocks. A timeout should mean the thing being waited for failed, not that a precondition had not arrived yet. Where a component waits for another to become ready, it gates on the readiness signal and the wait is unbounded; where the wait is genuinely unsafe to leave open, it escalates rather than expiring.

Deployment parameters are read, not assumed. Retention windows, intervals and limits set at deployment are checked against configuration at load time by querying them, never by hardcoding the current value. A constant copied into validation code is a constraint that silently stops holding the day the original changes.

A mechanism does not ship without saying how its failure would be noticed. Its signals go in Observability in the same commit that defines it.

Anything encoded as a string suffix will have to be parsed later. This applies to units, phases, aggregation and circuit position. It is promoted to a field instead.

Build order

Semantic model and vocabularies — point identity, quantity, unit, subject, quality, retention class
Event log, state projection, snapshot and replay, cold start
Driver contract, plus one push driver and one poll driver
Freshness watchdog and derived points
Command plane, including arbitration and the confirmation modes
Rules engine
Instrumentation and timeseries export

Steps 1 and 2 are the expensive ones to change later. Everything from 3 onwards is additive if those are right.

Each step is finished when the exit criteria in its owning document pass. They live there rather than here, because a criterion is the proof of that document's claim and only stays truthful next to it. A separate test plan would drift within a month, leaving two places describing what ingest guarantees.

Security is not a step. The subject layout and the publisher check land in step 3, bus credentials with the first driver host, and forward auth with the API when it exists. Deferring it to a step of its own is how it becomes a retrofit.

Neither is the runtime. The advisory lock and the startup ordering belong in step 2, with the first process that writes; the core epoch lands in step 3 alongside the command envelope.

Neither is observability. Each step adds its own self-points as it goes, which is also how the ingest path gets exercised before there are devices to exercise it.

Retention is split across both: the class field, the hypertable split and the ingest routing belong in steps 1 and 2, because they are physical; the policy values and the compaction job follow once volume is measured.

Ingest throughput is measured during step 2, against the design rate in Core Runtime. It is the term the stream sizing and the recovery time both depend on, and it is cheap to measure with synthetic load before any driver exists.

Open items

Sustainable ingest throughput. The only unmeasured term in the buffer and recovery arithmetic. Measured on per-observation validation, registry lookup and conversion in application code, which is where the ceiling sits — not on the batch insert.
Cold-start time. The planned-restart number, dominated by derived-point rehydration at a thousand-plus derived points. Measured rather than estimated, and kept small.
Retention policy values. The classes and the split are decided; the durations are not. They need measured volume, and the driver for measurement raw retention is how long it is plausible to want to reprocess.

All three are measurements rather than decisions, and the first two are taken during step 2.

Settled: scale targets and the four-hour tolerable outage, both in Core Runtime. Retention classes and the samples-versus-history distinction, in Retention & Compaction. Exit criteria, in each owning document.

Out of scope: alert routing, and the read API and UI.

Next Wednesday - The Semantic Model. Why the primary key for every point in this system is a dotted string you can paste into a grep, and never a UUID.

Notes to Self: The Interview Between an Issue and a Spec

Philip Shaw — Fri, 14 Aug 2026 12:25:24 +0000

On 1 August I opened an issue that was three sentences long. A hundred and one minutes later
the feature was merged, and the document that got it there ran to
457 lines.

I didn't write those 457 lines. In fact, I didn't have to write any more documentation, and not because I simply allowed Claude to run amok.

Here is the issue in full — control-api#265, 225 characters:

control-api#265 — Manifest-backed dashboard feeds

For each dashboard, auto create a manifest keyed by dashboard_id.

For each sensor the dashboard uses, tag it to be included in the manifest.
When a dashboard definition is updated, add / remove tags from sensors accordingly.

From that genesis moment, this is the lifecycle of the issue all the way through to landing:

Time (UTC)	Event
14:25	Issue #265 opened — 225 characters
14:54	`FEAT-0007` spec committed — 457 lines
15:35	Spec merged (PR #266)
15:51	Implementation committed
16:06	Implementation merged (PR #267, 15 files), issue closed

The interesting part isn't the speed. It's the step at 14:54 that landed a previously non-existent spec document, and what happened in the twenty-nine minutes before it.

The issue was never a specification

I often write issues like this one...the way most people write shopping lists. Actuator address is not ensured?
Baseline the trace correctly. With the pre-rolls, the frame-rate looks out. They're abbreviated to
the point of being cryptic to everyone else. I write them this way deliberately: I'm usually mid-something
else when I notice a problem, or have an idea for a better route to the solution. The cost of a full write-up right at that moment would be a fractured sense of flow. As most engineers will tell you, the transitions into and out of flow are the most disruptive parts of their working day.

This terse form of issue-writing can be all you need, and it's worth being precise about why it works and the trade-offs it includes. It is not because "the issues are good enough".

They aren't. When you pick one of these up later, you can reconstruct everything missing from memory.

In my case with control-api#265, it was a particular panel type I was thinking about, a constraint I hit last week with it, the reason the
obvious approach was off the table, etc... That reconstruction was free and, more importantly, it was
invisible to any external observer. When you are the issue filer and its eventual closer, you never experience this rehydration as filling a gap. You see the issue's sparse wording as an aide-memoire for the fuller context that is still cached in your head.

What bites here is when you work in a team (of other engineers or coding agents): something has to bridge this gap or the issue will get resolved using a foundation of well intentioned misinterpretation and apparently sensible assumptions.

An agent has something both better and worse than an engineer's recall — it can read
every file in the repository, including ones you have forgotten, and it can read exactly nothing
of your latent intentions.

Fortunately, the fix is not to write longer issues. Capture is working; it's the handoff that's broken, the context sync.

An agent as interviewer asks for decisions, not descriptions

The obvious version of this step — "the agent asks clarifying questions" — undersells it, and
would be annoying in practice. Being interrogated about your own bug report is worse than
writing the report properly in the first place.

What makes it worth doing is the ordering. The agent goes and reads first, then comes back
with what it found and asks you to decide — not to describe.

You can see the split in the spec that grew out of my dashboard issue. Its reference list is entirely material the
agent uncovered on its own:

FEAT-0007 §2 — References (excerpt)

- `app/repositories/sensor_repo.py:123-126` (`get_sensors_with_tag`, the JSONB
  `.contains()` query pattern this spec's reconciliation reuses)
- `app/services/sensor_service.py:42-84,131-174` (`upsert_sensor` /
  `_fan_out_upsert` — precedent for both the reserved-tag-prefix guard and the
  "best-effort, must never fail the primary write" fan-out posture this spec copies)

Nobody asked me where the JSONB containment query lived, or which existing service had already
settled the question of what a fan-out failure should do to the primary write. Those are
enrichment lookups, and lookups are the agent's half of the job.

What came back to me as decisions to make were the things no amount of reading resolves: whether one SSE stream per
dashboard was actually the point, whether a second repository was allowed to be changed for
this, where the boundary of the work sat. The spec's Out of Scope section is five such
rulings written down.

That's the trade that makes the interview cheap enough to bother with. The expensive half is
already done by the time it reaches you.

The best question contradicted the issue

Read my second sentence again: for each sensor the dashboard uses, tag it.

There's an assumption buried in there — that the control-api server can tell which sensors a dashboard
uses. It can't, and the spec says so at length:

FEAT-0007 — Design Decision (excerpt)

The obvious-looking alternative — have control-api decode `layout` itself and derive
the sensor set — does not actually work cleanly, for one panel type specifically.
`chart`/`gauge` panels store `series[].{deviceId,sensorRef}` plus a panel-level
`dataType`, which *would* parse cleanly. But a `health_stats` panel stores only
`healthStats: {bodyDeviceId?, heartRateDeviceId?}` (confirmed in
`serializeDashboardLayout`) — its 7 `sensor_ref`s are frontend constants
(`BODY_SENSOR_REFS`, `HR_SENSOR_REF` in `useHealthStatsPanel.ts`), and each role's
`data_type` is discovered at render time from `listDeviceSensors`, not stored anywhere.

When I wrote the issue I was picturing chart panels, where my sentence is true. Health-stats
panels store two device IDs and resolve the rest at render time in a different repository.
A server-side parser would have to duplicate a list of constants that already lives in the frontend,
with no shared source of truth and no error when it happens to drift.

So the spec proposes the opposite: the client sends an explicit sensors list alongside the
layout, and the server never parses the layout at all. That decision brings a new API rule
with it — a PATCH supplying layout without sensors is rejected with a 422, so a layout
change can't silently leave the tag set stale.

None of that is in my three sentences. One line of it directly contradicts them.

This is the part I want to defend hardest. An interview process that only elaborated what I wrote would be
formatting. This one went and checked, found my premise didn't hold, and brought back a
reversal — before any code existed to be wrong.

The numbering is the fossil record

Something else survives in the finished document. Its requirements run FR-1 through FR-10 —
but there's also an FR-6a, an FR-10a, an NFR-3a, an AC-8a and an AC-8b.

A list written straight through numbers 1 to n. Letters mean insertion into a list that had
already been numbered. Each one marks a place where a later question forced something in
between.

FR-6a is the one I'd point at. FR-6 had established that dashboard tags are reserved, so a
caller's tag edit must preserve them. Then:

FEAT-0007 — FR-6a (excerpt)

The prior-tags read [...] is already wrapped in `try/except Exception:
logfire.exception(...)` so it can never break the upsert — but under FR-6, silently
treating a failed read as `old_tags = []` would mean `preserved` comes back empty and
the caller's list wins outright, wiping every `dashboard-*` tag on that sensor: the
exact outcome FR-6 exists to prevent.

An existing try/except in unrelated code turns the new requirement into precisely the
failure it was written to stop. Follow FR-6 faithfully and you ship the bug. That is not a
question anyone thinks to ask about a three-sentence issue, and it was answered in a spec document
before any code was written rather than in a post-mortem.

The whole document is published alongside this article — FEAT-0007: Manifest-Backed Dashboard
Feeds — so the excerpts above can be read in context.

What the spec bought afterwards

Implementation merged thirty-one minutes after the spec did, across 15 files. What I want to
draw out isn't the speed — it's this line from the implementation PR's test plan:

AC-8b (migration's server_default behaving correctly against a pre-populated table) has no
automated test — this repo has no migration-test harness (integration tests use
metadata.create_all, not Alembic). Verified by construction: [...]

One acceptance criterion, honestly marked unmet, with the reason. That's only possible because
AC-8b existed to be unmet. Without the spec there'd be no gap — just a migration nobody had
thought to test, and no record that anyone noticed.

A spec's most underrated property is that it gives the work something to fail against.

Doing it on purpose

The shape I'd suggest, in the order that matters:

Keep writing bald issues. The three-sentence note is doing its job. Don't move the cost back to capture time; you'll just stop capturing.
Put the interview at triage pickup, not at capture. This is the whole trick. At pickup you're already thinking about the problem and the agent can read the code now, against the repo as it currently is rather than as it was when you filed.
Make the agent read before it asks. Questions that arrive with file paths attached are answerable in a sentence. Questions that arrive cold are homework.
Scale the output to the work. A small change wants an implementation-plan comment on the issue itself. A feature wants a numbered document — mine live at docs/specs/FEAT-NNNN-<slug>.md, and get their own PR ahead of the implementation.
Write it back where the issue lives. This one I learned by losing it. Enrichment that stays in the session dies with the session — the issue still reads like three sentences a month later, and the next pickup starts from nothing. FEAT-0007 survives because it was committed; plenty of equally good reasoning hasn't.

And the honest limits. This costs about half an hour of agent time and a handful of real
decisions from you, which is absurd overhead for a typo and roughly break-even for a
one-file fix. It leans hard on the agent being able to read everything relevant — the
insight that reversed my issue came out of a sibling repository, and an agent without that
repository would have cheerfully specified the parser. And it can't manufacture intent: if
you genuinely don't know what you wanted, the interview surfaces that rather than solving it.
Which is still better than finding out from the diff.

None of this is a claim that issues should be specifications. Mine aren't and won't be. The
issue is a pointer, written in the two minutes I could spare, and it's the right artifact for
that job.

The interview is just the step that dereferences it — the one I used to run silently, in my
own head, without ever noticing I was running it at all.

Applicable to Coding Agents and Engineers

None of the above is specific to working with coding agents. The agent-led interview that spawned the spec doc is recognisably a tech review with a peer, team lead or stakeholder.

AI assisted software development is not so much about learning entirely new paradigms as it is about having the immediate opportunity to enlist the expertise you need to do a job well.

DEV Community: Philip Shaw

Sentinel: The Command Plane

Desired and reported are separate points

Command state machine

Confirmation modes

Deliverability and re-drive

Arbitration

Safety bands

Which band a claim gets

Fail-safe versus fail-clear

Confirmation mode constrains what an interlock can protect

The intent API

Validation before the wire

Idempotency

What this buys the rules engine

Exit criteria

Sentinel Dev Diary: Auditing the Watchdog

Half a sentence

The fix was wider than the sentence

Then the document moved

Criteria that passed without a core

Wiring it found a collision

The stale point whose checkpoint never moved

Where the document changed

The fan-out

The propagation table

Flap damping

The wheel

A seam review, pointed back

An escape hatch that exists in one document

Two columns the vocabulary never had

What the step has in common

Sentinel: The Watchdog and Derived Points

Watchdog

When staleness is suppressed

Derived points

Windows are bounded by the retention of their inputs

Rehydration is the cold-start budget

The restricted expression grammar

The property that keeps the rules engine small

Exit criteria

Sentinel Dev Diary: Checks & Balances

A setting nothing read

What found it

Five instruments, five tenses

The one that points at the code

Proved by, or unverified

It rotted in two days

Different ways to rot

Sentinel: Ingest Path and Log Schema

Two logs, not one

Schema

Where ingested_at comes from

The registry is versioned data

Ingest pipeline

Projection

Restart

Reprocess

The horizon is per class, and partial reprocess is refused

Retention and rollups

Exit criteria

Sentinel Dev Diary: Where the Check Lives

The contract said both, and the code believed the one with a code block

The audit did not mention it

The step went amber for a day

Three checks, and one place the third can go

The register said otherwise, for three pull requests

Two smaller ones, and they rhyme

What the shape is

Sentinel: The Driver Contract

Process model

Assignment, not configuration

Lifecycle

Bus subjects

Stream limits

Northbound: the observation

Push and poll are the same thing

Buffering

Availability

Southbound: commands

Where `ingested_at` comes from

The register had no row for `addresses`