DEV Community: Philip Shaw

Sentinel: Ingest Path and Log Schema

Philip Shaw — Wed, 23 Sep 2026 10:33:04 +0000

Last week's sensor is still reporting 71.6 in Fahrenheit, and its driver has published that number untouched, as the contract requires. The conversion to Celsius happens here instead, at the eighth of thirteen stages between the bus and the database, and the row it writes keeps both numbers: 22.0 in the canonical column, and 71.6 in the native one beside the unit the device actually used.

Keeping both is what makes a conversion reversible. Suppose that weeks later the descriptor's scale factor turns out to have been wrong. Nothing collected in the meantime is lost, because the number the device produced was never overwritten; a reprocess recomputes every canonical value from native under a corrected registry version, rebuilds the current state of the affected points, and refreshes the aggregates built over them. It can only reach rows that still exist, and it refuses outright rather than correcting the part of a request it can reach. What it will not do is re-run the rules that fired on the wrong numbers, because those decisions were made on the data as it stood.

The other load-bearing decision is smaller and much easier to get wrong: where ingested_at comes from. Timescale requires the partitioning column in every unique index, so the dedup key cannot be just the three columns that identify a message; it has to carry the timestamp as well. Stamp that from the core's clock at processing time, and a message redelivered after a crash arrives with a new timestamp, slips past the index and inserts a duplicate, while the index goes on looking as though it enforces dedup. Take it from the bus instead, where it is fixed once and identical on every redelivery, and the same index turns at-least-once delivery into exactly-once effects without a watermark table of its own.

Those two decisions are the spine of this document. The two logs, the per-class hypertables, the thirteen stages and their ordering rules, the restart sequence and the reprocess command are what the rest of the schema has to look like once both are held fixed.

What this document owns: the split into an observation log and a control-event log, the observation schema and its physical split into one hypertable per retention class, where ingested_at comes from and why, the versioned and content-addressed registry, the thirteen-stage ingest pipeline and its three ordering rules, the in-memory projection and its checkpoint, the restart sequence, reprocess and its refusal to apply partially, and the properties of the continuous aggregates that belong with the schema rather than the policy. What it deliberately does not own is how long anything is kept, how each class rolls up and how state-class compaction works, which belong to Retention & Compaction; keeping cold-start rehydration bounded, which belongs to Watchdog & Derived Points; the core epoch, restarting against a backlog and the utilisation arithmetic the ingest ceiling feeds into, which belong to Core Runtime; and who a principal is and why a wrong_instance count is an alert rather than a dashboard figure, which belong to Security Model. Everything downstream that asks what a point's value is, or was, is reading a projection of the two logs defined here.

Two things drive the whole design here: the bus message's own identity doubles as the dedup key and the ingest watermark, and value_native in the log is what makes conversion reversible. Get those right and restart and reprocess both fall out.

Two logs, not one

The pure event-sourcing instinct is one ordered log. It is the wrong shape here. Observations and control events have different volumes, retention, and query shapes, and Timescale hypertables want homogeneous rows.

observation — high volume, append-only, compressed, aged out. Ordering matters only within a point. Physically several hypertables, one per retention class; see below.
control_event — commands, availability, quality transitions, assignment changes, reprocess operations. Low volume, globally sequenced, kept forever, not classed.

The projection is built from both. Quality transitions from the freshness watchdog go in control_event because they carry no new value; in a healthy system they are rare, and when they are not rare that is the thing most worth a permanent record.

Schema

CREATE TYPE quality AS ENUM (
  'unknown','restored','live','stale','unavailable','bad','assumed');

-- identical definition for each retention class:
--   observation_measurement, observation_state, observation_diagnostic
CREATE TABLE observation_<class> (
  ingested_at       timestamptz  NOT NULL,   -- bus-accept time; see below
  point_id          text         NOT NULL,
  observed_at       timestamptz  NOT NULL,
  device_time       timestamptz,

  value_num         double precision,   -- canonical
  value_bool        boolean,
  value_text        text,               -- canonical enum member / string
  native_num        double precision,
  native_text       text,               -- raw device representation

  unit_native       text,
  quality           quality      NOT NULL,
  reason            text,
  detail            jsonb,

  changed           boolean      NOT NULL,
  registry_version  integer      NOT NULL,
  driver_instance   text         NOT NULL,
  source_epoch      integer      NOT NULL,
  source_seq        bigint       NOT NULL,
  skew_flagged      boolean      NOT NULL DEFAULT false
);

SELECT create_hypertable('observation_<class>','ingested_at',
                         chunk_time_interval => INTERVAL '1 day');

CREATE UNIQUE INDEX ON observation_<class>
  (driver_instance, source_epoch, source_seq, ingested_at);
CREATE INDEX ON observation_<class> (point_id, observed_at DESC);

Typed columns rather than JSONB for values. JSONB costs space on the highest-volume table and blocks index-only scans; five nullable narrow columns cost less, with only one or two populated per row.

One hypertable per retention class. Timescale attaches retention, compression and continuous-aggregate policies to a table rather than to a row, so per-class policy means a physical split. The class comes from the resolved descriptor, which ingest already has by the time it writes. The policies themselves are in Retention & Compaction; what belongs here is that the split exists, that the schema is identical across classes, and that it cannot be deferred — adding the class field later is easy, splitting a populated table is a rewrite of everything ever collected.

An observation view over the three, as UNION ALL, exists for ad-hoc querying. The hot paths do not use it: rehydration and reprocess resolve the class from the descriptor and query the specific table.

Chunk on ingested_at, not observed_at. A driver host replaying a buffered partition writes rows with old observed_at values, which would land in chunks that are already compressed. ingested_at is monotonic by construction, so chunks close cleanly and never reopen. The cost is that analytical queries filtering on observed_at lose chunk exclusion, which the secondary index mostly covers at these volumes.

Every observation writes a row, including one that changed nothing. Suppressing unchanged rows at write time is a tempting optimisation and it breaks two mechanisms: derived-point windows rehydrate by querying these tables, so mean, count and min_samples need the samples that were suppressed; and reprocess can only reconvert rows that exist. Volume is managed afterwards, by retention and compaction, where the trade is explicit and bounded.

Where `ingested_at` comes from

It is the JetStream message timestamp, not now() at processing time. This is load-bearing and easy to get wrong.

Timescale requires the partitioning column in every unique index, so the dedup index is necessarily (driver_instance, source_epoch, source_seq, ingested_at) rather than the three-column key that would otherwise be correct. If ingested_at were assigned when the core happened to process the message, a redelivery after a crash would carry a different value, land outside the index, and insert a duplicate row. The index would look like it was enforcing dedup and would not be.

The bus timestamp is fixed when the message enters the stream and is identical on every redelivery, which makes the four-column index behave exactly like the three-column one. It also keeps the monotonicity that chunking depends on, because a single obs stream assigns timestamps in stream order. If observations ever arrive from more than one stream, that property weakens and this decision needs revisiting.

The cost is a change of meaning: ingested_at is when the bus accepted the observation, not when the core recorded it. That is the more useful of the two anyway — it is the moment the observation became durable, it comes from one clock rather than from whichever core process handled it, and it is stable across a core restart. Core processing lag belongs on a self point, not on a column of the highest-volume table in the system.

CREATE TABLE control_event (
  seq          bigserial PRIMARY KEY,
  occurred_at  timestamptz NOT NULL,
  kind         text        NOT NULL,   -- quality_changed, command_*, availability, assignment, reprocess
  point_id     text,
  subject_key  text,                   -- driver_instance / device / component
  principal    text        NOT NULL,   -- who caused it; see Security Model
  core_epoch   integer     NOT NULL,   -- which core instance; see Core Runtime
  payload      jsonb       NOT NULL,
  registry_version integer  NOT NULL
);

principal and core_epoch are both NOT NULL deliberately. A code path that cannot name who asked, or which core acted, cannot write a control event — which means it cannot change state.

The registry is versioned data

Vocabularies and point descriptors load from git into Postgres. Every registry load creates a version row, and descriptors are stored per version by content hash:

CREATE TABLE registry_version (
  version    integer PRIMARY KEY,
  loaded_at  timestamptz NOT NULL,
  git_sha    text        NOT NULL,
  content_hash text      NOT NULL
);

CREATE TABLE descriptor_blob (
  hash       text PRIMARY KEY,          -- of the canonicalised descriptor
  descriptor jsonb NOT NULL
);

CREATE TABLE point_descriptor (
  version   integer NOT NULL REFERENCES registry_version,
  point_id  text    NOT NULL,
  hash      text    NOT NULL REFERENCES descriptor_blob,
  PRIMARY KEY (version, point_id)
);

The indirection exists because of scale. At a few hundred points a full descriptor copy per version is a rounding error; at the four thousand points targeted in Core Runtime it is around 1.6 MB per commit, growing without bound, when a typical commit changes a handful of points. Content-addressing collapses that to one small row per point per version plus one blob per genuinely new descriptor.

It preserves the property that matters: any historical observation can be reinterpreted under the exact descriptor that produced it, or under any other version, because every version still names a complete descriptor set.

Ingest pipeline

Stages, in order, all core-side:

Consume from the JetStream durable consumer on obs.>, taking ingested_at from the message
Validate envelope — required fields, parseable timestamps, observed_at not absurdly in the future
Dedup on (driver_instance, source_epoch, source_seq) against an in-memory set of recently seen keys; this is a fast path, and the unique index is the authority
Resolve point — registry lookup; miss goes to quarantine and the message is acked
Check publisher — the host and instance assigned to this point must match the ones the message was published from; a mismatch goes to quarantine with reason wrong_instance and the message is acked
Unit check — driver-supplied unit_native disagreeing with the descriptor gives quality bad, reason unit_mismatch, no conversion attempted
Validate value — range check, enum codebook lookup; failures give bad with the offending value preserved in native_text
Convert — value_native × factor + offset, affine-aware
Skew check — device_time against observed_at, sets skew_flagged
Evaluate change — deadband against the in-memory projection, sets changed
Append to the hypertable for the descriptor's retention class, batched, as INSERT ... ON CONFLICT DO NOTHING ... RETURNING
Commit, then ack the JetStream messages
Update projection and publish transitions to state.{point_id}, for returned rows only

Stage 5 is what makes bus permissions mean anything. Subject scoping confines a host to publishing under its own prefix, but says nothing about which point IDs it puts inside those subjects. Without this check, one compromised driver host could publish readings for every point in the system, including the inputs to thermal interlocks, entirely within its permissions. It is one comparison against the assignment mapping the registry already holds.

The result is a security signal rather than a data-quality one. A unit_mismatch is a configuration error; a wrong_instance is a host claiming points that were never assigned to it, and the count belongs in the alerting path rather than on a dashboard. See Security Model.

Stage 11 batches per class, since a batch has to target one table. That is a grouping in the flush, not three pipelines — a batch is partitioned by class immediately before the insert and committed in one transaction.

Stages 2 to 10 are the throughput ceiling, not the insert. Per-observation validation, registry lookup, codebook resolution and conversion all happen once per row in application code, and that is where sustainable ingest rate is actually determined. It is the unmeasured term in the utilisation arithmetic in Core Runtime, and it is worth benchmarking against the design rate before sizing anything that depends on it.

Three ordering rules that are easy to get wrong and expensive to debug:

Ack after commit. The consumer's ack position is the ingest watermark. Crash between commit and ack, and the redelivered messages carry the same (instance, epoch, seq, ingested_at) and are swallowed by the unique index. Crash before commit, and they are redelivered and processed properly. This produces exactly-once effects from at-least-once delivery without a separate watermark table — but only because ingested_at is deterministic. If that ever changes, the whole property goes with it.

Publish after commit. Rules must never fire on data that could roll back. A relay that clicks because of an aborted transaction is a bad afternoon.

Project only what actually inserted. ON CONFLICT DO NOTHING makes a redelivered batch a no-op in the table, but the projection update and the transition publish are not idempotent in the way that matters: re-publishing a transition re-fires the rules engine on an edge that already happened. Stage 13 is driven from the RETURNING set rather than from the batch submitted. This is the redelivery case doing its job silently, and it is invisible in testing unless a core is crashed mid-batch on purpose.

Inserts are batched — multi-row INSERT flushed on 500 rows or 100 ms, whichever comes first. Per-row inserts will not keep up at this scale, and the batch window is invisible next to the polling intervals.

Failure at stages 6–8 still writes a row. A bad observation is data — it refreshes freshness, it feeds bad-data rule triggers, and it is the evidence when working out why a sensor went strange at 3am. Silence is the one outcome that helps nobody.

Failures at stages 4 and 5 do not write a row, because there is no descriptor to write one against, and therefore no class to route to. Both go to quarantine with the full native payload.

Projection

The authoritative copy is in memory in the core. Postgres holds a checkpoint:

CREATE TABLE point_state (
  point_id        text PRIMARY KEY,
  value_num       double precision,
  value_bool      boolean,
  value_text      text,
  quality         quality     NOT NULL,
  reason          text,
  observed_at     timestamptz,
  ingested_at     timestamptz,
  changed_at      timestamptz,          -- last value change past deadband
  quality_since   timestamptz NOT NULL,
  watermark_seq   bigint                -- last control_event applied
);

Write policy: immediately on quality change, at most every 5 s per point on value change, plus a full flush every 30 s. Per-observation UPDATE on a hot table generates MVCC bloat for no benefit — the value churn does not need durability because the log already has it. Quality changes are rare and operationally important, so those go through synchronously.

The value in point_state is never nulled when quality degrades. Consumers get (value, quality, observed_at) and decide.

Restart

Load vocabularies and current registry version
Load point_state, force every quality to restored (except rows already unknown)
Resume the JetStream consumer from its ack position; replay anything the checkpoint missed
Replay control_event above each point's watermark_seq
Rehydrate derived-point windows by querying the appropriate class table for the required lookback
Rehydrate durable timers from control_event
Start drivers; instances go connecting → connected → ready
Rules leave warmup as their inputs reach live, or when their warmup deadline expires

Step 3 can be blunt precisely because of the dedup key: anything already committed is re-consumed and discarded, so the consumer resumes from a conservative position without the core having to reason about how far it got.

Step 5 is the quiet payoff of log-first. A 15-minute moving average needs no separately persisted ring buffer — it is a query. Derived points become stateless to restart, which removes an entire category of subtle post-restart wrongness. It is also the step that dominates cold start, and Watchdog & Derived Points covers how to keep it bounded.

Step 2's blanket restored is deliberately pessimistic. Something that was bad before the restart might be fine now, and the driver will say so within one poll interval.

Where the restart follows an outage long enough to have built a backlog, steps 3 onward run against a draining queue rather than a current one. What that changes is in Core Runtime.

Reprocess

value_native is never rewritten. That is what makes this possible:

reprocess --point-glob 'boiler.*' --from 2026-06-01 --registry-version current

Recompute canonical values from native under a chosen registry version, update the affected rows, rebuild the projection for those points, refresh every dependent continuous aggregate over the affected range, and write a reprocess entry to control_event recording the scope, the old version, the new one, and the aggregate ranges refreshed.

ingested_at is never rewritten either. Reprocess changes what a row means, not when it arrived, and the dedup key has to keep pointing at the same message.

Refreshing the aggregates is part of the operation, not a side effect of it. Timescale records an invalidation when the underlying rows change, but the background refresh policy only revisits its own recent window — typically the last day or two. A correction to June's data leaves an invalidation nobody will ever process, and the aggregates keep serving the old numbers indefinitely while the raw table is right. That divergence is worse than the original error, because the raw table and the dashboard now disagree and nothing says so.

So the reprocess command calls refresh_continuous_aggregate explicitly for each aggregate over the affected range, after the row updates commit. Two details:

Round the range outward to bucket boundaries. A range that starts mid-bucket produces a partially recomputed bucket, which is a worse artefact than either the old or the new value.
The refresh is point-blind. Aggregates bucket by time, not by point glob, so refreshing recomputes every point in those buckets, not just the ones reprocessed. That is more work than the reprocess itself and it is the right trade: filtering by point would mean maintaining a parallel invalidation map, and the whole operation is rare.

The horizon is per class, and partial reprocess is refused

Reprocess can only reach rows that still exist, and raw retention differs by retention class. A point glob can therefore span classes with different horizons, and a range that is valid for a measurement point may be long gone for a diagnostic one.

The command validates the whole scope first and refuses outright if any point in it cannot be covered, reporting which points and which classes made it impossible. It does not correct the part it can reach.

That refusal is the important behaviour. Silently correcting two thirds of a glob leaves the system in a state where some points are right, some are wrong, and nothing distinguishes them — which is precisely the divide-and-be-wrong failure the operation exists to prevent. Narrowing the glob or the range is then a deliberate second command, taken with the horizon in view.

The practical consequence: a unit error found within a class's raw retention is fully correctable, and one found after that is permanent in the aggregates. If a horizon is wrong, the retention policy is the thing to change, not this.

Compressed chunks need decompress-update-recompress, which is slow, but this is a maintenance operation run rarely. It is built as a real command with a dry-run mode from the start, while the surface is small. Dry run reports the row count, the classes in scope with their horizons, the aggregate ranges it would refresh, and a time estimate, because the refresh is usually the expensive half.

Reprocess does not re-run rules. Historical automation decisions were made on the data as it stood, and rewriting them would make the control event log a lie.

Retention and rollups

The continuous aggregates are defined here as schema; how long anything is kept, which functions each class rolls up with, and how state-class compaction works are all in Retention & Compaction.

Three properties of the aggregate definitions belong with the schema rather than the policy:

Store sum and count, not avg. An average is a view expression over the two. The moment anything rolls an hourly bucket up to a day, or a minutely bucket up to an hour, an average of averages is wrong wherever the bucket counts differ — and it is wrong quietly, by a few percent, in exactly the direction nobody checks. Sum and count compose; averages do not.

Aggregates are defined directly on the raw class tables, never hierarchically. Building the hourly on the minutely would be cheaper to maintain and would introduce a refresh ordering dependency that reprocess has to get right every time. Defining both on raw costs some materialisation work and removes the ordering question entirely.

agg_safe is enforced in the aggregate definition, not only in derived points. An hourly mean of RSSI in a continuous aggregate is exactly as wrong as one computed live, and much harder to notice once it is sitting in a dashboard.

Only rows where quality = 'live' are aggregated, and live_count is carried alongside count so a gappy hour is visibly gappy rather than silently averaged over three samples.

Deleting data by age happens in Postgres, under the policies in Retention & Compaction. The observation stream on the bus is bounded by size instead, for the reasons in Driver Contract.

Exit criteria

This is the step whose claims are most expensive to discover are false, and most of them are otherwise only prose. Prose cannot fail loudly. Each of the following is a test that runs in CI with a fault injector, not a procedure performed by hand once.

Exactly-once effects from at-least-once delivery. Kill the core mid-batch, restart, and assert two things: no duplicate rows, and no transition re-published for a row that was already committed. The second half is the one that gets missed, because the table looks correct either way — it is the rules engine that would have fired twice.

Both crash windows, separately. Crash before commit: the messages are redelivered and processed normally. Crash after commit but before ack: the redelivered messages are swallowed by the unique index and produce no projection update. If the second case ever starts inserting duplicates, ingested_at has stopped being deterministic and the whole property has gone with it.

Publisher check. An observation for a point assigned to a different host or instance is quarantined with reason wrong_instance, no row is written, and the message is acked.

Class routing. One point per retention class, each landing in its own hypertable; a count over the UNION ALL view equals the sum of the three. A point whose class changes is not silently split across two tables.

Reprocess refuses rather than partially applies. A dry run over a glob spanning classes, with one point out of horizon, refuses and names the offending points and classes. Assert that no rows were changed and no aggregate was refreshed.

Reinterpretation under an older registry version reproduces the original canonical value from native, which is the property that makes the whole log-first argument true.

Two measurements to record rather than assert, because their thresholds are open:

Sustainable ingest throughput against the design rate, measured with synthetic load through stages 2 to 10 rather than against the insert alone.
Cold-start duration, broken down by startup step.

On Monday - the Dev Diary. Not one finding this time but the whole apparatus: the five instruments that have grown up around the specification, and the different thing that keeps each of them honest.

Then the Wednesday after - The Watchdog and Derived Points. One timer wheel for every freshness deadline, why a sensor that only reports on change cannot be told apart from a dead one unless it is made to heartbeat, and how the restart step that dominates cold start is kept bounded.

Start of the series: An Introduction. The map, the two decisions every later document is downstream of, and why a specification at this scale is being published in public while the system it describes gets built.

Sentinel Dev Diary: Where the Check Lives

Philip Shaw — Mon, 21 Sep 2026 08:10:58 +0000

Build step three is the driver contract: a shared conformance harness, one push module and one poll module, and the host process that supervises them. It was audited a fortnight ago against the documents that own it, and the first of those, The Driver Contract, went out last Wednesday.

Entry two was about silence. A specification that described things carefully, at length, and then never said whether they were required.

This one is the opposite failure, and I did not see it coming. The document is not silent. It says the thing. It says the thing twice, in two places, and the two do not agree.

Everything below is that same shape. Not a false statement about the system; a true statement in the wrong place, or two true statements that cannot both be the contract.

The contract said both, and the code believed the one with a code block

From Push and poll are the same thing:

The driver module implements one method: emit observations. Whether that is an MQTT callback or a poll loop is internal.

From Conformance, eighty lines further down, where the interface is printed:

read(points) → observations

Those are two different contracts. The first says a module emits, on its own initiative, whenever it has something. The second says a host asks and a module answers.

The driver host built the second. I would build the second again from the same document, and the reason is not subtle: it is the one with a code block beside it. A signature beats a sentence, every time, and it beats it silently.

So a push module latches the latest frame per point, and the host samples that latch on the assignment's interval; thirty seconds where a point declares nothing.

For a level reading that is fine. A temperature that arrives at an awkward moment gets read at the next tick, and it is still the temperature.

For a transient it is not fine at all. A button press. An alarm that raises and clears itself. A notification with no status counterpart. Those complete inside one interval and are never published. Not delayed; never published.

And the document has a sentence about exactly that, which it can no longer support. Buffering says of the bounded disk queue that

this is the single place in the whole path where data is dropped by design.

The latch is a second place. It has no drop count, and it has no self point. So the problem is not that a sentence went out of date; it is that the guarantee the sentence makes, that data is never lost quietly, is defeated by the other reading of the same document.

The audit did not mention it

This is the part that stings.

build-step-3.md is the audit of the step that built that interface. It runs to twenty thousand characters, it is unsparing elsewhere, and it does not mention the choice at all.

Not because anything was hidden. Because an audit asks whether the exit criteria pass, and "which of the document's two readings did you implement" was never an exit criterion. There was nothing there to fail.

Which means the pull shape has been load-bearing since step three with nothing behind it. No decision row. No open item. No line anywhere saying a fork had been reached and a side taken.

It is three rows now, because it is three different kinds of statement: the gap in the document, the reading the code took while it stood, and the build that closes it. Splitting them is not bureaucracy. The gap is discharged by the document saying which reading is the contract and by nothing else; the reading is history the moment the build lands; the build is work somebody has to do. One row would have to be wrong about two of those.

The step went amber for a day

The amendment landed over the weekend, and it brought four new exit criteria with it. None of them passed.

So step three, complete since a fortnight ago, went back to amber.

Nothing regressed. No code changed. Every test that passed before the amendment passed after it. The document simply started asking for more than had been built, and a step is finished when its owning document's exit criteria pass.

It was green again a day later, with the delivery path built and both hosts in the field running it.

I want to be precise about why that yo-yo is the machine working rather than the machine embarrassing itself, because a status marker that moves backwards looks like a mistake and this one is the opposite.

The marker follows the definition. The alternative is a marker that follows the work, and that quietly changes what the marker means: it stops recording whether the thing the document asks for exists, and starts recording how much effort has been spent. Those come apart precisely when a document is amended, which is the moment you most need the marker to be honest.

A step going amber because its document grew is not a regression. It is the only truthful thing the marker can do.

Three checks, and one place the third can go

Same shape, smaller blast radius, and it is sitting in the post that went out last Wednesday where anyone can check it. Conformance names

Three additions to the harness that come from later documents and belong in the same suite

and lists epoch rejection, publish scope, buffer bounds.

Only publish scope is in the harness. The other two are the host's, and the suite says so as a matter of design:

The suite is deliberately silent on anything the host owns. Backoff, buffering, epoch fencing, self-points, sequence numbers and subjects never reach a module, so a module cannot get them wrong and the harness does not ask.

That is the right call, and the argument for it comes from the document itself, in the section immediately before. What drivers must never do forbids a module to hold verification state or timers. A module that tracked the highest epoch would be holding a fencing token; a module that owned the durable queue would be holding buffering state. The queue is one per host, behind every instance, so there is not even a coherent thing a single module could be asked to demonstrate.

A check cannot be placed in a suite that runs against an interface which never sees the thing being checked. The code is right. The document put three checks in a place that can hold one of them.

And that is a gap in the document rather than in the code, which decides what happens next: shipping the code that routed around it does not close it. So it was filed against The Driver Contract, with the reading recorded separately so nobody would have to re-derive it meanwhile, and the document was amended a week later to say whose each check is. One of the three is the harness's; two are the host's.

The order that repair happened in is the part worth keeping. The requirements register carried a paraphrase of the original sentence, and the paraphrase was wrong in an extra way of its own: it said the check happens in the harness "as well as core-side", and buffer bounds has no core half at all. It was still left alone while the finding was open, and moved in the same commit as the document rather than ahead of it. Correcting a paraphrase before the thing it paraphrases does not fix anything; it just produces a second answer, and now you have two.

The register said otherwise, for three pull requests

This is the one I keep coming back to.

ROADMAP.md carried a sentence claiming that publish scope and buffer bounds "were module checks in the harness from the start". A docstring in the driver host repository said the same two live under tests/conformance/.

Buffer bounds never did. Not moved, not relocated: never there.

Epoch rejection's move out of the harness was noticed and recorded when it happened. Buffer bounds' move was not, and so a sentence asserting the opposite survived three pull requests, in two repositories, saying something false about the codebase it shipped with.

Entry one ended on the registers being under test. test_register_integrity.py asserts that ids are contiguous and unduplicated, that a requirement is never both a bare row and a set of sub-ids. Fifty-seven of those tests at the time of this audit. I wrote then that pointing the specification's own convention back at the process documents felt faintly ridiculous, and that it had caught things.

It had, and it does. It could not catch this, and the reason is worth saying plainly:

Not one of the fifty-seven can check a claim about where a test lives.

They check the shape of the registers, which is the thing a register can be wrong about on its own. A sentence asserting that a particular test sits in a particular file is a claim about the world outside the register, and the register has no way to reach out and look.

The same audit turned up two stale cross-reference anchors the same way: by a person reading, while all fifty-seven passed.

And those did have a fix available, which is what makes the boundary visible rather than merely assertable. An anchor points at a section inside the register, so a test can follow it and see whether the row is there. One was written, straight after the audit, and it found four more the moment it ran.

A sentence about which file a test lives in points outward, at a repository the register cannot see. There is nothing there for a test to follow.

So the idea holds, and its edge is now drawn, which is more useful than the idea was on its own. A register can be machine-checked for everything that is a statement about itself. Everything it says about the world outside is still something somebody has to go and look at, and the cadence at which that happens is the audit, not the test suite.

Two smaller ones, and they rhyme

The command dedup window is a fixed five minutes, deliberately not derived from the command's own deadline. Sizing it from the deadline is the obvious move and it is actively harmful: it puts the core's clock inside a driver-side decision, so a host running fast forgets a command id early and pulses the relay on exactly the redelivery the window exists to suppress. Wrong in the direction that costs memory, rather than wrong in the direction that moves a physical thing twice. The published copy still leaves the figure open as "a short window"; it is a number now, and fixed on purpose.

Two subjects were built and never called. Availability and quarantine each had a subject builder, an envelope type, and a publish path asserted against a real broker, and nothing in the driver host ever constructed a message for either. A test that proves the machinery works is not a test that proves anything runs it.

Availability has both ends now. The core subscribes at startup and narrows each envelope to what the declaring instance actually owns, and a driver declares on exactly two edges: losing the transport while it is serving, and stopping deliberately. Quarantine was lifted out of that row rather than carried along inside it, because its producer end turns out not to be buildable yet at all, and a resolved row still quietly carrying half its subject is a row nobody ever comes back for. That is entry two's distinction between deferred and forgotten, met again from the other direction: the danger this time was not an absence with no row, but a row marked done with an absence still in it.

What the shape is

Entry two found one root cause behind three symptoms. This step has a different unity, and it took me a while to see it.

Every finding is about placement. Which of two sentences is the contract. Which side of a process boundary a check belongs on. Which file a test is in, as asserted by a document that cannot see the file. Whether a built path has anything calling it.

None of that is the kind of error a test catches, because a test runs the code, and these are all statements about where things are.

And the drift check, which reads the code against the document and is the thing I would have expected to catch all of it, only works from one side. It finds where the code departs from the prose. When the prose departs from itself, both halves are documented, both are defensible, and the code picks one without recording that it picked.

The honest summary of the step: the specification was wrong in three places, none of them about the system, all of them about itself.

This Wednesday - Ingest Path & Log Schema. The specification's fourth part: what an observation does between arriving on the bus and being answerable, and the log that the current state is only ever a projection of.

Next Monday - the hole entry one ended on has been filled. Nothing reads the documents against each other was an open question a fortnight ago. It is a procedure now, and the first run of it broke its own rule on purpose. That entry waits for the documents it found things in to be published.

Start of the diary: Four Registers and a Drift Check. Why the registers exist at all, and the one rule about where a sentence is allowed to live.

Sentinel: The Driver Contract

Philip Shaw — Wed, 16 Sep 2026 14:31:10 +0000

A temperature sensor reports 71.6, and it means Fahrenheit. Meanwhile, the canonical unit for temperature in the system is Celsius. It is a simple conversion: a subtraction and a multiplication, and the driver bringing the sensor's reading into the system has everything it needs to perform it.

But it must not convert. What it publishes is 71.6, tagged with the unit the device actually used: the conversion happens on the far side of a wire boundary.

Two things break if the conversion happens at the driver. The number the sensor actually produced stops existing anywhere, so the reading cannot be reproduced once the unit tag or the scale factor turns out to have been wrong - and on a long enough timeline one of them is bound to be. The other is that the driver becomes a second place where the meaning of a point gets decided, when the registry is supposed to be the only one. A protocol adapter running on a Raspberry Pi in a cupboard is a poor place to keep half a semantic model.

Converting a unit is the first item on a list of eight things a driver is forbidden to do, and that list is the spine of this document. Everything else here - the process model, the assignment export, the subject layout, the buffering rules, the deliberately narrow command contract - is what remains once protocol ugliness has been confined to one side of that line and meaning to the other.

What this document owns: the driver host and instance process model, the assignment export and why it is pushed rather than queried, the instance lifecycle, the bus subject layout and the limits on the observation stream, the driver-supplied half of the observation envelope, local buffering and the one place in the path where data is dropped by design, availability declarations, driver-side command handling, discovery and quarantine, the prohibitions list, and the conformance harness every module is tested against. What it deliberately does not own is anything about what a value means, which belongs to The Semantic Model; command verification and the epoch whose token the driver checks, which belong to Core Runtime; and the permissions that turn the subject layout into an authorisation surface, which belong to Security Model. The boundary drawn here is what lets all three assume a device looks the same whatever protocol it speaks.

The driver contract has one job: make every device look identical to the core while keeping all protocol ugliness on the far side of a wire boundary. Everything below follows from drivers being dumb by design.

Process model

Driver host is the deployment unit — one container or systemd unit per network segment. It loads driver modules and supervises driver instances, one per configured device or device group.

driver host "rpi-c1"
├── modbus-tcp    → instance "boiler-relay"
├── mqtt          → instance "esp32-fleet"
└── ble           → instance "ble-local"

Supervision is per instance. A wedged Modbus poller must not take down MQTT ingest in the same host.

One northbound transport, for every protocol without exception. Every driver host reaches the core the same way: over the bus, in the observation envelope below. A second northbound path for any one protocol buys two failure models, two buffering implementations, and two places to fix backpressure, in exchange for convenience in a single module.

The temptation is strongest for BLE, because the radio work genuinely is bespoke — supervision, retry-on-connect, per-device connection state — and it is easy to let that bespokeness leak upward into how the host talks to the core. It stays below the line. Whatever a BLE host has to do to hold a connection is its own business; what it emits is an observation like any other.

Assignment, not configuration

Drivers must not read the registry, but they need to know what to poll. The registry exports assignments rather than having drivers query it.

On startup a driver host requests its assignment from the core and caches it to local disk. If the core is unreachable, it starts from cache. An assignment carries addressing only:

driver_instance: boiler-relay
module: modbus-tcp
transport: { host: 10.0.4.12, port: 502, unit_id: 3 }
points:
  - point_id: boiler.ch1.flow-temp
    read:  { fn: input_reg, addr: 30001, type: int16, scale: 0.1 }
    poll:  { interval: 30s }
  - point_id: boiler.ch1.pump
    read:  { fn: coil, addr: 1 }
    write: { fn: coil, addr: 1 }
    idempotent: true

No quantity, no unit, no subject, no freshness policy. The driver sees a point ID, an address, and a scale factor. scale is a raw-register-to-native adjustment, not a unit conversion — a Modbus register holding tenths of a degree is a wire encoding, not a unit.

Credentials are referenced, never carried. Where a transport needs one — an MQTT broker password, a vendor API key — the assignment names it:

transport: { host: mqtt.lan, port: 8883, credential: mqtt/fleet }

The driver host resolves that reference against its own local store, provisioned out of band. The bus never carries a secret and neither does the on-disk assignment cache, which matters because the cache sits on every host and the assignment subject is readable by anything that can subscribe to it. See Security Model.

Assignment changes arrive as a push on the bus with a version number. The driver applies them without restarting: new points start polling, removed points stop, changed addresses rebind. This is also the load-shedding lever — raising poll intervals across a fleet is an assignment change that takes effect in seconds, which is the manual response to sustained ingest lag described in Core Runtime.

Lifecycle

init → connecting → connected → ready → degraded → connecting
                                  ↓
                              stopping → stopped

connecting — transport not established, exponential backoff with jitter, capped around 60s
connected — transport up, initial reads in flight
ready — initial read pass complete, or explicitly skipped for points with no readable state
degraded — transport up, some points failing; instance stays up and keeps serving the rest

The connected → ready transition is what makes cold start work. Until an instance is ready, the core keeps its points in restored or unknown rather than promoting them, rule warmup gates on it, and the command plane treats every claim on its points as undeliverable rather than sending into the dark.

Instances publish their own lifecycle as points under subject: self — driver.boiler-relay.state, .reconnects, .poll-latency, and the queue and rejection counters below. Driver health is not a separate observability channel; it is the same pipeline, with the same freshness and alerting. Driver monitoring comes free, and the ingest path is exercised constantly as a side effect. The full catalogue of what a driver must publish is in Observability.

Bus subjects

obs.{host}.{instance}.{point_id}         observations
avail.{host}.{instance}                  device/component availability
quarantine.{host}.{instance}             unregistered points
cmd.{host}.{instance}.{point_id}         commands, core → driver
cmdres.{host}.{instance}.{command_id}    command outcome, driver → core
assign.{host}                            assignment push, core → host

Every driver-plane subject leads with the host, and that is what makes bus permissions workable: one credential grant per host covers every instance it will ever run, so credentials stop needing to change whenever an assignment does. Two places to keep in step is one too many, and the one that gets forgotten is the credential.

cmdres carries host and instance for the same reason. A subject of cmdres.{command_id} alone would force every driver to hold cmdres.>, and any driver could then forge an outcome for another driver's command. See Security Model.

Commands route by instance rather than by point, because the core knows the binding and the driver should not have to filter a firehose. Each instance subscribes to cmd.{its host}.{its own id}.>.

Stream limits

obs is a single JetStream stream, and it is the system's outage buffer. Three properties, all load-bearing:

Reject new publishes when full; never discard old. A full stream pushes back onto the driver's local queue, which is bounded and reports its drops. Discarding old messages would delete observations that had already been accepted, between two components that both believe the handoff succeeded.
Bounded by bytes and messages, with no max_age. An age limit deletes unconsumed messages silently, which is the same failure by a different route. Age-based deletion belongs only where dropping old data is the actual intent, which here is Postgres retention.
Sized from tolerable outage and utilisation, not from a duration that feels sufficient for a restart. The arithmetic is in Core Runtime, and the size cannot be computed until capacity targets exist.

Durability lives in Postgres. The stream exists so that a core outage costs nothing, not as an archive.

Northbound: the observation

Exactly the envelope from the semantic model:

point_id, value_native, unit_native?,
quality: live | bad | unavailable,
reason?, detail?,
device_time?, observed_at, source_seq,
driver_id, driver_instance, assignment_version

unit_native is populated only when the source self-describes — Shelly and UniFi JSON do, Modbus does not. When present it drives the mismatch check; when absent the registry is trusted.

source_seq is monotonic per instance and resets on restart, with the restart marked by an epoch counter. (epoch, seq) is what the core dedups on.

A driver may publish only for points its own assignment names. The core enforces this rather than trusting it — see the publisher check in Ingest Path & Log Schema — because subject permissions confine a host to its own subjects but say nothing about which point IDs it puts inside them.

Push and poll are the same thing

The driver module implements one method: emit observations. Whether that is an MQTT callback or a poll loop is internal.

Poll scheduling lives in the driver, never the core, for two reasons: the scheduler must survive a core outage, and it needs to be adjacent to the transport to do jitter, coalescing, and batching. A Modbus driver reading twenty registers should issue one multi-register read, not twenty round trips the core scheduled independently.

Every poll produces an observation. A poll that returns the same value still publishes, because the deadband decision is core-side and freshness needs the refresh. A poll that fails produces quality bad with the reason — not silence.

Buffering

When the bus is unreachable or full, the driver buffers to a local disk queue and replays with original observed_at values intact.

This is exactly why ingested_at exists as a separate field. A driver host that buffers through a twenty-minute partition and then floods must not have those readings evaluated as fresh. The core orders by ingested_at, computes freshness from observed_at, and a rules engine seeing a burst of old readings treats them correctly.

The queue is bounded by size and age. On overflow it drops oldest and publishes a self point recording the drop count — silent data loss is the failure mode nobody notices, and this is the single place in the whole path where data is dropped by design.

Availability

The driver declares unavailability; the core fans it out.

avail.{host}.{instance}: {
  scope: instance | device | component,
  target: "boiler" | "boiler.ch1",
  available: false,
  reason: transport_timeout,
  observed_at
}

The driver names the scope and the core marks every affected point unavailable. This is the difference between "the cause of the silence is unknown" (stale) and "the device is off the network" (unavailable), and only the driver can tell which.

A driver that dies without saying anything is caught by its own heartbeat going stale — the core marks the instance unavailable after a grace period. Both paths converge on the same state.

Southbound: commands

command_id, point_id, value, issued_at, deadline, idempotent, core_epoch

The driver's contract is narrow. It attempts the write and replies once:

command_id,
outcome: accepted | rejected | failed | timeout,
device_ack?: { value, observed_at },
reason?, detail?

accepted means the transport accepted it. device_ack is populated only when the protocol returns confirmation inline. Everything else the core owns.

core_epoch is a fencing token. The driver tracks the highest epoch it has seen and rejects any command carrying a lower one, with outcome rejected and reason stale_epoch. It needs to understand nothing else about the value, only that it never goes backwards. The reasoning is in Core Runtime; the effect is that a core which has already been replaced cannot still be driving relays through a partition it has not noticed yet.

Verification is core-side, without exception. poll_verify is the core issuing a read command after a delay. async_event is the core correlating a later observation. optimistic is the core marking the point assumed. The driver has no concept of confirmation modes, no timers, and no memory of what it was asked to do.

That boundary matters because verification needs durable timers and registry knowledge, and because a driver that retries on its own behalf will eventually double-fire a non-idempotent command. Drivers retry transport establishment, never command delivery — unless idempotent: true, which permits one retry within the deadline.

Commands are deduplicated on command_id within a short window, so a bus redelivery does not pulse a relay twice.

Discovery and quarantine

An observation for an unknown point_id goes to quarantine.{host}.{instance} with its full native payload, never to obs.*. The core records it, deduplicates by point ID, and surfaces it for triage. Nothing auto-registers.

This gives discovery as a side effect of normal operation. Connect a new device, let the driver run, and watch what accumulates in quarantine. Promoting a point is a registry commit supplying quantity, subject, unit, and freshness — the four things no driver can guess correctly.

For protocols with real discovery (mDNS, vendor announce messages, device lists), the driver publishes candidates to the same quarantine subject in the same shape. One triage path regardless of how the point was found.

What drivers must never do

Worth writing down and enforcing in review, because each of these is a plausible-looking shortcut that dissolves the boundary:

Convert units, or emit anything but value_native
Assert quality other than live, bad, or unavailable
Invent, rename, or auto-register points
Publish for a point its assignment does not name
Implement debounce, deadband, moving averages, or hysteresis
Retry a non-idempotent command
Hold verification state or timers
Read the registry directly

Conformance

The module interface is small enough to test properly, and with six or more modules to write, a shared harness pays for itself immediately.

connect() → transport
read(points) → observations
write(point, value) → outcome
close()

A conformance suite is built alongside the first driver: a mock transport that produces malformed payloads, unmapped enum values, out-of-range readings, mid-poll disconnects, slow responses that exceed a deadline, and clock skew. Every module runs the same suite. The failure modes are protocol-independent even though the protocols are not.

mqtt and modbus-tcp come first — push and poll, self-describing and not, verifiable and not. Between them they exercise every branch of this contract. Other protocols are then variations rather than new problems.

Exit criteria

The conformance suite above is this step's exit criterion, not a nice-to-have. A driver module is done when it passes the shared harness, and mqtt and modbus-tcp both pass it before anything else is written.

Three additions to the harness that come from later documents and belong in the same suite:

Epoch rejection. A command carrying a core_epoch below the highest seen is rejected with stale_epoch, and the driver does not attempt the write. Tested by replaying an old command after a higher epoch has been observed.
Publish scope. The module publishes only under its own host and instance prefix, and only for point IDs its assignment names. This is checked here as well as core-side, because a driver that gets it wrong should fail its own tests rather than fill quarantine.
Buffer bounds. Fill the local queue past its limit and assert that the oldest entries are dropped, the drop count is published, and nothing is lost silently.

The step as a whole is done when both modules pass, and when an instance taken from connecting to ready to degraded and back publishes the lifecycle transitions that cold start and deliverability depend on.

On Monday - the Dev Diary. The third entry of what the build has had to send back to the specification.

The next part - Ingest Path and Log Schema. Two logs rather than one, and the column that makes the unit conversion a driver is forbidden to do reversible once it happens in the right place.

Sentinel Dev Diary: What the Specification Left Unsaid

Philip Shaw — Mon, 14 Sep 2026 16:52:49 +0000

Build step one is the semantic model: identity, the closed vocabularies, units, quality, retention class. It was marked complete a fortnight ago. Then it was audited against The Semantic Model, which is the document that owns it and which went out last Wednesday, and the audit is the part worth writing about.

Here is the thing I did not expect. The specification was not wrong anywhere. Not once. Every delta between what it says and what got built came from the same place: it described something carefully, at length, with the reasoning attached... and then never said whether it was optional.

Silence is not neutral. Silence resolves, and it resolves to whatever the code does by default.

Six rejections became twenty-eight

The exit criteria name six ways the registry must refuse bad input. app/registry/errors.py defines twenty-eight.

Most of that is not interesting. malformed_point_id, duplicate_point_id, canonical_unit_missing, kind_mapped_twice: these are finer-grained versions of what the criteria already asked for, and they need no justification beyond a load reporting what is wrong rather than that something is. If you are going to refuse, refuse specifically.

Two of the twenty-eight are different. They are new rules, in the sense that the specification does not state them.

A numeric point must declare a deadband (change_policy_required). The descriptor carries change_policy, and the document argues hard for why the deadband exists; without one "a noisy analogue sensor generates an edge per sample and the rules engine spends its life on 0.01 °C". Having made that argument, it never says the field is required.

So what happens if it is absent? Every sample becomes an edge. The exact failure the document spent a paragraph warning about, arrived at by leaving a field out. Booleans, enums and strings still default to any, because a boolean that changes is an edge by construction; but a float with no deadband is a mistake wearing the costume of a default.

A writable point must declare a confirmation mode (confirmation_mode_required). Same shape, worse consequence. There is no safe default here, and that is the whole argument: the mode an unset field would most naturally mean is optimistic, which is the mode that verifies nothing. A default that quietly means "assume the write worked" is not a default; it is a trap with good manners.

Both could have been solved with a default value. I want to be clear about why they were not, because "make it mandatory" is the more annoying choice and it needs defending.

A default is a decision made once, by whoever wrote the schema, on behalf of every point that will ever exist. It is invisible at the point of use. Nobody authoring a descriptor sees the deadband they did not write. Whereas a load-time refusal is a decision made by the person who knows what the point actually is, at the moment they are thinking about it, with a named error telling them precisely which of the twenty-eight things went wrong.

It costs a line of YAML per point. It buys the guarantee that no point in the system has a policy nobody chose.

Turning prose into types found a missing number

The Semantic Model writes its policies as strings: periodic(30s, grace=15s), absolute(0.2), poll_verify(500ms), assume(v). They are readable, they are compact, and they are exactly what Introduction warns against: "anything encoded as a string suffix will have to be parsed later."

So they became discriminated unions in app/schemas/descriptor.py. That is a straightforward application of a rule the specification already made, and it would not be worth a paragraph except for what fell out of it.

Writing the type forced a number the document never gives. periodic has a grace shown as optional, with no statement of what it defaults to. In prose that reads fine; you skim past it. In a typed model you cannot skim past it, because the field either has a default or it does not, and you have to type one or the other.

PeriodicFreshness.effective_grace now resolves it to half the interval, capped at 60 s, at load. Not at every read; the distinction matters, because a value computed at read time is a value that can differ between two readers of the same descriptor.

That is the second-order benefit of turning prose into types, and I had not really appreciated it before this step. The type system does not just enforce the document. It interrogates it, and it asks about precisely the things a careful reader glides over.

Three smaller decisions, one pattern

Units are nested under quantity rather than becoming a sixth vocabulary. The specification names five closed vocabularies and gives the unit table no home of its own, so it could have gone either way. It went inside quantity because the quantity owns its canonical unit and there is no useful reading in which the conversions into that unit live somewhere else. Nesting keeps the count at five as specified, makes unit_native convertibility a membership test rather than a lookup across files, and lets affine: false reject an offset at load; which, as the document itself points out, is where a 5 °C rise mistaken for 41 °F gets caught.

Derived fields are declared anyway, so authoring one is a named refusal. unit_canonical and interlock_protected are both derived. The document says so and says nothing about how a hand-authored one should fail. They are declared on PointDescriptor regardless, so writing one yourself fails as derived_field_authored rather than being swallowed as an unknown key. A typo and a misunderstanding deserve different errors; the first wants a spell-checker, the second wants an explanation.

dBm was kept rather than "corrected" to strict UCUM dB[mW]. The quantity table spells it dBm, which is not a valid UCUM code. It is, however, what every datasheet, vendor API and engineer writes for RSSI. The owning document was followed rather than quietly corrected in code, and the divergence was recorded in the register instead. This one is a small rule with a big blast radius: canonical values are frozen in the log at ingest, so changing that spelling stops being a YAML edit and becomes a migration over the highest-volume table in the system, the moment the first driver publishes.

The pattern in all three: where the document had made a choice, follow it and record the disagreement. Where the document had made no choice, make one and record the reasoning. Never quietly improve a document from inside the code.

What the step did not deliver

This is the half of an audit that earns it. Three findings, and the first two share a root cause: nothing compared registry version N against version N−1 at load.

A guard that existed as a function with a fixture

vocabulary_member_removed_while_referenced is one of the six exit-criteria rejections, and its fixture passed. Green. Ticked.

The check needs a baseline, and only the tests supplied one. Both real call sites passed none: the runtime at startup, and the CI check that runs against the actual registry. So removing heatsink from the subject vocabulary would have failed as unknown_vocabulary_member; a different error, from a different check, that happens to look exactly like the guard working.

That is the bit that took a moment to sit with. The guard was not missing. It was not broken. It was unarmed, and the symptom of it being unarmed was a passing test and a plausible-looking error message.

The fix was not "call it with a baseline". The fix was to make the resolver refuse when it cannot find a previous revision, rather than returning "no baseline" and carrying on. A fallback would have reproduced the original failure exactly where it matters most, which is a shallow CI checkout: no previous revision available, so no comparison, so no guard, so green. The cost is that CI now needs fetch-depth: 0, and the failure message says so in as many words.

A guard that degrades gracefully is not a guard. It is a guard-shaped hole that reports success.

The register had no row for `addresses`

The Semantic Model is the document that places addresses in the registry:

The registry holds addresses separately: MQTT topic, Modbus unit ID and register, BLE MAC, UniFi site path. These change freely without touching identity.

There is no field on PointDescriptor, no file under registry/, and no table. Nothing was built.

And that is defensible, because the document that consumes them is later in the series and the work is genuinely blocked behind it. What is not defensible is that the step's ledger had no row saying so. The gap between "deliberately deferred" and "forgotten" is invisible from outside; both look like an absence. The only thing that distinguishes them is a row somebody wrote at the time.

The transition criterion was delivered in half

The criterion asks that the quality transition table "exists as a checked enum, with a test asserting that no code path can produce a transition outside it".

The first half landed: the full 7×7 matrix is asserted, including the two an implementation gets wrong, which are that unavailable has only two exits and that nothing returns to a startup state. The second half could not land, because at that point nothing in the system wrote a quality at all, and the deferral was written into a docstring honestly.

The interesting part is what happened next. A later step added the first real writer, and a test that it cannot write an illegal transition. What was never added is the structural assertion that no writer bypasses the function; and the docstring still reads as though nothing writes a quality, which is no longer true. So the deferral note itself has rotted, in the specific way that deferral notes rot: not by being wrong when written, but by being right about a world that moved on.

The shape of the whole thing

One root cause produced three symptoms across two build steps, and none of the three looked related while I was in them. That is what an audit buys that a code review does not: not a closer look at a change, but a look at the step from the outside, against the document rather than against the commits.

The step was already marked complete when the audit ran. It stayed complete; nothing found here unmade it. What changed is that "complete" now means something checkable, because the enumeration of what is missing exists and is written down where it can be argued with.

I would rather ship a step with three named gaps than one with none I have looked for.

This Wednesday - The Driver Contract. The specification's third part: making every device look identical to the core while keeping the protocol ugliness on the far side of a wire boundary.

The next entry - Where the Check Lives. The driver contract turned out to say a module emits and to say a host reads, and only one of those was built; plus three conformance checks placed where one of them can go.

Start of the diary: Four Registers and a Drift Check. Why the registers exist at all, and the one rule about where a sentence is allowed to live.

Sentinel: The Semantic Model

Philip Shaw — Wed, 09 Sep 2026 09:08:10 +0000

A UUID is the obvious primary key. It is opaque, globally unique, immune to renames, and every tutorial reaches for it. It is also the wrong choice here, and the reason has nothing to do with database design.

boiler.ch1.flow-temp shows up in a NATS subject, a Postgres row, a rule definition, a log line and a stack trace. Every one of those is a place a person reads under time pressure. f47ac10b-58cc-4372-a567-0e02b2c3d479 is correct in all five and useful in none of them: it moves the question "which point is this?" one join away, and one join away is far enough that nobody makes the trip at eleven at night.

So the identifier is a dotted slug, and it is the primary key rather than a label hanging off one. That choice has a price, and it is paid at rename time and nowhere else. The rest of this document is the same kind of trade made about twenty more times - what a point is, what it is allowed to mean, and what the system is required to admit it does not know.

What this document owns: point identity and hierarchy, the descriptor fields, the five closed vocabularies, units and conversion, the seven quality states and their legal transitions, the three timestamps, and the observation envelope. Everything downstream reads the registry defined here - the log schema, the watchdog, the rules engine and the command plane all resolve against it. It is the first build step for the same reason it is the first document: nothing later can correct a semantic model that was wrong, only migrate away from it.

Identity

point_id is a dotted path: {device}.{component}.{point}, each segment [a-z0-9-]+. This maps 1:1 onto NATS subjects, survives being pasted into logs and grep, and reads correctly in a stack trace.

The slug is the primary key in Postgres, not a UUID with the slug as a label. Stability comes from the ID being assigned at registration and never derived from anything the network can change. A rename is a deliberate migration event, and it should feel like one.

The registry holds addresses separately: MQTT topic, Modbus unit ID and register, BLE MAC, UniFi site path. These change freely without touching identity.

Hierarchy

Device → Component → Point, where Component may be trivial (main) so a one-sensor device does not pay ceremony for structure it does not have.

Component earns its place because it is the unit of independent failure and hot-swap: the four channels of a multi-channel relay, a sensor behind one leg of an I²C multiplexer. When the multiplexer leg goes dead, one component is marked unavailable, not the device.

Points are scalar. If a device emits a struct, the driver explodes it into points. No exceptions — the moment a point can hold a composite, every downstream consumer has to understand payload shapes, which is the coupling this model exists to prevent.

Point descriptor

The declarative half, living in the registry:

id                 boiler.ch1.flow-temp
device_id          boiler
component_id       ch1
direction          in | out | inout
kind               measurement | state | setpoint | command | diagnostic
value_type         float | int | bool | enum | string

quantity           temperature
unit_native        Cel                    # UCUM
unit_canonical     Cel                    # derived from quantity
subject            water
subject_ref        zone:upstairs-ch       # optional, nullable
modifiers          { aggregation: instant }
tags               [ heating, primary-loop ]

valid_range        { min: -10, max: 120 }
precision          1
change_policy      absolute(0.2)
freshness          periodic(30s, grace=15s)
retention_class    measurement            # defaults from kind

# out / inout only
init_strategy      read_on_connect | restore_from_log | assume(v) | assert(v)
confirmation_mode  inline | async_event | poll_verify(500ms) | optimistic
default_value      false                  # optional
require_intent     false
interlock_protected false                 # derived at rule load, not authored

kind and direction are not redundant. kind describes what the value means; direction describes flow. A setpoint is inout; a command is out with no retained value; a diagnostic is in but must never appear in a measurement dashboard or share a freshness policy with real sensors.

default_value and require_intent are the point's half of the arbitration model in the command plane. default_value is optional; when present, the registry materialises a non-expiring claim at the lowest band, which is the resting state the point falls back to when every other claim is released. When absent, releasing the last claim leaves the point alone rather than driving it anywhere.

require_intent: true says the point must never be without a resolved intent, and registry load fails if it has no default_value. interlock_protected is set at rule load rather than authored, and implies require_intent. Between them, a protective point is one that cannot accidentally be left with nothing asserting a state.

Retention class

retention_class is a closed vocabulary — measurement, state, diagnostic — and it decides how the point's data is stored, aged and summarised. The policy behind each class is in Retention & Compaction.

It is rarely authored, because it defaults from kind:

`kind`	default class
`measurement`	measurement
`state`, `setpoint`, `command`	state
`diagnostic`	diagnostic

Two properties belong here rather than in the retention document, because they are facts about the descriptor:

The default is resolved at registry load and stored explicitly. The class determines physical placement, and placement must not change silently because kind was edited for an unrelated reason. What the registry stores is the resolved class, not the rule that produced it.

Changing it afterwards is a migration, in the same category as a point rename: existing rows live in a table chosen by the old class and have to be moved. A deliberate operation, not a config edit.

The semantic axes

quantity is a closed vocabulary, and it owns the canonical unit. One table, one entry per quantity, and the canonical unit is a property of the quantity rather than a per-point decision.

Decision: pragmatic canonical units (°C, Wh) rather than strict SI ones (K, J). Strict would buy clean dimensional algebra — divide energy by time, get power, units fall out. Pragmatic buys rules and dashboards that read without conversion. This system does not do dimensional algebra, so pragmatic is the right choice.

unit uses UCUM codes (Cel, mA, V, W.h, %) rather than invented strings, with a separate display_symbol. UCUM is a real standard with libraries, and it permanently kills the °C / degC / C / celsius ambiguity. The point stores unit_native; the observation stores both native and canonical values.

subject is the class of thing measured: air, water, heatsink, core_body, skin, battery-cell, mains. Also a closed vocabulary.

Subject alone is underspecified. Two air-temperature sensors in different rooms have the same subject, and that difference is location rather than subject. So the axis splits:

subject — the class of thing (closed vocabulary)
subject_ref — an optional typed reference to a specific modelled entity: zone:loft, device:rpi-t2, person:x

subject_ref is nullable and unresolved for now. It is the hook that allows spatial or logical modelling to be added later without reworking every point. Declaring the field costs nothing; retrofitting it costs everything.

modifiers is a small typed map for things that qualify a reading without changing quantity or subject:

current_type: ac | dc
phase: l1 | l2 | l3 | n
aggregation: instant | mean | min | max | sum | delta — this is what distinguishes a daily energy total from an instantaneous power reading, and it is a real semantic difference that must not hide in the point name

aggregation earns its place twice over: it is also what tells the rollup layer that a cumulative counter must not be averaged. See Retention & Compaction.

tags are free-form and exist to hold things not yet formalised. One governance rule, enforced strictly: tags are never load-bearing. No rule, no unit conversion, no aggregation may branch on a tag. The moment one needs to, the tag is promoted to a field. Without that rule, tags become a shadow schema within six months.

Vocabularies are closed

quantity, subject, enum_type, retention_class, and the modifier value sets live in the registry as versioned tables, not free strings. Adding heatsink is a deliberate registry change. This is the only thing that prevents accumulating heatsink, heat_sink, and heatSink alongside a rule that silently matches nothing.

Enums and codebooks

A point with value_type: enum references an enum_type:

enum_type: switch_state
members:   [ on, off, unknown-position ]

Drivers map native representations (Modbus 0/1, Shelly "on"/"off", UniFi status strings) to canonical members. A native value that maps to nothing produces quality bad with the unmapped value in the detail field. It does not invent a member and does not throw.

Quality

Seven states, plus a machine-readable reason and free-text detail:

State	Meaning
`unknown`	never observed
`restored`	loaded from log at startup, not yet reconfirmed
`live`	observed within its freshness window
`stale`	observed, now past its window, cause unknown
`unavailable`	driver or device known-disconnected
`bad`	last attempt gave an invalid or device-reported-error value
`assumed`	written, and unverifiable

stale and unavailable are separate because the difference is actionable: stale means the cause of the silence is unknown, unavailable means it is known. Alerting and rule tolerance differ.

Two states are not entered by any runtime transition:

unknown is the state a point is in from registration until its first observation.
restored is entered only at core start, when the checkpointed projection loads. A point checkpointed as unknown stays unknown; everything else becomes restored regardless of what it was.

Legal runtime transitions:

unknown     → live | bad | unavailable | assumed
restored    → live | stale | bad | unavailable | assumed
live        → stale | bad | unavailable | assumed
stale       → live | bad | unavailable | assumed
bad         → live | stale | unavailable | assumed
assumed     → live | stale | bad | unavailable
unavailable → live | bad
*           → unavailable          (driver reports disconnect)

Three of those need stating explicitly, because they are the ones an implementation gets wrong:

Anything readable can become assumed. A write with confirmation_mode: optimistic asserts a value irrespective of what the point held before, so unknown → assumed (a cold-boot relay with init_strategy: assert(v)), restored → assumed, stale → assumed and bad → assumed are all reachable. The prior state has no bearing on it; that is exactly what assumed means.

unavailable has only two exits, and both require real data. A point known to be off the network is never promoted synthetically, so there is no unavailable → stale and no unavailable → live by timer expiry. It leaves only when an observation arrives. There is deliberately no unavailable → assumed either: writing to a device known to be disconnected must not manufacture confidence in the result.

No transition returns to unknown or restored. Both are startup states. A point that has been observed once has been observed forever, and restored is a claim about a particular process start.

The rule that matters most: when quality is not live, the value field still holds the last known value. It is never nulled. Consumers receive (value, quality, observed_at) and decide for themselves. Clearing the value on staleness destroys information and reintroduces the null-means-three-things problem this model exists to escape.

Drivers may only assert live, bad, or unavailable. Everything else is the core's to derive. This keeps the driver contract small and prevents a badly-behaved driver from declaring things it cannot know.

Time

Three timestamps. Collapsing them to two destroys information that later mechanisms depend on:

device_time — nullable, untrusted. Devices without an RTC will lie. Used for analysis only, and flagged when skew against observed_at exceeds a threshold.
observed_at — when the driver saw it. This is what freshness is computed against.
ingested_at — when the bus accepted the observation. This is what the log orders by.

ingested_at is taken from the bus message itself, not from the core's clock at the moment of handling. It is part of the dedup key, so it must be identical on every redelivery of the same message; the reasoning is in Ingest Path & Log Schema and it is not safe to change here in isolation. The practical reading is that it means "when this observation became durable in the system", which is also the more useful of the two candidate meanings.

The second and third diverge exactly when a driver host buffers through a network partition and then floods. Any host on a wireless link or a segment that drops makes this routine rather than exceptional, and BLE and radio hosts are the obvious candidates. Collapsed together, a burst of buffered readings all look fresh at the moment they land.

source_seq, monotonic per driver instance, exists for dedup and out-of-order detection.

Change vs refresh

change_policy (any | absolute(delta) | relative(pct) | none) belongs in the semantic model because it determines what the rules engine sees as an edge.

Every observation refreshes freshness. Only observations that clear the change policy emit a value_changed event. Without the deadband, a noisy analogue sensor generates an edge per sample and the rules engine spends its life on 0.01 °C.

The changed flag this produces has a second life in storage: it is what state-class compaction keeps and what it drops. See Retention & Compaction.

The observation envelope

What a driver publishes:

point_id, value_native, unit_native,
quality (live|bad|unavailable), reason, detail,
device_time?, observed_at, source_seq,
driver_id, driver_instance

What the core adds on ingest:

event_id, ingested_at,
value_canonical, unit_canonical,
prev_value, prev_quality,
changed (bool, per change_policy)

That second block is the full transition tuple, which is exactly what the rules engine consumes. Edge detection, staleness triggers, and bad-data triggers all read the same record.

Normalisation is a write-time decision

Core-side normalisation plus a log-first core means canonical values are computed once, at ingest, and then frozen in the log. If a wrong unit_native is corrected in the registry six months later, every historical event stays wrong.

The fix is cheap now and impossible later: the log stores native and canonical side by side, plus the registry version that did the conversion.

value_native, unit_native, value_canonical, unit_canonical, registry_version

Reprocessing then becomes a real operation — replay the log, reconvert from native under the current registry, rebuild the projection. Without value_native in the log, a unit error is permanent data loss. This is codified as a reprocess command from day one rather than a script written in anger later.

Registry authority and the two failure cases

With drivers dumb, the core needs explicit answers for two things that happen in week one:

Unregistered point. A driver publishes boiler.ch1.return-temp and the registry has never heard of it. It is neither dropped nor auto-registered. It goes to a quarantine stream with full native payload, raised for triage. Auto-registration guesses at semantics, and a guessed quantity is worse than no point at all. Quarantine also provides discovery as a side effect: connect a new device, let the driver run, and watch what accumulates.

Unit mismatch. Some sources self-describe (Shelly and UniFi JSON carry units), most do not. Where a driver supplies unit_native and it disagrees with the registry, the observation gets quality bad with reason unit_mismatch. Neither claim is converted from — either could be the wrong one, and silently picking is how a heatsink reading of 90 °F becomes a value a rule treats as 90 °C.

The quantity table

Pragmatic canonicals. Two columns that are easy to omit and expensive to add later:

quantity	canonical (UCUM)	affine	agg_safe
temperature	`Cel`	yes	yes
temperature_delta	`Cel`	no	yes
relative_humidity	`%`	no	yes
pressure	`kPa`	no	yes
voltage	`V`	no	yes
current	`A`	no	yes
power_active	`W`	no	yes
power_apparent	`V.A`	no	yes
power_factor	`1`	no	yes
frequency	`Hz`	no	yes
energy	`W.h`	no	yes
charge	`A.h`	no	yes
illuminance	`lx`	no	yes
flow_volumetric	`L/min`	no	yes
volume	`L`	no	yes
signal_strength	`dBm`	no	no
sound_level	`dB`	no	no
duration	`s`	no	yes
ratio	`%`	no	yes
count	`1`	no	yes

affine exists because °F → °C needs an offset, not just a scale factor. A multiplicative-only conversion layer is correct for every quantity except temperature, which is the one most likely to be converted. The conversion is value × factor + offset from the start.

agg_safe: false marks logarithmic quantities. The mean of two dBm values is not the dBm of the mean, and a naive moving average over RSSI produces a number that looks plausible and means nothing. Derived points refuse to compute mean on a non-agg-safe quantity unless the definition explicitly opts into log_mean, and the same flag governs which functions a rollup may use.

temperature_delta as a distinct quantity is the same class of trap. A 5 °C rise is not 41 °F.

Two notes on the table. pressure in kPa is awkward for both atmospheric and hydraulic readings, since neither is the natural unit; one quantity handled through display_symbol is preferred to splitting, and the decision is worth revisiting once real pressure points exist. And quantity is required for kind: measurement and null for boolean and enum state points — a state pseudo-quantity must not be invented to fill the column.

Subject and subject_ref

Starter vocabulary, flat, to be edited rather than accepted as given:

air, water, surface, heatsink, enclosure, soil, mains, circuit, battery, core_body, skin, ambient, self

self covers diagnostics — a device reporting its own uptime, free heap, or signal strength. Those are real points with real freshness needs and must not be forced into a physical subject.

subject_ref format is {kind}:{slug}, with kinds zone, device, circuit, asset, person. The format is validated, the string stored, and nothing resolved. No entity tables until a second consumer needs them.

The signal that flat subjects have stopped working is a point declared as subject: water with tags: [flow] and another with tags: [return], and then a rule branching on the tag. That is the tag governance rule firing as designed. When it happens the distinction gets promoted — either to a circuit_position modifier or to a nested subject — and by then there is real evidence for which.

Vocabularies as code

All five closed vocabularies (quantity, subject, enum_type, retention_class, modifier value sets) live as versioned files in the repository, loaded into Postgres by a migration. The registry version stamped on every observation refers to this.

Three checks belong in CI rather than at runtime:

every point's unit_native is convertible to its quantity's canonical unit
no point references a vocabulary member that does not exist
no vocabulary member is removed while points still reference it

The last is the migration guard. Renaming heatsink to heat-sink fails the build rather than silently orphaning forty points.

Exit criteria

This step is done when a test suite demonstrates that the registry rejects what it is supposed to reject. Not that it accepts good input — that will be obvious — but that bad input fails at load rather than in production.

A fixture set of deliberately invalid descriptors, each expected to fail with a named reason:

unit_native not convertible to its quantity's canonical unit
a reference to a vocabulary member that does not exist
on_change freshness with neither heartbeat nor an explicit never
a retention_class whose rollup functions cannot apply to the point's value_type or quantity
require_intent: true with no default_value
removing a vocabulary member while points still reference it

Plus two positive checks that are cheap and catch real errors:

Conversion round-trip against known pairs, affine included: °F → °C at 32 and 212, mA → A, W.h at scale. A multiplicative-only conversion layer passes every case except the one that matters, so the affine cases must be in the table.
The quality transition table exists as a checked enum, with a test asserting that no code path can produce a transition outside it. The prose table above is for readers; this is what keeps it true.

On Monday - the Dev Diary will look at what needed to change when the model spec'ed above met reality.

The next part - The Driver Contract. Why every driver speaks one northbound transport no matter what protocol it talks to the device in, and why converting a unit is the first item on a list of eight things a driver is forbidden to do.

Sentinel Dev Diary: Four Registers and a Drift Check

Philip Shaw — Tue, 08 Sep 2026 10:21:49 +0000

The Sentinel specification is ten documents, and it was finished before a line of the system was written. That was the whole idea: decide what the thing is while deciding is cheap, then build it.

Changes to the spec were anticipated. I have a series of Dev Diary look-backs planned that address exactly this set of deltas starting next week. But barely had the first lines of code been written before a wholly different gap surfaced: registers!

Correcting specifications, recording decisions, creating issues to come back to things later, hand-written notes...these are all fine in their own right, but they rot so fast... This is where registers earn their keep from day one.

Two false starts

The registers were the third arrangement, not the first. Let me be honest about that up front. First the story of the short-shelf-life solutions I mentioned.

Stage one: PR bodies. Anything getting deferred, assumed or decided-in-passing was written in the pull request. This is a natural default and it is not nothing; it is at least written down, next to the diff that motivated it. Where it fails is on retrieval and discovery. Nobody greps forty merged PRs to find out whether a thing was deliberate.

Stage two: one file. So it went into docs/deferred.md: blocked work, overdue work, measurements, and findings about the specification, all in one place. Wonderfully arranged by category with notes and even some commit hashes. Better. It lasted... hmm... a few days if I am generous. Why?

It was carrying four different kinds of statement in one file. Because its content was sourced from PRs merged, it organised along similar lines and mixed findings and deferrals into one because it lacked a set of organising principles to keep it disciplined.

A file that holds four kinds of thing has no rule for what belongs in it, so it also has no rule for what has fallen out of it. Nothing was lost dramatically. Things were lost quietly, which is worse, because a register you have stopped trusting is one you stop reading.

And it had a twin. The driver-host repository had grown its own docs/deferred.md, on the same convention and for the same reasons, so the same mistake was now being made twice, in two repositories, about one specification. It was time to do things properly. Enter, the Registers...

Any deferral, every decision or a freshly discovered risk goes in the register that covers the specification, naming the repository it belongs to. Numbered, immutably and durably, and referenced. Importantly, never repeated elsewhere - when the temptation to do so arises, squash it, and link back to the register that owns it instead.

One set of registers spans both repositories, deliberately. Two registers on one specification is two places to look and one to forget, and the seam between two repositories is exactly where a row goes missing; build step three is split across both, so its deferrals were too. Code does not move; documentation does.

The four registers

docs/
  spec/              ← normative. Ten documents. Nothing outside this defines anything.
  REQUIREMENTS.md    ←  SR-*  what the system must do, grouped by owning document
                        CN-*  the constraints that may never be violated
  COMPONENTS.md      ←  C-*   components, each with a build status and its package
                        I-*   integration contracts
  DECISIONS.md       ←  D-*   every choice taken where the spec was silent, superseded ones kept
  ROADMAP.md         ←  CP-*  checkpoints: evidence that does not exist yet
                        A-*   assumptions
                        R-*   risks
                        OI-*  open questions - gaps in the specification
                        DW-*  deferred work - code that is owed
  spec-deltas/       ← one audit per completed build step

Everything except spec/ is derivative. That word matters rather more than it looks:

Nothing in a register defines anything. They point at docs/spec/, which is normative; a register that restates a mechanism has created a second home for it, which is the failure the specification exists to prevent.

The registers number what the specification already says, so that a component, a commit, an Issue or a review can cite it. SR-14 is not a requirement written in REQUIREMENTS.md; it is a pointer with a stable handle, and the requirement itself lives in the document that owns the mechanism. Where the two disagree, the row is the bug. Always the row.

Then a precedence order, because two documents will disagree eventually and I would rather settle it now than in the moment:

On what a mechanism is, or what a term means - the owning specification document wins, always.
On why a decision was taken where the specification is silent - DECISIONS.md wins.
On what is built right now - COMPONENTS.md and ROADMAP.md win over any document's description of intent.
On implementable work - the GitHub Issue wins over all of the above.

Without that written down, every disagreement gets settled by whoever happened to notice it, on the day, in whatever mood they were in. And half of those settlements go the other way next month.

The distinction that is actually load-bearing

If you take one thing from any of this: an open question is not a task, and shipping code does not close one.

An OI-* is a gap in the specification - something that could have been decided and was not. A DW-* is code that is owed. In the moment they feel identical. They are not, and getting them confused is filed as a risk against my own process:

R-2 - A specification gap is "closed" by the commit that routes around it. The work that reveals a gap is almost always the work that works around it, so a finding filed as a task is struck by the commit that proves it real.

Which is exactly how it goes. You are building, you find the document never said which way a case falls, you pick a reading, you ship. If that finding was filed as a task, the task is now done. The specification is still silent - and nobody will ever notice, because the code works. That last part is the one that gets you.

So an OI-* is discharged by amending the owning document and by nothing else. Where a gap was routed around in code, the reading taken gets recorded as a D-* and the open question stays open, pointing at it. Two rows, two lifecycles. One says what the code does; the other says the document never said.

Snippets from the first three steps

Some of what actually landed in the registers as the build got going, quoted as written. The shape of the thing rather than the content of it.

The register vocabulary managed to break the specification's own citation rule, more or less immediately:

OI-1 - Introduction, Retention & Compaction. There is no build step that owns retention policy values. In the absence of one, this repo invented "step 10" - the document's reading-order number wearing a step's clothes - and it has spread into the register, the self-point catalogue test's deferral map and several docstrings. It also directly contradicts Introduction's rule that nothing cites a document by number.

Which is why every specification document here is cited by title, in italics, and never by number or path. The rule pre-dates the incident. The incident is what made it enforceable, because there is now a row explaining precisely what happens when it lapses.

The constraints had the opposite problem. Real, load-bearing, and entirely uncitable:

OI-17 - Introduction. The constraints have no numbered home in the document that states them. Introduction carries twenty-two Conventions - the properties everything else is measured against, each extracted from a specific error - in a document that says of itself "this note is the map and the rationale. It defines nothing." A definition found there is stated to be a bug in the note, and yet these are the closest thing the system has to invariants. They are also uncitable: every reference to one quotes its prose.

They are indexed as CN-1 to CN-22 now, which is an index and not a copy. The argument for each one stays exactly where it was.

An assumption is a different animal again, because work is already resting on it before anyone thinks to check:

A-1 - The test environment has TimescaleDB, not plain Postgres, and NATS started with JetStream. If it is wrong: ~95 tests skip rather than fail and the suite reports green; the tell is the skip count and coverage near 80% rather than ~97%. Tested by: every run - a correct one has 0 skipped.

A green suite that is green because a hundred tests did not run is the most expensive kind of wrong. And it is invisible unless somebody wrote down what the number is supposed to be.

A good many decisions turn out not to be about the system at all, but about the ground it is standing on:

D-07 - Enum members are quoted in YAML. YAML 1.1 resolves bare on, off, yes and no to booleans, which would have turned the switch_state codebook into [true, false] silently. Not a specification matter - a trap in the serialisation format the specification chose.

No specification was ever going to catch that one. It is the format's trap rather than the design's, and without the codebook check it would have been found eventually by something behaving very strangely at runtime.

And occasionally a decision is really a finding about the tests:

D-14 - A guard with no caller is not a guard. The check passed its fixture and neither real call site supplied a baseline, so the operation it was meant to refuse failed with a different error, from a different check, that happens to look like the guard working.

That one is worth considerably more than its two lines. A test that exercises a function directly, plus a codebase where nothing calls that function in anger, gives you a green build and an unarmed guard - and the failure presents as the guard working correctly. I have no idea how many more of those there are. Presumably not zero.

Then the category for numbers that do not exist yet. A checkpoint says what produces it, what it must yield, what it unblocks, and one more field that is the whole point of the exercise:

CP-1 - Ingest throughput against a populated database. Current value: 4,369 obs/s, taken 2026-08-31 against freshly-truncated hypertables with the three aggregates defined and no refresh policy. Steady state is 220/s, so utilisation ≈ 5%. Do not pre-decide: that the margin holds. Re-taking can only move the figure down.

Do not pre-decide is the field that earns the register its keep. A number measured under favourable conditions reads as headroom, and the caveat only survives if it travels in the same row as the figure. That is R-3 - and R-3 is retired by re-taking the measurement, not by anyone deciding it is probably fine.

The status marker that lies

COMPONENTS.md and ROADMAP.md mark every entry built, partial, or not started. Partial is the interesting one:

A 🟡 is a claim about what is missing, not only about what is there. It is the marker most likely to be wrong, because it is the one that requires someone to have looked.

So partial is only legal when what is missing is enumerated, with a deferred-work row behind each gap. A 🟡 with one named gap and three real ones is worse than an honest "not started"; it has spent the credibility of a status marker to buy an inaccuracy.

Which is where the list of things reviewers are asked to attack begins: any 🟢 that is really a 🟡, and any 🟡 whose account of what is missing looks suspiciously short.

The drift check

None of this stays true on its own. Registers rot exactly like the notes they replaced, unless something makes them false loudly. So there is a ritual, and it does not vary.

At the end of each build step: enumerate the owning document's exit criteria and map each one to a test. A criterion with no test is a finding, not something to quietly fix - all three audits so far have turned one up. Then audit the code against the documents rather than against the commit history, because the commits record what I intended and only the code records what I did. Write it up as spec-deltas/build-step-N.md, in two halves: what changed during implementation, and what the step did not deliver. Then route every finding into exactly one register, which is the part that actually matters.

On each pull request: if it defers something, the DW-* row lands in the same commit. If it adds a mechanism, that mechanism's signals go into the observability document in the same commit. That rule exists in writing because I broke it once.

The audits keep earning their place. The third one found a paragraph in a register asserting the opposite of what the code did - not drift between the code and the specification, but drift between the register and reality, in a document whose entire job is to describe reality. Humbling.

Why this shape, specifically

Sentinel is built almost entirely with coding agents, and that has shaped the registers more than anything else here has.

An agent opening a session has read the repository and none of my recollection. It cannot know that a field is mandatory because of an argument three weeks ago, or that a number is provisional, or that a gap in a document was spotted and deliberately left open. Every one of those is something I would have supplied from memory without noticing - and, if I am honest, supplied differently on different days.

So the project's CLAUDE.md does not explain the registers. It points at them, and says to read the index before adding to any of them. What that buys is that a decision can be cited rather than re-derived. A decision that gets re-derived every time gets decided differently at least once.

Superseded decisions stay for the same reason:

A decision that was overridden is more useful than one that was erased, because the override is itself evidence about what the design gets wrong.

And the registers are themselves under test. test_register_integrity.py asserts that ids are contiguous and unduplicated, and that a requirement is never both a bare row and a set of sub-ids. That is the specification's own convention - every claim a document makes has a test that would fail if it stopped being true - turned around and pointed back at the process documents. It felt faintly ridiculous to write. It has caught things.

What is still unowned

One gap is worth ending on, because it is open and I do not have an answer for it:

OI-7 - Introduction. Nothing reads the documents against each other. All three cross-document gaps found so far were found by building rather than by reading, and each sat between two documents that were each internally consistent.

The single-owning-document discipline optimises for consistency within a document, and it works. The price is that the seams between documents are where the gaps actually are, and nothing currently looks there. The drift check runs per build step, so every seam does get inspected eventually - by the build, at the point where it costs the most to find.

A real hole in the process, then. Filed as an open question, in the register whose job is to hold the questions the process cannot yet answer. Which is at least the machine working as designed.

Next - The Semantic Model. Part two of the specification proper: why the primary key for every point in this system is a dotted string you can paste into a grep, and never a UUID.

The next entry - What the Specification Left Unsaid. What the first audit found in a step that had already been called complete: six named rejections that had become twenty-eight, and a guard that existed as a function with a fixture and no caller.

Sentinel: An Introduction

Philip Shaw — Wed, 02 Sep 2026 09:22:19 +0000

import PlaneArchitecture from '../../assets/sentinel-plane-architecture.svg';

Does it matter if a soil moisture sensor goes quiet for three days?

On its own, probably not. But if an irrigation decision is derived from that reading, three days of silence is three days of watering against a number that stopped being true, and nothing in the system said so.

Move up the scale. Indoor air quality is degrading because the air conditioning has stopped working — not thrown an error, stopped, which from the outside looks identical to an air conditioner with nothing to do. Move up again. A battery bank charging off solar, in an enclosed space, begins to climb in temperature. Do you want to know about that now, or the next time you happen to look at a dashboard?

Those three questions have the same shape but completely different answers and priorities. What separates them is not technical, it's a judgement about what matters, how much, and how quickly. That call gets made somewhere regardless. Written down, it is a specification. Left unwritten, it is whatever the code or user happened to do.

Problem Statement: Underneath all three questions is the same problem: a system that cannot tell a reading from a memory of one. A value that arrived three days ago still presents as a value, a device that has stopped reporting looks like a device with nothing to report, and a command that was sent looks like a command that worked — none of it announcing itself, all of it producing plausible numbers and unbroken graphs right up until a decision gets made against something that stopped being true. Sentinel exists to make that difference explicit at every hop, so that acting on a number the system cannot vouch for is a choice rather than an accident.

There is an obvious objection, which is that this is a solved problem and you can buy it. However, I have spent a long time cajoling systems and their processes to fit around real world situations they were not designed to address — reframing the problem until it fitted the tool. That works until the reshaping costs more than the tool saves, and at that point it is worth exploring something with the flexibility those use-cases actually need.

So this design is written down, and in public.

A specification at this scale is not trivially derived from a set of working assumptions; the assumptions are the easy part, and they usually create a false sense of simplicity that unravels the moment you think the work is finished.

Publishing holds me to the choices I made during the design. Not as choices I must live with, but ones that I must litigate clearly when I adopt or change them. What I decided, what I claimed they would achieve, and what had to change when the design met the thing it describes.

That pressure is the same in a one-maintainer repository as it is in a corporate monorepo, and so are the failure modes. Arguing the changes in public makes each one a worked example rather than a commit message.

The spec is here to be read and argued with.

What is Sentinel?

A platform for monitoring and controlling devices across several network segments: collecting readings over mixed protocols, normalising them into a single semantic model, deriving values from them, evaluating automation rules, and driving physical outputs with verification and safety interlocks.

The name is the job. A sentinel stands watch, knows the difference between quiet and absent, and raises an alarm rather than acting on a guess. Two properties of this design follow that description more closely than they follow any conventional monitoring platform: data quality is a first-class field, so the system is explicit about what it does not know; and absence of data is a signal in its own right rather than a gap to be interpolated over.

It is written for a system built and operated by one person, and several decisions follow from that constraint rather than from first principles. Where they do, the constraint is stated alongside the decision.

Components take the system name as a prefix: sentinel-core, sentinel-driver-modbus, sentinel-registry.

How this series is organised

Eleven parts, published weekly. This one is the map and the rationale for Sentinel's design; the ten that follow are the documents that define it.

Every mechanism in this system has exactly one owning document. Where this part mentions a state set, a policy name, or a lifecycle, it names it only to say where it is defined.

Each part is published as it was drafted, rather than as a tidied-up retrospective. The system is being built while the series runs, so where the plan meets reality and loses, a follow-up says what changed and why. Three figures in the design are unmeasured today — sustainable ingest throughput, cold-start time, and the retention policy values — and all three get measured during the build, inside the window this series runs in. You will see the numbers when I do.

Reading order

The Semantic Model
Driver Contract
Ingest Path & Log Schema
Watchdog & Derived Points
Command Plane
Rules Engine
Core Runtime
Security Model
Observability
Retention & Compaction

Core Runtime, Security Model and Observability come after the mechanisms they constrain: what enforces single-instance operation and what the system is sized for, which boundary carries the authorisation weight, and what signals every mechanism must expose.

Retention & Compaction reads last because it uses descriptor fields defined across four earlier documents, but it is a mechanism rather than a policy appendix — it constrains the log schema, the derived-point layer and the reprocess operation, and part of it cannot be deferred without a data migration.

Scope

These ten documents cover the core and the data pipeline: collection, ingest, state, derivation, automation, control, runtime, security, instrumentation and storage lifecycle.

Two things have been deliberately left out, because neither changes anything above: alert routing and notification delivery, and the read API and UI. Observability says what must be watchable and what should raise an alarm; but how an alarm reaches a person is not specified here. Rules Engine names actions with a channel; the channel model is not specified here either.

The one constraint those inherit: alerting runs outside the core, because alerting inside the core cannot report the core's absence.

The two decisions that shape everything else

1. Log-first core. Current state is a projection of an append-only event log, never the primary artefact. Every observation, quality transition, command and command outcome is an appended event. Restart is snapshot plus replay of the tail. This is the whole answer to restarting without losing state, and retrofitting it later is painful, so it goes in first.

2. Hard driver/core boundary. The core has zero protocol knowledge. Drivers have zero knowledge of rules, units or storage. The boundary is exactly two contracts: a normalised observation flowing north, and a command envelope flowing south. The failure this prevents is an observability or collection concern leaking outward into the layers around it, so that every part of the system ends up coupled to whichever part was built first. Making the boundary a wire protocol between processes means it cannot be casually violated.

Everything else in these documents is downstream of those two choices, and most of it follows fairly directly.

Non-goals

This is a system built from scratch, with no migration path in or out. No document should acquire a migration section, a dual-run plan or a compatibility shim.

Multiple concurrent cores are not a goal. Exactly one core instance runs at a time, enforced rather than assumed. The reasoning, the enforcement mechanism, and the two conditions that would reopen the question are in Core Runtime.

Keeping raw data indefinitely is not a goal. Nothing is retained in original sample detail forever; measurements are rolled up and their samples dropped. State transitions are a different thing and are kept — the distinction is samples versus history, and it is in Retention & Compaction.

A second storage tier is not a goal. No object-store archive, no cold tier. Data not worth keeping in Postgres is not worth keeping.

Nothing here describes existing infrastructure. These documents specify what to build. Where they name a protocol, a device or a deployment shape, that is a design target rather than an inventory, and no document should assert that any component currently exists.

The planes

The diagram is illustrative. This section is the normative description of the decomposition; where the two disagree, this text wins and the diagram should be corrected.

Devices are anything that produces readings or accepts writes, over any protocol, push or poll.

Driver plane. Protocol adapters, deployed as driver hosts, one per network segment, each supervising per-device driver instances. Drivers translate a protocol into observations and attempt writes. They hold no semantics, no verification state and no automation. Defined in Driver Contract.

Bus. Durable subject streams carrying observations, availability, quarantine, commands and command results. Every driver-plane subject leads with the host, which is what makes per-host credentials workable. The observation stream is the system's outage buffer, bounded by size rather than age and sized from tolerable outage and utilisation — see Core Runtime. Subject layout is defined in Driver Contract; it is also the primary authorisation surface, so the two are designed together — see Security Model.

Core. A single process — see Core Runtime — containing four stages that are logically distinct and physically together:

Normalise — validate, deduplicate, resolve against the registry, check the publisher, convert to canonical units, evaluate the deadband. Defined in Ingest Path & Log Schema.
Event log — the durable append-only record, split by volume, retention class and query shape. Schema in Ingest Path & Log Schema; the class split in Retention & Compaction.
State and registry — the current projection, plus the versioned registry that says what every point means. Projection defined in Ingest Path & Log Schema; registry content defined in The Semantic Model.
Derivation and watchdog — freshness deadlines and derived points, both of which emit into the same pipeline as device data. Defined in Watchdog & Derived Points.

The diagram places derived points and timers inside the rules engine box. That is an over-simplification, and it is the one worth flagging, because it survived several drafts before the distinction it blurs turned out to matter. Derived points are ordinary points, computed before rules evaluate; the watchdog that ages them is not part of the engine at all. That separation is exactly what keeps the rules engine small, and it is argued properly in Watchdog & Derived Points.

Rules engine. Evaluates predicates over transitions and produces claims. It sends no commands, holds no state between evaluations, and has no visibility into whether a claim reached hardware. Defined in Rules Engine.

Command plane. Owns the durable claim set and its resolution into a desired value, deliverability and reconciliation of reported against desired, confirmation and verification, and the safety and interlock model. Returns to the bus as commands to driver instances. Defined in Command Plane.

The single closed loop is: device → driver → bus → core → rules → command plane → bus → driver → device, with the log recording every hop.

Scale

The design targets, stated once here and derived from in Core Runtime: 200+ devices across 16+ network segments, 2,000+ device timeseries points, 1,000+ derived points, and roughly 700 self-points that fall out of instrumenting the rest — about 4,000 points in total, at ~220 observations per second, designed for 500.

Two figures matter more than their size suggests. Self-points are a fifth of the registry and are easy to omit when sizing anything. And sustainable ingest throughput is the one term still unmeasured, which makes it the term everything else waits on.

Where things are defined

Concern	Owning document
Point identity, hierarchy, descriptor fields	The Semantic Model
Closed vocabularies: quantity, subject, enum types, retention class, modifiers	The Semantic Model
Units, canonicalisation, affine conversion, aggregation safety	The Semantic Model
Quality states and legal transitions	The Semantic Model
Timestamps and sequence numbers	The Semantic Model
Observation envelope	The Semantic Model (shape), Driver Contract (driver-supplied half)
Registry authority, quarantine and unit-mismatch handling	The Semantic Model
Driver host and instance model, lifecycle, assignments	Driver Contract
Bus subjects, stream limits, buffering, availability	Driver Contract
Driver-side command handling and the prohibitions list	Driver Contract
Log schema, hypertable split, ingest pipeline ordering, dedup, publisher check, projection, restart, reprocess	Ingest Path & Log Schema
Registry versioning and descriptor storage	Ingest Path & Log Schema
Freshness policies, watchdog behaviour, staleness suppression, flap damping	Watchdog & Derived Points
Derived point descriptors, operators, quality propagation, evaluation, rehydration	Watchdog & Derived Points
Restricted expression grammar, and its two binding contexts	Watchdog & Derived Points
Claim model, bands, resolution, suspension	Command Plane
Desired/reported, deliverability, convergence, command state machine, confirmation modes	Command Plane
Fail-safe versus fail-clear, interlock constraints	Command Plane
Intent API, validation, rate limits, idempotency, escape hatch	Command Plane
Rule shape, triggers, evaluation, claims, warmup, loop protection	Rules Engine
Rule and registry load-time validation, rule testing	Rules Engine
Single-writer enforcement, core epoch, startup and shutdown ordering	Core Runtime
Scale targets, outage figures, buffer sizing, drain behaviour	Core Runtime
Backpressure and ingest lag	Core Runtime
Availability while the core is down, restart time, geographic split	Core Runtime
Threat model, principals, credentials, bus permissions	Security Model
Human authentication, capabilities, what each principal may do	Security Model
Secret handling, audit requirements, exposure assumptions	Security Model
Self-point catalogue, naming, what raises an alarm	Observability
Retention classes, policy values, rollup derivation, compaction	Retention & Compaction
Exit criteria for each build step	the document owning that step's mechanism

Conventions

Most of these were extracted from a specific error rather than asserted as principle, and each names the failure it prevents.

One definition per concept. Cross-reference rather than restate, including in this note. A concept that appears in two documents has one normative home and one pointer.

Vocabularies and schemas are code. The closed vocabularies, point descriptors and rule definitions live as versioned files in the repository and load into Postgres by migration. Prose in these notes explains them; it does not define them once the code exists.

State machines are enums with tests. Quality transitions, driver lifecycle, command lifecycle, convergence and arbitration bands appear as tables in prose here and must exist as checked enums in code, with a test that no code path produces an illegal transition. Prose tables are for readers and will rot.

Every claim a document makes has a test that would fail if it stopped being true, and that test lives in the same document. Prose cannot fail loudly. The exit criteria at the end of each document are that test set, and they are what says a build step is finished — not a feature checklist, but a demonstration that the step's central property actually holds.

A boolean that has to carry three meanings is an enum. Quality was the first instance and convergence the second. When a field needs to say "not yet", "no", and "not applicable", write the enum the first time.

Store what composes, not what is convenient to read. Native alongside canonical, sum and count rather than average. Anything derived can be recomputed from a composable base; a base that has already been reduced cannot be recovered, and the reduction is usually wrong at the next level up.

Storage optimisations that discard raw rows are not optimisations. Derived-point windows and reprocess both read the raw tables, so a row not written is a mechanism quietly broken. Every observation writes a row; volume is managed by retention and compaction, which are deliberate and bounded, rather than by suppression at write time.

Behaviour that varies by data type is derived from the descriptor, not authored per point. Which rollup functions are valid follows from value_type, agg_safe and the aggregation modifier; which retention class applies follows from kind. Restating what the registry already knows is an invitation to state it wrongly.

One number per concept. Where an ordering exists, it is expressed once. Bands are an ordered enum and there is no parallel priority integer; the same discipline applies anywhere else a rank appears.

Two numbers that differ by an order of magnitude are two numbers. Planned restart and tolerable unplanned outage were briefly treated as one quantity, which would have sized the outage buffer against the restart case. Before reusing a figure in a second place, check that the two cases actually coincide.

Identity comes from the message, not from the moment of handling. Anything a component assigns to an event at processing time is different on a retry, and therefore cannot be part of an identity or dedup key. This is why ingested_at is taken from the bus rather than from the core's clock.

Authority comes from the connection, never from the payload. Who a request or a message is from is established by the credential that carried it or the binding it arrived on, never by a field inside it. Owner and band are derived from the authenticated principal; a driver's right to publish for a point is checked against the registry rather than taken from the subject it chose.

Assumptions that several documents rely on are enforced somewhere. Single-writer was assumed by five mechanisms before anything checked it. An assumption held by convention is one that gets broken on a quiet afternoon, and the failure modes that follow are usually silent.

Nothing expires data by age except a retention policy chosen for it. Age limits on a buffer delete messages that were never consumed, silently, between components that both believe the handoff succeeded. Buffers are bounded by size and refuse new work when full; age-based deletion is confined to the places where dropping old data is the actual intent.

Physical layout that varies by policy is decided before there is data. Retention attaches to a table rather than to a row, so a per-class policy is a table split, and a table split after the fact rewrites everything ever collected. The values can wait; the shape cannot.

An operation that cannot cover its whole scope refuses, rather than covering part of it. Reprocess spans retention classes with different horizons, and correcting the reachable two thirds would leave right and wrong data indistinguishable. The scope is validated first, the refusal carries a reason, and narrowing it is a deliberate second command.

A signal about a component must not depend on that component. Core self-points are injected at the projection rather than published through the queue they describe, and alerting runs outside the core so it survives the core being down. A monitor that fails with the thing it monitors is not a monitor.

Never report a fact about the core as a fact about a device. Ingest lag would otherwise surface as fleet-wide staleness, and the system would take protective action across the estate because it was slow. Where the cause is internal, the system says so and suppresses the derived signal — staleness and warmup deadlines both pause on the same condition.

Wait on conditions, not on clocks. A timeout should mean the thing being waited for failed, not that a precondition had not arrived yet. Where a component waits for another to become ready, it gates on the readiness signal and the wait is unbounded; where the wait is genuinely unsafe to leave open, it escalates rather than expiring.

Deployment parameters are read, not assumed. Retention windows, intervals and limits set at deployment are checked against configuration at load time by querying them, never by hardcoding the current value. A constant copied into validation code is a constraint that silently stops holding the day the original changes.

A mechanism does not ship without saying how its failure would be noticed. Its signals go in Observability in the same commit that defines it.

Anything encoded as a string suffix will have to be parsed later. This applies to units, phases, aggregation and circuit position. It is promoted to a field instead.

Build order

Semantic model and vocabularies — point identity, quantity, unit, subject, quality, retention class
Event log, state projection, snapshot and replay, cold start
Driver contract, plus one push driver and one poll driver
Freshness watchdog and derived points
Command plane, including arbitration and the confirmation modes
Rules engine
Instrumentation and timeseries export

Steps 1 and 2 are the expensive ones to change later. Everything from 3 onwards is additive if those are right.

Each step is finished when the exit criteria in its owning document pass. They live there rather than here, because a criterion is the proof of that document's claim and only stays truthful next to it. A separate test plan would drift within a month, leaving two places describing what ingest guarantees.

Security is not a step. The subject layout and the publisher check land in step 3, bus credentials with the first driver host, and forward auth with the API when it exists. Deferring it to a step of its own is how it becomes a retrofit.

Neither is the runtime. The advisory lock and the startup ordering belong in step 2, with the first process that writes; the core epoch lands in step 3 alongside the command envelope.

Neither is observability. Each step adds its own self-points as it goes, which is also how the ingest path gets exercised before there are devices to exercise it.

Retention is split across both: the class field, the hypertable split and the ingest routing belong in steps 1 and 2, because they are physical; the policy values and the compaction job follow once volume is measured.

Ingest throughput is measured during step 2, against the design rate in Core Runtime. It is the term the stream sizing and the recovery time both depend on, and it is cheap to measure with synthetic load before any driver exists.

Open items

Sustainable ingest throughput. The only unmeasured term in the buffer and recovery arithmetic. Measured on per-observation validation, registry lookup and conversion in application code, which is where the ceiling sits — not on the batch insert.
Cold-start time. The planned-restart number, dominated by derived-point rehydration at a thousand-plus derived points. Measured rather than estimated, and kept small.
Retention policy values. The classes and the split are decided; the durations are not. They need measured volume, and the driver for measurement raw retention is how long it is plausible to want to reprocess.

All three are measurements rather than decisions, and the first two are taken during step 2.

Settled: scale targets and the four-hour tolerable outage, both in Core Runtime. Retention classes and the samples-versus-history distinction, in Retention & Compaction. Exit criteria, in each owning document.

Out of scope: alert routing, and the read API and UI.

Next Wednesday - The Semantic Model. Why the primary key for every point in this system is a dotted string you can paste into a grep, and never a UUID.

Notes to Self: The Interview Between an Issue and a Spec

Philip Shaw — Fri, 14 Aug 2026 12:25:24 +0000

On 1 August I opened an issue that was three sentences long. A hundred and one minutes later
the feature was merged, and the document that got it there ran to
457 lines.

I didn't write those 457 lines. In fact, I didn't have to write any more documentation, and not because I simply allowed Claude to run amok.

Here is the issue in full — control-api#265, 225 characters:

control-api#265 — Manifest-backed dashboard feeds

For each dashboard, auto create a manifest keyed by dashboard_id.

For each sensor the dashboard uses, tag it to be included in the manifest.
When a dashboard definition is updated, add / remove tags from sensors accordingly.

From that genesis moment, this is the lifecycle of the issue all the way through to landing:

Time (UTC)	Event
14:25	Issue #265 opened — 225 characters
14:54	`FEAT-0007` spec committed — 457 lines
15:35	Spec merged (PR #266)
15:51	Implementation committed
16:06	Implementation merged (PR #267, 15 files), issue closed

The interesting part isn't the speed. It's the step at 14:54 that landed a previously non-existent spec document, and what happened in the twenty-nine minutes before it.

The issue was never a specification

I often write issues like this one...the way most people write shopping lists. Actuator address is not ensured?
Baseline the trace correctly. With the pre-rolls, the frame-rate looks out. They're abbreviated to
the point of being cryptic to everyone else. I write them this way deliberately: I'm usually mid-something
else when I notice a problem, or have an idea for a better route to the solution. The cost of a full write-up right at that moment would be a fractured sense of flow. As most engineers will tell you, the transitions into and out of flow are the most disruptive parts of their working day.

This terse form of issue-writing can be all you need, and it's worth being precise about why it works and the trade-offs it includes. It is not because "the issues are good enough".

They aren't. When you pick one of these up later, you can reconstruct everything missing from memory.

In my case with control-api#265, it was a particular panel type I was thinking about, a constraint I hit last week with it, the reason the
obvious approach was off the table, etc... That reconstruction was free and, more importantly, it was
invisible to any external observer. When you are the issue filer and its eventual closer, you never experience this rehydration as filling a gap. You see the issue's sparse wording as an aide-memoire for the fuller context that is still cached in your head.

What bites here is when you work in a team (of other engineers or coding agents): something has to bridge this gap or the issue will get resolved using a foundation of well intentioned misinterpretation and apparently sensible assumptions.

An agent has something both better and worse than an engineer's recall — it can read
every file in the repository, including ones you have forgotten, and it can read exactly nothing
of your latent intentions.

Fortunately, the fix is not to write longer issues. Capture is working; it's the handoff that's broken, the context sync.

An agent as interviewer asks for decisions, not descriptions

The obvious version of this step — "the agent asks clarifying questions" — undersells it, and
would be annoying in practice. Being interrogated about your own bug report is worse than
writing the report properly in the first place.

What makes it worth doing is the ordering. The agent goes and reads first, then comes back
with what it found and asks you to decide — not to describe.

You can see the split in the spec that grew out of my dashboard issue. Its reference list is entirely material the
agent uncovered on its own:

FEAT-0007 §2 — References (excerpt)

- `app/repositories/sensor_repo.py:123-126` (`get_sensors_with_tag`, the JSONB
  `.contains()` query pattern this spec's reconciliation reuses)
- `app/services/sensor_service.py:42-84,131-174` (`upsert_sensor` /
  `_fan_out_upsert` — precedent for both the reserved-tag-prefix guard and the
  "best-effort, must never fail the primary write" fan-out posture this spec copies)

Nobody asked me where the JSONB containment query lived, or which existing service had already
settled the question of what a fan-out failure should do to the primary write. Those are
enrichment lookups, and lookups are the agent's half of the job.

What came back to me as decisions to make were the things no amount of reading resolves: whether one SSE stream per
dashboard was actually the point, whether a second repository was allowed to be changed for
this, where the boundary of the work sat. The spec's Out of Scope section is five such
rulings written down.

That's the trade that makes the interview cheap enough to bother with. The expensive half is
already done by the time it reaches you.

The best question contradicted the issue

Read my second sentence again: for each sensor the dashboard uses, tag it.

There's an assumption buried in there — that the control-api server can tell which sensors a dashboard
uses. It can't, and the spec says so at length:

FEAT-0007 — Design Decision (excerpt)

The obvious-looking alternative — have control-api decode `layout` itself and derive
the sensor set — does not actually work cleanly, for one panel type specifically.
`chart`/`gauge` panels store `series[].{deviceId,sensorRef}` plus a panel-level
`dataType`, which *would* parse cleanly. But a `health_stats` panel stores only
`healthStats: {bodyDeviceId?, heartRateDeviceId?}` (confirmed in
`serializeDashboardLayout`) — its 7 `sensor_ref`s are frontend constants
(`BODY_SENSOR_REFS`, `HR_SENSOR_REF` in `useHealthStatsPanel.ts`), and each role's
`data_type` is discovered at render time from `listDeviceSensors`, not stored anywhere.

When I wrote the issue I was picturing chart panels, where my sentence is true. Health-stats
panels store two device IDs and resolve the rest at render time in a different repository.
A server-side parser would have to duplicate a list of constants that already lives in the frontend,
with no shared source of truth and no error when it happens to drift.

So the spec proposes the opposite: the client sends an explicit sensors list alongside the
layout, and the server never parses the layout at all. That decision brings a new API rule
with it — a PATCH supplying layout without sensors is rejected with a 422, so a layout
change can't silently leave the tag set stale.

None of that is in my three sentences. One line of it directly contradicts them.

This is the part I want to defend hardest. An interview process that only elaborated what I wrote would be
formatting. This one went and checked, found my premise didn't hold, and brought back a
reversal — before any code existed to be wrong.

The numbering is the fossil record

Something else survives in the finished document. Its requirements run FR-1 through FR-10 —
but there's also an FR-6a, an FR-10a, an NFR-3a, an AC-8a and an AC-8b.

A list written straight through numbers 1 to n. Letters mean insertion into a list that had
already been numbered. Each one marks a place where a later question forced something in
between.

FR-6a is the one I'd point at. FR-6 had established that dashboard tags are reserved, so a
caller's tag edit must preserve them. Then:

FEAT-0007 — FR-6a (excerpt)

The prior-tags read [...] is already wrapped in `try/except Exception:
logfire.exception(...)` so it can never break the upsert — but under FR-6, silently
treating a failed read as `old_tags = []` would mean `preserved` comes back empty and
the caller's list wins outright, wiping every `dashboard-*` tag on that sensor: the
exact outcome FR-6 exists to prevent.

An existing try/except in unrelated code turns the new requirement into precisely the
failure it was written to stop. Follow FR-6 faithfully and you ship the bug. That is not a
question anyone thinks to ask about a three-sentence issue, and it was answered in a spec document
before any code was written rather than in a post-mortem.

The whole document is published alongside this article — FEAT-0007: Manifest-Backed Dashboard
Feeds — so the excerpts above can be read in context.

What the spec bought afterwards

Implementation merged thirty-one minutes after the spec did, across 15 files. What I want to
draw out isn't the speed — it's this line from the implementation PR's test plan:

AC-8b (migration's server_default behaving correctly against a pre-populated table) has no
automated test — this repo has no migration-test harness (integration tests use
metadata.create_all, not Alembic). Verified by construction: [...]

One acceptance criterion, honestly marked unmet, with the reason. That's only possible because
AC-8b existed to be unmet. Without the spec there'd be no gap — just a migration nobody had
thought to test, and no record that anyone noticed.

A spec's most underrated property is that it gives the work something to fail against.

Doing it on purpose

The shape I'd suggest, in the order that matters:

Keep writing bald issues. The three-sentence note is doing its job. Don't move the cost back to capture time; you'll just stop capturing.
Put the interview at triage pickup, not at capture. This is the whole trick. At pickup you're already thinking about the problem and the agent can read the code now, against the repo as it currently is rather than as it was when you filed.
Make the agent read before it asks. Questions that arrive with file paths attached are answerable in a sentence. Questions that arrive cold are homework.
Scale the output to the work. A small change wants an implementation-plan comment on the issue itself. A feature wants a numbered document — mine live at docs/specs/FEAT-NNNN-<slug>.md, and get their own PR ahead of the implementation.
Write it back where the issue lives. This one I learned by losing it. Enrichment that stays in the session dies with the session — the issue still reads like three sentences a month later, and the next pickup starts from nothing. FEAT-0007 survives because it was committed; plenty of equally good reasoning hasn't.

And the honest limits. This costs about half an hour of agent time and a handful of real
decisions from you, which is absurd overhead for a typo and roughly break-even for a
one-file fix. It leans hard on the agent being able to read everything relevant — the
insight that reversed my issue came out of a sibling repository, and an agent without that
repository would have cheerfully specified the parser. And it can't manufacture intent: if
you genuinely don't know what you wanted, the interview surfaces that rather than solving it.
Which is still better than finding out from the diff.

None of this is a claim that issues should be specifications. Mine aren't and won't be. The
issue is a pointer, written in the two minutes I could spare, and it's the right artifact for
that job.

The interview is just the step that dereferences it — the one I used to run silently, in my
own head, without ever noticing I was running it at all.

Applicable to Coding Agents and Engineers

None of the above is specific to working with coding agents. The agent-led interview that spawned the spec doc is recognisably a tech review with a peer, team lead or stakeholder.

AI assisted software development is not so much about learning entirely new paradigms as it is about having the immediate opportunity to enlist the expertise you need to do a job well.

Trust, Doubt, and a Production Rate-Limit Incident

Philip Shaw — Sun, 12 Jul 2026 00:00:00 +0000

A production incident post-mortem where the debugging partner was an AI agent — the real story is when I believed it, when I didn't, and why that mattered.

Quieting 4xx noise in FastAPI + Logfire without losing your 500s

Philip Shaw — Wed, 01 Jul 2026 00:00:00 +0000

A one-line APIRoute fix for a non-obvious interaction between FastAPI's exception handling and Logfire's tracing instrumentation.

Your CLAUDE.md Is an Architecture Document — Write It Like One

Philip Shaw — Tue, 30 Jun 2026 00:00:00 +0000

The patterns worth stealing from a FastAPI project whose agent-instructions file quietly became its architecture doc — layering, config quarantine, the DI trap that breaks transactions, and more.

DEV Community: Philip Shaw

Sentinel: Ingest Path and Log Schema

Two logs, not one

Schema

Where ingested_at comes from

The registry is versioned data

Ingest pipeline

Projection

Restart

Reprocess

The horizon is per class, and partial reprocess is refused

Retention and rollups

Exit criteria

Sentinel Dev Diary: Where the Check Lives

The contract said both, and the code believed the one with a code block

The audit did not mention it

The step went amber for a day

Three checks, and one place the third can go

The register said otherwise, for three pull requests

Two smaller ones, and they rhyme

What the shape is

Sentinel: The Driver Contract

Process model

Assignment, not configuration

Lifecycle

Bus subjects

Stream limits

Northbound: the observation

Push and poll are the same thing

Buffering

Availability

Southbound: commands

Discovery and quarantine

What drivers must never do

Conformance

Exit criteria

Sentinel Dev Diary: What the Specification Left Unsaid

Six rejections became twenty-eight

Turning prose into types found a missing number

Three smaller decisions, one pattern

What the step did not deliver

A guard that existed as a function with a fixture

The register had no row for addresses

The transition criterion was delivered in half

The shape of the whole thing

Sentinel: The Semantic Model

Identity

Hierarchy

Point descriptor

Retention class

The semantic axes

Vocabularies are closed

Enums and codebooks

Quality

Time

Change vs refresh

The observation envelope

Normalisation is a write-time decision

Registry authority and the two failure cases

The quantity table

Subject and subject_ref

Vocabularies as code

Exit criteria

Sentinel Dev Diary: Four Registers and a Drift Check

Two false starts

The four registers

The distinction that is actually load-bearing

Snippets from the first three steps

The status marker that lies

The drift check

Why this shape, specifically

What is still unowned

Sentinel: An Introduction

What is Sentinel?

How this series is organised

Reading order

Scope

The two decisions that shape everything else

Non-goals

The planes

Where `ingested_at` comes from

The register had no row for `addresses`