import PlaneArchitecture from '../../assets/sentinel-plane-architecture.svg';
Does it matter if a soil moisture sensor goes quiet for three days?
On its own, probably not. But if an irrigation decision is derived from that reading, three days of silence is three days of watering against a number that stopped being true, and nothing in the system said so.
Move up the scale. Indoor air quality is degrading because the air conditioning has stopped working — not thrown an error, stopped, which from the outside looks identical to an air conditioner with nothing to do. Move up again. A battery bank charging off solar, in an enclosed space, begins to climb in temperature. Do you want to know about that now, or the next time you happen to look at a dashboard?
Those three questions have the same shape but completely different answers and priorities. What separates them is not technical, it's a judgement about what matters, how much, and how quickly. That call gets made somewhere regardless. Written down, it is a specification. Left unwritten, it is whatever the code or user happened to do.
Problem Statement: Underneath all three questions is the same problem: a system that cannot tell a reading from a memory of one. A value that arrived three days ago still presents as a value, a device that has stopped reporting looks like a device with nothing to report, and a command that was sent looks like a command that worked — none of it announcing itself, all of it producing plausible numbers and unbroken graphs right up until a decision gets made against something that stopped being true. Sentinel exists to make that difference explicit at every hop, so that acting on a number the system cannot vouch for is a choice rather than an accident.
There is an obvious objection, which is that this is a solved problem and you can buy it. However, I have spent a long time cajoling systems and their processes to fit around real world situations they were not designed to address — reframing the problem until it fitted the tool. That works until the reshaping costs more than the tool saves, and at that point it is worth exploring something with the flexibility those use-cases actually need.
So this design is written down, and in public.
A specification at this scale is not trivially derived from a set of working assumptions; the assumptions are the easy part, and they usually create a false sense of simplicity that unravels the moment you think the work is finished.
Publishing holds me to the choices I made during the design. Not as choices I must live with, but ones that I must litigate clearly when I adopt or change them. What I decided, what I claimed they would achieve, and what had to change when the design met the thing it describes.
That pressure is the same in a one-maintainer repository as it is in a corporate monorepo, and so are the failure modes. Arguing the changes in public makes each one a worked example rather than a commit message.
The spec is here to be read and argued with.
What is Sentinel?
A platform for monitoring and controlling devices across several network segments: collecting readings over mixed protocols, normalising them into a single semantic model, deriving values from them, evaluating automation rules, and driving physical outputs with verification and safety interlocks.
The name is the job. A sentinel stands watch, knows the difference between quiet and absent, and raises an alarm rather than acting on a guess. Two properties of this design follow that description more closely than they follow any conventional monitoring platform: data quality is a first-class field, so the system is explicit about what it does not know; and absence of data is a signal in its own right rather than a gap to be interpolated over.
It is written for a system built and operated by one person, and several decisions follow from that constraint rather than from first principles. Where they do, the constraint is stated alongside the decision.
Components take the system name as a prefix: sentinel-core, sentinel-driver-modbus, sentinel-registry.
How this series is organised
Eleven parts, published weekly. This one is the map and the rationale for Sentinel's design; the ten that follow are the documents that define it.
Every mechanism in this system has exactly one owning document. Where this part mentions a state set, a policy name, or a lifecycle, it names it only to say where it is defined.
Each part is published as it was drafted, rather than as a tidied-up retrospective. The system is being built while the series runs, so where the plan meets reality and loses, a follow-up says what changed and why. Three figures in the design are unmeasured today — sustainable ingest throughput, cold-start time, and the retention policy values — and all three get measured during the build, inside the window this series runs in. You will see the numbers when I do.
Reading order
- The Semantic Model
- Driver Contract
- Ingest Path & Log Schema
- Watchdog & Derived Points
- Command Plane
- Rules Engine
- Core Runtime
- Security Model
- Observability
- Retention & Compaction
Core Runtime, Security Model and Observability come after the mechanisms they constrain: what enforces single-instance operation and what the system is sized for, which boundary carries the authorisation weight, and what signals every mechanism must expose.
Retention & Compaction reads last because it uses descriptor fields defined across four earlier documents, but it is a mechanism rather than a policy appendix — it constrains the log schema, the derived-point layer and the reprocess operation, and part of it cannot be deferred without a data migration.
Scope
These ten documents cover the core and the data pipeline: collection, ingest, state, derivation, automation, control, runtime, security, instrumentation and storage lifecycle.
Two things have been deliberately left out, because neither changes anything above: alert routing and notification delivery, and the read API and UI. Observability says what must be watchable and what should raise an alarm; but how an alarm reaches a person is not specified here. Rules Engine names actions with a channel; the channel model is not specified here either.
The one constraint those inherit: alerting runs outside the core, because alerting inside the core cannot report the core's absence.
The two decisions that shape everything else
1. Log-first core. Current state is a projection of an append-only event log, never the primary artefact. Every observation, quality transition, command and command outcome is an appended event. Restart is snapshot plus replay of the tail. This is the whole answer to restarting without losing state, and retrofitting it later is painful, so it goes in first.
2. Hard driver/core boundary. The core has zero protocol knowledge. Drivers have zero knowledge of rules, units or storage. The boundary is exactly two contracts: a normalised observation flowing north, and a command envelope flowing south. The failure this prevents is an observability or collection concern leaking outward into the layers around it, so that every part of the system ends up coupled to whichever part was built first. Making the boundary a wire protocol between processes means it cannot be casually violated.
Everything else in these documents is downstream of those two choices, and most of it follows fairly directly.
Non-goals
This is a system built from scratch, with no migration path in or out. No document should acquire a migration section, a dual-run plan or a compatibility shim.
Multiple concurrent cores are not a goal. Exactly one core instance runs at a time, enforced rather than assumed. The reasoning, the enforcement mechanism, and the two conditions that would reopen the question are in Core Runtime.
Keeping raw data indefinitely is not a goal. Nothing is retained in original sample detail forever; measurements are rolled up and their samples dropped. State transitions are a different thing and are kept — the distinction is samples versus history, and it is in Retention & Compaction.
A second storage tier is not a goal. No object-store archive, no cold tier. Data not worth keeping in Postgres is not worth keeping.
Nothing here describes existing infrastructure. These documents specify what to build. Where they name a protocol, a device or a deployment shape, that is a design target rather than an inventory, and no document should assert that any component currently exists.
The planes
The diagram is illustrative. This section is the normative description of the decomposition; where the two disagree, this text wins and the diagram should be corrected.
Devices are anything that produces readings or accepts writes, over any protocol, push or poll.
Driver plane. Protocol adapters, deployed as driver hosts, one per network segment, each supervising per-device driver instances. Drivers translate a protocol into observations and attempt writes. They hold no semantics, no verification state and no automation. Defined in Driver Contract.
Bus. Durable subject streams carrying observations, availability, quarantine, commands and command results. Every driver-plane subject leads with the host, which is what makes per-host credentials workable. The observation stream is the system's outage buffer, bounded by size rather than age and sized from tolerable outage and utilisation — see Core Runtime. Subject layout is defined in Driver Contract; it is also the primary authorisation surface, so the two are designed together — see Security Model.
Core. A single process — see Core Runtime — containing four stages that are logically distinct and physically together:
- Normalise — validate, deduplicate, resolve against the registry, check the publisher, convert to canonical units, evaluate the deadband. Defined in Ingest Path & Log Schema.
- Event log — the durable append-only record, split by volume, retention class and query shape. Schema in Ingest Path & Log Schema; the class split in Retention & Compaction.
- State and registry — the current projection, plus the versioned registry that says what every point means. Projection defined in Ingest Path & Log Schema; registry content defined in The Semantic Model.
- Derivation and watchdog — freshness deadlines and derived points, both of which emit into the same pipeline as device data. Defined in Watchdog & Derived Points.
The diagram places derived points and timers inside the rules engine box. That is an over-simplification, and it is the one worth flagging, because it survived several drafts before the distinction it blurs turned out to matter. Derived points are ordinary points, computed before rules evaluate; the watchdog that ages them is not part of the engine at all. That separation is exactly what keeps the rules engine small, and it is argued properly in Watchdog & Derived Points.
Rules engine. Evaluates predicates over transitions and produces claims. It sends no commands, holds no state between evaluations, and has no visibility into whether a claim reached hardware. Defined in Rules Engine.
Command plane. Owns the durable claim set and its resolution into a desired value, deliverability and reconciliation of reported against desired, confirmation and verification, and the safety and interlock model. Returns to the bus as commands to driver instances. Defined in Command Plane.
The single closed loop is: device → driver → bus → core → rules → command plane → bus → driver → device, with the log recording every hop.
Scale
The design targets, stated once here and derived from in Core Runtime: 200+ devices across 16+ network segments, 2,000+ device timeseries points, 1,000+ derived points, and roughly 700 self-points that fall out of instrumenting the rest — about 4,000 points in total, at ~220 observations per second, designed for 500.
Two figures matter more than their size suggests. Self-points are a fifth of the registry and are easy to omit when sizing anything. And sustainable ingest throughput is the one term still unmeasured, which makes it the term everything else waits on.
Where things are defined
| Concern | Owning document |
|---|---|
| Point identity, hierarchy, descriptor fields | The Semantic Model |
| Closed vocabularies: quantity, subject, enum types, retention class, modifiers | The Semantic Model |
| Units, canonicalisation, affine conversion, aggregation safety | The Semantic Model |
| Quality states and legal transitions | The Semantic Model |
| Timestamps and sequence numbers | The Semantic Model |
| Observation envelope | The Semantic Model (shape), Driver Contract (driver-supplied half) |
| Registry authority, quarantine and unit-mismatch handling | The Semantic Model |
| Driver host and instance model, lifecycle, assignments | Driver Contract |
| Bus subjects, stream limits, buffering, availability | Driver Contract |
| Driver-side command handling and the prohibitions list | Driver Contract |
| Log schema, hypertable split, ingest pipeline ordering, dedup, publisher check, projection, restart, reprocess | Ingest Path & Log Schema |
| Registry versioning and descriptor storage | Ingest Path & Log Schema |
| Freshness policies, watchdog behaviour, staleness suppression, flap damping | Watchdog & Derived Points |
| Derived point descriptors, operators, quality propagation, evaluation, rehydration | Watchdog & Derived Points |
| Restricted expression grammar, and its two binding contexts | Watchdog & Derived Points |
| Claim model, bands, resolution, suspension | Command Plane |
| Desired/reported, deliverability, convergence, command state machine, confirmation modes | Command Plane |
| Fail-safe versus fail-clear, interlock constraints | Command Plane |
| Intent API, validation, rate limits, idempotency, escape hatch | Command Plane |
| Rule shape, triggers, evaluation, claims, warmup, loop protection | Rules Engine |
| Rule and registry load-time validation, rule testing | Rules Engine |
| Single-writer enforcement, core epoch, startup and shutdown ordering | Core Runtime |
| Scale targets, outage figures, buffer sizing, drain behaviour | Core Runtime |
| Backpressure and ingest lag | Core Runtime |
| Availability while the core is down, restart time, geographic split | Core Runtime |
| Threat model, principals, credentials, bus permissions | Security Model |
| Human authentication, capabilities, what each principal may do | Security Model |
| Secret handling, audit requirements, exposure assumptions | Security Model |
| Self-point catalogue, naming, what raises an alarm | Observability |
| Retention classes, policy values, rollup derivation, compaction | Retention & Compaction |
| Exit criteria for each build step | the document owning that step's mechanism |
Conventions
Most of these were extracted from a specific error rather than asserted as principle, and each names the failure it prevents.
One definition per concept. Cross-reference rather than restate, including in this note. A concept that appears in two documents has one normative home and one pointer.
Vocabularies and schemas are code. The closed vocabularies, point descriptors and rule definitions live as versioned files in the repository and load into Postgres by migration. Prose in these notes explains them; it does not define them once the code exists.
State machines are enums with tests. Quality transitions, driver lifecycle, command lifecycle, convergence and arbitration bands appear as tables in prose here and must exist as checked enums in code, with a test that no code path produces an illegal transition. Prose tables are for readers and will rot.
Every claim a document makes has a test that would fail if it stopped being true, and that test lives in the same document. Prose cannot fail loudly. The exit criteria at the end of each document are that test set, and they are what says a build step is finished — not a feature checklist, but a demonstration that the step's central property actually holds.
A boolean that has to carry three meanings is an enum. Quality was the first instance and convergence the second. When a field needs to say "not yet", "no", and "not applicable", write the enum the first time.
Store what composes, not what is convenient to read. Native alongside canonical, sum and count rather than average. Anything derived can be recomputed from a composable base; a base that has already been reduced cannot be recovered, and the reduction is usually wrong at the next level up.
Storage optimisations that discard raw rows are not optimisations. Derived-point windows and reprocess both read the raw tables, so a row not written is a mechanism quietly broken. Every observation writes a row; volume is managed by retention and compaction, which are deliberate and bounded, rather than by suppression at write time.
Behaviour that varies by data type is derived from the descriptor, not authored per point. Which rollup functions are valid follows from value_type, agg_safe and the aggregation modifier; which retention class applies follows from kind. Restating what the registry already knows is an invitation to state it wrongly.
One number per concept. Where an ordering exists, it is expressed once. Bands are an ordered enum and there is no parallel priority integer; the same discipline applies anywhere else a rank appears.
Two numbers that differ by an order of magnitude are two numbers. Planned restart and tolerable unplanned outage were briefly treated as one quantity, which would have sized the outage buffer against the restart case. Before reusing a figure in a second place, check that the two cases actually coincide.
Identity comes from the message, not from the moment of handling. Anything a component assigns to an event at processing time is different on a retry, and therefore cannot be part of an identity or dedup key. This is why ingested_at is taken from the bus rather than from the core's clock.
Authority comes from the connection, never from the payload. Who a request or a message is from is established by the credential that carried it or the binding it arrived on, never by a field inside it. Owner and band are derived from the authenticated principal; a driver's right to publish for a point is checked against the registry rather than taken from the subject it chose.
Assumptions that several documents rely on are enforced somewhere. Single-writer was assumed by five mechanisms before anything checked it. An assumption held by convention is one that gets broken on a quiet afternoon, and the failure modes that follow are usually silent.
Nothing expires data by age except a retention policy chosen for it. Age limits on a buffer delete messages that were never consumed, silently, between components that both believe the handoff succeeded. Buffers are bounded by size and refuse new work when full; age-based deletion is confined to the places where dropping old data is the actual intent.
Physical layout that varies by policy is decided before there is data. Retention attaches to a table rather than to a row, so a per-class policy is a table split, and a table split after the fact rewrites everything ever collected. The values can wait; the shape cannot.
An operation that cannot cover its whole scope refuses, rather than covering part of it. Reprocess spans retention classes with different horizons, and correcting the reachable two thirds would leave right and wrong data indistinguishable. The scope is validated first, the refusal carries a reason, and narrowing it is a deliberate second command.
A signal about a component must not depend on that component. Core self-points are injected at the projection rather than published through the queue they describe, and alerting runs outside the core so it survives the core being down. A monitor that fails with the thing it monitors is not a monitor.
Never report a fact about the core as a fact about a device. Ingest lag would otherwise surface as fleet-wide staleness, and the system would take protective action across the estate because it was slow. Where the cause is internal, the system says so and suppresses the derived signal — staleness and warmup deadlines both pause on the same condition.
Wait on conditions, not on clocks. A timeout should mean the thing being waited for failed, not that a precondition had not arrived yet. Where a component waits for another to become ready, it gates on the readiness signal and the wait is unbounded; where the wait is genuinely unsafe to leave open, it escalates rather than expiring.
Deployment parameters are read, not assumed. Retention windows, intervals and limits set at deployment are checked against configuration at load time by querying them, never by hardcoding the current value. A constant copied into validation code is a constraint that silently stops holding the day the original changes.
A mechanism does not ship without saying how its failure would be noticed. Its signals go in Observability in the same commit that defines it.
Anything encoded as a string suffix will have to be parsed later. This applies to units, phases, aggregation and circuit position. It is promoted to a field instead.
Build order
- Semantic model and vocabularies — point identity, quantity, unit, subject, quality, retention class
- Event log, state projection, snapshot and replay, cold start
- Driver contract, plus one push driver and one poll driver
- Freshness watchdog and derived points
- Command plane, including arbitration and the confirmation modes
- Rules engine
- Instrumentation and timeseries export
Steps 1 and 2 are the expensive ones to change later. Everything from 3 onwards is additive if those are right.
Each step is finished when the exit criteria in its owning document pass. They live there rather than here, because a criterion is the proof of that document's claim and only stays truthful next to it. A separate test plan would drift within a month, leaving two places describing what ingest guarantees.
Security is not a step. The subject layout and the publisher check land in step 3, bus credentials with the first driver host, and forward auth with the API when it exists. Deferring it to a step of its own is how it becomes a retrofit.
Neither is the runtime. The advisory lock and the startup ordering belong in step 2, with the first process that writes; the core epoch lands in step 3 alongside the command envelope.
Neither is observability. Each step adds its own self-points as it goes, which is also how the ingest path gets exercised before there are devices to exercise it.
Retention is split across both: the class field, the hypertable split and the ingest routing belong in steps 1 and 2, because they are physical; the policy values and the compaction job follow once volume is measured.
Ingest throughput is measured during step 2, against the design rate in Core Runtime. It is the term the stream sizing and the recovery time both depend on, and it is cheap to measure with synthetic load before any driver exists.
Open items
- Sustainable ingest throughput. The only unmeasured term in the buffer and recovery arithmetic. Measured on per-observation validation, registry lookup and conversion in application code, which is where the ceiling sits — not on the batch insert.
- Cold-start time. The planned-restart number, dominated by derived-point rehydration at a thousand-plus derived points. Measured rather than estimated, and kept small.
- Retention policy values. The classes and the split are decided; the durations are not. They need measured volume, and the driver for measurement raw retention is how long it is plausible to want to reprocess.
All three are measurements rather than decisions, and the first two are taken during step 2.
Settled: scale targets and the four-hour tolerable outage, both in Core Runtime. Retention classes and the samples-versus-history distinction, in Retention & Compaction. Exit criteria, in each owning document.
Out of scope: alert routing, and the read API and UI.
Next, on 9 September — The Semantic Model. Why the primary key for every point in this system is a dotted string you can paste into a grep, and never a UUID.
Top comments (0)