Sentinel: The Rules Engine

Watchdog & Derived Points turned staleness and derived values into ordinary transitions on ordinary points, and The Command Plane took commands away from rules altogether. That leaves the rules engine very little to do. A rule watches a few points, evaluates a condition when one of them changes, and holds a claim while the condition is true.

A rule keeps no state between evaluations. It sees the current value and quality of each input, without timestamps or history, and a rule that needs a fifteen-minute average or a debounce takes a derived point as an input instead. So every rule is a pure function of what it reads. A restart can't leave a latch out of step with the world, and a test is a snapshot of inputs and the claims expected from it.

A rule has two kinds of output. A claim is idempotent, and the engine diffs it against what the rule already holds, so a rule evaluated every five seconds sends nothing to arbitration until its answer changes. An action, such as a notification, is not idempotent, which is why every action must have a cooldown. Without one it will eventually fire in a loop.

Restarts are the hard case. Values restored from before an outage look like real readings, so a rule waits in warmup until its inputs are live again instead of reacting to edges that never happened. Interlocks are the exception. A fail-safe interlock asserts its protective value as soon as the core starts, because warmup is when the system knows least. The engine also limits how often a rule can evaluate and how often a point's claims can change. Those limits are the backstop for a pump whose own running switches it off again, a loop that no static check will find.

What this document owns: the shape of a rule, the four trigger types, level-triggered evaluation and the single pass after derived points settle, the claim diff, assertion and release, actions and their mandatory cooldown, the three input-loss policies and the bound on hold, the quality a rule requires of its inputs, the warmup policies and why interlocks skip them, the per-rule evaluation limit and the per-point limit on claim changes, load-time validation of a rule set, and fixture and replay testing. It does not own the expression grammar or the derived points a rule takes as input, which belong to Watchdog & Derived Points; what a band means in arbitration, how a rule's band is checked at registry load, whether a claim reaches a device, the escalation of an interlock claim that can't be enforced, or the command rate limit, which belong to Command Plane; the ingest lag that pauses a warmup deadline, which belongs to Core Runtime; or the self point that flips when a rule is suspended, which belongs to Observability. The channel an action notifies is not defined anywhere in this specification.

The engine is small by construction. Almost every trigger type a control system needs is already a derived point or a quality transition, so what remains is a predicate over transitions and a claim as output.

Rule shape

- id: pump-run-on
  band: automation

  inputs:
    flow:   boiler.ch1.flow-temp
    demand: boiler.ch1.demand

  trigger:
    on: transition
    of: [flow, demand]

  when: |
    demand.value == false and flow.value > 35

  claims:
    - point: boiler.ch1.pump
      value: true
      expires_in: 20m

  on_input_loss: release
  warmup: all_inputs_live

Declarative in YAML, in git, loaded into the registry like everything else, versioned the same way. band is the rule's whole position in arbitration; there is no priority number alongside it.

when is written in the restricted expression grammar, which is defined in Watchdog & Derived Points and owned there. The only thing that differs in this context is the binding: names resolve to the full (value, quality) pair rather than to a bare value, so flow.value and flow.quality are both available and neither is implied. A rule that reads a value without considering its quality is usually a bug, and here it has to say so.

Beyond value and quality there is nothing. No timestamps, no history, no lookback. A rule needing the average of the last fifteen minutes takes a derived point with a name, not an inline window. This is the constraint that keeps the engine from growing a query language.

Triggers

trigger:
  on: transition            # any transition of the listed inputs
  of: [flow, demand]

trigger:
  on: edge                  # value crossing, direction-specific
  of: demand
  edge: rising | falling

trigger:
  on: quality               # quality transition
  of: flow
  to: [stale, unavailable, bad]

trigger:
  on: interval              # periodic re-evaluation
  every: 5m

edge is sugar over transition plus a prev/new comparison, and it exists because rising and falling are the terms people actually think in. It is defined on the changed flag, so it respects the point's deadband — an edge is a real change, not a resampled one.

Thresholds are not a trigger type. A rule that wants "above 35 °C" either compares in when, or, where it needs hysteresis, takes a hysteresis derived point as input. The second is correct for anything driving a relay.

interval exists for rules whose condition is time-dependent rather than data-dependent, and it should be rare. If most rules use it, something is wrong with the derived-point layer.

Evaluation

Level-triggered, not edge-triggered, in the sense that matters: the trigger decides when to evaluate, when decides what the world looks like, and the claim is derived from the current evaluation. The engine does not accumulate state between evaluations.

That means a rule is a pure function of its inputs' current values and qualities. Restart-safe by construction, testable by feeding it a state snapshot, and free of the class of bug where a rule's internal latch drifts out of sync with reality. Anything that genuinely needs memory — debounce, hysteresis, rate — is a derived point, which is durable and rehydratable from the log.

Evaluation order: transitions arrive in batches from ingest, derived points settle in topological order, then rules evaluate. One pass. A rule cannot trigger on a claim made by another rule in the same pass, which is deliberate — that path leads to ordering dependence, and the intermediate is better made an explicit derived point.

Claims are diffed against the rule's existing claims. Unchanged claims are not re-asserted, so a rule evaluating every 5 s does not generate command churn. Only the diff reaches arbitration.

Claims and release

A rule holds claims. when true means assert; when false means release. Release is the default and should be, because a rule that asserts and never withdraws permanently occupies its band.

A rule's output ends at the claim. Whether the claim reached the device, whether it converged, and whether the driver was even up are the command plane's concerns, and the rule can see none of them. That is deliberate: a rule that could observe delivery would eventually be written to retry, and retry logic scattered across a rule set is how two things end up pulsing the same relay.

Some rules genuinely want to fire and forget — send a notification, pulse a door strike. Those are actions, not claims:

  actions:
    - kind: notify
      channel: ops
      message: "Flow temp above 80 in {{ zone }}"
      cooldown: 15m

Actions need a cooldown, mandatory. An action without one will eventually fire in a loop, and the difference between a claim and an action is precisely that a claim is idempotent and an action is not.

expires_in on a claim is a safety net, not a mechanism. It bounds the damage if a rule stops evaluating for a reason nobody anticipated.

Input loss

The same fork as interlocks, required on every rule:

• release — withdraw claims, fall through to the next band. The right default for ordinary automation.
• hold — keep existing claims, do not re-evaluate. For cases where the last known good decision is better than nothing, and where the point has other protection.
• assert(value) — take a defensive position.

hold needs a bound: hold_max: 30m, after which it degrades to release. Holding indefinitely on stale data is the failure mode where the system looks fine and is operating on a picture of the world from three days ago.

What counts as loss is per rule: required_quality: live by default, live_or_stale where the rule tolerates it. A rule reading a 24-hour average should not lose its footing because one input was late.

Input loss is about the rule's inputs. It says nothing about the point the rule claims, which may be unreachable at the same moment for the same underlying reason without that changing what the rule should decide.

Warmup

A rule starts in warmup and does not evaluate. It leaves warmup when its policy is satisfied:

• all_inputs_live — every input has reached live since core start
• all_inputs_known — inputs are live or stale, but not unknown or restored
• deadline(60s) — a hard cap; on expiry, evaluate under the rule's on_input_loss policy

Warmup exists because restored values look like real values. Without gating, a restart evaluates every rule against a snapshot of the world as it was before the outage, generates a burst of edges that never happened, and issues commands based on them.

Warmup deadlines pause while ingest lag is high, on the same condition that suppresses staleness in Watchdog & Derived Points. The deadline exists to stop a rule waiting forever for an input that is never coming; during a drain the input is coming, just not yet. Letting it expire would push rules into their input-loss policy while the data they need sits in the queue — safe, but premature and for the wrong reason. A core-internal condition must not drive device-level decisions.

The practical effect after a long outage is that the system declines to automate until it has caught up, rather than making hours of retrospective decisions in ninety seconds. That is the intended behaviour, not a stall.

Interlocks skip warmup, and this is the wrinkle worth being explicit about. A fail_safe interlock asserts its fail_value the instant the core starts, before any inputs are live, and releases it only once inputs are genuinely live and the condition evaluates clear. That is the correct reading of fail-safe: the protective state is the default, and warmup is exactly the window with the least information.

That assertion is a claim, not a command. At core start no driver instance is up, so the claim does not reach hardware immediately, and it does not need to: the command plane records it as undeliverable and issues it the moment the instance is ready. The rule does not wait, does not retry, and does not get to know. An interlock claim that stays unenforced is escalated by the command plane, which is where the visibility into delivery actually exists.

A fail_clear interlock does the opposite — it holds no claim during warmup, same as any other rule. The asymmetry is the point.

Loop and flap protection

Three limits, all enforced by the engine rather than left to rule authors:

• Evaluations per rule per window. Past the threshold, the rule is suspended, a control event is written, and a self point flips. A suspended rule releases its claims under its on_input_loss policy.
• Claim changes per point per window. Independent of which rule caused them. This catches two rules fighting at the same band, which the load-time lint warns about but cannot always predict.
• Command rate, already enforced in the command plane per point descriptor. This is the last line and it protects hardware regardless of what the rule layer does.

A rule whose own claim feeds back into its inputs — pump on, flow temp rises, rule turns pump off, temp falls — is a real physical loop that no static analysis catches. Hysteresis derived points are the primary defence; the evaluation limit is the backstop.

Load-time validation

All of this runs in CI, because a rule set is small enough to check exhaustively:

• Every input point exists in the current registry version
• when parses under the shared grammar and references only declared inputs
• Claim target points are out or inout, and claim values pass type, enum, and range checks
• on_input_loss and warmup are present on every rule
• Interlock rules carry interlock_class and, if fail_safe, a fail_value
• No two rules at the same band claim the same point with different values (warning, not error — sometimes deliberate)
• No two interlock rules claim the same point with different values (error)
• Actions have cooldowns

Testing

The shape that makes the constraints worth it: a rule takes a state snapshot and produces claims, so a test is a fixture and an assertion.

- rule: pump-run-on
  given:
    boiler.ch1.flow-temp:  { value: 42, quality: live }
    boiler.ch1.demand:     { value: false, quality: live }
  expect_claims:
    - { point: boiler.ch1.pump, value: true }

No device, no bus, no database. The whole rule set runs against a table of fixtures in CI. The cases covered by convention are: condition true, condition false, each input stale, each input unavailable, and warmup.

A second layer, once the first works: replay. Take a window of real observations from the log, run the rule set against it, and diff the claims against what actually happened. That is how a rule change is validated against real conditions before it touches hardware — and it is only possible because the log kept value_native and the rules are pure.

Exit criteria

The fixture table and the replay layer above are this step's exit criterion. A rule set is done when the whole of it runs against fixtures in CI, covering by convention: condition true, condition false, each input stale, each input unavailable, and warmup.

Four cases worth adding to that convention, because they are the ones where a rule does something plausible and wrong:

• A rule that never leaves warmup because an input is permanently unknown — assert it holds no claims rather than evaluating on absent data.
• Warmup deadline pausing under lag, so a long drain does not push rules into their input-loss policy while the data they need is still in the queue.
• Release on input loss actually releases, and hold degrades to release at hold_max rather than holding indefinitely.
• A fail-safe interlock skipping warmup asserts its fail_value immediately, and releases only once inputs are live and the condition evaluates clear — not on the first good reading.

Load-time validation is the other half, and it is a test rather than a hope: a fixture rule set containing each of the listed violations fails the build with the right message, including the two that are errors rather than warnings.

Replay is the acceptance gate for a rule change rather than for the step: take a window of real observations, run the modified rule set, diff the claims against what actually happened. Nothing touches hardware until that diff is understood.

---

On Monday - the Dev Diary. The latest on where the specification and the build disagree as the system takes shape, and which of the two had to change.

The next part - The Core Runtime. Why only one core may run at a time, what enforces it (a Postgres advisory lock, not a lease), what breaks silently if two run at once, and what the system does while no core is running.

Start of the series: An Introduction. The map, the two decisions every later document is downstream of, and why a specification at this scale is being published in public while the system it describes gets built.