Two terminals. The left one is scrolling so fast it's basically a waterfall: my plant simulator pushing ten thousand sensor readings a second at the Rust gateway I've spent the last few weeks building. The right one is calm. It has one command typed and waiting:
ssh db_host 'sudo systemctl stop postgresql'
I'm about to kill my own database. At full load. During a simulated gas leak.
If I built this thing right, nothing important will happen. If I built it wrong, I'm about to watch alarms from a methane emergency vanish into the void, and everything I wrote in Part 1 about "safety-critical design" was just decoration.
I hit enter.
Where We Left Off
Part 1 built the world: a private OpenStack cloud on a mini PC under my desk, five public domains through a zero-trust tunnel, a time-series database with its survival machinery armed. Good bones. But bones don't do anything.
Part 2 is the nervous system: hazshield-ingest, a single Rust binary on a 2-vCPU VM with a 512MB memory cap. Every sensor reading in the plant passes through it, and in about six microseconds it answers three questions: do I know you, are you lying, and are you on fire?
Then it routes you by what you deserve:
The entire architecture lives in that fork, and honestly, in one method call. Bulk telemetry rides the warm lane with try_send: if the system is drowning, bulk politely gets shed. Violations ride the hot lane with send().await: if the alarm pipeline is saturated, the caller waits. Slow truth beats fast silence. I will die on this hill, and this post is me showing you the receipts.
Teaching a Running System New Rules
First problem: the gateway has to know 3,000 sensors and their thresholds, on the hottest path in the system. A database lookup per reading? Absurd. A mutex around a shared map? A traffic jam wearing a safety vest.
Instead, the whole plant lives in memory as an immutable snapshot behind an atomic pointer. Reads cost nothing. When someone edits a threshold in Postgres, a message on a Redis channel rings a doorbell, the gateway rebuilds the snapshot, and swaps it in atomically. Requests already in flight finish against the old rules; the next request sees the new ones. Nobody stops.
The first time I updated a threshold and watched the config generation tick from 1 to 2 in the health endpoint, mid-traffic, no restart, I just sat there grinning at a JSON blob. You know the feeling.
Sabotage Night, Act I: The Haywire Sensor
Industrial sensors fail in creative ways, and one of them is screaming. So before the big run, I gave one 1Hz temperature sensor a hundred readings in a single batch, like a device with fried firmware.
The gateway checked its registry, saw the sensor had declared itself at 1Hz, did the math, and stored exactly five readings, the polite burst allowance. Ninety-five came back stamped rate_limited.
Then the test that actually matters. Same flood, but with critical values. A broken sensor that might also be reporting a real emergency. Result: storage accepted zero of the fifty readings. The alarm stream received all fifty.
Sit with that for a second. The rate limiter and the safety check disagree about the same fifty readings, and they're both right, because they guard different things. The limiter protects the disk. Nothing protects the disk from the alarms, because nothing is allowed to.
Act II: The Leak
The main event needs a worthy adversary, so the simulator runs three thousand virtual sensors on mean-reverting physics, tuned so that steady state produces exactly zero violations. That tuning is sneaky-important: it means every alarm in the run is caused, and therefore countable. Remember that for the ending.
At the four-minute mark, the script leaks methane. Thirty-four CH4 sensors in one zone have their baseline dragged toward 1.5x critical over ten seconds, and then physics does the rest, sensor by sensor, each one crossing its own threshold as its random walk drifts up:
Nineteen criticals. Then 686. Then 1,817. Then 4,625, still climbing, while the gateway keeps judging ten thousand readings a second with a client-side p99 of three milliseconds. The flood sensor is still screaming in the background this whole time, by the way. Over a million of its readings got politely declined by the end of the run.
Act III: The Execution
And then, terminal two. Postgres dies at full load, mid-cascade.
Here's what ten thousand readings a second hitting a gateway with no database behind it looks like from the outside: nothing. The request stream doesn't blip. req_errors=0, before, during, after. Which sounds anticlimactic until you realize that boring was the engineering goal, and every piece of it was built in calm daylight for exactly this moment.
Inside, three things happen at once. The writer starts buffering rows and retrying with climbing backoff, a quarter second doubling toward five, patient as a debt collector. The channel watermark trips and degraded mode engages: bulk storage drops to 1-in-10 sampling, every surviving row stamped with a quality flag so future-me knows this data is sparse and why. And threshold evaluation, the thing that actually keeps miners alive in the story this system tells, continues completely untouched. The criticals keep counting up right through the outage.
Forty-five seconds later I resurrect the database. One fat recovery COPY lands the entire buffer. And my favorite artifact of the whole night: the degraded-batch counter, which climbed during the outage, simply stops. Frozen at 883 for the rest of the run. The system bent exactly as long as it had to, then un-bent, and nobody had to tell it to do either.
(Redis got the same treatment on a different evening, because the alarm transport deserves its own assassination attempt. Violations spilled to an append-only file on local disk, complete JSON records written while their transport was dead, and when Redis returned, the file replayed into the stream and deleted itself. The proof of success was an absence.)
The Audit
Load tests lie. Pretty dashboards during a demo prove nothing about loss, because loss is invisible unless you make it countable. This is why the simulator's zero-ambient-violations tuning matters: at the end of the run, the arithmetic must close.
alarms in the Redis stream == warn + crit the gateway counted exact match
hot_violations_lost_total == 0 the sacred number
client p99 == 3.0 ms target was 10
request errors == 0 through a database murder
Every alarm that occurred, accounted for, to the digit, through the death of the primary datastore. Not "we believe no data was lost." Counted.
What I'm Keeping
Three convictions came out of this phase. Guarantees belong to data, not to systems. The moment I stopped asking "is my pipeline reliable?" and started asking "what does this reading deserve?", most hard decisions collapsed into one-liners. Degradation is a feature you write, not a failure you suffer. The sampling mode, the quality flags, the shed-oldest policy: all authored before they were needed, so the 3 AM incident had somewhere soft to land. And an audit beats a thousand assertions. Design your load test so that loss has nowhere to hide, or you haven't tested anything.
Next Time: The Machines Start Thinking
There are twenty-five thousand alarm events sitting in a Redis Stream right now, and nothing is reading them. Phase 3 gives them a reader: an async Python worker cluster that folds raw violations into alarm episodes, builds hazard context from the plant's zone graph, and hands it to a locally hosted LLM with one job: plan the isolation. Which conveyor stops. Which zone seals. Which way the air goes.
The reflexes work. Next we build the brain.
Part 3 lands when the machines start planning. Subscribe if you want to watch an 8B parameter model learn mine safety.
Top comments (0)