I Built a Private Cloud That Runs an AI Safety System Under My Desk

Picture a mineral processing plant at 3 AM. A methane sensor in the crusher hall starts climbing. Within 5 milliseconds, the breach is flagged at the edge. Within 50, it's glowing red on the control room screen. And within seconds, a locally hosted AI has studied the plant's layout, worked out which zones connect, what equipment is running, and where the airflow goes, and produced a step-by-step isolation blueprint: shut down conveyor C4, reroute ventilation through shaft B, seal zone 7.

No cloud bill. No third-party AI API. No open ports on the network.

That's HazShield AI, a distributed industrial safety system I'm building end-to-end, and the entire thing runs on a single mini PC under my desk. This series documents the build. Part 1 is about forging the platform: a genuine private cloud, carved out of 28GB of RAM, exposed to the world through a zero-trust tunnel.

The System in One Picture

HazShield simulates a full industrial safety ecosystem: thousands of concurrent sensor streams for gas, temperature, vibration, airflow, and seismic activity, pouring into a Rust ingestion gateway, evaluated against safety thresholds on every single reading, stored in a time-series database built for the firehose, and escalated to an AI planner when things go wrong.

The design revolves around one idea I consider the heart of the whole project: not all data deserves the same promises. Every reading entering the system is routed into one of three lanes.

🔥 The Hot Lane carries threshold violations. Sub-50ms from sensor to screen. Never dropped, ever. If the message broker is unreachable, the edge spills violations to local disk and replays them when it recovers. A safety system that loses alarms isn't a safety system.

🌊 The Warm Lane carries bulk telemetry, batched and bulk-loaded into TimescaleDB thousands of rows at a time. Under extreme load, this lane deliberately degrades by sampling 1-in-N readings for storage, while threshold evaluation still runs on 100% of them. Storage fidelity bends; safety never does.

🧠 The Cold Lane is AI planning. When an alarm opens, workers assemble the hazard context (which sensor, which zone, what's adjacent, what's running) and queue it for a quantized LLM running on Ollama. Local inference is precious, so identical hazard scenarios are deduplicated by prompt hash. An alarm storm of fifty related sensors produces one plan, not fifty.

That asymmetry between what bends under pressure and what never does drives every technical decision in the stack.

The Stack

Layer	Technology
Private cloud	OpenStack (Kolla-Ansible) on a single mini PC
Provisioning	Terraform + Ansible, 100% infrastructure-as-code
Edge ingestion	Rust · Axum · Tokio
Orchestration	Python · FastAPI · async workers
Data	PostgreSQL 16 + TimescaleDB · Redis Streams
AI	Ollama, quantized local models
Control room	React · TypeScript · WebSockets
Public ingress	Cloudflare Tunnel → Caddy

Three VMs, three jobs. An edge node that owns the public doorway and the ingestion gateway. A state node running the time-series engine and the real-time message layer. A compute node where the AI thinks. Each is sized to the byte after profiling what the workload actually needs, because the whole cloud has 28GB to spend and every gigabyte is contested.

Squeezing a Cloud Into a Mini PC

Here's the fun part nobody tells you about private clouds: OpenStack is built to manage thousands of hypervisors, and it assumes it deserves production resources. A stock deployment spun up five API workers per service (five Neutron servers, five Keystone workers) to serve requests from exactly one human: me.

After profiling the control plane container-by-container, I tuned it down to single-worker mode and stripped the services my toolchain makes redundant (Terraform replaces Heat entirely). Three lines of configuration reclaimed over 4GB of RAM, the difference between a cloud that thrashes and one that hums:

enable_heat: "no"
openstack_service_workers: 1

My favorite discovery came from the storage layer. While validating the scheduler's capacity math through OpenStack's Placement API, I found the hypervisor reporting a fraction of the disk I knew the machine had. The trail led all the way down the stack to the LVM layer, where Ubuntu Server's default install had left 828GB of the SSD sitting unallocated, invisible to everything above it. One online resize later, the cloud had nine times the storage, with no downtime and no reinstall:

sudo lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv
sudo resize2fs /dev/ubuntu-vg/ubuntu-lv

If you run Ubuntu anywhere, check your volume group. You might be richer than you think.

Five Public Domains. Zero Open Ports.

The part of this build I show people first: HazShield has five public hostnames.

• app.… is the control room UI
• api.… handles orchestration and asset management
• stream.… carries live telemetry over WebSockets
• ingest.… is the sensor firehose
• telemetry.… fronts Grafana observability

And my home router forwards nothing. Not one port.

The trick is a Cloudflare Tunnel: a lightweight daemon inside the edge VM dials outbound to Cloudflare's network and holds the connection open. Public traffic rides Cloudflare's edge (TLS included), flows down the tunnel, and lands on a local Caddy instance that routes by hostname. From the internet's perspective, my house doesn't exist.

One tunnel, one Caddy, one config file. That means CORS policy, security headers, and request-size limits live in exactly one place instead of being smeared across four services. Even the tunnel daemon itself runs in a hardened, resource-capped systemd sandbox (MemoryMax=256M, no privileges, read-only filesystem) so nothing can starve the ingestion hot path that will share its node.

The network inside the cloud is just as locked down. Security group rules are expressed as intent ("the database accepts connections from members of the edge group") rather than as IP addresses. The state node's Postgres and Redis ports are unreachable from the internet, from my LAN, from anywhere except the two internal services that need them. Defense in depth, enforced twice: once at the cloud layer, once again in Postgres's own access control.

A Database Designed to Survive the Firehose

Do the math on "thousands of readings per second" and it gets scary fast: 5,000 readings/sec is 430 million rows a day. Store that naively and a 1TB SSD is dead in weeks.

So the data layer is built as a lifecycle, not a table.

• Raw readings land in a TimescaleDB hypertable in 2-hour chunks, compressed after 4 hours and deleted after 48. Raw data is ephemeral by design.
• Continuous aggregates roll everything into 10-second and 5-minute summaries, refreshed incrementally in the background. The dashboard reads these and never touches raw. Reads and writes are physically separated, which is why the UI stays fast while the firehose blasts.
• Alarm episodes are stored forever. A sensor breaching at 1Hz for an hour produces one alarm row with an updated timestamp, not 3,600 rows. The audit trail is permanent and compact.

The engine is tuned for its 6GB VM with one deliciously counterintuitive setting: synchronous_commit = off. Losing half a second of bulk telemetry in a crash is a fine trade for write throughput, and alarm writes will override it per-transaction, because durability in this system is a per-write decision, not a global one. Same philosophy on Redis: maxmemory-policy noeviction, because a message layer that quietly discards data under pressure has no business carrying safety alarms. When Redis is full, it refuses loudly, and that refusal is precisely the backpressure signal the edge uses to trigger its local spill.

Right after deployment, one query confirms the whole lifecycle machine is alive:

SELECT job_id, proc_name FROM timescaledb_information.jobs;
-- compression ✓  retention ✓  aggregate refresh ✓  all scheduled before the first reading exists

The Moment It All Clicked

Phase 1's finish line was a single request from my phone, on mobile data, standing in my kitchen:

GET https://api.<mydomain>/health
→ HTTP/2 200
→ {"status":"ok","service":"hazshield-api","phase":"1-stub"}

The response headers tell the story: server: cloudflare (it crossed the public internet), cf-ray: …-PER (through Cloudflare's Perth edge), via: 1.1 Caddy (down the tunnel, through the proxy), into a hardened service, out to a database-backed platform, and back to my hand. Every hop encrypted, every hop authenticated, zero ports open.

What exists after Phase 1: a memory-tuned private cloud, a fully codified tenant (every network, firewall rule, and VM is a terraform apply away from resurrection), a zero-trust public ingress, and a data platform with its survival machinery already running. Not a hello-world. A foundation with opinions.

Next Time: 10,000 Readings Per Second

Phase 2 is where the system grows its heart: the Rust ingestion gateway. A lock-free sensor registry that hot-swaps configuration without pausing the pipeline. Threshold evaluation inlined into the ingest path, costing microseconds per reading. A batching engine bulk-loading TimescaleDB while violations race down the hot lane.

And then I try to kill it. A Python-based plant simulator with thousands of virtual sensors running realistic physics, scripted gas-plume scenarios spreading across zones, and a malicious sensor flooding at 100Hz, ramped to 10,000 readings per second while I shoot the database mid-run, to prove the one promise this whole architecture makes: bulk storage may bend, but the safety lane never breaks.

The targets are already on the wall: p99 ingest latency under 10ms. Violations in the stream within 5ms. Zero alarms lost, no matter what fails.

Subscribe / follow along. Part 2 is where the Rust starts.