DEV Community: Sumit

I Built a Private Cloud That Runs an AI Safety System Under My Desk

Sumit — Tue, 07 Jul 2026 11:56:17 +0000

Picture a mineral processing plant at 3 AM. A methane sensor in the crusher hall starts climbing. Within 5 milliseconds, the breach is flagged at the edge. Within 50, it's glowing red on the control room screen. And within seconds, a locally hosted AI has studied the plant's layout, worked out which zones connect, what equipment is running, and where the airflow goes, and produced a step-by-step isolation blueprint: shut down conveyor C4, reroute ventilation through shaft B, seal zone 7.

No cloud bill. No third-party AI API. No open ports on the network.

That's HazShield AI, a distributed industrial safety system I'm building end-to-end, and the entire thing runs on a single mini PC under my desk. This series documents the build. Part 1 is about forging the platform: a genuine private cloud, carved out of 28GB of RAM, exposed to the world through a zero-trust tunnel.

The System in One Picture

HazShield simulates a full industrial safety ecosystem: thousands of concurrent sensor streams for gas, temperature, vibration, airflow, and seismic activity, pouring into a Rust ingestion gateway, evaluated against safety thresholds on every single reading, stored in a time-series database built for the firehose, and escalated to an AI planner when things go wrong.

The design revolves around one idea I consider the heart of the whole project: not all data deserves the same promises. Every reading entering the system is routed into one of three lanes.

🔥 The Hot Lane carries threshold violations. Sub-50ms from sensor to screen. Never dropped, ever. If the message broker is unreachable, the edge spills violations to local disk and replays them when it recovers. A safety system that loses alarms isn't a safety system.

🌊 The Warm Lane carries bulk telemetry, batched and bulk-loaded into TimescaleDB thousands of rows at a time. Under extreme load, this lane deliberately degrades by sampling 1-in-N readings for storage, while threshold evaluation still runs on 100% of them. Storage fidelity bends; safety never does.

🧠 The Cold Lane is AI planning. When an alarm opens, workers assemble the hazard context (which sensor, which zone, what's adjacent, what's running) and queue it for a quantized LLM running on Ollama. Local inference is precious, so identical hazard scenarios are deduplicated by prompt hash. An alarm storm of fifty related sensors produces one plan, not fifty.

That asymmetry between what bends under pressure and what never does drives every technical decision in the stack.

The Stack

Layer	Technology
Private cloud	OpenStack (Kolla-Ansible) on a single mini PC
Provisioning	Terraform + Ansible, 100% infrastructure-as-code
Edge ingestion	Rust · Axum · Tokio
Orchestration	Python · FastAPI · async workers
Data	PostgreSQL 16 + TimescaleDB · Redis Streams
AI	Ollama, quantized local models
Control room	React · TypeScript · WebSockets
Public ingress	Cloudflare Tunnel → Caddy

Three VMs, three jobs. An edge node that owns the public doorway and the ingestion gateway. A state node running the time-series engine and the real-time message layer. A compute node where the AI thinks. Each is sized to the byte after profiling what the workload actually needs, because the whole cloud has 28GB to spend and every gigabyte is contested.

Squeezing a Cloud Into a Mini PC

Here's the fun part nobody tells you about private clouds: OpenStack is built to manage thousands of hypervisors, and it assumes it deserves production resources. A stock deployment spun up five API workers per service (five Neutron servers, five Keystone workers) to serve requests from exactly one human: me.

After profiling the control plane container-by-container, I tuned it down to single-worker mode and stripped the services my toolchain makes redundant (Terraform replaces Heat entirely). Three lines of configuration reclaimed over 4GB of RAM, the difference between a cloud that thrashes and one that hums:

enable_heat: "no"
openstack_service_workers: 1

My favorite discovery came from the storage layer. While validating the scheduler's capacity math through OpenStack's Placement API, I found the hypervisor reporting a fraction of the disk I knew the machine had. The trail led all the way down the stack to the LVM layer, where Ubuntu Server's default install had left 828GB of the SSD sitting unallocated, invisible to everything above it. One online resize later, the cloud had nine times the storage, with no downtime and no reinstall:

sudo lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv
sudo resize2fs /dev/ubuntu-vg/ubuntu-lv

If you run Ubuntu anywhere, check your volume group. You might be richer than you think.

Five Public Domains. Zero Open Ports.

The part of this build I show people first: HazShield has five public hostnames.

app.… is the control room UI
api.… handles orchestration and asset management
stream.… carries live telemetry over WebSockets
ingest.… is the sensor firehose
telemetry.… fronts Grafana observability

And my home router forwards nothing. Not one port.

The trick is a Cloudflare Tunnel: a lightweight daemon inside the edge VM dials outbound to Cloudflare's network and holds the connection open. Public traffic rides Cloudflare's edge (TLS included), flows down the tunnel, and lands on a local Caddy instance that routes by hostname. From the internet's perspective, my house doesn't exist.

One tunnel, one Caddy, one config file. That means CORS policy, security headers, and request-size limits live in exactly one place instead of being smeared across four services. Even the tunnel daemon itself runs in a hardened, resource-capped systemd sandbox (MemoryMax=256M, no privileges, read-only filesystem) so nothing can starve the ingestion hot path that will share its node.

The network inside the cloud is just as locked down. Security group rules are expressed as intent ("the database accepts connections from members of the edge group") rather than as IP addresses. The state node's Postgres and Redis ports are unreachable from the internet, from my LAN, from anywhere except the two internal services that need them. Defense in depth, enforced twice: once at the cloud layer, once again in Postgres's own access control.

A Database Designed to Survive the Firehose

Do the math on "thousands of readings per second" and it gets scary fast: 5,000 readings/sec is 430 million rows a day. Store that naively and a 1TB SSD is dead in weeks.

So the data layer is built as a lifecycle, not a table.

Raw readings land in a TimescaleDB hypertable in 2-hour chunks, compressed after 4 hours and deleted after 48. Raw data is ephemeral by design.
Continuous aggregates roll everything into 10-second and 5-minute summaries, refreshed incrementally in the background. The dashboard reads these and never touches raw. Reads and writes are physically separated, which is why the UI stays fast while the firehose blasts.
Alarm episodes are stored forever. A sensor breaching at 1Hz for an hour produces one alarm row with an updated timestamp, not 3,600 rows. The audit trail is permanent and compact.

The engine is tuned for its 6GB VM with one deliciously counterintuitive setting: synchronous_commit = off. Losing half a second of bulk telemetry in a crash is a fine trade for write throughput, and alarm writes will override it per-transaction, because durability in this system is a per-write decision, not a global one. Same philosophy on Redis: maxmemory-policy noeviction, because a message layer that quietly discards data under pressure has no business carrying safety alarms. When Redis is full, it refuses loudly, and that refusal is precisely the backpressure signal the edge uses to trigger its local spill.

Right after deployment, one query confirms the whole lifecycle machine is alive:

SELECT job_id, proc_name FROM timescaledb_information.jobs;
-- compression ✓  retention ✓  aggregate refresh ✓  all scheduled before the first reading exists

The Moment It All Clicked

Phase 1's finish line was a single request from my phone, on mobile data, standing in my kitchen:

GET https://api.<mydomain>/health
→ HTTP/2 200
→ {"status":"ok","service":"hazshield-api","phase":"1-stub"}

The response headers tell the story: server: cloudflare (it crossed the public internet), cf-ray: …-PER (through Cloudflare's Perth edge), via: 1.1 Caddy (down the tunnel, through the proxy), into a hardened service, out to a database-backed platform, and back to my hand. Every hop encrypted, every hop authenticated, zero ports open.

What exists after Phase 1: a memory-tuned private cloud, a fully codified tenant (every network, firewall rule, and VM is a terraform apply away from resurrection), a zero-trust public ingress, and a data platform with its survival machinery already running. Not a hello-world. A foundation with opinions.

Next Time: 10,000 Readings Per Second

Phase 2 is where the system grows its heart: the Rust ingestion gateway. A lock-free sensor registry that hot-swaps configuration without pausing the pipeline. Threshold evaluation inlined into the ingest path, costing microseconds per reading. A batching engine bulk-loading TimescaleDB while violations race down the hot lane.

And then I try to kill it. A Python-based plant simulator with thousands of virtual sensors running realistic physics, scripted gas-plume scenarios spreading across zones, and a malicious sensor flooding at 100Hz, ramped to 10,000 readings per second while I shoot the database mid-run, to prove the one promise this whole architecture makes: bulk storage may bend, but the safety lane never breaks.

The targets are already on the wall: p99 ingest latency under 10ms. Violations in the stream within 5ms. Zero alarms lost, no matter what fails.

Subscribe / follow along. Part 2 is where the Rust starts.

From Click-Ops to Code: Managing My Self-Hosted OpenStack Cloud with Terraform

Sumit — Thu, 02 Jul 2026 06:31:29 +0000

Part 2 of my homelab OpenStack series. I’d built a private cloud on a mini PC and provisioned VMs by hand. This is how I turned that entire tenant into reproducible Infrastructure as Code — and the wall I hit connecting Terraform to a self-hosted cloud.

Where Part 1 Left Off

In the first post, I built a full OpenStack private cloud on a single mini PC: every service in Docker via Kolla-Ansible, the dashboard public through a Cloudflare Tunnel, a Neutron router with floating IPs, and direct SSH from my laptop into a real Ubuntu VM.

But everything was built by hand. Every network, router, security group, and VM came from typing openstack ... create one command at a time. If my cloud got wiped, I'd be rebuilding from memory.

That’s the “pet” model — nurse each resource individually. The next step up is the “cattle” model: describe what you want in code, and let a tool make reality match. Destroy everything, run one command, and the whole tenant rebuilds identically in about a minute.

That tool is Terraform. This post is how I codified the entire tenant — and why connecting Terraform to a self-hosted OpenStack was trickier than the official docs suggest.

The Mental Model: Two Layers

The thing that finally made this click for me: there are two completely different layers of cloud automation, and people conflate them.

Building the cloud itself — installing Keystone, Nova, Neutron. That was Kolla-Ansible’s job. It’s the layer AWS hides from you entirely.
Using the cloud to create resources — networks, VMs, floating IPs on the running cloud. That’s Terraform’s job.

On AWS you only ever see the second layer, because Amazon already did the first. Running your own private cloud means you’re suddenly responsible for both. Terraform talks to OpenStack’s APIs — the same ones the CLI uses — so everything I’d done manually had a direct Terraform equivalent.

Key insight: the Terraform you write against OpenStack is structurally identical to Terraform against AWS, GCP, or Azure. Same init → plan → apply workflow, same HCL language — only the provider changes. Learn it once, use it everywhere.

Setting Up (the Safe Way)

I’m cautious about anything that could touch a working setup, so I leaned hard on Terraform’s safety features:

terraform plan is a dry run. It shows exactly what it would create/change/destroy and changes nothing. You can run it endlessly.
terraform apply asks for confirmation. Nothing happens until you type yes.
Terraform only manages what it created. It keeps a state file of its own resources. My hand-built VM and networks were invisible to it — it literally cannot touch what it doesn’t know about.

So I named every new resource tf-* and used a different tenant CIDR (10.10.0.0/24 vs my existing 10.0.0.0/24), guaranteeing it would sit alongside my manual setup, never replace it.

Install + repo skeleton

# Terraform via HashiCorp's apt repo
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install -y terraform

# git-ready project folder
mkdir -p ~/openstack-terraform && cd ~/openstack-terraform
git init

Critical first step — the **.gitignore.** Terraform generates state and credential files that contain secrets and must never hit GitHub:

*.tfstate
*.tfstate.*
.terraform/
.terraform.lock.hcl
*.tfvars
clouds.yaml

Pushing a terraform.tfstate full of passwords is a classic, dangerous beginner mistake. Block it from line one.

The Wall: Connecting Terraform to a Self-Hosted Cloud

This is where I lost the most time, and it’s the part tutorials gloss over because they assume a “normal” cloud. Here’s the chain of issues I hit, in order:

1. Wrong cloud name

The provider reads credentials from clouds.yaml. My first error:

Error: cloud admin does not exist in clouds.yaml

Kolla names its cloud entry kolla-admin, not admin. Fixed the provider block to match.

2. The public FQDN trap

Next, terraform plan hung for a full minute then failed:

Get "http://dashboard.example.com:9696/v2.0/networks": connection error

The problem: my clouds.yaml pointed at my public Cloudflare domain. But Cloudflare only tunnels the dashboard (ports 80/443) — not the OpenStack API ports like 9696 (Neutron) or 5000 (Keystone). So Terraform was trying to reach Neutron through a tunnel that has no route for it.

The fix: point Terraform at the internal VIP directly. Since Terraform runs on the host, it can reach the internal endpoints with no tunnel:

# clouds.yaml
auth_url: http://192.168.0.250:5000

3. The service catalog still pointed public

Even after fixing auth_url, it failed again on :9696 at the public domain. Why? After authenticating, Terraform asks Keystone "where are the other services?" via the service catalog — and the catalog had the public FQDN registered for Neutron.

Checking the endpoints confirmed it:

openstack endpoint list --service network
# public → http://dashboard.example.com:9696 (the one that hangs)
# internal → http://192.168.0.250:9696 (the one I want)

The clean fix is endpoint overrides in the provider, forcing Terraform straight to internal URLs:

provider "openstack" {
  cloud = "kolla-admin"

  endpoint_overrides = {
    "identity" = "http://192.168.0.250:5000/v3/"
    "network" = "http://192.168.0.250:9696/v2.0/"
    "compute" = "http://192.168.0.250:8774/v2.1/"
    "image" = "http://192.168.0.250:9292/"
  }
}

Lesson: the public FQDN is for the human-facing dashboard only. Programmatic API access (Terraform, CLI) uses internal endpoints. That separation is normal and correct — but you have to wire it explicitly when the catalog defaults to public.

Once that was right, terraform plan read my existing external network and returned its ID — authenticated, talking to my cloud, creating nothing. The "hello world" of IaC.

Writing the Whole Tenant as Code

With the connection solid, I codified everything. A clean Terraform project splits into files by purpose:

main.tf — provider + all resources
variables.tf — configurable inputs with defaults
outputs.tf — values printed after apply (the floating IP, an SSH command)

The resources mirror exactly what I’d built by hand:

# Tenant network + subnet
resource "openstack_networking_network_v2" "tenant_net" {
  name = "tf-net"
  admin_state_up = true
}

resource "openstack_networking_subnet_v2" "tenant_subnet" {
  name = "tf-subnet"
  network_id = openstack_networking_network_v2.tenant_net.id
  cidr = "10.10.0.0/24"
  ip_version = 4
  dns_nameservers = ["1.1.1.1", "8.8.8.8"]
}

# Router bridging tenant <-> external
resource "openstack_networking_router_v2" "router" {
  name = "tf-router"
  external_network_id = data.openstack_networking_network_v2.external.id
}

resource "openstack_networking_router_interface_v2" "router_iface" {
  router_id = openstack_networking_router_v2.router.id
  subnet_id = openstack_networking_subnet_v2.tenant_subnet.id
}

# Security group + SSH/ICMP rules, keypair, VM, floating IP...

The “aha” moment

Two things make this feel like magic the first time:

Dependencies are automatic. The subnet references tenant_net.id, the router references the subnet, the floating IP associates to the VM. Terraform reads these references, builds a dependency graph, and creates everything in the correct order — I never specify it. Run apply and you watch the keypair, security group, network, subnet, router, and router-interface all materialize in seconds, in dependency order.

Helper functions. cidrhost("10.10.0.0/24", 1) computes .1 of the range. Change the CIDR once and the gateway and allocation pool follow. That's the power of code over clicking.

A terraform plan showed the full picture:

Plan: 11 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + ssh_command = (known after apply)
  + vm_floating_ip = (known after apply)

Eleven resources — my entire tenant — defined, validated, changing nothing until I said so.

The Bugs I Hit Applying It

Because I was working with a current provider version, a few things had drifted from older examples:

Provider version drift. The resource I first used to attach a floating IP didn’t exist anymore:

The provider does not support resource type
"openstack_compute_floatingip_associate_v2".

In the v3 provider, floating IP association moved to the networking side. Swapped to openstack_networking_floatingip_associate_v2, which attaches via a port_id instead of an instance ID. Lesson: provider resources get renamed across major versions; the errors are precise, so fixes are quick once you know the replacement.

The floating IP wouldn’t attach. Even after the swap, the association silently didn’t happen — the vm.network[0].port attribute came back empty. The robust fix was looking up the port explicitly with a data source:

data "openstack_networking_port_v2" "vm_port" {
  device_id = openstack_compute_instance_v2.vm.id
  network_id = openstack_networking_network_v2.tenant_net.id
}

resource "openstack_networking_floatingip_associate_v2" "fip_assoc" {
  floating_ip = openstack_networking_floatingip_v2.fip.address
  port_id = data.openstack_networking_port_v2.vm_port.id
}

This queries “which port belongs to this VM on this network” — far more reliable than the array-index attribute.

Copy-paste mangling. A couple of errors came from values losing their variable {} wrapper or a URL dropping a colon (192.168.0.250.9696 instead of :9696). The lesson there: when pasting multi-line config, heredocs (cat > file <<'EOF') preserve formatting that a terminal paste sometimes mangles.

The Payoff: Idempotency

The moment it all clicked was a partial failure. My first apply created 10 of 11 resources, then failed on the VM (an image-name mismatch). I fixed the name and ran apply again.

Terraform didn’t recreate the 10 existing resources. It saw they already existed, left them alone, and built only the missing VM. That’s idempotency — re-running converges to the desired state without redoing finished work. It’s what makes Terraform safe to run over and over.

Final result: apply builds the whole tenant and prints:

ssh_command = "ssh ubuntu@192.168.0.233"

One command, ~60 seconds, the entire manual process — reproducible, version-controlled, in a git repo. And terraform destroy cleanly removes only the tf-* resources, nothing else.

Lessons Worth Keeping

Public FQDN ≠ API endpoint. Tunnels expose the dashboard; Terraform and the CLI use internal endpoints. Override them explicitly when the catalog defaults to public.
Check the service catalog (openstack endpoint list) when connections hang — the URL Terraform actually uses comes from there, not just auth_url.
Provider versions drift. Resources get renamed across major versions. Read the precise error, find the replacement.
Use data sources for reliable lookups — vm.network[0].port can be empty; an explicit port data source isn't.
.gitignore your state and credentials first. Never commit *.tfstate or clouds.yaml.
plan is free and safe. Run it constantly. Name new resources distinctly (tf-*) so they sit beside, never replace, existing infra.
Idempotency is the whole point. Re-running apply fixes only what's missing.

Why This Was Worth It

I went from “I can click together a cloud” to “I can declare a cloud in code, version it, reproduce it, and destroy it on demand.” That second skill is the one that transfers directly to any cloud platform and any real infrastructure job.

The most valuable part wasn’t the working result — it was debugging the connection layer myself. Self-hosted OpenStack doesn’t hand you a clean managed endpoint like AWS does; you have to understand auth, the service catalog, and internal-vs-public routing to make Terraform work. That understanding is exactly what’s missing when you only ever consume someone else’s cloud.

What’s the trickiest thing you’ve had to wire Terraform into? Drop it in the comments.

Building a Self-Hosted OpenStack Private Cloud on a Mini PC (Behind CGNAT, No Public IP)

Sumit — Sat, 27 Jun 2026 08:54:07 +0000

A hobbyist’s end-to-end build log: from a bare GMKtec mini PC to a public-facing OpenStack cloud you can SSH into — including every wall I hit and how I got past it.

Why I Did This

I’d run Proxmox before and exposed services from VMs to the internet, so I understood virtualization and home networking. What I wanted was the next step up: a real cloud orchestration platform — multi-tenancy, an API, self-service provisioning — the kind of thing AWS and GCP are built on, but running on hardware sitting on my desk.

The goal was concrete:

A working OpenStack private cloud on a single mini PC.
The dashboard reachable from the public internet.
The ability to provision VMs and SSH into them.

I had one complication that turns out to be very common: my home internet is behind CGNAT — no public IP, no port forwarding possible. Spoiler: that didn’t stop anything. It just shaped the architecture.

The Hardware

GMKtec mini PC — Intel CPU with VT-x, 32 GB RAM , 1 TB NVMe
Two onboard NICs (one 2.5GbE, one 1GbE)
A JetKVM for out-of-band access (KVM-over-IP) — this turned out to be a lifesaver
A MacBook as my workstation
A domain managed in Cloudflare

32 GB RAM is genuinely the practical floor for a single-node OpenStack that still has room to run real VMs. The control plane alone eats 12–16 GB.

Proxmox vs OpenStack: The Mental Model

If you’re coming from Proxmox like I was, the key reframe is this: the hypervisor layer is identical (KVM/QEMU + libvirt). Everything above it is what’s different. On Proxmox you click “create VM.” On OpenStack, that same action is decomposed across services that each own one job:

What you did on Proxmox OpenStack equivalent Service Web UI / API Horizon dashboard / CLI Horizon Users & permissions Tokens, projects, roles Keystone ISO / templates Cloud images (qcow2) Glance Create VM, size it Instance from a flavor Nova Bridges, NAT, VLANs Virtual networks, routers, ports Neutron Manual per-VM firewall Security groups Neutron Port-forward to a VM Floating IP (1:1 NAT) Neutron

Two things bit me coming from Proxmox, so internalize them early:

Cloud images, not ISOs. OpenStack boots pre-built cloud images that self-configure on first boot via cloud-init. You do not click through an installer. (This caused my single biggest time sink — more later.)
Security groups default to deny. A new VM blocks all inbound until you explicitly open ports.

Architecture: Working Around CGNAT

Since I had no public IP, the design splits cleanly into two planes:

Admin plane — me reaching the dashboard and SSHing into VMs.
Service plane — public web traffic to services (future).

Both reach the outside world through Cloudflare Tunnel , which dials outbound from my network. Because the connection originates from inside, CGNAT is irrelevant — nothing inbound needs to be routable. This is essentially the zero-trust / BeyondCorp pattern, and it’s arguably more modern than the classic “public IP + port forward” approach.

Key realization: Cloudflare Tunnel is perfect for HTTP/HTTPS (the dashboard, web services), but it cannot give a VM raw arbitrary-port public access. A true public VPS needs a real routable IP. For a homelab/POC where you reach your own VMs, tunnels + a bastion pattern are exactly right. Real multi-user VPS hosting is a hardware/IP upgrade later, not a re-learn.

Deployment Method: Why Kolla-Ansible

Three real options:

DevStack — dev/test only, explicitly not for anything you keep. Rejected.
OpenStack-Ansible — production-grade but heavy and complex; wrong place to start.
Kolla-Ansible — production-grade, runs every service as a Docker container , great single-node “all-in-one” support, real upgrade path.

For a container-comfortable beginner who wants something real that can grow, Kolla-Ansible is the clear pick. Debugging is just docker logs.

The Build, Step by Step

1. Install Ubuntu Server (the right version!)

This is where I hit my first real lesson. My instinct was “use the newest LTS.” Wrong. OpenStack and Kolla only support specific, tested host OS + release combinations, and the matrix moves fast.

After checking the live support matrix (not a guide, not memory):

Ubuntu 24.04 LTS as the host
Kolla-Ansible / OpenStack 2025.1 “Epoxy”

I initially installed 22.04 based on outdated guidance, then discovered 2024.1 had been retired to unmaintained/ and 2025.1 had dropped 22.04 support. The fix without reinstalling everything: an in-place upgrade.

sudo apt update && sudo apt upgrade -y
sudo apt install -y update-manager-core
sudo do-release-upgrade

Lesson: Always match your host OS to what the deployer officially tests today. “Latest” is a liability in infrastructure, not a feature.

BIOS prerequisites (do these first, via JetKVM):

Enable Intel VT-x / virtualization (non-negotiable — VMs won’t run without it)
Disable Secure Boot

Verify virtualization after boot:

egrep -c '(vmx|svm)' /proc/cpuinfo # >0 means enabled

2. Pin a Static IP

Identify your NIC (modern names like eno1, not eth0):

ip -br addr

Set a static IP in netplan outside your router’s DHCP range, with JetKVM open in case you lock yourself out:

# /etc/netplan/00-installer-config.yaml
network:
  version: 2
  ethernets:
    eno1:
      dhcp4: false
      addresses: [192.168.0.200/24]
      routes:
        - to: default
          via: 192.168.0.1
      nameservers:
        addresses: [1.1.1.1, 8.8.8.8]

sudo netplan apply

3. Install Kolla-Ansible

# host prep
sudo apt install -y git python3-dev libffi-dev gcc libssl-dev python3-venv python3-pip
sudo timedatectl set-ntp true

# isolated venv
python3 -m venv ~/kolla-venv
source ~/kolla-venv/bin/activate
pip install -U pip
pip install 'ansible-core>=2.16'

# clone the EXACT supported branch with git (more reliable than pip's git+ shallow clone)
cd ~
git clone --branch stable/2025.1 https://opendev.org/openstack/kolla-ansible
pip install ./kolla-ansible
kolla-ansible install-deps

Lesson: pip install git+...@branch does a shallow clone that can fail to find the branch. A plain git clone --branch is more reliable. And confirm the branch actually exists first: git ls-remote --heads .

4. Configure globals.yml

sudo mkdir -p /etc/kolla && sudo chown $USER:$USER /etc/kolla
cp -r ~/kolla-venv/share/kolla-ansible/etc_examples/kolla/* /etc/kolla/
cp ~/kolla-venv/share/kolla-ansible/ansible/inventory/all-in-one ~/
kolla-genpwd

Key settings (single-node, single-NIC NAT to start):

kollabasedistro: "ubuntu"
openstack_release: "2025.1"
kollainternal_vipaddress: "192.168.0.250" # a SPARE LAN IP, not the host's
network_interface: "eno1"
neutronexternalinterface: "dummy1" # see note below
neutronpluginagent: "ovn"
enable_haproxy: "yes"

The single-NIC trick: Neutron’s external interface must be a “naked” (no-IP) interface, and it must not be the one your host IP lives on (or you lock yourself out). With one NIC, create a dummy interface for Neutron to bridge onto so it never touches your real NIC:

sudo ip link add dummy1 type dummy
sudo ip link set dummy1 up

5. Bootstrap → Prechecks → Deploy

kolla-ansible bootstrap-servers -i ~/all-in-one # installs Docker, etc.
kolla-ansible prechecks -i ~/all-in-one # validates config
kolla-ansible deploy -i ~/all-in-one # pulls images, starts everything (~30 min)

Prechecks are your friend — they fail loudly and specifically. I hit a chain of them, each a quick fix:

No module named 'docker' → pip install docker
No module named 'dbus' → sudo apt install -y libdbus-1-dev libdbus-glib-1-dev pkg-config && pip install dbus-python
Please enable at least one backend for Cinder → I set enable_cinder: "no" for the POC (VMs run fine on ephemeral disk; add block storage later)

Also needed passwordless sudo for the deploy user so Ansible can escalate non-interactively:

echo "$USER ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/$USER

A successful run ends with failed=0. Mine: ok=449 changed=273 failed=0. 449 tasks, a full cloud, in containers.

6. Verify

pip install python-openstackclient
kolla-ansible post-deploy -i ~/all-in-one
source /etc/kolla/admin-openrc.sh
openstack service list # keystone, nova, neutron, glance, placement... all there

Horizon dashboard: http:// — admin password via:

grep keystoneadminpassword /etc/kolla/passwords.yml

Provisioning the First VM

# Image — CirrOS first to test the pipeline (tiny, 16MB)
wget http://download.cirros-cloud.net/0.6.2/cirros-0.6.2-x86_64-disk.img
openstack image create "cirros" --file cirros-0.6.2-x86_64-disk.img \
  --disk-format qcow2 --container-format bare --public

# Flavor (CPU/RAM/disk)
openstack flavor create --vcpus 2 --ram 2048 --disk 20 m1.small

# Private tenant network
openstack network create tenant-net
openstack subnet create tenant-subnet --network tenant-net \
  --subnet-range 10.0.0.0/24 --gateway 10.0.0.1 \
  --dns-nameserver 1.1.1.1 --allocation-pool start=10.0.0.10,end=10.0.0.200

# Security group (open SSH + ping)
openstack security group create vm-access
openstack security group rule create vm-access --protocol tcp --dst-port 22 --remote-ip 0.0.0.0/0
openstack security group rule create vm-access --protocol icmp --remote-ip 0.0.0.0/0

# SSH key + launch
openstack keypair create --public-key ~/.ssh/id_ed25519.pub mykey
openstack server create --image cirros --flavor m1.small \
  --network tenant-net --security-group vm-access --key-name mykey my-first-vm

ACTIVE with a 10.0.0.x IP = a VM running on your own cloud. 🎉

Networking: The Keystone Concept (Router + Floating IPs)

VMs on a private 10.0.0.x network can reach out (via NAT) but aren't reachable in. To SSH into them cleanly, you need a Neutron router bridging the tenant network to an external network , plus floating IPs.

# External (provider) network
openstack network create --external --provider-network-type flat \
  --provider-physical-network physnet1 external-net
openstack subnet create external-subnet --network external-net \
  --subnet-range 192.168.0.0/24 --gateway 192.168.0.1 \
  --allocation-pool start=192.168.0.224,end=192.168.0.254 --no-dhcp

# Router wiring both sides
openstack router create main-router
openstack router set main-router --external-gateway external-net
openstack router add subnet main-router tenant-subnet

# Floating IP → VM
openstack floating ip create external-net
openstack server add floating ip my-first-vm <FLOATING-IP>

The Single-NIC Limitation (and the real fix)

With OVN, there’s no qrouter namespace to hop through, and the dummy1 interface has no physical uplink — so floating IPs weren't reachable from the LAN. The proper fix was using the second physical NIC as the external interface:

# globals.yml
neutronexternalinterface: "enp3s0" # real NIC, cabled, no IP

kolla-ansible reconfigure -i ~/all-in-one --tags neutron

Gotcha: the reconfigure updated the config but didn’t actually swap the physical port on the OVS bridge — it still had dummy1. I had to move it manually (OVS runs inside a container):

sudo docker exec openvswitch_vswitchd ovs-vsctl list-ports br-ex # showed dummy1
sudo docker exec openvswitch_vswitchd ovs-vsctl del-port br-ex dummy1
sudo docker exec openvswitch_vswitchd ovs-vsctl add-port br-ex enp3s0

After that, the floating IP pinged from the whole LAN. Lesson: when you change neutron_external_interface, verify the physical port actually moved on br-ex — don't assume reconfigure did it.

Making the Dashboard Public via Cloudflare Tunnel

Two halves that must match: the tunnel and telling OpenStack its external hostname.

Tunnel:

wget https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64.deb
sudo dpkg -i cloudflared-linux-amd64.deb
cloudflared tunnel login
cloudflared tunnel create openstack
cloudflared tunnel route dns openstack dashboard.example.com

# ~/.cloudflared/config.yml
tunnel: openstack
credentials-file: /home/user/.cloudflared/<TUNNEL-ID>.json
ingress:
  - hostname: dashboard.example.com
    service: http://192.168.0.250:80
  - service: http_status:404

Tell OpenStack the FQDN (in globals.yml) and reconfigure:

kollaexternalfqdn: "dashboard.example.com"

kolla-ansible reconfigure -i ~/all-in-one --tags horizon,haproxy

The “Cookies may be turned off” fix: Horizon behind an HTTPS-terminating proxy needs to trust the forwarded protocol. Create /etc/kolla/config/horizon/_9999-custom-settings.py:

SECUREPROXY_SSL_HEADER = ("HTTP_X_FORWARDEDPROTO", "https")
CSRFTRUSTEDORIGINS = ['https://dashboard.example.com']
CSRFCOOKIESECURE = True
SESSIONCOOKIESECURE = True

Then kolla-ansible reconfigure -i ~/all-in-one --tags horizon.

Make the tunnel permanent (systemd, survives reboots):

sudo mkdir -p /etc/cloudflared
sudo cp ~/.cloudflared/config.yml ~/.cloudflared/*.json /etc/cloudflared/
# point credentials-file in the copied config at /etc/cloudflared/<UUID>.json
sudo tee /etc/systemd/system/cloudflared.service >/dev/null <<'EOF'
[Unit]
Description=Cloudflare Tunnel
After=network.target
[Service]
ExecStart=/usr/bin/cloudflared --config /etc/cloudflared/config.yml tunnel run
Restart=on-failure
User=root
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl enable --now cloudflared

Dashboard now public at https://dashboard.example.com, over HTTPS, through CGNAT, surviving reboots.

The Bug That Cost Me the Most: Cloud Image vs Installer ISO

I spent a long time convinced SSH key injection was broken. VMs were ACTIVE, the network worked, metadata was healthy, keys were correct — but SSH always fell back to asking for a password.

The culprit, revealed only when I opened the noVNC console : the VM was sitting at the Ubuntu installer’s language-selection screen. I had registered an installer ISO in Glance, not a cloud image. An installer never runs cloud-init, never injects your key — it’s just waiting for someone to install an OS.

The fix:

# The REAL cloud image — note "cloudimg" in the name, ~600MB .img
wget https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.img
openstack image create "ubuntu-24.04" --file ubuntu-24.04-server-cloudimg-amd64.img \
  --disk-format qcow2 --container-format bare --public

The keeper lesson:

Cloud images = cloudimg, ~500–700 MB .img (qcow2). Use these for OpenStack.
Installer ISOs = live-server, ~2–3 GB .iso. These are for installing on bare metal. Never for cloud VMs.

After swapping to the real cloud image, cloud-init ran, injected the key, and:

ssh-keygen -R <floating-ip> # clear stale host key from a previous VM on that IP
ssh ubuntu@<floating-ip> # straight into Ubuntu 24.04, passwordless

(That REMOTE HOST IDENTIFICATION HAS CHANGED warning is expected when you recreate a VM on a reused IP — ssh-keygen -R clears it.)

What I Ended Up With

A bare-metal OpenStack private cloud (Kolla-Ansible, 2025.1, Ubuntu 24.04), every service containerized
Horizon publicly accessible via Cloudflare Tunnel, no public IP, surviving reboots
Full Neutron networking: tenant network, external network, router, floating IPs on a real NIC
Direct, key-authenticated SSH from my laptop into a real Ubuntu VM

Lessons Worth Tattooing

Cloud images, not installer ISOs. cloudimg .img, never live-server .iso. This one cost the most.
Match versions to the live support matrix — newest ≠ supported. Check today, not from memory or an old blog.
Verify OVS bridge changes actually applied — ovs-vsctl list-ports br-ex after changing the external interface.
Prechecks are your friend — they fail specifically; read the error, fix, re-run.
Have out-of-band access (JetKVM/IPMI). Networking changes will eventually drop your SSH, and the console saves you.
CGNAT isn’t a blocker — outbound tunnels (Cloudflare/Tailscale) sidestep it entirely.
Get it working simply first, then add the clean architecture. Single-NIC NAT → then the router and second NIC.

Where to Go Next

A bastion + ingress architecture for secure multi-VM access
Public web services on VMs via more Cloudflare tunnels (subdomain-per-service)
Terraform with the OpenStack provider to codify the whole tenant as reproducible infrastructure
Re-enable Cinder for persistent block storage

If you’ve got a spare mini PC and a weekend (okay — maybe two), this is one of the most educational projects you can do. You don’t just use a cloud; you understand every layer because you had to fight through each one yourself.

Got questions or hit a different wall? Drop a comment below.