About five on-premise deployments a year, and one dedicated engineer for every two of them. That was the shape of our setup when we realised on-premise was going to be most of our business.
I'm Panth, and I lead the software team at Oizom, where we build Envizom, an air quality monitoring platform. This post is about the deployment problem that came with selling it, and On-Prem Manager (OPM), the tool I built and open-sourced under Apache-2.0 to fix it.
Why on-premise
Many organisations don't want to hand their data to a cloud. Governments most of all, and air quality is first a government responsibility. So when a government picks software like Envizom, it also needs that software on its own servers, for its own security reasons.
The more we sold, the clearer it got: local deployment is the one real selling point for an air quality platform, and per-client customisation is what wins the client.
What a deployment actually looks like
The ideal case: the client gives you exactly what you asked for, and if something breaks on their side, a firewall rule or an IP config, they own it.
That is not what happens. One client wants a DC/DR setup. Another wants high availability. Another wants a single server. Another wants DC/DR eventually, but for now on one server, with a second one later and two more after that.
Then there are the integrations: pull data from one of their systems, or expose an API in a schema they hand us for a public display. None of that belongs in our production code.
What we had
An end-to-end setup on Docker Compose, with Portainer as the GUI.
After enough deployments it was clear Compose wasn't enough. Every client is different, so every install meant a lot of configuration and setup afterwards. A simple issue took hours to close, because debugging a remote server through that stack is slow.
That setup is fine at five deployments a year with one engineer for every two. It is the bottleneck the moment you want more.
Where the idea came from
Oizom started as an IoT company. It ships devices with software inside, and every device sits at a remote location where nobody can do anything by hand. For that, Oizom uses a device manager, Aikaan.
So: what if our deployment team had the same thing for on-premise servers?
What the team needs to run a deployment is a short list. Automated volume backups. A way to make a new version live. Environment and config per site. Variables, with separate secrets for each deployment. A GUI to see the containers. I wrote down every problem we hit in operations and built a tool around the list.
What OPM is
Two parts.
| Part | What it does |
|---|---|
| Cloud console | The catalog of images, deployment templates, variables, secrets and files. You commit a change once and review the diff per site. Self-host it on a VM (SQLite and local files) or run it on Firebase, same frontend build. |
| Agent | One Go binary per server, with its own GUI built in. It applies commits to that server's Docker, runs an embedded Caddy as the edge proxy, snapshots volumes, and rolls back. |
The README puts it as: like Portainer, plus a git-style commit history for every server's config.
Install is one line
The cloud's Setup page gives a command per server:
curl -fsSL -H "Authorization: Bearer opi_…" <apiBase>/v1/install/script | sudo sh
It runs preflight checks, uses the Docker that is already there (20.10 or newer) or asks before installing a pinned version, downloads the agent and checks its sha256 and Ed25519 signature, writes the config, sets up a systemd service, and waits up to 60 seconds for a health check, rolling itself back if that fails. Everything it installs is pinned when the script is generated.
Images without a registry
You register images in the cloud's catalog, and the cloud keeps its own copy of their layers. The agent loads images from those blobs with docker load and never connects to a Docker registry, so it never sees a registry password. Sites in the same deployment copy layers from each other first, and only one site with internet fetches from the cloud.
Commits, not proposals
Templates, deployments and each site are branches of commits. The latest commit on a deployment's branch is what its sites should run. You can compare any two branches and cherry-pick between them.
Nothing changes on a site until a person on that site merges it, in the agent GUI. A merge runs one fixed sequence:
LOCK -> SNAPSHOT -> APPLY -> VERIFY -> COMMIT
VERIFY waits for each container to pass its health check, or to run 10 seconds without a restart, within 300 seconds by default. Any failure rolls back, and a failed merge leaves no commit. The failed container is kept for 24 hours under a -failed-<time> name, with its last 200 log lines, so the post-mortem doesn't depend on someone grabbing logs in time.
The dangerous act isn't the merge, it's restoring data. So rollbacks and volume restores get a 60-second countdown and a typed ACCEPT. The spec's reasoning: "the commit message + plan review are the deliberate act… restoring data is still the dangerous act".
Backups by default
Before every update and rollback, the agent snapshots config, image references and the volumes marked for backup, as .tar.zst archives. It keeps the last 10. A scheduled snapshot runs daily at 02:00 site time and keeps the last 7. A snapshot refuses to start if free disk is under 1.2 times its estimate. Snapshots download as signed .opmsnap files.
Docker stays the source of truth
People still change things in Docker directly. The agent scans every 60 seconds and on Docker events. A hand edit to an image, env, port, mount or file shows up as uncommitted drift, and merges are refused until it is committed or discarded. A site can also auto-commit drift once it has been stable for 30 seconds, authored as "changed in Docker".
Standalone first
Every agent GUI can do everything locally with no cloud at all: create containers, change images, edit files, variables and secrets. The cloud is how you run many sites, not something a site needs to keep running.
When the client cuts the internet
Some clients take the server off the internet for security. OPM has three routes to the cloud, in order:
- The server's own internet.
- A peer site in the same deployment that has internet, relaying.
- The browser of the person using the agent GUI. The GUI tab long-polls the agent and runs the cloud requests from the browser, so an engineer's laptop on the client network becomes the tunnel.
The agent only ever talks HTTPS on 443 to one API domain, which is one line in a client's firewall request.
And for a client that allows neither the agent nor the cloud, the cloud can mirror every branch to a GitHub repository. A machine checks out its site's branch, fills in site.env and runs ./deploy.sh: plain docker compose, Caddy as the router, Portainer as the GUI, no OPM code on the box. A real-Docker test compares that output with the agent's and reports 0 differences. You lose drift detection, snapshots and rollback, which is why it's the last resort.
The size of it
The full specification, SPEC-final.md, is 894 KB, and its decision log has 116 numbered entries, each with the reason. The code is about 97,000 lines of Go, 96,000 of TypeScript and 60,000 of Svelte, with 105 Go test files and 157 TypeScript test files. There is an end-to-end harness that runs real agents against real Docker.
Integrations got their own tool too. They run in codeshell, a small browser editor in a container whose code is one volume. OPM's "Persist to cloud" versions a volume's content in the cloud, the docs name "client integration code" as the use, so that code is reviewed and kept there instead of in a git repository. codeshell gets its own post.
What I'd ask you
If you ship software that customers run on their own servers: what does one deployment cost you in engineer-hours, and which part of it did you automate first?
Top comments (0)