During a routine proxy upgrade, every site on my server stopped answering for ninety seconds. The config hadn't changed. The upgrade had gone through cleanly. The fix turned out to be renaming one file, and why that worked is a story for another post. What stayed with me was what happened afterwards: the explanation went into the repo's README, a check for the same failure now runs twice a day, and both got there as commits — the same way everything else on that machine does.
That's the idea behind docker-gitops: run real servers as if they were a program. Git is the single source of truth for what runs on them, and changes go in as commits and pushes, not SSH sessions spent poking at things. It started as one box. There are three now. This post is a map of the repo, by way of the four corners where the idea actually paid for itself.
The repo directory is the runtime directory
Each host gets a directory, and each compose stack gets a directory inside its host. That directory is where the stack runs from: you cd into it and docker compose up -d. There is no separate deploy path for the repo and the server to drift apart on. The file you edit is the file the container starts from.
Which means nothing done by hand stays a secret. If I patch something on the box at two in the morning, it shows up in git status the next day, and it stays there until I either commit it or throw it away. There's no third option where the server quietly remembers something the repo doesn't.
The other two hosts don't have a clone at all. They're driven from the first one over SSH, which is its own post.
Four corners
The firewall is a shell script with a git history. It didn't start that way. I used to persist rules the usual way — iptables-save to a file, replayed at boot. The snapshots were taken while Docker was running, so they froze Docker's own per-bridge rules along with mine. After one reboot, a recreated network reused an old subnet, rules for bridges that no longer existed came back to life, and containers on it couldn't reach each other. Now only the rules I wrote by hand persist, in one script that a oneshot unit applies at boot. Docker, k3s and Cilium rebuild their own chains. A rule change is a commit.
Adding a service is a skill, not a wiki page. The repo carries an agent skill — step-by-step instructions my coding agent follows — that takes a new service end to end: write the compose stack, create the reverse-proxy entry, add the dashboard card, commit. The routine steps are delegated. The judgment calls, like which network it joins and which access list guards it, stay with me. I used to have a "how to add a service" doc I kept meaning to update. The skill can't drift that way, because it gets run every time.
The host fixes the small stuff and tells me about the big stuff. An inspector runs at 09:00 and 21:00 and works through thirty-odd checks across the three hosts and the cluster. Some clean up on their own: week-old stopped containers, dangling images, stale build cache. The rest only raise an alert, because the fix is a human call. My favourite came out of an incident where two containers were OOM-killed and quietly restarted. The pods stayed Running, so nothing watching pod status ever noticed. There's a check for exactly that now.
The README keeps a map of the landmines. The root README repeats one warning in bold: every Custom Location bakes its hostname into proxy_pass, and a literal upstream must resolve at config load time or nginx refuses to start, taking every reverse-proxied site down at once. That's the ninety seconds from the top of this post, written down so the next person trips over the warning instead of the bug. A warning only protects people who read it, though, so one of the inspector's checks runs nginx -t against the live config. The landmine shows up in a report before the next restart finds it.
The line underneath
Running all this left me with one rule: software you only run belongs in Compose; software you write belongs in Kubernetes. Compose is where you run someone else's thing and leave it alone. Kubernetes is for things you're going to keep changing. The repo is mostly Compose, because most of what I run I didn't write, plus a small k3s cluster, because the things I am writing need somewhere to practice. I learned that the expensive way, by moving six services onto Kubernetes and then moving them back.
Where to start
The repo is public. Read the Custom Location warning in the README first, then docs/incidents/. Those two are where most of the rules came from. After that, pick a host. If you run a single box and you're tired of it being a pile of things you once did to it by hand, this is one way out.
Top comments (0)