The short version
The state of my company's development infrastructure used to live in separate places. The continuous-integration runners, the security scans that fire on every code push, the vulnerability ledger, the metrics-and-logs monitoring, the 24/7 high availability that keeps everything up when any node dies, the local LLM running on a GPU node — each one had its own place to look and its own way of looking. I aggregated all of that state into a single dashboard.
That dashboard is Infra Portal. But the thing I most want to convey in this article isn't the screen itself. It's a way of thinking: what matters in dev infrastructure is not "where you run it" but the mechanism of "choosing the information you decide on, gathering it into one dashboard, and backing those numbers with tests on real hardware."
Here are the key points up front.
- What I aggregated is a real platform — Infra Portal only renders the display; behind it run my own CI/CD, security scanning, monitoring, high availability, a local LLM, and infrastructure-as-code.
- The screen returns a verdict, not raw numbers — not "12 of 12 runners" but "is capacity short right now, or is routing misassigned?" The screen takes on the work of interpreting the numbers so a human doesn't have to.
- The passing line is a real-hardware test, not a design doc — I can only say high availability is "achieved" when I actually kill one node and everything survives. Even with every design review and unit test green, there were bugs that only showed up when I dropped a node for real.
- Both the monitoring and the safety mechanisms grew after the fact, incident by incident — I didn't design this perfectly from the start. Every time something broke, I added the one thing I hadn't been watching. What I have now is the accumulation of that. This article traces those additions by the date of each failure.
It's about a 20-minute read. It's long, so feel free to jump to whichever layer interests you (CI / security / monitoring / high availability / local LLM).
The body (about a 20-minute read)
What runs behind the dashboard
This is not "I moved GitHub Actions to a home server to cut costs," and it's not "a dashboard that just collects scattered information." A single repository (an infra/ops monorepo) holds all of the following, living together:
- My own CI/CD runners — running GitHub Actions on my own servers
- A security-scanning platform and a vulnerability ledger — inspecting for vulnerabilities on every push and aggregating results into a ledger
- Fully self-hosted monitoring — metrics, logs, and traces, held entirely by me with nothing outsourced to a SaaS
- 24/7/365 high availability — if any one node dies, the rest keep every service alive
- A local LLM — models running on a GPU node, monitored, routed by purpose, and used in actual development
- Infrastructure as code — reproducing a second (and later) server wholesale with one command
In numbers: this repository has 62 architecture decision records (ADRs), about 930 commits, and covers 7 hosts in total — 2 CI servers, 1 GPU node, and several Macs. I've written down architecture decisions 62 times because if I don't leave "why I did it this way" for my future self (and whoever eventually takes over), I won't be able to read my own work six months later.
Infra Portal is the dashboard that aggregates this whole thing into one. It aggregates 26 products across 10 tabs, and it opens on an internal subdomain (inside Cloudflare Access — unreachable from outside). It's a hand-written static page with no framework and no build step: about 3,900 lines of front-end script and about 1,800 lines of API server. Changes ship by copying files, not by building. To keep grasping the whole thing on one screen, this kind of simplicity breaks less.
Why I built it — gathering decision inputs into one place, riding on the CI server
The company I run stands on "run every business function with AI." Humans concentrate on judgment; the actual work moves to AI. For that, the material you decide on has to be right in front of you, with the fewest steps. If I'm SSH-ing into dashboards all over the place to scrape numbers together, I spend more time on work than on judgment. So I needed a dashboard that gathers the state of my dev infrastructure into one place and returns not raw numbers but "what's the bottleneck right now." I didn't stand up a new box for it — I rode it on the local server where CI was already running, because that was simply more efficient.
The trigger was a mundane cost cut. I wanted to reduce the bill for GitHub's hosted runners, so I stood up my own runners on a home server. But that kept exposing operational gaps, one after another. A runner ran away and ate the whole server. A security gate turned out never to have run at all. A spot I thought I was monitoring turned out to be one I "wasn't watching" — I only learned that when it broke. One machine going down took a service with it. The back half of this article is the record of fixing those gaps one at a time. I'm not dismissing the cost cut itself. On the contrary, the essential operational gaps came out of it, and fixing them is what grew this into a platform.
Triage — telling causes apart without SSH or LAN-only Grafana
The most important design in Infra Portal is that panels return a verdict, not raw numbers.
Take a stalled CI build. An ordinary dashboard lines up numbers — "3 queued, 0 runners busy" — and leaves the rest to a human. Portal is different. Looking only at two axes, queue and idle, a pure function decides:
- queued, and idle is 0 → capacity (not enough machines; add runners)
- queued, but idle exists → routing (there are machines that could run it but the work isn't flowing; suspect a label mismatch)
- queued, and 0 machines running → offline (no executor at all)
- nothing queued → healthy
And at the very top of the screen it says, in one line, "capacity is the bottleneck right now." This decision logic is pinned against regression by 16 tests, confirmed not to break as I vary the combinations of labels and queues. Grafana is only visible on the internal LAN, but even on the road, this dashboard alone lets me tell "should I add runners, or fix the config?" apart.
Portal's settings tab has one more thing I was strict about: it never displays a secret. For a linked service's token or URL, it shows only a diagnosis — "does it exist," "do I have permission" — never the value itself, which can be neither displayed nor entered. Not putting secrets on screen even when I'm the only viewer is a structural way to prevent screen-share and screenshot accidents.
Layer ① CI/CD runners — closing the gaps that going free opened, failure by failure
On my own runners, every single safety setting is there as a response to an accident that already happened.
The first accident was a runaway. Back when I hadn't set a CPU ceiling, one runner for lightweight jobs monopolized 8 cores and ballooned to 830% CPU usage. One container running away drags the whole server in with it. That's when I started enforcing per-container CPU and memory ceilings at the kernel level. Cross the ceiling and only that container gets throttled; its neighbors are fine.
Today the runners are split into 6 role-based pools (light, security, browser, Docker, cost, backup). Only the Docker pool is handed the Docker socket. Every runner is ephemeral — thrown away after a single job — isolated with tmpfs and no privilege escalation. On a 16-core CI server, 13 containers run in the standard configuration, and in this area alone I've stacked up 8 architecture decisions.
I build the runner images myself, too. Once, GitHub's code-download infrastructure had an outage and Actions checkouts failed all at once. So I baked the commonly used dependencies (runtime, package manager, cloud CLI, browser automation, IaC tooling) into the image ahead of time and hold my own caches. So that builds keep going even when something external is down, the cache layer is "pass through if you can't fetch (fail-open)," while the security-relevant dependency mirror is the opposite — "stop if you can't fetch (fail-closed)." I change the behavior by nature.
Onboarding CI to a new repo is self-service from Portal. Edit a runner's count or CPU ceiling on the screen, and the change to the configuration-management (Ansible) inventory becomes a pull request automatically. Instead of hand-touching a server, it always lands as a code change in the history. The fewer people you operate with, the more that "you can trace it later" property pays off.
Layer ② Security — the gate that kept failing 100% of the time
This is the failure I should write about most honestly.
Up to a certain point, pull requests had a security-check step, and green checks lined up. I believed I was protected by it. But the reality was that the step was invoking a subcommand that didn't exist. Because the command didn't exist, it failed with an error every time, regardless of the code's contents. Even a docs-only change went red. And the same broken line kept running across all 29 repos where it was installed.
So it wasn't "green = safe" — the gate had never once held. Red became normal, and people started manually ignoring the red and merging. This is the worst state a gate can be in. It's worse than not existing, because it creates the illusion of "being protected."
I recorded this failure as an architecture decision, deleted the broken gate, and rebuilt a meaningful one in its place. It now runs in three lanes: a "stop only newly introduced issues" gate for pull requests, re-ingestion on push to main, and a daily scheduled scan. Four open-source scanners (static analysis, secret detection, vulnerability scanning, dependency audit) run on my own runners, so the SaaS bill is zero.
The pull-request gate has a trick to it. An existing repo always has a backlog of findings piled up from the past. Block all of them and no one can merge. So I take the before/after diff of the pull request and fail only what the change newly introduced, while showing the existing backlog non-blocking. Stop new defects; make old defects visible — both at once.
The results aggregate into the vulnerability ledger (DefectDojo). That ledger is LAN-only, unreachable from the internet. I keep it as an audit-trail set with ISO 27001 and SOC 2 in view, and I triple its backups: a daily dump → off-site storage → a daily restore into a standby DB.
Layer ③ Monitoring — the spots I wasn't seeing, I only noticed when they broke
The monitoring stack brings up Grafana, Prometheus, Loki, Tempo, Alloy, various exporters, and alert management from a single definition, with every image version pinned. On top of that, I keep a separate full-text search stack for cross-cutting search by request ID or trace ID. There are 24 alert rules in 8 groups and 5 dashboards.
But this configuration wasn't designed from the start. You can't know in advance "what you should be monitoring." This platform's alerts are the trace of adding, after the fact, "the metric I wasn't watching at the time" on each real incident. It's something like a museum of monitoring.
The one that hurt most was on 2026-08-07. A split in the synchronous disk replication (DRBD) went unnoticed by anyone for 18 hours, and during that time the active role bounced back and forth 4 times. The standard exporters weren't watching this sync state at all. So I turned DRBD's state into metrics with my own exporter.
I have 5 home-grown exporters that fill the monitoring stack's blind spots (disk sync, external probing, fencing, backup, runner exhaustion) — the spots the standard exporters don't see (the GPU/model-server exporters below are a separate set). And I've put one important design into them: on a collection failure, stop emitting the metric rather than leaving a stale value (fail-loud). Further, each exporter has a paired alert for "has the exporter itself gone stale?" Monitoring the death of monitoring. It's the monitoring of monitoring, to prevent "I thought I was watching, but the monitoring itself had stopped."
There are 2 CI servers, and both Prometheus instances collect independently of each other. If one dies, the other keeps monitoring. Monitoring, of all things, must not be a single point of failure.
Layer ④ The mechanism that never stops for 24 hours — even with every test green, it isn't achieved yet
The mechanism that keeps running no matter which node dies was the hardest area to judge in this whole platform.
Let me state the principle first. The passing verdict is not a design doc or a unit test — it's a kill-test that drops one node on real hardware. Stop any one node, and every endpoint responds on the surviving node, no data is lost, no double write (split-brain) occurs, and it recovers properly. Only when that goes green on real hardware do I call it "high availability achieved."
I learned this principle from a brutal failure. It began with a split-brain incident that ran for 26 hours. Both nodes became active (and both were returning 200 on health checks, no less), two workers fought over the same file-backed queue, writes were truncated mid-way, and job data was corrupted. Health checks green, data corrupt inside. A surface 200 is no proof of health.
As the fix, I consolidated the sole backing for "who is active" into a single lease. Only the lease holder can be active. It's a scheme that makes dual-active impossible in principle by design. The disk aims for zero loss through synchronous replication, and the GPU node acts as a third voter so a split is decided by 2/3 majority. The edge side, too, has a doubled tunnel that switches over automatically.
And then the real-hardware kill-test. In the test on 2026-07-12, I force-stopped the active CI server container-and-all, and the standby switched to active automatically in 18 seconds with zero human involvement. The stop and the confirmed-up times are kept as evidence in real UTC timestamps. In the same test I confirmed self-fencing and takeover from a network partition, agreement of the data diff, and fail-back.
One honest caveat here. This 18 seconds is a measured value on the NFS-based configuration of the time (2026-07-12). I later migrated the lease foundation to DynamoDB on 2026-08-22, so the old configuration's 16–18 seconds does not carry over to the current one as-is. Keeping the number as "a record of the configuration at the time" and not asserting it as a current value is a discipline I hold on this platform.
One more honest note. There were 3 bugs that the design and unit tests couldn't detect and that appeared only in the real-hardware kill-test: a livelock during the grace period, a problem where a hard-mount's I/O blocks infinitely so self-fencing misfires, and a problem where blindly grabbing the lease rolls the generation number backward. All were green on paper. They only came out when I dropped a node for real. That's why I say "until you drop it for real, I don't call it achieved."
And I'll write the limit that still remains. In the reboot-involving kill-test on 2026-09-03, I measured that automatic recovery fails. The causes are startup races and an asymmetry under packet loss where "reads succeed but acquires fail." I've recorded this honestly as an architecture decision and put it on the ledger as a weakness I haven't fully fixed. High availability isn't a binary of "achieved / not achieved" — it's the work of counting weaknesses with evidence and clearing them incrementally. In this area alone there are 196 automated tests for real-hardware behavior.
Layer ⑤ Local LLM — a GPU that is both the monitored and the worker
On the GPU node I run a local LLM. It has two faces here. It is the monitored side and, at the same time, the side that actually works.
I wrote 2 exporters for monitoring myself: about 100 lines for the GPU and about 230 lines for the model server. Both use only the Python standard library, so they need no extra install. On a platform operated by a small number of people, not adding dependencies is itself availability.
I also face the not-so-clean parts head-on. This GPU has a unified-memory configuration, and the standard GPU-monitoring command won't return the model's memory usage. So I made the value the model server's API returns the source of truth. The model server has no metrics-output port, so I pick up the processing-time lines that appear in the logs with a regex and measure tokens generated per second for real. If there's no clean API, measure from what you have.
I route by purpose, too. Light purposes like classification go to one endpoint; heavy purposes like writing code go to the GPU node. The classifier side has passed the HA kill-test as well — I confirmed on real hardware that dropping one Mac switches transparently to the other.
And here's the most symbolic part. The large model running on this GPU actually writes commits to this infrastructure's repository. As the execution backend for automation, the AI maintains its own infrastructure. In the log, 5 commits remain that you can tell were written by the AI, by their prefix. The company's thinking of "run every business function with AI" appears here at once, both in a glowing card on the monitoring screen and in the commit history.
Reproducing everything from code, so it continues even when the person changes
Anything operated by a small number of people stops if that person falls. So I was thorough about "reproducible from code." The thoroughness comes in two stages.
Stage one is building the second and later servers. One configuration-management command brings up every service. The configuration roles (24 Ansible, 17 playbooks, 14 Compose service layers, 11 Terraform stacks) all derive from a single source-of-truth file that lists 25 products. The catalog shown in Portal is a derivative of that source, too. Because there's a single source, discrepancies don't arise.
Stage two goes further. It's a "clean-room export" that lets me export this whole platform safely without leaving any company-internal values. Even as nodes increase (ci3, ci4…), or when someone takes over, it can be handed off safely by the same procedure. It makes a copy with the history zeroed out, keeps only what's needed via an allow-list, mechanically substitutes company-internal values (domains, real IPs, real names, colleagues' names, task-tracker IDs), and finally runs it through a secret scan. If that scan catches even one item, it exports zero bytes (fail-closed). It's a one-way pipeline that doesn't leave unless it passes both gates. The same thinking as the security lesson in the earlier chapter runs through here.
Backing the numbers on real hardware — the kill-test I keep as a record, and the hard cap on cost
Let me return once more to this article's core. A number only becomes a number once you back it on real hardware.
The HA kill-test becomes the record directly. When, which node, dropped how, switched over in how many seconds. Kept with a UTC timestamp. Not "probably fine" but "on this day, on this configuration, it was this many seconds."
Cost is the same — held down by measurement and structure. The log platform stores the full volume on AWS, but I put a 10 GB (about 10.7 billion bytes) scan cap per query on the search platform, enforced so the user side can't lift it. Billing is metered per TB, so however large a single query is, it converges to about $0.05. I designed "store," "search," and "ship" as separate mechanisms: store long-term on S3, search short-term on the local search platform, and ship sorted by importance.
Honest here too. The policy says "forward only important logs," but the running forwarder configuration actually ships container logs and system logs in full, with no filter. There's a gap between policy and current implementation. When you match numbers against design on real hardware, discrepancies like this always surface. Measuring for real is how you find and fix them.
Updating what kinds of information you keep and how you verify them — scripts that do nothing without certainty
Finally, this is about operation that doesn't cause accidents even when no human is glued to it.
The scripts that run automatically share a common principle: do nothing without certainty. Destructive operations default to a dry run, place a two-stage grace period, and require typing an uppercase "YES" to execute.
Take disk cleanup. On 2026-07-02, a CI server's root filesystem filled to 97% (the running runners' write layers summed to over 60 GB), and out of that the disk-guardian script was born. This script cleans only when a threshold is crossed, and it absolutely protects any runner holding a running job. Deleting a live job just to clean up would defeat the purpose.
The runner reapers split into 3, dividing responsibility strictly. The first is the disk guardian, protecting live jobs. The second deletes only the offline ephemeral ghosts. The third reaps deadlocks that look like they're running but have actually frozen. Even the same "broken runner" has a different owner depending on its state. Making one script do all of it makes the collateral damage large when it misjudges, so I split them on purpose.
The HA switchover verdict follows the same idea. When it can't decide "which one is active," it tips both to the stopped side. Stopping and calling a human is safer than forcing it into a double write. fail-safe — when in doubt, tip to the safe side — is something I hold to thoroughly.
Holding infrastructure with a small number of people might sound like the height of a bus-factor problem. But what I'm actually doing is the opposite: gather the decision material into one dashboard and return a verdict, back the passing line on real hardware, make everything reproducible from code, and have the scripts stop when in doubt — adding "hard to break" one piece at a time is the result. Each one of those pieces corresponds to a concrete failure date: the day it ran away, the day the gate wasn't running, the day 18 hours went unnoticed, the day data was corrupt for 26 hours, the day the disk hit 97%, the day a reboot didn't recover.
Infrastructure doesn't become safe because the design is beautiful. It becomes safe only by however much you drop it, break it, count the gaps you weren't noticing, and fix them one at a time with evidence. A single dashboard is the foundation for continuing that work.
Top comments (0)