DEV Community: Tomasz Wostal

I Let an AI Re-Platform My CI Pipeline. Here's What Broke.

Tomasz Wostal — Sun, 02 Aug 2026 18:32:45 +0000

Originally published at wostal.eu.

TL;DR: I handed a CI re-platform — GitHub Actions → Argo Workflows, GitHub → Codeberg — to an AI agent. The rewrite looked correct and passed lint, but it didn't account for one thing: the runtime environment had changed completely. The old runner was an external VM; the new one is a pod inside the cluster, where Tailscale IPs aren't routable. Nobody ran the pipeline end-to-end, so every difference stayed as a landmine — six of them. The worst part wasn't the AI's mistake. It was that I spent hours blaming the network when the real culprit was a single missing newline in an SSH key.

This is the companion to a different incident. While I was migrating my homelab k3s control plane from SQLite to etcd, the pipeline that ran that migration turned out to be broken in ways that taught me more than the migration itself.

In this post I'll cover:

Where the mess came from — an AI-assisted CI re-platform that nobody validated end-to-end
The phantom CNI bug — hours of chasing MTU, Tailscale, firewall, and conntrack
The real culprit — a single missing newline in an SSH key
Why early failures hide later ones — six latent bugs, surfaced one at a time
The lesson about delegating migrations — to a human or an AI

Where the mess came from

The entire CI had just been migrated from GitHub Actions to Argo Workflows, and from GitHub to Codeberg — and that migration was handed to an AI agent 🙂.

The re-platforming rewrote the logic roughly 1:1, but it didn't catch that the runtime environment had changed fundamentally. The old runner (GitHub Actions / ARC) behaved one way; the new one is a pod in the cluster — from which Tailscale IPs aren't routable, but private IPs are. Nobody (human or AI) ran the pipeline end-to-end, so all of those differences stayed as landmines.

That's why there were six latent bugs and the hang I'm about to describe — not random faults, but traces of a GHA→Argo translation that was never validated in the new environment.

The phantom CNI bug

After the migration, it turned out the deploy pipeline (CI running in the cluster) had never passed end-to-end — ansible-configure was hanging on gather_facts. I spent hours on it and was wrong repeatedly:

"It's MTU" → a test with MTU 1240 hung too.
"It's the Tailscale overlay" → the private IP hung as well (in the test).
"It's the firewall" → I opened the firewall, the public IP hung too.
"It's asymmetric routing / conntrack in the CNI" → a hostNetwork pod hung too.

Every one of those tests pointed at the network. Every one of them was lying to me.

The real culprit

Each test was contaminated by a single bug: in the test pods, I mounted the SSH key with cp from a secret — without a trailing newline. That produces error in libcrypto → the key never loads → ssh behaves randomly (sometimes a hang, sometimes "permission denied"). I was chasing phantoms, because the test rig was broken, not the network.

The breakthrough only came when I did it properly: tcpdump + conntrack on the node, and the key written exactly as CI writes it (printf '%s\n'). Then the picture was clean and consistent: from a pod, the private IP (10.0.1.x) works (ssh + sftp + full ansible -m setup), while the Tailscale IP (100.x) hangs — because the pod has no route into the tailnet. The fix: CI uses the private IP (the firewall doesn't filter the private network anyway). It worked instantly.

Lesson #1: before you blame the network/CNI/MTU, prove that your test rig isn't broken. One missing newline in a key cost me hours chasing a non-existent CNI problem. Reproduce identically to production (here: exactly how CI writes the key), and for "packets vanish midway" use tcpdump/conntrack, not guesses. "Works from my laptop, not from a pod" isn't always the network — sometimes it's a test-rig artifact.

Why early failures hide later ones

Here's the second trap. The deploy pipeline never reached apply+ansible, so each subsequent bug only surfaced once the previous one started working. There were six in total:

missing openssh in the runner image
missing python3 (ansible needs it on the control node)
a kubeconfig artifact that wasn't passed between steps
an approve gate with duration:0 that didn't actually gate anything
ansible reaching over Tailscale instead of the private network (the phantom-CNI bug above)
git auth failing in semantic-release

Lesson #2: a pipeline that fails early masks every downstream bug. You fix one, run again, and the next one — previously unreachable — finally executes and fails. The first green end-to-end run is the moment everything hidden surfaces at once. Plan for that run; don't assume "it lints, so it works."

The lesson about delegating migrations

Lesson #3: re-platforming CI (GHA→Argo, GitHub→Codeberg) requires an end-to-end run — whether a human or an AI did it. A migration rewritten "1:1" looks correct and passes lint, but it loses the runtime-environment differences: an external runner vs an in-cluster pod, the GitHub API vs Forgejo. Those differences won't show up in any static review — only in one full pass. If you delegate such a migration (to an AI agent 🙂), plan a smoke-run of the whole path immediately, or you'll discover the delta months later, at 3am.

This isn't an argument against using AI for migrations. The rewrite was mostly correct, and it saved real time. It's an argument about where the human stays in the loop: not in re-typing the YAML, but in validating the thing actually runs in its new home. The AI translated the language; it couldn't translate the environment, because it never executed there. Neither did I — until it was breaking.

The finale — full GitOps

After the fixes, the pipeline runs the way it should: merge to main → sensor → deploy pipeline → plan → [approve: suspend] → apply → ansible → release, with a manual gate (argo resume = "after approval"). I validated it on a live merge — it cut release v1.4.1 cleanly. Control plane on etcd, DR (local + S3), config via GitOps with a human gate in the middle.

Two incidents, one weekend. One taught me that defaults become landmines when a lab grows up. This one taught me that a migration isn't done until something has actually run end-to-end in its new environment — no matter how confident the thing that wrote it.

When Your Homelab Grows Up: How SQLite Took Down My k3s Control Plane

Tomasz Wostal — Sun, 02 Aug 2026 18:32:01 +0000

Originally published at wostal.eu.

TL;DR: My Hetzner k3s lab quietly became a platform. Dozens of operators with leader-election leases hammered the default datastore — SQLite via kine — until compaction entered a death-spiral: 1.36M rows, a 13.8 GB WAL that wouldn't checkpoint, CPU pinned at 99%, load average 79 on 8 cores. I stopped the bleeding by truncating the WAL, then migrated the control plane to embedded etcd (7.5 GB SQLite → 313 MB etcd, load 79 → 5). This is the full postmortem — and the lessons.

This is a war story, not a tutorial. It's about the moment a homelab stops being a homelab and starts behaving like production — without ever announcing it. The cluster in question, homelab, is the Hetzner k3s setup I wrote about previously. It started small. It did not stay small.

In this post I'll cover:

How an overgrown lab broke the default datastore — the kine/SQLite compaction death-spiral
The firefight — measuring instead of guessing, and the fix that actually worked
The permanent fix — migrating the control plane to embedded etcd, and the honest caveats
The meta-lesson — how to recognize when your lab has become a platform
A diagnostic runbook — so next time it's minutes, not hours

There's a companion piece to this incident. The CI pipeline that ran this etcd migration was itself freshly — and badly — migrated, and debugging it cost me hours over a single missing newline. I split that into its own post: I Let an AI Re-Platform My CI Pipeline. Here's What Broke.

Context: it's "just a homelab" — except it isn't

homelab began like any homelab: one k3s node on Hetzner, a few things to play with. The problem is that over months it quietly became a platform. A single master node (cx43, 8 vCPU / 16 GB, untainted, and also carrying Longhorn and workloads) now runs: ArgoCD, Kargo, Crossplane/Upbound, CloudNativePG, EMQX, Longhorn, trivy-operator, kubescape, Gatekeeper, Goldilocks/VPA, VictoriaMetrics, Loki, OpenTelemetry, Argo Workflows/Events/Rollouts, kgateway, and more.

Each of those is a solid, production-grade operator. But all of them together hammer the control plane on k3s's default settings — which means the datastore is SQLite, accessed through kine, a shim that translates the etcd API into SQL. This works beautifully… until the lab crosses an invisible threshold and starts behaving like a platform. At that point you get production-grade failure modes on lab-grade infrastructure. That's what this post is about.

Part 1 — The firefight: kine's compaction death-spiral

The symptom

The control plane was pinned: master at 99% CPU, ~42% of it in the kernel (sys), load average climbing 32 → 79 on 8 cores. The apiserver started spewing:

"Failed to get storage metrics" storage_cluster_id="etcd-0" err="context deadline exceeded"

The datastore was so saturated it couldn't even answer a query about its own metrics.

The wrong leads (and the most important lesson: measure, don't guess)

My first suspects were "obvious" — and all wrong:

Gatekeeper audit ran with audit-from-cache=false every 60s (full live LISTs from the apiserver). I disabled it and measured: CPU moved by noise.
Goldilocks/VPA — the recommender was writing ~7 checkpoints/s across 142 VPAs. Disabled it → CPU unchanged.

Lesson #1: verify every "it's definitely X" by scaling it to zero and measuring. Cutting individual API clients didn't move CPU, because the bottleneck wasn't any client — it was the datastore engine itself.

The actual root cause

Only by getting onto the node (SSH over Tailscale — public SSH is firewalled) did the truth show up:

state.db:      7.5 GB
state.db-wal: 13.8 GB   ← WAL larger than the DB and NOT checkpointing
select count(*) from kine → 1,361,474 rows

Broken down by key:

/registry/leases/cnpg-system/...      55,709 revisions
/registry/leases/monitoring/...       55,684
/registry/leases/upbound-system/...   55,654
/registry/leases/kgateway-system/...  55,621
... (every leader-election lease ~55,000 dead revisions)

The mechanism: leader-election leases are updated every ~2 seconds by each operator. kine is supposed to delete old revisions (compaction). Here, ~55,000 dead revisions accumulated per lease (≈31 hours with no compaction) → 1.36M rows → a 7.5 GB database.

This is the classic kine death-spiral: the table grew so large that the compaction query itself began to time out — so compaction never caught up, so the table kept growing. A feedback loop. My own remediation from the day before added fuel (a full trivy rescan, deleting 876 reports, a CNPG resync after moving pods).

Dozens of operators (leader-election leases every ~2s)
   → high write rate to kine/SQLite
   → dead revisions pile up  ←──────────────┐
   → kine table grows (rows in the millions)│
   → compaction query times out             │
   → old revisions never deleted ───────────┘   (feedback loop)

side effect: CPU 99% · load 79 · storage metric timeouts

(the original post has this as an interactive diagram)

Lesson #2: on a saturated datastore, every "cleanup" action generates writes that add to the spiral. Tidying up is the opposite of helping.

The fix

A clean window only exists with k3s stopped (the only moment kine isn't holding the database open):

systemctl stop k3s
apt install -y sqlite3                                # NOT installed by default
sqlite3 state.db "PRAGMA wal_checkpoint(TRUNCATE);"   # WAL 13.8 GB → 0
systemctl start k3s

The trap I fell into: I first ran PRAGMA quick_check before the checkpoint — an integrity scan that churned forever on a 7.5 GB database and blocked the operation I actually needed, needlessly extending the downtime.

Lesson #3: on a large, degraded database, skip the expensive full scans — do the minimum required to reach your goal.

The moment of terror (and Lesson #4)

Right after the restart, load jumped to 79 — it looked like the fix had failed. It was a reconciliation storm: after the apiserver restarts, every controller and ArgoCD simultaneously rebuilds its cache with full LISTs. Layered on top of compaction catching up, that produced a temporary spike.

Lesson #4: a control-plane restart always triggers a thundering herd. Don't read the peak as failure — give it a few minutes.

And indeed — once the WAL was truncated, kine's compaction came back to life and started eating the backlog:

Time	kine rows	load (1-min)
start	1,361,474	79
+5 min	646,895	36
+10 min	14,565	3.0

End result: CPU 99% → 30%, load 79 → 3, rows 1.36M → ~14k, storage timeouts → 0. A 99% keyspace reduction; a healthy control plane with headroom.

What remained

The WAL grew to 16.7 GB during the compaction storm (a high-water mark from 1.35M DELETEs), and state.db was still 7.5 GB despite holding only 14k rows — SQLite doesn't release space without VACUUM. Worth cleaning in a window, but not urgent (disk at 47/150 GB).

The most important lesson of Part 1: this was a band-aid. The compaction death-spiral will come back under load, because kine-on-SQLite is fragile at this write profile. We put out the fire — we didn't remove the fuel.

Part 2 — The permanent fix: moving to etcd

Why SQLite/kine hit a wall here

Not because SQLite is "bad" — because an overgrown homelab generates a profile kine can't tolerate:

lots of writes (leases every 2s × dozens of operators, VPA checkpoints, events),
lots of objects and revisions to compact,
one process, one writer — kine serializes everything into SQLite, and compaction is a single point that can choke.

This isn't "production at millions of users" scale. It's overgrown-lab scale — and that's exactly the boundary where the default datastore stops being enough.

Why etcd

native to the apiserver (no SQL translation layer), MVCC, range-ready from an in-memory tree,
real, working compaction plus defragmentation — no lease death-spiral,
better write concurrency.

The honest caveats (Lesson #5: etcd is not magic)

etcd is extremely sensitive to fsync latency → it needs a fast, dedicated local disk (NVMe), never a networked Hetzner Volume.
"let's just add a second master" is an antipattern — etcd needs an odd quorum (1/3/5). The target is 3 nodes, not 2.
migrating a bloated SQLite carries risk (etcd quota) — so compact/VACUUM first, then migrate.
in the migration playbook I found two real bugs before running anything: detecting etcd by directory alone (the master has an empty db/etcd/name stub from provisioning day → a false "already migrated"), and a copy task that would overwrite the existing /etc/rancher/k3s/config.yaml, breaking the master on restart.

Lesson #6: read the real state of the node before you trust the assumptions baked into your automation.

The plan (two phases)

Phase 1 — in-place --cluster-init on the existing master (k3s performs a one-time SQLite → embedded etcd migration). No Terraform, no new nodes. Backup → migrate → verify readyz → etcd snapshots to S3 (Hetzner Object Storage).
Phase 2 — HA: 3 control-plane nodes (a separate Terraform resource — not refactoring the existing master onto count, because changing its address in state would destroy the live master), a stable API endpoint (6443 on the LB), NoSchedule on the masters, and moving Longhorn off the control plane.

Part 3 — Execution log (the real work)

I keep this section as a live journal — so you see not just the analysis, but the actual execution.

Step 1 — fixing the migration playbook (done)

Before touching the cluster, a pre-flight on the live node caught two bugs in the migration playbook (confirming Lesson #6):

False "already migrated." etcd detection checked for the db/etcd directory — but a SQLite cluster keeps a vestigial stub there (just a name file from provisioning day). The playbook would skip the migration as done. Fix: check db/etcd/member (which exists only after real etcd initialization).
Config overwrite. A task used copy on /etc/rancher/k3s/config.yaml — which already exists with node-ip/flannel-iface/tls-san. Overwriting it would break the node on restart. Fix: merge cluster-init: true via lineinfile (idempotently, with a file backup), leaving the rest untouched.

Lesson #7: migration automation written "dry" almost always assumes a clean, textbook node state. A real node, months into its life, has stubs, hand-edited files, and leftovers. A pre-flight that reads the actual state is cheaper than debugging a migration halfway through.

Step 2 — VACUUM + WAL reset (skipped — deliberately)

Originally planned as a separate window. Skipped, because it became moot: after compaction recovered, the logical data was already small (~15k rows), and the 7.5 GB state.db was just dead pages. The etcd migration (Step 3) reads only the current state, so the fresh etcd is born small regardless of the SQLite file size — VACUUM would have been wasted downtime.

Lesson #8: don't do cosmetic cleanup right before an operation that renders it irrelevant.

Step 3 — migration to etcd, Phase 1 (done)

In-place --cluster-init on the existing master via the fixed playbook (locally, over Tailscale, in a maintenance window): stop k3s → wal_checkpoint(TRUNCATE) → back up state.db/token/tls on the node → merge cluster-init into config.yaml → start (k3s does the one-time SQLite → embedded etcd migration) → verify db/etcd/member + readyz.

Result:

Metric	Old kine (baseline)	Death-spiral	etcd (post-migration)
Master CPU	85%	99%	37%
Load	~12	79	~5
Datastore	7.5 GB sqlite	7.5 GB + 13.8 GB WAL	313 MB etcd, stable
Storage-metric timeouts	sporadic	12 / 3 min	0

The node role now reads control-plane,etcd,master. etcd doesn't balloon (307 → 313 MB) and has working MVCC compaction — the compaction death-spiral is structurally impossible. The pre-migration backup sits on the node (/root/k3s-pre-etcd-backup/), and k3s removed the original state.db after the successful conversion.

Lesson #9: the momentary CPU figure (37%) looks similar to a freshly restarted kine — but that's not the point. etcd's value is durability: kine under load would fall back into the compaction death-spiral; etcd can't. Measure resilience, not just a CPU snapshot.

Step 4 — etcd snapshots to S3 (done)

Off-site DR for the new datastore: bucket homelab-etcd-snapshots (Hetzner Object Storage) + --etcd-s3 in config.yaml (creds chmod 600, schedule every 12h, retention 5). Validated with a manual snapshot (landed in the bucket, 147 MB) and a k3s restart (readyz in 4s — etcd starts instantly, none of the SQLite-style storm). Codified: bucket in Terraform + an opt-in playbook (creds via -e, never in git). Nice surprise: k3s already takes local snapshots by default — S3 just adds an off-node layer.

Phase 2 — HA + dedicated control plane (later)

3 etcd nodes (odd quorum), a stable API endpoint, NoSchedule on the masters, Longhorn off the control plane. This will cut CPU further and add redundancy.

The meta-lesson: recognize when a lab becomes a platform

The most important takeaway isn't about SQLite or etcd. It's that a homelab crosses the line silently. k3s's defaults — SQLite, one untainted master, everything co-located — are perfect to start with and harmless for a long time. But once you pile on ArgoCD, Crossplane, CloudNativePG, Longhorn, two observability stacks, and half a dozen operators with leader-election, those same defaults become production-grade landmines. The outage looks production-grade while the foundation is still lab-grade.

Signs your lab just grew up:

the number of operators with leader-election keeps rising (each one = a lease every 2s = compaction load),
GitOps appears, managing everything (reconciliation storms on restart),
the control plane shares a node and a disk with storage.

That's when it's time to pull the control plane out of the defaults: etcd instead of SQLite, a dedicated/tainted master, snapshots, monitoring of datastore size and compaction — before the death-spiral does it for you at 3am.

Appendix: diagnostic runbook

If the symptoms ever return — minutes instead of hours:

# datastore + WAL size (red flag: WAL in GB / not shrinking)
ls -la /var/lib/rancher/k3s/server/db/state.db*

# kine row count (healthy: ~10–30k; spiral: hundreds of thousands / millions)
sqlite3 -readonly /var/lib/rancher/k3s/server/db/state.db "select count(*) from kine;"

# which keys are ballooning (usually leader-election leases)
sqlite3 -readonly /var/lib/rancher/k3s/server/db/state.db \
  "select name, count(*) c from kine group by name order by c desc limit 10;"

# is compaction working / is storage timing out
journalctl -u k3s --since "3 min ago" | grep -iE "compact|Failed to get storage metrics"

If you're running an overgrown lab on k3s defaults, do yourself a favor: check your state.db size before it checks you.