A Stranger's Pull Request Almost Stole My Cloud Credentials. So Every CI Job Gets a Fresh microVM Now.

My self-hosted CI runner was the least trusted machine I owned. Now every job runs in a disposable Firecracker microVM on Krova Cloud — here's the setup, the scripts, and the trade-offs.

Earlier this year a boring dependency-bump PR came in to one of my open-source projects. One hunk touched the CI config, just slightly. I almost merged it with one eye on the diff.

Then I actually read it: one extra step on every build. And my self-hosted runner — a cheap VPS I'd set up to save CI minutes — ran jobs from forks. It also had cloud credentials sitting in a config file some past version of me had left there "for convenience."

A malicious PR wouldn't have needed to escape anything. It could have curled my environment to a pastebin and the runner would have obliged.

The real lesson wasn't about that PR. It's that a self-hosted runner is a machine that executes strangers' code, and then keeps living. My cheapest machine was my most dangerous one.

Why I didn't just containerize the runner

containers are the standard answer, and they're not nothing. But on a runner they're a screen door where you need a wall:

shared kernel with the host
mounted caches that outlive the job (a poisoned cache hands itself to the next build)
often a Docker socket
a filesystem that persists between builds
I didn't want politeness. I wanted a boundary. So now every CI job gets a fresh Firecracker microVM — own kernel, own rootfs — and the machine dies when the job ends.

The per-job lifecycle

On Krova Cloud each microVM is a Cube. Provisioning is one command and boots in under a second:

npm i -g @krovacloud/cli

krova cubes create ci-job-812 --cpu 2 --ram 4 --disk 40 --image ubuntu-24.04
# ✓ Cube provisioned  ·  booted in 0.9s

The wrapper my pipeline calls per job:

#!/bin/bash
set -euo pipefail

CUBE="ci-job-$BUILD_ID"

# fresh box per job — nothing carries over, ever
krova cubes create "$CUBE" --cpu 2 --ram 4 --disk 40 --image ubuntu-24.04

# secrets injected at runtime as short-lived, scoped tokens.
# never in the image, never on disk after teardown
krova ssh "$CUBE" "DEPLOY_TOKEN=$DEPLOY_TOKEN ./run-build.sh"

# teardown is the security feature
krova cubes delete "$CUBE"

When the orchestrator does it directly (retries included — note the idempotency key so a retried request doesn't double-provision):

curl -X POST https://krova.cloud/api/v1/spaces/$SPACE/cubes \
  -H "X-API-KEY: $KROVA_KEY" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{
    "name": "ci-job-812",
    "image": "ubuntu-24.04",
    "resources": { "vcpu": 2, "ramGb": 4, "diskGb": 40 }
  }'

What the runner looks like from inside

This is the part that changed my threat model. The job's machine has no public IP — it lives on a private NAT'd network:

root@ci-job-812:~# ip -brief addr
lo               UNKNOWN        127.0.0.1/8
eth0             UP             10.0.x.x/24   # private. that's it.

root@ci-job-812:~# ss -tlnp
# nothing listening that you didn't start.
# no SSH on the internet, no scannable surface.

Inbound is default-deny; only ports you explicitly open are reachable, and those can be IP-allowlisted. A CI box opens none.

So the three properties that matter:

1. Nothing persists. No poisoned cache, no leftover cron, no creds on disk for the next build.
2. The boundary is real. Own kernel per Cube (Firecracker — same tech behind AWS Lambda). A kernel bug popped in a build lands in a box that's seconds from deletion.
3. No address to attack. The runner isn't a scannable VPS with port 22 on the internet.

Cost: billed by the minute, so a 12-minute job on a 2 vCPU / 4 GB Cube is a few cents. Ephemeral only works when teardown is cheap — that's the whole reason this architecture is affordable.

What this doesn't fix (read this part)

Isolation ≠ safety. If you hand a job a long-lived production credential, a fresh VM just means the credential gets stolen from a brand-new machine. So on top of the VM:

• short-lived, scoped tokens per job — injected at runtime, gone at teardown
• no shared volumes between jobs
• egress restricted where it matters

Krova gives you the blast radius. Secret hygiene is still your job, on any platform. Anyone selling "secure CI" as a checkbox is selling something.

The reframe

That PR moved CI from "infrastructure" to "supply chain" in my head. The code your pipeline runs is code you didn't write; the machine that runs it should be disposable, bounded, and unreachable.

Give every job its own box. Kill the box when it's done.

Has a PR ever targeted your CI instead of your code? And if you run ephemeral runners — microVMs, containers, or bare-metal chaos — what does your teardown look like? Genuinely comparing notes, comments open.