A while back a colleague introduced our team to terraform-aws-github-runner.
Cool project, job comes in, it spins up an EC2 instance for that one job, job runs, instance dies, nothing sits around idle. Dead simple model.
It got me thinking about doing the same kind of thing a different way, packing a bunch of jobs onto the same host at once using Firecracker microVMs, instead of one instance per job. None of this is some new idea I came up with either, ARC (actions-runner-controller) already does something similar with Kubernetes pods. I just wanted to see what it looked like with actual microVMs instead of containers, if it would scale even better and mostly for fun and to learn something.
What happened is I learned a lot, broke some stuff, got a surprise AWS bill, and found a bug in my own code that was working against itself. Figure'd I'd share here partly so I remember it, partly and also, I’m genuinely curious if anyone else is looking into stuff like this. If you know something I don’t (very likely), see mistakes in my understanding please drop it in the comments.
Quick definitions of things to know before we go further so the rest of this makes sense.
- Firecracker - a lightweight virtual machine monitor made by AWS, it’s what actually runs your code when you use Lambda or Fargate under the hood. A microVM boots way faster than a normal VM, we’re talking under 200ms instead of the usual 30-60 seconds.
- KVM - the Linux kernel feature that lets you run virtual machines with real hardware isolation. Firecracker needs direct access to this.
- Bare metal instance - a
.metalEC2 instance type, meaning there’s no hypervisor between you and the actual physical server. This is the only way to get KVM access on AWS, a regular EC2 instance can’t give you that. - Bin packing - fitting as many things as possible into a fixed amount of space efficiently, in this case fitting as many microVMs as you can onto one host based on how much CPU/memory each job needs.
What I built
I call the project BurstGrid still very experimental. A scheduler receives GitHub’s workflow_job webhooks and holds a queue. Worker hosts (the bare metal ones) run a pile of Firecracker microVMs, and the scheduler dispatches jobs to whichever worker has room. Each job gets its own microVM, its own kernel, its own disk, destroyed the second the job finishes.
Every microVM needs 3 things to boot, the Firecracker binary itself, a kernel image, and a disk image (the rootfs) with whatever tools the job actually needs, Docker, Node, Python, whatever. That rootfs is built once (via Docker export) and has to physically live on the worker host's own filesystem before Firecracker can hand it to a microVM at all, either downloaded from S3 at boot, or pre baked into the host's AMI as an optimization. Each microVM that actually boots just gets its own private copy of that host side file so concurrent VMs don't step on each other's writes.
Building a new rootfs image is something you end up doing pretty often since it changes every time you change what's inside the job environment, so instead of baking all that into something slow to rebuild, I just uploaded them to an S3 bucket and had worker hosts pull whatever's currently in there when they boot. That's why S3 keeps coming up later in this post, it's basically the deploy target for everything a worker needs.
“Wait Sebastian, why bin pack at all, why not just boot a fresh microVM for every job the same way terraform-aws-github-runner boots a fresh EC2 instance for every job?”
Good question, and I almost didn’t think about this hard enough myself. Firecracker needs KVM, and under the bare metal only assumption I was working with, that means the host underneath every microVM has to be bare metal.
A c6g.metal costs about $2.18/hr for 64 vCPUs whether it’s running 1 microVM or 40. Booting a fresh host per job would mean paying for 64 vCPUs on a job that might only need 2, plus eating that same 30-60 second EC2 cold boot I was trying to avoid in the first place.
Firecracker’s fast boot only matters once the host is already warm. So bin packing isn’t really a style choice here, it’s what makes the financing work, you keep the expensive host warm and pack multiple microVMs onto it.
That assumption might be changing though. AWS added a NestedVirtualization CPU option for some 8th gen Intel instances, c8i, m8i, r8i, and their flex variants, no bare metal required, officially announced and documented here.
If that exposes a KVM device Firecracker can use cleanly, you could keep smaller, cheaper instances warm instead of one big bare metal box. You'd still need warm hosts, the cold start problem doesn't disappear, but you could scale the fleet more granularly and rely less on aggressive bin packing. Haven't tested it myself yet, so I don't know if nested KVM has any performance or feature gaps compared to real KVM.
You’d still need warm hosts, the cold start problem doesn’t disappear, but you could scale the fleet more finely and rely less on aggressive bin packing. Haven’t tested it myself yet, so I don’t know if nested KVM has any performance or feature gaps compared to real KVM.
The AMI thing I didn’t think through at first
Here’s an honest one. I picked Firecracker specifically because of the fast boot time, and then sort of forgot that the bare metal host underneath still needed a completely normal, boring boot process before any of that speed mattered. It still needs Firecracker installed on it, the GitHub Actions runner installed, a kernel image and a disk image downloaded from somewhere. None of that is instant just because the microVMs running on top of it are fast. Felt kind of obvious in hindsight but it genuinely didn’t click until I was writing the boot script and thought “why is this taking so long” (Also AI made me think my idea came from God himself lmao... ALWAYS read the docs).
Also early on I assumed if I installed Docker on the host’s AMI, jobs running inside the Firecracker VM could just use it somehow. They can’t. The guest VM only gets vCPU, memory, a block device, and a network device from Firecracker, nothing else. No shared filesystem, no shared kernel. Whatever a job needs has to be baked into that job’s own disk image, completely separate from anything on the host. Understandable mistake if you’re new to this but definately one I should’ve caught sooner.
The $64 bill
After about 2 days of messing with it on and off, I got a $64 AWS bill. Not huge but more than I expected for what felt like light usage. Part of the reason is just that I didn’t know AWS had started supporting nested virtualization on some non bare metal instance types, so bare metal felt like my only option and I never even looked for a cheaper path. But the bigger reason is the bare metal worker, plus a NAT instance, plus the scheduler, were all just sitting there running between my test sessions instead of scaling down the way I assumed they would. The autoscaler config existed, it just wasn’t actually being enforced end to end the way I thought it was.
“So does bin packing bare metal actually save money then?”
Depends entirely on how much you’re actually running on it. terraform-aws-github-runner has close to zero idle cost since nothing exists until a webhook shows up. My setup has the opposite shape, a real fixed cost that only pays for itself once there’s enough sustained job volume to keep that host busy most of the time. For 2 days of me poking around occasionally, that fixed cost just sat there. For a team running CI constantly, the math might flip since you’re not paying EC2 cold start overhead per job anymore. I don’t have real numbers proving that yet, that’s reasoning no real metrics yet.
My own dispatch logic was fighting my own autoscaler
This one I’m kind of proud I caught. My job dispatch logic picked, out of all the workers that could fit a job, whichever one had the MOST free capacity left. Felt like reasonable load balancing at the time.
Except my scale down logic only kills a worker once it’s completely empty. If I’m always spreading new jobs onto whichever host has the most room, no host ever actually empties out while there’s a steady trickle of jobs. You end up with 5 hosts each running 2 jobs instead of 1 host running 10 jobs and 4 hosts sitting empty ready to shut down. My dispatch algorithm and my scale down algorithm were working against each other and I didn’t notice until I actually traced through what happens under sustained load.
Fixed it by switching to best fit, pack each job onto the tightest fitting worker that still has room, so load concentrates onto fewer hosts and the rest can go idle. Added a density cap too (80% utilization) so I’m not packing one host so tight that co-located VMs start fighting over disk io, if the tightest fit would push a host past that cap I look elsewhere first. Is 80% the right number? Honestly I picked it because it felt reasonable, not because I measured anything. If you’ve done real testing on Firecracker VM density on metal instances, please tell me.
Borrowing ideas from Karpenter
Once I noticed the dispatch vs scale down issue, I looked at how Karpenter (the Kubernetes node autoscaler) handles this exact tension, since it’s had years to solve it.
Stuff that translated well:
disruption budgets cap: how many workers can get killed in one scale down pass so a burst of idle workers doesn’t wipe out your whole fleet at once.
consolidation ordering: prefer killing the emptiest workers first.
expiration: force recycle a worker past a certain age (only once it’s idle, never mid job) so stale builds don’t stick around forever.
Cooking and Uncooking AMI's
To cut boot time I built a Packer pipeline that baked Firecracker, the runner, the guest kernel, and the default disk image directly into the worker’s AMI, so it wouldn’t need to pull all that from S3 on every single boot.
Then I actually sat down and did the math before making this the default, and I’m glad I did. EC2 instance launch itself (allocate the host, set up networking, boot the kernel, run cloud init) is already 30-60+ seconds before my script even starts.
The S3 download I was trying to skip was only adding maybe 10-20 seconds on top of that. Real savings, sure, but way smaller than I first assumed.
Worse, baking it created a new problem. The boot script only checks whether a baked copy already exists, not whether it's still current.
So if I update the disk image and push a new version to S3 but forget to rebuild the AMI, every new worker keeps booting from the AMI's now outdated baked copy, no error, no warning, just silently running the wrong version until someone remembers to rebake. Fixed it by recording a fingerprint (S3's ETag) of exactly which version got baked in at build time, then having the boot script do one cheap metadata check against the current S3 object before trusting the baked copy.
Match means it's still current, skip the download, keep the speed. Mismatch means someone pushed something newer since this AMI was built, so it falls back to downloading fresh instead of quietly running stale code
Stuff I genuinely don’t know yet
Being honest here because I think it matters more than pretending this is all done and proven.
I have zero expertise with this type of work and I'm really just
learning as I go.
not all of the fixes above have been tested under real sustained concurrent load yet and still many edge cases to cover
The 80% density cap is a guess, not a measured number
packing more jobs onto fewer hosts means a spot interruption now takes out more jobs at once than spreading would have, haven’t fully reasoned through that tradeoff
whether nested virtualization on the newer non-metal instance types (c8i/m8i/r8i and their flex variants) actually works with Firecracker, and if it does, whether a fleet of small warm non-metal hosts beats one big warm bare metal host
Top comments (0)