The playgrounds on https://labs.iximiuz.com have always been amazing, but they recently got an upgrade to enable creation with a custom image for their root filesystem. Not only is this super cool, but it unlocks the ability for authors to significantly speed up the boot times of their playgrounds and labs by pre-installing and configuring everything to achieve massive startup speed improvements.
I recently went on a journey to try this out for myself, since I'd never done it before, and this is a writeup of where I made silly mistakes, so you don't have to.
The New Project
I'm heading to Kubecon EU next month, and I thought it would be cool to have a new project to demo. The committee didn't accept my proposal to present, but the joke's on them, since obviously, with an iximiuz Labs playground, I can demo it to anybody with a web browser.
Only problem is that when I run this locally, it downloads and compiles the kubernetes operator from source, plus a bunch of other stuff that slows down the initialization and basically means the user has to wait for about 5 minutes.
The Challenge
I knew that recent work had been done to allow you to bring your own rootFS to an iximiuz Labs playground, and that sounded like exactly what I needed, but how to actually do it?
The things that make this easy:
- you can just set a link to an image in an OCI compliant container registry as the root file system for a VM
there are a ton of examples that have been opensourced on what these look like for the current playgrounds
when trying it out, you can just run a playground and see what happens (if/when it fails, there's some helpful error messages to show you what went wrong)
The things that make this hard:
- conceptually, I didn't initially get how a container image and a VM could be the same (wasn't the whole Docker/VM thing like "the great divide" in how to run software?!?!)
- the examples can be used as base images, but how do I add my stuff on top of it?
- when you try out a new custom image as a rootFS mount...and it doesn't work...how do you debug it? (* - this actually has been improved since I started drafting this article...that's how awesome it is to be in the iximiuz discord and ask for a feature!)
The Stuff I Did
So I started out with one of the rootFS images, the one for Ubuntu 24.04. Then I took some of the stuff from my custom playground and added it to the Dockerfile...basically the apt install -y stuff, and also the curl -s https://raw.githubusercontent.com/kubescape/kubescape/master/install.sh | /bin/bash stuff, cause I gots to have my kubescape scanning in this playground.
That built okay, and so I pushed to GitHub's container registry (editor's note: this is preferable to just pushing to dockerhub because the rate limiting is less of a pain).
The VM immediately blew up though, and wouldn't even boot.
Lesson: Github packages, which container images are, are private by default, and you have to go through the whole rigmarole of switching their visibility to public before they can be pulled by anyone but you (this is why just pulling it locally isn't a good test!).
Now having switched my image to public, it finally pulled and built fine, but nothing worked, because there was no kubectl installed, nor any of the stuff I actually needed to run k3s. I needed to use the same underlying image as the actual k3s playground I had based on this Dockerfile.
Lesson: base the custom rootFS on the base image that actually has the stuff you need.
There was another kerfuffle when I tried to copy in the kubernetes operator (y'know, the WHOLE POINT of the playground) into the image, but it kept disappearing.
Lesson: don't copy stuff into /tmp, cause just like in AWS ECS (which you think I would have remembered by now cause it's happened to me about a bajillion times) it gets wiped when the system mounts it. It works in the playground configuration because all that stuff happens after mounting.
I was also trying out different things, pushing images to ghcr, but not seeing any differences in the lab VMs. Turns out there is some pretty aggressive image caching going on, and if you use the same tag, it might not get picked up.
Lesson: while prototyping, you might just need to cycle through different tags, and that's how I ended up with v4 as my (current) working tag.
First Result
So with all my efforts the startup time went from about 6 minutes to...about 5 minutes. It turns out that the package installation was already fairly quick, and the real slowdown was happening when I compiled all the kubernetes operator stuff (there's a LOT of stuff that goes on there!).
So the next obvious move was to move all that compilation to build time, swap in a multi-stage build so that I could get rid of the entire golang toolchain in the final image, and just copy in the binary and CRD yaml.
Second Result
Now with the binary all ready to go on boot, the VM startup went down to around 1-2 minutes. The remaining time is starting up all the pods in the k3s cluster and making sure things are running smoothly before we're all ready for the user to start doing thing.
Lesson: If you want to actually make sure things are ready to interact with, don't get rid of all your init scripts, cause those are what make the fancy "Warming up playground" screen, and then users are immediately dumped into the terminal and might think things are ready, but they might not be.
So I added a
kubectl wait --for=condition=Ready --timeout=60s pods --all -n test-lab
to my lab just to make sure that screen kept the user waiting while things get ready.
I also was wondering if it would be necessary to create two different versions of the playground with different specs (obviously, more cpu/RAM would mean that it can start up and be ready even faster), but thankfully that's not even an issue, since you can just set it to the max of the premium tier, and when a free tier user starts it, it defaults to free tier limits (important - test out your playground as a free user if you want to make sure this actually works)
The End Result
So I've got a playground where you can practice running scans with kubescape to find security vulnerabilities in a running k3s cluster, then test out your fix (it's usually just updating the deployment), and the cluster will reset to another random vulnerability.
The end result is a combo of Dockerfile, playground manifest, and some operator CRD stuff, which you can see here on GitHub.
the reason I got confused initially between docker/VM is that in this case, we're only creating an image to use as a root volume, which is kind of what docker also does, but docker also layers on all of the fancy namespace (network/file/process/etc) stuff, which we don't use here, so a volume is a volume is a volume.
I'll definitely be using this custom rootFS method going forward, and I encourage you to give it a try if you find yourself in need of some speed!
Top comments (0)