DEV Community: Vandit Singh

What Hardening a Container Image Actually Removes

Vandit Singh — Tue, 01 Sep 2026 14:00:00 +0000

Here's the fact that reframes container security once it lands: most of the CVEs a scanner screams about in your image are not in your code. They're in the operating system packages you inherited from your base image and never once use at runtime. Your Go binary doesn't need bash, apt, perl, curl, a full glibc userland, or the seven libraries those drag in, but if they're in the image, every vulnerability in them is now your vulnerability as far as the scanner is concerned.

I spent a chunk of time at RapidFort hardening open-source container images, more than a dozen of them, including images that had to meet IronBank / DoD standards, and the headline result was cutting CVE counts by roughly 80%. The thing nobody tells you up front is how. You don't patch your way to that number. You remove your way to it. This post is what hardening actually does, why the removal is legitimate and not scanner-gaming, and the real costs it comes with.

Where the CVEs actually live

Pull a normal application image and scan it with Trivy or Grype. You'll get a wall of findings. Read them closely and a pattern jumps out: the overwhelming majority are attached to OS packages, not your application. A Node app on node:latest ships a complete Debian userland. A Python app on python:3 ships the same. That's a shell, a package manager, coreutils, compression libraries, TLS libraries, sometimes a compiler toolchain, plus their transitive dependencies. Every one of those is code maintained by someone else, scanned against the same CVE feeds, and counted against your image.

So the image is mostly not your program. It's a small application sitting on top of a full Linux distribution that came along for the ride. And a full Linux distribution has a lot of surface for a CVE database to hit. Your 40MB of app is buried under a few hundred megabytes of things you're being held accountable for.

The core move: the most secure package is the one that isn't there

Hardening is, at its heart, one idea applied relentlessly: remove everything the application doesn't actually need to run. Not patch it. Remove it. A CVE in bash is completely irrelevant to an image that has no bash. You didn't fix the vulnerability, you deleted the code that had it, which is strictly better because now it can't be exploited, can't be a foothold, and can't show up in next week's scan when a new CVE lands in that same package.

This flips the mental model from "keep everything updated" to "ship as little as possible." Every binary you don't include is a vulnerability you'll never have to patch, an attack-surface reduction, and one fewer thing in your SBOM. The question stops being "how do I fix these 200 CVEs" and becomes "why is any of this in my image at all."

The reframe

Patching is a treadmill: new CVEs land in your dependencies forever, and you chase them forever. Removal is permanent: a package that isn't in the image can never generate another finding. Hardening trades an endless patching task for a one-time subtraction. That's why the CVE count doesn't just drop, it stays down.

The toolkit, roughly in order of impact

None of these are exotic. The craft is in applying them without breaking the thing.

Start from a smaller base. The single biggest lever. The ladder runs full → slim → distroless → scratch. Google's distroless images contain your language runtime and nothing else: no shell, no package manager, no coreutils. scratch is literally empty, perfect for a static Go binary. The newer option worth knowing is Chainguard's Wolfi images, built from the ground up to be minimal and near-zero-CVE. Swapping a node:latest base for a distroless or Wolfi runtime can erase most of your findings before you've done anything else.

Use multi-stage builds. Build in a fat stage with all the compilers, headers, and package managers you need, then COPY only the finished artifact into a minimal runtime stage. The build toolchain is where a huge amount of CVE surface hides, and with multi-stage it never ships. Your users get the binary, not the gcc that made it.

Kill the shell and the package manager. Removing bash/sh and apt/apk/yum does double duty. It drops their CVEs, and it removes the attacker's toolkit: if someone gets code execution in your container, a distroless image gives them no shell to spawn and no package manager to pull down their tools. This is one of those rare moves that improves the scan number and the actual security posture at the same time.

Drop root, drop setuid. Run as a non-root USER, and strip setuid/setgid bits off any binaries that keep them, since those are classic local privilege-escalation vectors. Remove docs, man pages, package caches, and test fixtures while you're at it. They're pure dead weight and occasionally a finding.

What that looks like concretely

Here's the transformation in the smallest honest example. The naive image, a full base with the whole toolchain baked into the shipped layer:

# before: app + a complete Debian userland + the build toolchain, all shipped
FROM node:20
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
CMD ["node", "server.js"]

That final image carries apt, bash, npm, the compilers node-gyp pulled in, and a full glibc userland, none of which server.js touches at runtime. Now the hardened version, multi-stage so the build junk stays in the builder, and a distroless runtime with no shell and no package manager:

# stage 1: build with everything you need
FROM node:20 AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .

# stage 2: ship only the runtime and the artifact
FROM gcr.io/distroless/nodejs20-debian12
WORKDIR /app
COPY --from=build /app /app
USER nonroot
CMD ["server.js"]

Same app, same behavior. The second image has no apt, no bash, no build toolchain, and runs as non-root. The scanner findings that were attached to all of that vanish, because all of that is gone. Nothing about server.js changed; everything about what surrounds it did.

Why the number really does drop ~80%

Put those together and the arithmetic is not mysterious. You removed the packages the CVEs were counted against. If 80% of the findings were in the OS userland and the build toolchain, and you ship neither, 80% of the findings are gone. Not suppressed, not marked as false positives, gone, because the vulnerable code is no longer in the image. That's the honest mechanism behind the headline number, and it's why a hardened image tends to stay clean as new CVEs get disclosed: the disclosures land in packages you already don't have.

The honest part: is this just gaming the scanner?

It's the first objection everyone raises, and it deserves a straight answer. Deleting packages to make Trivy show a smaller number would be gaming if the packages were actually used, because then you've shipped a broken image that happens to scan well. The line between legitimate hardening and cheating the scanner is exactly one thing: is the removed package genuinely unused at runtime?

That's where the real work is, and it's less glamorous than it sounds. You have to prove the image still does its job with the package gone. That means actually running it, exercising the real code paths, and watching for the thing that breaks because some library shelled out to a binary you removed, or loaded a locale file, or needed a CA bundle you cleaned up too aggressively. Hardening that isn't verified by running the workload isn't hardening, it's wishful deletion. The scan number is the easy part; the proof that the image still works is the job.

This is exactly the problem RapidFort's runtime intelligence approach automates: instead of guessing what's unused, it profiles what the workload actually loads and touches while it runs, then removes what it never reaches, which is how it reduces attack surface by 60 to 90% without breaking the app. The manual version of this, doing it by hand for one image, is what teaches you why the runtime signal matters. You cannot safely remove a package by reading the Dockerfile. You can only safely remove it by watching the container run and confirming it never needs it. (RapidFort's resource center has the deeper write-ups on the methodology if you want the vendor-grade version.)

And the final acceptance test is the least clever and most convincing one: actually use the hardened image. Not scan it and move on, run it as the real image, in the real workflow, and exercise the things it's supposed to do. If hardening stripped something the app quietly needed, using it is where you find out, immediately and unambiguously, instead of shipping a "secure" image that falls over the first time it hits a code path the scanner never made it touch. A hardened image that nobody ran is a hypothesis, not a result.

We did that by hand first, then automated it with Robot Framework acceptance tests (the Python-based keyword-driven testing framework). Each hardened image gets booted and put through its real behaviors by a test suite, so every rebuild is auto-verified instead of manually poked, and the framework produces a pass/fail report you can attach to the image as evidence it still works. Manual dogfooding proves the idea once; automated acceptance tests prove it on every rebuild, which is exactly what you need when you're hardening a dozen images that keep getting new base layers over time. The scan proves the CVEs are gone; the acceptance suite proves you didn't remove anything that mattered.

The cost

Removing the shell is great for security and annoying for you at 3am. kubectl exec -it pod -- sh to poke around a running container doesn't work when there's no sh. You debug hardened images differently: ephemeral debug containers (kubectl debug) that attach a temporary toolbox to the pod, or a separate, fatter debug image you deploy deliberately. The convenience you're giving up is real. Budget for it instead of being surprised by it.

IronBank, and hardening as a standard rather than a vibe

Some of the images had to meet IronBank requirements. IronBank is the US Department of Defense's hardened-container registry (part of Platform One), and it raises the bar from "make the scanner happy" to "satisfy a documented standard." That means an approved base image, a justification for the packages that remain, continuous scanning, and an accepted-findings process for the CVEs you genuinely can't remove yet (with a reason attached, not a shrug).

Working to that standard changed how I thought about the whole exercise. "Zero CVEs" is not really the goal, because you can't always hit zero and a number alone doesn't prove much. The goal is a defensible image: everything in it is there for a reason you can state, everything that's gone was removed because it was unused, and the handful of findings that remain are known, tracked, and justified. That's a stronger claim than a green checkmark, and it's the mindset that survives contact with an actual auditor.

TL;DR

Most CVEs in your image are in OS packages you never use, inherited from a fat base image, not in your application code.
Hardening's core move is remove, don't patch. The most secure package is the one that isn't in the image. Removal is permanent; patching is a treadmill.
The toolkit: smaller base (distroless / Wolfi / scratch), multi-stage builds so the toolchain never ships, kill the shell + package manager (drops CVEs and the attacker's tools), non-root, no setuid.
The ~80% CVE cut is just arithmetic: you deleted the packages the CVEs lived in. And it stays down because new disclosures land in packages you no longer carry.
It's not scanner-gaming if the removed package is truly unused, and proving that (by running the workload) is the actual work.
Costs are real: no shell means debugging via kubectl debug or a separate debug image. Standards like IronBank push you from "clean scan" to "defensible image," which is the better goal anyway.

Go deeper

Google distroless and Chainguard / Wolfi images, the two minimal-base families worth knowing
Multi-stage builds, the cheapest way to keep the build toolchain out of your runtime
Trivy and Grype, the scanners you're hardening against
IronBank / Platform One, the DoD hardened-image standard
RapidFort's software attack surface use case and resource center, for the runtime-intelligence approach to automating all of the above
Reducing attack surface noise with runtime intelligence, on separating the CVEs that matter from the ones in code you never run
SBOMs, because the smaller the image, the smaller the thing you have to account for

Fun fact: the most satisfying hardening result isn't the CVE number, it's opening a shell in the "before" image, then trying the same thing in the "after" image and getting "no such file or directory." That error message is the whole point. There's nothing there to attack.

I Wrote a Container Runtime From Scratch to Find Out What a Container Even Is

Vandit Singh — Tue, 25 Aug 2026 14:00:00 +0000

For the longest time "container" lived in my head as a magic word. You type docker run alpine, a tiny Linux appears out of nowhere, and you just accept it, the way you accept that the fridge light turns off when you close the door. At some point I got tired of not knowing. So I wrote my own little runtime in Go, called dockerium, for three reasons: I wanted to get better at Go, I wanted to see the container trick with my own eyes, and I wanted to understand why Docker itself is written in Go in the first place. That last one turned out to have the most interesting answer.

Here's the spoiler that ruined the magic in the best way: a container is not a thing. It's just a normal process that has been lied to about what it can see. That's it. No tiny VM, no special "container" object in the kernel. Just a process with a very sheltered upbringing. Let me show you the actual code that does the lying.

The one sentence version

A container is a regular Linux process where you've flipped a few switches so it thinks it's alone in the world:

Its own filesystem (it can't see yours).
Its own process list (it thinks it's PID 1, the first process).
Its own hostname, network, users.

Those switches are namespaces, and the kernel hands them to you for free. dockerium's whole job is to start a process with the right switches flipped, then drop it into a filesystem we downloaded from Docker Hub. Here's the shape of it:

The entire runtime in one picture. The interesting part is the dashed box: a fresh namespace set is what makes a process feel like a container.

Flipping the switches

This is the beating heart of the whole project, and it's shorter than you'd think. When we launch the child process, we hand the kernel a pile of CLONE_NEW* flags. Each one says "give this process its own private copy of X."

cmd.SysProcAttr = &syscall.SysProcAttr{
    Cloneflags: syscall.CLONE_NEWNS | // mount namespace: its own filesystem view
        syscall.CLONE_NEWUTS | // its own hostname
        syscall.CLONE_NEWIPC | // its own inter-process comms
        syscall.CLONE_NEWPID | // its own process tree (hello, PID 1)
        syscall.CLONE_NEWNET | // its own network stack
        syscall.CLONE_NEWUSER, // its own user table (this is the rootless bit)
    UidMappings: []syscall.SysProcIDMap{
        {ContainerID: 0, HostID: os.Getuid(), Size: 1},
    },
    GidMappings: []syscall.SysProcIDMap{
        {ContainerID: 0, HostID: os.Getgid(), Size: 1},
    },
}

↗ main.go

Read that comment column top to bottom and you have basically read the definition of a container. Six flags. That's the "magic."

The last one, CLONE_NEWUSER, is the part I'm proudest of, because it's what lets the whole thing run without root. The UidMappings block says: "the user that is root (ID 0) inside this namespace is actually just me, my normal unprivileged user, out here." So inside the container you're root and can do root-y things, but the kernel still treats you as boring old you on the host. Root in there, not-root out here. That is the entire rootless trick in one struct.

The reframe

Docker feels like it "creates" an isolated machine. It doesn't. The kernel already knows how to give a process a private filesystem, process list, and network. Docker (and dockerium) just asks nicely with the right flags. The isolation was always sitting there in the kernel waiting for someone to turn it on.

The weird part: the program has to run itself

Here's the first thing that genuinely surprised me. You'd think you'd just set those flags on your own process and start doing container stuff. You can't, not cleanly. A Go program is multithreaded from the moment it starts, and some namespaces (PID especially) only really apply to a fresh child. So the classic move, the one every real runtime uses, is: the program re-executes itself. It launches a second copy of its own binary, and that copy is the thing that gets the new namespaces and becomes PID 1.

dockerium does this with Docker's own reexec helper, and finding it is half the fun of this project. I went digging through the moby/moby source (moby is the open source engine underneath Docker) expecting the "make a container" code to be some huge scary subsystem. Instead, tucked away in pkg/reexec, was this tiny, almost cheeky helper that does exactly the self-re-execution trick. The billion-dollar container company and my weekend toy use the same little function. In init() we register a named entrypoint, and check whether we're the re-exec'd child:

func init() {
    // register the function that will run *inside* the new namespaces
    reexec.Register("initialisation", initialisation)
    // if we ARE the re-exec'd child, run it and stop here
    if reexec.Init() {
        os.Exit(0)
    }
}

↗ main.go

So the same binary plays two roles. First run: "I'm the parent, let me pull the image and spawn a child with the clone flags." Second run (the child it spawned): "I'm initialisation, I'm inside the fresh namespaces now, let me set up the container." One program, two personalities, split by a single if.

So why is Docker written in Go?

This was the question I actually built dockerium to answer, and the reexec trick turns out to be the clue. Go is a strange choice for a container runtime if you only look at the good parts. Here's the honest ledger.

The pros are real and big. Go compiles to a single static binary with no runtime to install, which is exactly what you want for a tool that has to run on every Linux box on earth. It has first-class access to raw Linux syscalls through the syscall package, so clone, pivot_root, mount, and sethostname are all right there. It cross-compiles trivially, and its concurrency model makes juggling many containers pleasant. For a fleet-of-processes tool, that's a fantastic fit.

But reexec exists because of Go's one genuine con here, and it's a sharp one. A Go program is multithreaded the moment it starts (the runtime spins up threads for the scheduler, garbage collector, and so on). Some namespace operations, especially entering a new PID namespace, are defined per-thread or only apply cleanly to a fresh single-threaded process. You cannot reliably just call unshare inside a running Go program and expect a clean container, because you're not one process, you're a threaded runtime. So the whole "re-exec yourself as a fresh child with the flags set at spawn time" dance isn't a cute pattern. It's a workaround for Go's threading model, and Docker's own engineers hit the exact same wall, which is why reexec is sitting in moby's source in the first place.

The lesson

The most interesting thing I learned building this wasn't how containers work. It was that even the "right" language choice comes with a tax, and good engineering is often a clever workaround for your own tools. reexec is Docker admitting, in code, that Go's concurrency (its headline feature) is the exact thing that makes low-level namespace work awkward. They shipped the workaround and moved on. So did I.

Building the container's fake world

Once we're the child, inside the new namespaces, we set the place up. The order matters and each line is doing real work:

func initialisation() {
    limitCPUandMemory(rootfsPath, memorylimit, cpulimit) // (see the honest note below)
    mountProc(rootfsPath)                                // give it a private /proc
    defer syscall.Unmount(filepath.Join(rootfsPath, "/proc"), 0)
    pivotRoot(rootfsPath)                                // swap the whole filesystem
    syscall.Sethostname([]byte("dockerium"))             // its own name
    nsRun()                                              // finally, exec /bin/sh
}

↗ main.go

The star of this function is pivotRoot. This is how the container gets a filesystem that isn't yours. You might expect chroot here, the old classic. Real runtimes use pivot_root instead because chroot is escapable and pivot_root actually swaps out the root mount and lets you unmount the old one entirely, so there's no path back to the host filesystem:

func pivotRoot(newRoot string) error {
    putOld := filepath.Join(newRoot, "/.pivot_root")
    // pivot_root needs newRoot to be a mount point, so bind-mount it to itself
    syscall.Mount(newRoot, newRoot, "", syscall.MS_BIND|syscall.MS_REC, "")
    os.MkdirAll(putOld, 0700)
    syscall.PivotRoot(newRoot, putOld) // new root is live; old root parked at putOld
    os.Chdir("/")
    putOld = "/.pivot_root"
    syscall.Unmount(putOld, syscall.MNT_DETACH) // cut the cord to the host
    os.RemoveAll(putOld)
    return nil
}

↗ rootfs.go

Two little gotchas hide in there that cost me real time. One: pivot_root refuses to run unless the new root is a proper mount point, which is why we bind-mount the directory onto itself first. Two: the place you park the old root (putOld) has to live underneath the new root, or the kernel throws EINVAL and you sit there confused. Both are the kind of thing no tutorial mentions until you hit the wall yourself.

Why /proc

That mountProc step isn't decoration. Tools like ps read the /proc filesystem to list processes. Give the container a fresh /proc and ps inside it shows only the container's own processes, which is why your process genuinely believes it's PID 1 and alone. This only works because CLONE_NEWNS gave us a private mount namespace, so mounting /proc here doesn't leak out onto your real machine.

Where the image comes from

Before any of that, we need a filesystem to pivot into. That's just Alpine (or whatever) pulled straight from Docker Hub with its plain HTTP registry API: grab an auth token, fetch the image manifest, download the layer blob, and untar it into a folder. The extraction is refreshingly dumb, it literally shells out to tar:

func ExtractLayer(filepath string) error {
    cmd := exec.Command("tar", "-xvf", filepath, "-C", "/tmp/dockerium/rootfs/")
    return cmd.Run()
}

↗ docker/docker.go

A Docker image, demystified, is a tarball of a filesystem plus a JSON file describing it. Unpack the tar, and you have a root directory you can pivot into. That's the whole "image" concept.

The honest part: what I faked

I promised myself I'd be straight about this, because pretending a learning project is production is how you fool exactly one person, yourself. Two things in dockerium look done but aren't:

The resource limits are theatre. There's a limitCPUandMemory function that dutifully writes 500MB and some CPU shares into files like memory.limit_in_bytes. Looks legit. It does nothing. It writes those files inside the container's own rootfs directory, as plain text files the kernel never reads. Real cgroups mean writing into an actually-mounted cgroup hierarchy, and doing it under a user namespace needs delegated permissions on top. So my container will happily eat all your RAM. Lesson burned into me forever: writing a file named memory.limit_in_bytes is not the same as setting a memory limit. The name is not the mechanism.

There's no networking. I set CLONE_NEWNET, which gives the container its own network stack, which sounds great until you realize a fresh network namespace has nothing in it but a loopback interface. To actually reach the internet you have to build a virtual ethernet pair, wire it to a bridge, set up NAT. dockerium does none of that. So the container is isolated on the network in the most complete way possible: it can talk to no one.

Gotcha

There's also a real bug I left in as a monument to my own hubris: the layer download writes every layer to the same fixed path and overwrites it each time, so a multi-layer image only ever unpacks its last layer. Single-layer images like Alpine work fine, which is exactly why I didn't notice for way too long. The demo working is not the same as the thing working.

What this actually taught me

The point was never to replace Docker. Docker has a decade of people solving the exact "theatre" problems I just admitted to. The point was to delete the magic. And it worked: I can't hear "container" anymore without seeing six clone flags and a pivot_root.

If you've got a black box in your own stack that you've just been accepting, this is my pitch: go build the crappiest possible version of it. Two hundred lines that barely work will teach you more than a hundred blog posts, because the kernel doesn't accept vibes. It makes you get the flags right.

TL;DR

A container is a normal process with a few kernel switches flipped: namespaces for filesystem, PID, network, hostname, users. There is no "container" object.
The program re-execs itself so the child gets a clean set of namespaces and becomes PID 1. One binary, two roles, split by an if.
pivot_root, not chroot, swaps the filesystem and cuts the cord to the host. Mind the two gotchas: new root must be a mount point, old root must live underneath it.
CLONE_NEWUSER mapping your uid to 0-inside-only is the whole rootless trick.
Be honest about the theatre. Writing to a file called memory.limit_in_bytes limits nothing. A fresh net namespace has no network.

Go deeper

The whole thing is ~440 lines of Go: github.com/Vandit1604/dockerium
Docker's own pkg/reexec in the moby source, the tiny helper this whole thing (and Docker itself) leans on
man 2 pivot_root and man 7 namespaces, the two pages that explain the real machinery
Liz Rice's "Containers From Scratch" talk, which is the canonical version of this exercise and where a lot of the shape comes from

Fun fact: the moment it clicked was watching ps inside the container show a process list of exactly one, my shell, sitting there as PID 1, completely convinced it was the only thing alive on the machine. A whole process, blissfully lied to. That's a container.

A Load Balancer Is 200 Lines of Go, and Five Bugs You Won't See Coming

Vandit Singh — Tue, 18 Aug 2026 14:00:00 +0000

"Load balancer" is one of those phrases that sounds like it should live in a data center behind a locked door. You have used a dozen of them without thinking: every time nginx sits in front of your app, every time a Kubernetes Service fans traffic across pods, every time an AWS ALB spreads requests, something in the middle is quietly picking a backend for you. I built a tiny one, go-lb, because I wanted to stop hand-waving the one question underneath all of that: how does a single incoming request find its way to one specific server out of many? You hit one address, but behind it are five identical backends. Something has to sit in the middle and decide. I wanted to be that something, in code, so the routing stopped being magic.

The answer, it turns out, fits in a couple hundred lines of Go. A load balancer takes incoming HTTP requests and spreads them across a pool of backend servers. That's the job. That's the whole job.

But this post has a twist, and it's the honest kind. Writing the happy path was easy. The interesting part is the bugs sitting quietly in my own code, the kind that pass every casual test and would absolutely bite you in production. There are five of them. Four you'd never notice by running the thing. One is loud enough that Go's own tooling flags it, and I shipped it anyway. Let me build the thing and then show you where it lies to you, because that is the actual lesson about load balancers.

What actually sits in the middle

Before the code, the shape. Strip away the marketing and every layer-7 load balancer is two pieces: a picker that chooses which backend handles the next request, and a forwarder that relays the request to it and streams the response back. nginx calls the pool an upstream block. HAProxy calls it a backend. Envoy calls it a cluster. Kubernetes hides it inside kube-proxy and calls the front door a Service. Different words, same two pieces.

The picker is where the named strategies live: round-robin, least-connections, IP-hash, weighted. The forwarder is mostly plumbing: rewrite the request headers, open a connection, copy bytes both ways. What surprised me building go-lb is how lopsided the difficulty is. The forwarder is nearly free in Go. The picker is trivial to write and very easy to write subtly wrong. Almost every bug below lives in the picker or in the state it leans on.

The core: round-robin in one function

The simplest way to spread load is round-robin: hand request 1 to backend A, request 2 to backend B, request 3 to backend C, then back to A. Just rotate through the list. Here's the real rotation from go-lb:

func (s *ServerPool) getNextBackend() *Backend {
    s.mux.Lock()
    defer s.mux.Unlock()
    if len(s.backends) == 0 {
        return nil
    }
    s.current = (s.current + 1) % len(s.backends)
    return s.backends[s.current]
}

↗ serverpool.go

The % len(s.backends) wraps the index back to zero, so it cycles forever. The sync.Mutex matters more than it looks: a load balancer is hammered by many requests at once, so two goroutines could try to bump current at the same instant and both grab the same backend, or worse, index out of bounds. The lock makes the rotation atomic. Miss that and you've got a race condition that only shows up under load, which is the worst kind. (nginx sidesteps this entirely by keeping its round-robin counter per-worker-process rather than shared, so there's nothing to lock. Different concurrency model, same problem to solve.)

Then the actual forwarding is almost free, because Go's standard library already has a reverse proxy:

rp := httputil.NewSingleHostReverseProxy(url)

↗ main.go

That one line is doing a lot. NewSingleHostReverseProxy returns a *httputil.ReverseProxy whose Director function rewrites each incoming request: it swaps in the backend's scheme and host, joins the paths, and leaves the rest alone. When a request arrives, ServeHTTP runs the Director, dials the backend (reusing pooled connections through the default http.Transport), and then streams the response back to the client as it arrives rather than buffering the whole thing in memory. That streaming detail is why a reverse proxy can relay a 2GB download without using 2GB of RAM. Writing that correctly, with trailers, flushing, and hop-by-hop header stripping handled to spec, is the genuinely hard part of a proxy, and Go hands it to you for free.

So the "balancer" is really just: pick a backend, hand the request to its proxy. Pick, forward, repeat.

The reframe

A load balancer isn't a mysterious appliance. It's a loop that picks a backend and a reverse proxy that forwards to it. Go ships the hard part (the proxy) in the standard library. What's left, the picking, is the part you actually design, and it's where all the interesting decisions and bugs live.

Bug one: it skips the first backend

Look at that rotation again. s.current starts at 0, and the function does (s.current + 1) % len before returning the backend. So the very first request doesn't go to backends[0]. It goes to backends[1]. Backend zero doesn't get a request until the counter wraps all the way around.

With three backends it's a small unfairness. But it's a real off-by-one, and it's the perfect example of a bug that passes every test you'd casually write. You fire ten requests, see them spread across all three servers, go "yep, round-robin works," and ship it. The skipped-first-backend detail only matters at the edges: a single request, a pool of two, a health check that always hits the "wrong" one, a canary you just added at index 0 that quietly gets no traffic. Increment-then-use versus use-then-increment is a one-character difference that changes behavior, and you will not see it unless you're looking. The fix is to read backends[s.current] first and increment afterward, so the sequence starts at 0.

Bug two: the health check that never runs

Here's the one that actually matters. A load balancer's real value isn't spreading load, it's not sending traffic to a dead server. If backend B falls over, requests routed to B should stop. go-lb even has the code for it. There's a function that TCP-dials a backend to see if it's alive:

func IsBackendAlive(ctx context.Context, url *url.URL) bool {
    var d net.Dialer
    conn, err := d.DialContext(ctx, "tcp", url.Host)
    if err != nil {
        return false
    }
    _ = conn.Close()
    return true
}

↗ serverpool.go

Looks great. There's a SetAlive setter, an IsAlive getter, and the routing code even walks the pool with GetNextValidServer, checking IsAlive() on each candidate and correctly returning a 503 if every backend is down. The whole liveness system is wired up... except nothing ever calls IsBackendAlive or SetAlive. No background loop pings the servers. So every backend is marked alive when it's created, in NewBackend, and stays "alive" forever, even after it's been on fire for an hour. The safety net is fully built and never hung up.

This is the real gotcha

This is the single most important thing a real load balancer does, and it's the easiest to leave half-finished, because a demo with three healthy backends works perfectly without it. The failure only appears when a backend actually dies, which never happens on your laptop and always happens at 3am in prod. "The code exists" and "the code runs" are different claims. Grep for the callers.

The grown-up version is a whole subsystem. HAProxy runs active checks on a timer (a TCP connect or an HTTP GET /health every few seconds) and, separately, passive checks that watch real traffic and eject a backend after N consecutive failures. Envoy calls the passive version outlier detection and will pull a host out of rotation the moment it starts returning 5xx, then slowly let it back in. Kubernetes splits the concept in two: a readinessProbe decides whether a pod gets traffic at all, and a livenessProbe decides whether to restart it. go-lb has exactly the dialing primitive those systems are built on, and never schedules it. That gap is the difference between a demo and a load balancer.

Bug three: the counter that's lying about its own name

go-lb has a field called aliveConnections and a getter GetActiveConnections. Sounds like it tracks how many requests a backend is currently handling, which is exactly what you'd need for a smarter "least-connections" strategy (send the next request to whoever's least busy, the thing nginx spells least_conn and Envoy calls LEAST_REQUEST). Here's how it's updated:

func (b *Backend) Serve(rw http.ResponseWriter, req *http.Request) {
    b.reverseProxy.ServeHTTP(rw, req)
    b.mux.Lock()
    b.aliveConnections++
    b.mux.Unlock()
}

↗ backend.go

Spot it? The counter goes up after the request finishes, and it never comes back down. There's no matching aliveConnections--. So it's not "active connections" at all, it's a lifetime total of requests ever served. A number that only grows. To actually track active connections you'd increment before ServeHTTP and decrement after, in a defer so it survives a panic:

func (b *Backend) Serve(rw http.ResponseWriter, req *http.Request) {
    b.mux.Lock(); b.aliveConnections++; b.mux.Unlock()
    defer func() { b.mux.Lock(); b.aliveConnections--; b.mux.Unlock() }()
    b.reverseProxy.ServeHTTP(rw, req)
}

As written, the field's name is a promise the code doesn't keep, and any least-connections logic built on it would be quietly, confidently wrong: it would always route to the backend that has served the fewest requests since startup, which after a restart means dumping everything on whichever server booted last. The name lies, and the lie is the kind that survives code review because everyone reads the name instead of the arithmetic.

Bug four: the config file is a decoration

go-lb reads a config.yaml on startup and validates it. Here's the file:

port: 3332
strategy: round-robin
backends:
  - "http://localhost:8080"
  - "http://localhost:8081"
  - "http://localhost:8082"

↗ config.yaml

The config loader even validates that a port was provided, erroring out if config.Port == 0. So it insists you give it a port. Then main.go throws it away and hardcodes a different one:

log.Println("Starting load balancer on port :9000")
if err := http.ListenAndServe(":9000", mux); err != nil {

↗ main.go

You set port: 3332, the validator demands it, and the server binds :9000 anyway. The strategy: round-robin line is worse: nothing in the codebase ever reads config.Strategy. The struct field exists, YAML happily unmarshals into it, and it is referenced exactly zero times. So the one knob that would make this a configurable load balancer, the choice of routing algorithm, is decorative. Round-robin isn't selected; it's the only thing there is.

This is the most human bug of the five, and the most common one in real codebases. The config file reads like a contract. Half of it isn't wired to anything. Nobody notices because the defaults happen to match what the ignored values say, so the demo works and the file looks authoritative. The tell is always the same: grep for where a config field is read, not where it's declared. A field that's declared and never read is a comment wearing a costume.

Bug five: the one go vet catches, that I shipped anyway

This is the loud one. Run go vet ./... on go-lb and it reports three copies of the same complaint:

loadbalancer.go:12: InitLoadBalancer passes lock by value:
    ServerPool contains sync.Mutex
main.go:33: call of InitLoadBalancer copies lock value:
    ServerPool contains sync.Mutex

Here's the crime. ServerPool embeds a sync.Mutex by value, and then I pass the whole ServerPool by value into the constructor and copy it again into a struct literal:

func InitLoadBalancer(serverPool ServerPool) *LoadBalancer {
    return &LoadBalancer{
        serverPool: serverPool,
    }
}

↗ loadbalancer.go

Copying a sync.Mutex is a genuine sin in Go, because a mutex's whole job is to be a single shared piece of state that every goroutine agrees on. Copy it and you now have two mutexes that don't know about each other; a goroutine locking one provides no protection against a goroutine reading through the other. go-lb gets away with it only by luck: the copy happens once, at startup, before any request goroutines exist, and from then on every request goes through the same lb.serverPool copy, so there's effectively one mutex in play. It works. It's still wrong, and it's exactly the kind of "works today" that becomes a heisenbug the day someone adds a second code path that touches the original ServerPool. The fix is to hold *ServerPool (a pointer) everywhere and never copy the struct. go vet told me this before I ever pushed. I shipped it anyway, which is its own small lesson about how easy it is to tune out the tooling that's trying to help you.

Why I'm showing you my own bugs

Because this is the real education. Anyone can write the round-robin. What building go-lb actually taught me is that a load balancer is defined by its failure handling and its honesty about its own state, and both are exactly the stuff that's invisible when everything's healthy. The five bugs aren't embarrassing accidents, they're a map of where the real difficulty lives: concurrency (the mutex, the off-by-one, the copied lock), liveness (the check nobody calls), and honest state (the counter that grows forever, the config that's ignored).

Notice that not one of them is a crash. They're all cases where the code runs, the demo passes, and the behavior is quietly not what the names claim. That's the texture of infrastructure bugs generally. The system stays up and does the wrong thing calmly, and you find out from a graph three weeks later.

Production load balancers, HAProxy, Envoy, nginx, Traefik, are mostly that hard part: active and passive health checks, connection draining, timeouts, retries with budgets, circuit breaking, outlier ejection, weighted and least-request routing, sticky sessions. The routing loop is the 10%. The other 90% is what happens when a backend misbehaves, and every one of my five bugs is a tiny hole in that 90%. Writing the toy version is what made the shape of the real thing finally obvious to me.

TL;DR

A load balancer's core is a picker plus a forwarder. In Go the forwarder is httputil.ReverseProxy, which streams responses and handles the hard proxy spec for free. The picker is the part you design.
Lock the rotation with a mutex, or you get a race under load that never shows up in testing.
Bug 1: increment-then-index skips backends[0] on the first request. A one-character off-by-one.
Bug 2: the health-check code exists but nothing calls it, so dead backends keep getting traffic. This is the whole point of a load balancer, and the easiest thing to leave unwired.
Bug 3: aliveConnections counts up after each request and never down, so it's a lifetime total, not active connections. The name lies.
Bug 4: config.yaml sets a port and a strategy that the code never reads. It binds a hardcoded :9000 and only ever does round-robin. Config that isn't wired up is a comment in a costume.
Bug 5: go vet flags a copied mutex (ServerPool passed by value). It works by luck because the copy happens once at startup. Still wrong. Hold a pointer.
Real load balancers are 90% failure handling. The routing loop is the easy 10%.

Go deeper

The whole thing, ~230 lines: github.com/Vandit1604/go-lb
Go's httputil.ReverseProxy, the standard-library workhorse, and its Director/Rewrite internals
How HAProxy does health checks, active and passive, for what the grown-up version of bug two looks like
Envoy outlier detection, the passive "eject a misbehaving host" pattern
go vet's copylocks check, the analyzer that caught bug five

Fun fact: I found most of these bugs by reading my own code back to write this post, not while writing the code. The go vet one I'd seen and ignored. Nothing teaches you a system like having to explain it out loud to a stranger.

How to Contribute to Open Source

Vandit Singh — Thu, 13 Aug 2026 20:56:14 +0000

Disclaimer: Contributing to open source requires patience—lots of it. If you lack the time or willingness to persevere, open source may not be the right choice for you. Instead, consider building impactful projects that demonstrate your skills. The goal is to showcase proof of work, especially if you're pursuing a job.

Getting started with open-source projects can feel overwhelming, but with the right approach, it becomes manageable. Here’s a straightforward guide to help you contribute effectively:

1. Choose an Organization

Explore organizations using resources like gsocorganizations.dev or the GSoC Project/Organization list.
Filter projects by the programming languages you know or want to contribute to.

2. Use the Project

Use the software you plan to contribute to. This helps you gain essential context and familiarize yourself with terminology and concepts reflected in the codebase and software interface.

3. Read the CONTRIBUTING.md

The README.md provides an overview of the project, while CONTRIBUTING.md outlines how you can contribute.
It is the essential guide for contributing. It includes steps for setting up the project locally, guidelines for issue assignment, testing procedures, and any separate issue trackers.

4. Set Up the Project Locally

Follow the instructions in CONTRIBUTING.md to set up the project on your local environment.
Build the project and familiarize yourself with the file structure, understanding where different components of the code are stored.

5. Analyze Frequently Updated Files

git log --pretty=format: --name-only | sort | uniq -c | sort -rg | head -10

Identify files that are updated most frequently; these are likely critical components of the project. This aligns with the 80/20 rule—most impactful work happens in a minority of files.

6. Understand the Repository Structure

Gain a high-level overview of the directory structure. For example:

  - Repository
  |
  |- directory1 # Contains X
  |- directory2 # Contains Y
  |- directory3 # Contains Z

Dive into specific directories to inspect files and understand their purpose. Validate your assumptions with project maintainers if you're making significant changes.

7. Focus on Your First PR

Choose a manageable issue and commit at least three weeks to it. Research thoroughly and familiarize yourself with the project context.
Look up unfamiliar terms or concepts related to the issue in context of the organization you're contributing to.
Check commit history and git blame when figuring out something in the codebase via the PR description that added the changes you're trying to wrap your head around.
Try to find file(s) where you might need to do the changes for the bug or the feature

8. Use Debugging Tools

While debugging, simple print statements often work best for gaining insights quickly.
Create and utilize your debugging hacks as needed.

Conclusion

Contributing to open source takes effort and time, but it’s a rewarding experience. Start with a project aligned with your skills, understand its structure and culture, and focus on delivering a meaningful first PR. Be patient, persistent, and open to learning—the process becomes easier as you grow. 🚀