Icinga had to release the same patch seven times. I mean not seven features. Seven releases of Icinga DB 1.5.1, just to keep chasing Go stdlib CVEs. There is no bug in how they operate. It's the agreement you made when you delivered a static binary. 📦
The superpower has a subscription fee
The experience of static linking is unparalleled the first time you use it. Just one binary. You don't have to go through the apt install process, or make excuses saying "it works on my machine", or play shared library version roulette. You scp it and then it's ready to run. That's really awesome. But here's the catch that's not written on the box. You are now responsible for every library baked in, forever. A vulnerability has been identified in the Go standard library. Perhaps it is found in net/http or in crypto/tls, but the specific location is not important. You didn't write that code. You still have to ship a fix for it.
Dynamic linking quietly did your homework
For many years, those in favor of dynamic linking had a dull response to this. The distribution patches libssl, you apt upgrade, and all binaries on the system are automatically fixed. The programmer who created your application is not physically alert. The maintenance burden has always been on the distribution and not on you. Static linking changes that completely. Because there is no shared object, there is nothing to patch. The vulnerable bytes are ingrained in every binary you have ever put out. You have to rebuild and redeploy everything to update one stdlib CVE. That is the tax. It isn't loud. It just manifests as a constant flow of "security release, no new features" that you are now responsible for.
Seven releases of nothing new
If we look back at the reality of the situation, one patch version, namely 1.5.1, had to go through the release pipeline seven times. Seven new versions were published, and interestingly, the release notes were almost identical for all of them. → Your users see churn that isn't progress
→ Your CI burns on rebuilds you didn't choose
→ Your security posture is now tied to Go's release cadence, not yours
Icinga is not at fault for any of that. In fact, they're doing everything right. And that's what hurts the most. We paid for the convenience in advance.
I'm not saying go back to shared libraries
I always choose static binaries. Keeping my deployment process sane is important, and I'm not ready to compromise that just to avoid a few rebuilds. I wish people would be more upfront about the trade-off they're making. "Just ship a static binary" is an easy thing to say with a bunch of written and unwritten caveats. The footnote means that you become the distribution. You become the person who must deploy security updates for software you didn't write. It doesn't matter for a hobby tool. But if there are users relying on it who anticipate regular updates and fixes, you're in it for the long haul. Additionally, Go exacerbates this problem in a subtle manner. The standard library is so extensive that a significant portion of Go binaries on the internet have the same attack surface. A single crypto/tls bug resulted in half of the ecosystem rebuilding simultaneously.
The honest way to think about it
When you use static linking, the work involved in maintaining dependencies is not eliminated. It is simply shifted elsewhere. In the past, the distribution would cover this expense so that you wouldn't have to worry about it. Today, this cost lands on the developer's shoulders, without you even knowing that it exists and that you will need to handle it for life. Well, it's a fair trade if you knew what you were getting into. It's a bad surprise if somebody presented you with a static binary as all good. Plan for this in advance. Automate the rebuild pipeline before your first Common Vulnerabilities and Exposures, not after your seventh release. 🧰
Just consider "recompile and re-release" as part of regular upkeep, not as a crisis. I have a question for the Go community. Are you taking into account the "I'm the distro now" tax when selecting static, or do you realize it after getting a heap of security-only releases?
Top comments (0)