DEV Community: Vivian Voss

ssh-agent

Vivian Voss — Tue, 14 Apr 2026 05:50:34 +0000

Technical Beauty — Episode 31

You have typed your passphrase four times this morning. Once to pull from GitHub. Once to deploy to staging. Once to SSH into production. Once because you mistyped it the third time and had to start over. By lunch, you will have typed it twelve more times, and by the end of the week you will have created a key without a passphrase because life, one feels, is too short.

Congratulations. Your private key is now a plaintext file on disk. Anyone who reads ~/.ssh/id_ed25519 owns every server you can reach. This is not a hypothetical. This is a Tuesday.

In 1995, a password-sniffing attack hit the network at Helsinki University of Technology. Tatu Ylonen, a researcher there, decided this was unacceptable and wrote SSH that same year. As part of that implementation, he wrote ssh-agent: a process that holds your private keys in memory and signs authentication challenges on your behalf. The key never leaves the process. Not to the client. Not to the server. Not to the wire. Never.

Thirty-one years later, that agent is still the authentication backbone of modern software delivery. It has been rewritten, hardened, extended, and maintained by the OpenSSH team (forked from Ylonen's ssh 1.2.12 in 1999, first stable release with OpenBSD 2.6 on 1 December 1999). The current source is at version 1.324, with contributions from Markus Friedl, Aaron Campbell, and Theo de Raadt, among others. The design, however, has not fundamentally changed. It did not need to.

The Design

The entire agent is 2,624 lines of C in a single file: ssh-agent.c. Key storage, socket management, the full agent protocol, PKCS#11 smart card support, FIDO/U2F hardware key support, agent forwarding with destination constraints, and locking. In one file. Shorter than most React components one has had the pleasure of reviewing.

The API is a Unix domain socket. When the agent starts, it creates a socket, sets its permissions to owner-only (umask(0177)), and prints two environment variables:

SSH_AUTH_SOCK=/tmp/ssh-XXXXXXXXXX/agent.12345
SSH_AGENT_PID=12345

That is the entire interface. No configuration file. No service manager. No daemon registration. No YAML. The socket exists. Programs that know SSH_AUTH_SOCK can talk to it. Programs that do not, cannot. ls -la $SSH_AUTH_SOCK tells you everything you need to know about your authentication state. One does find this rather refreshing.

The protocol is a packetised request-response exchange with five core operations:

Operation	Client sends	Agent returns
List keys	`REQUEST_IDENTITIES`	List of public keys
Sign data	`SIGN_REQUEST` (data + key reference)	Signature bytes
Add key	`ADD_IDENTITY` (private key + constraints)	Success/failure
Remove key	`REMOVE_IDENTITY`	Success/failure
Lock/unlock	`LOCK` / `UNLOCK` (passphrase)	Success/failure

Five operations. The IETF draft describing the protocol (draft-ietf-sshm-ssh-agent, progressing to RFC) is shorter than most framework tutorials. The entire specification fits in your head, which is, one suspects, rather the point.

The Elegance

The authentication flow:

Your SSH client connects to a server.
The server sends a challenge (data to be signed).
The client forwards the challenge to the agent via the Unix socket.
The agent signs the challenge with the private key using sshkey_sign().
The agent returns only the signature.
The client sends the signature to the server.

Step four is where the beauty lives. The agent calls the signing function internally. The result, a sequence of signature bytes, is sent back to the client. The private key material never crosses a process boundary. It is never serialised to any output channel. It exists in exactly one place: the agent's memory, in an in-memory linked list of Identity structures.

When the agent process terminates, the operating system reclaims the memory. The keys are gone. No cleanup script. No cache file to purge. No "secure delete" to trust. No residual state. The process was the vault, and the vault is demolished.

The agent also monitors its parent process. If the parent shell dies (detected by getppid() returning 1), the agent cleans up its socket and exits. No orphan processes accumulating in your process table. One does appreciate software that tidies up after itself.

The Constraints

ssh-add is the interface for managing keys in the agent:

ssh-add ~/.ssh/id_ed25519          # Add a key (passphrase prompt)
ssh-add -t 3600                     # Key expires after 1 hour
ssh-add -c                          # Confirm each signing operation
ssh-add -l                          # List loaded key fingerprints
ssh-add -D                          # Delete all keys

The -t flag stores an absolute expiry time in the identity's death field. After that time, the key is automatically removed. No cron job. No external timer. The constraint is part of the key's identity.

The -c flag requires interactive confirmation (via ssh-askpass) before every signing operation. The agent calls confirm_key() before executing sshkey_sign(). You see who is asking, and you decide whether to sign. For forwarded agents, this is the difference between convenience and a security incident.

The locking mechanism (ssh-agent -x or the LOCK protocol message) makes all keys inaccessible until the agent is unlocked with the correct passphrase. The passphrase is stored as a salted hash, not plaintext. For stepping away from your desk, this is rather more civilised than terminating the agent and re-adding all keys when you return.

The Security

The agent takes security seriously at the implementation level:

Anti-tracing: platform_disable_tracing(0) at startup prevents other processes from attaching a debugger to read key material from agent memory.
Privilege dropping: setegid(getgid()) and setgid(getgid()) at startup. The agent runs with minimal privileges.
Memory hygiene: sshkey_free() for key material. freezero() for PINs (zeroes memory before freeing). explicit_bzero() for lock password hashes, which prevents the compiler from optimising away the zeroing because the variable is "no longer used."
Socket permissions: Created with umask(0177). Owner-only access. The directory is created with mkdtemp() for additional protection.

These are not features listed on a marketing page. These are manners. The kind of quiet, disciplined engineering that distinguishes software built by people who understand what they are protecting.

Agent Forwarding

Agent forwarding (ssh -A or ForwardAgent yes) allows a remote machine to use your local agent. The remote sshd creates a Unix socket, sets SSH_AUTH_SOCK, and tunnels requests back through the SSH connection to your local agent. You can hop from server to server without copying keys.

The risk: anyone with root on the remote machine can access the forwarded socket while your session is active. They can use your agent to authenticate to any server your keys can reach.

The mitigations are characteristically minimal. ssh-add -c requires confirmation per use. ProxyJump avoids forwarding entirely by routing through an intermediate host without giving it agent access. Recent OpenSSH versions add destination constraints: keys can be restricted to specific hosts, so a compromised jump host cannot use the forwarded agent to reach arbitrary targets.

The design philosophy is consistent: provide the mechanism, make the risks visible, let the operator decide. No hand-holding. No default that pretends to be safe. Honest tooling for people who read the man page.

The Proof

Every CI/CD system on earth uses ssh-agent. GitHub Actions has a dedicated webfactory/ssh-agent action. GitLab's official documentation recommends eval $(ssh-agent -s) as the standard pattern for pipeline SSH key management. Jenkins, CircleCI, Buildkite, Travis CI: all use the agent protocol.

macOS integrated ssh-agent into the system keychain with Leopard in 2007. ssh-add --apple-use-keychain stores passphrases in Keychain, bridging the Unix tool with Apple's credential infrastructure. GNOME and KDE auto-start agent-compatible daemons. Windows 10 added an OpenSSH agent as a system service in 2018. 1Password and Bitwarden now implement the agent protocol natively.

An API designed by a Finnish researcher in 1995, implemented in a single C file, communicating over a Unix socket named by an environment variable, is the authentication backbone of modern software delivery. One does find this rather beautiful.

The Principle

ssh-agent is 2,624 lines of C. One file. One socket. One environment variable. No configuration file. No YAML. No cloud dependency. No subscription. The private key never leaves the process, the process never outlives the session, and the session never trusts more than it must.

Tatu Ylonen wrote it because typing passphrases was tedious. The best Unix tools often begin this way: someone finds a task annoying, writes a small programme to solve it, and designs it with enough discipline that it still works thirty-one years later. No rewrite. No framework migration. No breaking changes.

Technical beauty emerges from reduction. ssh-agent reduced authentication to five operations, one socket, and the guarantee that the secret never leaves the room.

Read the full article on vivianvoss.net →

By Vivian Voss — System Architect & Software Developer. Follow me on LinkedIn for daily technical writing.

periodic

Vivian Voss — Mon, 13 Apr 2026 06:03:35 +0000

The Unix Way — Episode 12

The problem: you need scheduled maintenance. Disk cleanup, log rotation, security audits, database backups, certificate renewal checks. Every server has them. Every sysadmin has cron jobs for them. And every sysadmin has, at some point, discovered a critical cron job that died silently three months ago because nobody checked the output.

cron is a scheduler. It runs commands at specified times. It does not care what those commands do, whether they succeed, whether they produce output, or whether anyone reads that output. It is a clock with a trigger. Nothing more.

The real problem is not scheduling. Scheduling is trivial. The problem is knowing what happened. And cron, by design, does not answer that question.

Scatter your jobs across per-user crontabs and root's crontab, and within six months you have a system where nobody knows what runs when, what produces output, what has been silently failing, or who added the job in the first place. This is not a theoretical scenario. This is every production server one has ever inherited.

FreeBSD: periodic(8)

FreeBSD solved this problem in the 1990s with periodic(8), a framework that sits on top of cron and provides structure, configuration, and output management for recurring system tasks.

The architecture is straightforward. Scripts live in directories:

/etc/periodic/daily/
/etc/periodic/weekly/
/etc/periodic/monthly/
/etc/periodic/security/

Each script is a self-contained shell script that performs one task and exits with a status code:

Exit Code	Meaning	Output Handling
0	Nothing notable	Shown only if `daily_show_success="YES"`
1	Notable information	Shown only if `daily_show_info="YES"`
2	Configuration warning	Shown only if `daily_show_badconfig="YES"`
>2	Critical, always shown	Cannot be suppressed

One configuration file controls the entire system:

# /etc/periodic.conf
daily_clean_tmps_enable="YES"
daily_backup_passwd_enable="YES"
daily_status_security_enable="YES"
daily_accounting_enable="YES"
daily_show_success="NO"
daily_show_info="YES"
daily_show_badconfig="YES"
daily_output="/var/log/daily.log"

Enable a task: set it to "YES". Disable it: "NO". Change the output destination: set daily_output to a file path or an email address. One file. Every task. Every setting.

The default configuration in /etc/defaults/periodic.conf documents every available option. Your overrides go in /etc/periodic.conf. The pattern is identical to rc.conf: system defaults are never modified, local changes are applied on top.

The output is collected from all scripts, filtered by severity, and delivered as a single report. If daily_output is an email address, you receive one email every morning with the complete state of your system: which temp files were cleaned, whether the password database was backed up, what the security check found, and whether anything went wrong. You read one email with your morning coffee and know the state of the machine.

Adding a custom task is trivial: write a shell script, make it executable, drop it into /etc/periodic/daily/. That is the entire API. No registration, no configuration reload, no daemon restart. The framework discovers it on the next run.

newsyslog(8) already knows about /var/log/daily.log, /var/log/weekly.log, and /var/log/monthly.log. If those files exist, they are rotated automatically. The logging of your maintenance framework is itself maintained. One does appreciate the recursion.

cron still does the scheduling. Three lines in /etc/crontab:

0  2  *  *  *  root  periodic daily
0  3  *  *  6  root  periodic weekly
0  5  1  *  *  root  periodic monthly

cron triggers periodic. periodic runs the scripts. The scripts report their status. The framework collects the output. The administrator reads one email. The separation of concerns is, one must note, rather elegant.

OpenBSD: daily(8)

OpenBSD takes a characteristically minimal approach. Three shell scripts ship with the base system: /etc/daily, /etc/weekly, and /etc/monthly. These are system scripts. You never modify them.

Your additions go into /etc/daily.local, /etc/weekly.local, and /etc/monthly.local. These local scripts run first, before the system scripts, which makes it convenient to define variables, perform cleanup, or prepare state that the system scripts depend on.

The daily script performs a comprehensive set of checks:

Removes scratch and junk files from /tmp
Purges accounting records and reports processes killed by pledge(2) or unveil(2) violations
Checks daemon status: lists any daemons enabled in rc.conf.local that are not actually running
Reports which file systems need to be dumped
Runs the security(8) check script
Optionally backs up the root file system to /altroot
Optionally runs calendar(1) and file system checks

The security integration is notable. OpenBSD's daily maintenance does not merely check disk space and rotate logs. It reports on pledge and unveil violations: processes that attempted to exceed their declared capabilities or access files outside their declared scope. Security is not an add-on, not an agent, not a third-party tool. It is a shell script that runs every morning and tells you who misbehaved.

Output is mailed to root. The OpenBSD handbook strongly recommends that root's mail is aliased to a real user account. Otherwise, root's mail simply accumulates until the partition runs out of space. One does note the pragmatism.

Linux: systemd timers

Linux offers systemd timers as the modern alternative to cron. Each scheduled task requires two files: a .service unit file defining what to run, and a .timer unit file defining when to run it.

A daily cleanup requires a service file:

# /etc/systemd/system/cleanup.service
[Unit]
Description=Daily cleanup

[Service]
Type=oneshot
ExecStart=/usr/local/bin/cleanup.sh

And a timer file:

# /etc/systemd/system/cleanup.timer
[Unit]
Description=Run cleanup daily

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

Then enable and start it:

systemctl daemon-reload
systemctl enable --now cleanup.timer

For ten maintenance tasks, this produces twenty files and ten reload-and-enable sequences. The output goes to the journal, retrievable via journalctl -u cleanup.service. There is no collected daily report. There is no severity filtering. There is no single email summarising the state of the system. Each task lives in its own universe.

systemd timers do offer features that cron does not: monotonic timers (relative to boot), persistent timers (catch up missed runs), calendar expressions with second-level granularity, and dependency ordering via the unit system. These are genuine capabilities. Whether the complexity is justified for "run this script every morning" is, one suspects, a matter of taste. And file count tolerance.

The Point

cron tells you when. periodic tells you what happened.

The difference between a scheduler and a maintenance framework is the difference between "the job ran" and "the job ran, succeeded, found nothing unusual, and here is the proof." One of those lets you sleep. The other requires you to check.

FreeBSD built a framework: structured directories, a single configuration file, severity-coded output, collected daily reports. OpenBSD built simplicity: three scripts, three local overrides, security baked into the daily routine. Linux built a general-purpose process management system and asked it to also handle cron jobs.

All three work. The BSDs understood, decades ago, that the problem was never scheduling. It was accountability. One configuration file. One email. One morning coffee. Rather civilised, really.

Read the full article on vivianvoss.net →

By Vivian Voss — System Architect & Software Developer. Follow me on LinkedIn for daily technical writing.

Why the Cloud Is the Default

Vivian Voss — Sun, 12 Apr 2026 07:00:00 +0000

On Second Thought — Episode 03

In February 2011, US Chief Information Officer Vivek Kundra published the Federal Cloud Computing Strategy under the Obama administration's 25-Point IT Reform Plan. The mandate was explicit: agencies must evaluate a cloud computing option first, and come up with a damn good reason not to use it. Kundra claimed $20 billion of the government's $80 billion IT budget could move to cloud.

By 2014, Gartner's Magic Quadrant showed AWS with more than five times the cloud IaaS compute capacity of the next fourteen providers combined. By 2018, "on-premise" had become a dirty word in enterprise IT. The axiom was established.

Not discovered. Not proven. Established. By a government memo, an analyst report, and three poster-child migrations from companies (Netflix, Spotify, Airbnb) whose elastic-demand requirements bear no resemblance to the vast majority of software running in production today.

The Axiom

"Of course we use the cloud. Everyone does." Nobody questions this in a board meeting. Nobody loses a promotion for recommending AWS. CTOs choose it not because it is optimal, but because it is defensible. If something goes wrong with AWS, it is AWS's fault. If something goes wrong with your own infrastructure, it is your fault. This is not engineering. This is career insurance.

The startup ecosystem reinforces this. AWS Activate offers $1,000 to $100,000 in free credits to new companies. Incubators and accelerators distribute AWS credits as standard onboarding. The credits expire after twelve to twenty-four months. By then, your architecture is built on AWS services, your team knows AWS tooling, your monitoring assumes CloudWatch, and your deployment pipeline assumes CodeDeploy. Migration cost exceeds staying cost. The business model is identical to the IBM mainframe playbook of the 1970s: make switching costs higher than the cost of staying. The technology changed. The economics did not.

Harvard Business School research from 2018 documented the effect: after AWS launched in 2006, first-round VC funding for cloud-benefiting startups dropped 20% because infrastructure costs fell dramatically. VCs responded by funding more startups with less diligence. The cloud did not just change infrastructure. It changed who gets funded and how. And it locked in AWS as the default infrastructure for an entire generation of companies that never evaluated the alternative.

The Cost

Flexera's 2025 State of the Cloud Report found that 27% of all cloud spend is wasted. At $675 billion in global cloud infrastructure spending, that is $182 billion per year evaporating into unused resources, over-provisioned instances, and forgotten development environments. Two-thirds of organisations report waste from idle or underused resources.

The utilisation numbers are worse. The median EC2 instance runs at 7 to 12% CPU utilisation. Kubernetes clusters average 10% CPU and 20% memory utilisation. You are paying for seven to ten times the compute you actually use. In no other industry would this be considered acceptable. In cloud computing, it is considered normal.

37signals, the company behind Basecamp and HEY, left AWS in 2023. David Heinemeier Hansson documented the entire process publicly. The hardware investment: approximately $700,000 in Dell servers, fully recouped during the first year. The storage migration: 10 petabytes moved from S3 to Pure Storage, with an upfront cost of $1.5 million and annual operating costs under $200,000 (replacing $1.3 million per year in S3 charges alone). The total annual savings: $2 million. The five-year projection, revised upward from the original $7 million: over $10 million. With faster hardware and considerably more storage. AWS reportedly comped a quarter-million-dollar egress bill. One does appreciate the parting gift.

Dropbox moved 90% of its customer data off AWS to custom colocation in 2015 and 2016. The investment: $53 million in its own data centres. The savings: $75 million over two years. The return on investment was achieved before the infrastructure was fully operational.

Ahrefs, the SEO analytics company, never went to cloud. They run 850 servers in a Singapore colocation. Monthly cost per server: $1,550 on-premises versus $17,557 for the AWS equivalent. AWS would cost 11.3 times more. Over 2.5 years, the estimated savings: $400 million. Ahrefs' total revenue for 2020 to 2022 was $257.5 million. The cloud would not have reduced their margin. It would have eliminated their company.

GEICO, Warren Buffett's insurance subsidiary, spent a decade migrating over 600 applications to Microsoft Azure. Costs ballooned to 2.5 times expectations. Reliability declined. In 2024, they announced repatriation of at least 50% of workloads to an OpenStack-based private cloud, projecting 50% savings per compute core and 60% per gigabyte of storage. Completion: 2029. A decade to get in, half a decade to get out.

The Sovereignty Problem

The CLOUD Act, signed into US federal law in March 2018, allows US law enforcement to compel American technology companies to hand over data stored anywhere in the world. If your data is hosted in Frankfurt or Paris, and the infrastructure is managed by AWS, Azure, or Google Cloud, it can legally be accessed by US authorities. This directly conflicts with GDPR Article 48, which requires international agreements for third-country data access. Cloud providers face an impossible choice: comply with US warrants and breach European law, or refuse and face US legal penalties.

Europe's response was Gaia-X, a federated cloud initiative launched six years ago. What happened: US hyperscalers lobbied to be included. Once inside, they, in the words of Nextcloud founder Frank Karlitschek, "flooded it with documents and regulations." Founding member Scaleway withdrew in 2021. Day-one member Agdatahub received EUR 4.8 million in funding, then went into liquidation. Karlitschek called it a "paper monster." A fundamental failure of strategy, vision, and political will.

The NIS2 Directive, transposed in October 2024, now requires critical-sector organisations to assess cybersecurity risks from cloud providers. DORA, effective January 2025, forces financial institutions to manage ICT risk, including cloud dependency. France's "Doctrine Cloud" mandates government data stays in French-controlled facilities. The regulatory environment is turning, slowly. The infrastructure, however, remains American.

The $1,000 Test

For $1,000 per month on AWS, you get approximately four mid-tier instances (8 vCPU, 16 GB each) with no bandwidth budget. Egress costs extra. Storage costs extra. Monitoring costs extra.

For $1,000 per month on Hetzner, you get seven dedicated AX102 servers. 16 cores each. 128 GB RAM each. 20 TB bandwidth included per server. Totals: 112 cores, 896 GB RAM, 140 TB bandwidth. Independent benchmarks show Hetzner delivering 76% better multi-core performance than AWS and 11 times more IOPS.

The price difference is not a rounding error. It is a factor of seven to ten. For workloads that do not require elastic scaling (which is most of them), the cloud is not a premium for convenience. It is a tax on the assumption that you had no other choice.

The Question

86% of CIOs now plan to move some workloads back from public cloud, according to Barclays' CIO Survey. The highest figure ever recorded, up from 43% in late 2020. The axiom is cracking.

Modern servers handle 500,000 HTTP requests per second. PostgreSQL delivers 70,000 IOPS. A single well-configured machine handles 50,000 concurrent users with proper caching. The vast majority of software in production does not need elastic scale. It needs reliability, predictable costs, and control over its own data.

The cloud was never the only answer. It was the only answer nobody got fired for choosing. The career insurance premium was paid by every company that did not question the default.

On second thought: what if the default is wrong?

Read the full article on vivianvoss.net →

By Vivian Voss — System Architect & Software Developer. Follow me on LinkedIn for daily technical writing.

The Feature Creep

Vivian Voss — Sat, 11 Apr 2026 05:00:00 +0000

Beta Stories — Episode 07

Notion launched in 2016 as a note-taking app with a clever block-based editor. By 2018, it had databases and project management. By 2023, AI. By 2024, a calendar. By 2025, an email client. 180 feature updates shipped in 2024 alone. The desktop app, an Electron wrapper, consumes 200 MB on disk. A note-taking app that now sends your email and manages your calendar. One does wonder when it will offer dental insurance.

This is the industry's favourite disease. And it has two symptoms that appear contradictory but share the same root cause.

Symptom One: Adding What Nobody Asked For

Jira was created in 2002 by two engineers in Sydney with a $10,000 credit card. A bug tracker. Simple, effective, named after Godzilla. Twenty-four years later, it serves Software Teams, DevOps, Product Managers, IT, HR, and Marketing through 3,000 marketplace plugins. Teams reportedly spend more time configuring their workflow tool than doing the work it tracks. Linear, which explicitly positions itself as "the anti-Jira," loads in about half the time. One does note the marketing angle with a certain sympathy.

Slack launched in February 2014 as a team chat application. By 2017, a single workspace consumed 130 MB to 960 MB of RAM. By 2019, users reported 5 GB consumption. The desktop app ran each workspace in its own Electron webview, complete with its own DOM state, JavaScript engine, and GPU resources. In 2019, Slack rebuilt the application, claiming 50% less memory. The application still consumes 200 to 500 MB on disk. For text messages.

Microsoft Teams consumed 5 to 6 GB of RAM in its Electron incarnation. The 2023 WebView2 rewrite claimed "2x faster, 50% less memory, 70% less disk space." Independent benchmarking confirmed the claims. Teams now idles at approximately 1 GB. Progress, certainly. Though one does note that "only 1 GB for a chat application" is a sentence that would have been considered satire in 2005.

Evernote may be the most instructive case. It launched as a simple note-taking application using 200 to 300 MB of RAM. Over the years, it added business card scanning, reminders, a presentation mode, a work chat feature, and, memorably, a marketplace selling branded backpacks and socks. The version 10 rewrite consumed over 1 GB with four notes open. The app fell from market dominance to near-irrelevance, overtaken by Notion (which is, of course, busy repeating the same trajectory at a higher velocity).

iTunes is the poster child. Apple's music player accumulated video playback, TV shows, podcasts, app management, ebook purchases, iPhone synchronisation, iCloud integration, a radio service, and social features. Apple killed it in 2019 with macOS Catalina, splitting it into three separate applications. The WWDC presentation was openly critical of the bloat. One does appreciate a company acknowledging its own mistakes, even if it took fourteen years.

Symptom Two: Removing What Everyone Used

Google has discontinued 299 products and services. Sixty apps. Two hundred and fourteen services. Twenty-five hardware products. The kill rate averaged twelve per year early on, rising to twenty-two per year between 2011 and 2021. 2019 was the bloodbath: over twenty-five products eliminated in a single year.

Google Reader, killed in July 2013, generated 100,000 petition signatures on Change.org within 48 hours, representing 24% of the platform's total traffic. Google proceeded regardless. Feedly gained 500,000 new users in 48 hours. The users were there. The will to serve them was not.

macOS Lion replaced "Save As" with "Duplicate" in 2011. Apple's support forums received hundreds of complaints per week. Users downgraded their operating systems. Apple partially reversed course in Mountain Lion, hiding "Save As" behind Option+Shift+Command+S. Four modifier keys to save a file. Progress.

Windows 11, released in October 2021, removed taskbar drag-and-drop, taskbar repositioning, icon ungrouping, small taskbar icons, and the full right-click context menu. A GitHub project providing a drag-and-drop workaround accumulated 1,500 stars and 173 forks. Microsoft restored partial drag-and-drop support in September 2022, nearly a full year later. Ungrouping returned in May 2023. Taskbar repositioning has not returned at all. The upgrade removed more than it added.

Reddit priced its API at $12,000 per 50 million requests in June 2023, translating to approximately $20 million per year for Apollo, a beloved third-party client with 1.5 million monthly active users. Apollo shut down on 30 June. So did Sync, BaconReader, and Reddit is Fun. Over 8,800 subreddits went dark in protest, representing a collective subscriber count of 2.8 billion. Reddit's CEO called the developer's objections "blackmail." The developer published audio recordings disproving the claim. Reddit proceeded regardless.

Firefox killed its entire XUL extension ecosystem with Firefox 57 in November 2017. Approximately 15,000 legacy extensions, 75% of all add-ons, became incompatible overnight. DownThemAll, Greasemonkey, Firebug, ScrapBook: gone. The browser lost 46 million users over the following three years, declining from 256 million in 2016 to below 200 million by 2020. Mozilla's rationale was sound (security, performance), but the execution was a masterclass in alienating your most loyal users.

Sonos redesigned its mobile application in May 2024, removing sleep timers, alarms, volume control, and accessibility options. The company's stock dropped 13%. Revenue declined 16% in fiscal Q4 2024. Approximately 100 employees were laid off. The CEO, Patrick Spence, resigned in January 2025. The company called the redesign "courageous." One does rather admire the vocabulary.

Skype, once the default for video calling with 405 million users in 2008, added Snapchat-style stories, flashy emoji themes, and colourful chat boxes in a 2017 redesign. App store ratings dropped from 3.5 to 1.5 stars. Usage declined to 36 million by 2023. Microsoft shut it down in May 2025, twenty-two years after launch. The features nobody wanted did not save it. The features everyone relied on were already gone.

The Mechanism

Adding features is how product managers justify the next sprint. Adding features is how startups justify the next funding round. Adding features is how enterprise sales closes the next contract ("Yes, we have that checkbox too"). The incentive structure rewards addition. Nobody gets promoted for keeping a feature count stable.

Removing features is how engineering teams reduce maintenance cost. Removing features is how companies push users toward newer, more profitable products (Google Reader died so Google+ could fail in peace). Removing features is how redesigns happen: strip the interface, call it "modern," ship it before anyone notices what is missing.

Neither decision involves asking the person using the software. The adding is driven by business metrics. The removing is driven by engineering metrics. The user is in neither equation.

The Signal

VLC has been a media player since 2001. No account required. No cloud sync. No AI assistant. No subscription. It plays everything, on every platform, and it has never once tried to send your email. The traffic cone icon has not changed. The feature set has barely changed. The software works. One does find this rather refreshing.

SQLite has maintained backwards compatibility since 2004 and runs on an estimated four billion devices. It did not achieve this by adding features every quarter. It achieved this by being finished. The file format has not changed in twenty-two years. The API is stable. The documentation is complete. The software does what it does, and it does not aspire to do more.

These are not legacy projects clinging to relevance. They are proof that restraint is a feature. The decision not to add a calendar, an email client, or a marketplace selling branded backpacks is itself a design decision, and it is the one that ages best.

Notion now sends your email. Skype added Snapchat stories and shut down eighteen months later. Google Reader served millions and was killed because it did not serve the company's social media ambitions. Windows 11 removed features that users relied on daily and spent two years putting some of them back.

One pattern adds until the product collapses under its own weight. The other removes until the users collapse under their frustration. Both end the same way: the user leaves.

The best software is not the one with the most features. It is the one that never added the wrong ones.

Read the full article on vivianvoss.net →

By Vivian Voss — System Architect & Software Developer. Follow me on LinkedIn for daily technical writing.

The Certification Industrial Complex

Vivian Voss — Fri, 10 Apr 2026 05:00:00 +0000

The Invoice — Episode 17

"Get certified! Industry-recognised credentials! Advance your career! Prove your expertise!"
Splendid. Let us examine what you are actually paying for.

In late 2024, CompTIA's certification business was acquired by Thoma Bravo and H.I.G. Capital, two private equity firms. The nonprofit membership organisation kept its 501(c)(6) tax-exempt status under a new name. The certification arm, the part that generates $168 million per year, became a separate for-profit entity. The quiet part, said aloud: certifications are a product, and you are the customer.

The Renewal Invoice

Cisco CCNP: $700 in exam fees. Expires in three years. Recertification: retake the exams or pay $300 plus Continuing Education courses at $99 to $500 each. AWS Solutions Architect Professional: $300. Expires in three years. Retake at full price. Your knowledge does not expire every thirty-six months. Your certification does. That is not education. That is a subscription.

SAFe, the Scaled Agile Framework, offers thirteen certification levels for a single methodology. Not three. Not five. Thirteen. Initial training: $500 to $3,500. Annual renewal: $195 to $995 per person, depending on level. Two million professionals have been certified. If even a quarter maintain active renewals at an average of $295, that is approximately $145 million per year flowing into one framework's ecosystem. One does admire the arithmetic, if not the pedagogy.

ITIL Foundation: $680 to $850. PeopleCert, which now owns ITIL, requires 20 CPD points annually and a $129-per-year subscription to maintain certification. PMP: $555 for the exam, $139 per year for PMI membership, and 60 Professional Development Units per cycle. Commercial PDU providers charge $25 to $100 per unit, totalling roughly $3,000 per three-year cycle. Your project management certification costs more to maintain than many of the tools you manage projects with.

A team of ten engineers holding mixed Cisco, AWS, and SAFe certifications can easily spend $50,000 to $100,000 per year on certification maintenance alone. Not on training. Not on learning. On keeping the certificates valid.

The Competence Invoice

Foote Partners has tracked IT skills pay premiums across 3,305 employers since 2007. Their finding: non-certified technical skills have earned nearly 2% more of base salary than certified ones for seventeen consecutive years. In 2023, the average pay premium for certifications declined nearly 1% overall, hitting its lowest point since 2015. The market is telling you something. Rather clearly.

Academic research on Microsoft MCP certification found it "does not predict job performance outcomes." Despite MCP-certified professionals reporting significantly higher job competencies than non-certified peers, the certification itself had no predictive power for actual performance on the job. A separate meta-analysis concluded that estimates of the "causal impact of certification on long-term labour market outcomes are not significantly different from zero," though certification can help with initial job-finding. It opens the door. It does not prepare you for what is behind it.

Christian Espinosa, a cybersecurity author, coined the term "cybersecurity paper tigers": people holding penetration testing certifications who cannot explain what nmap is. He reports: "Certifications often only require passing multiple-choice questions that people can memorise, which does not translate to the real world. Incidents do not present people with four options to avert a breach."

Caveon, a test security company, estimates that 15 to 25 percent of IT certification exams show indicators of cheating. The certificate proves you sat in a room. Not necessarily that you learned anything in it.

The Gatekeeping Invoice

"Must have three AWS-certified architects." A single line in a Request for Proposals that eliminates every competitor whose team uses alternative technology. The certification is not measuring competence. It is filtering for vendor loyalty. If your entire team holds Cisco certifications, switching to Juniper means writing off years of investment. The certs are worthless outside the vendor's ecosystem. That is not a side effect. That is the design.

SAFe in enterprise procurement is the purest expression of this pattern. Large organisations, particularly government contractors, require "SAFe-certified teams" in their statements of work. The framework's owner, Scaled Agile Inc., sells the certification. The certification creates the procurement requirement. The requirement sells more certifications. A self-reinforcing loop generating $25 million per year.

Kent Beck, creator of Extreme Programming and co-author of the Agile Manifesto, called the certification structure "dishonest," "a pyramid scheme," and "cancer." Robert C. Martin described agile certifications as "a complete joke and utter absurdity." Martin Fowler offered his assessment at a GOTO conference: "SAFe stands for Shitty Agile For Enterprises." Ken Schwaber, co-creator of Scrum, published "unSAFe at any speed" in 2013. Ten of the seventeen Agile Manifesto co-authors advise against adopting SAFe.

The framework persists regardless. The invoices must be paid.

The Scale Invoice

Pearson VUE delivers 21 million certification exams annually across 20,000 test centres in 180 countries. Contract renewal rate: 99 percent. Revenue: approximately $537 million per year.

Entity	Annual Revenue	Model
Pearson VUE	~$537M	Exam delivery
CompTIA	~$168M	Certification (now PE-owned)
Scaled Agile Inc.	~$25M	Framework + certification
PMI	Not disclosed	Membership + certification
IT Training Market (total)	~$79B	Training + certification
Cybersecurity Cert Market	~$3.9B	Growing to $7.5B by 2030

The total IT training and certification market was valued at $79 billion in 2025, projected to reach $107 billion by 2033. Private equity is not acquiring CompTIA because it believes in professional development. It is acquiring a recurring revenue stream with a 99 percent renewal rate.

The Alternative

Read the documentation. Build something. Contribute to open source. Write about what you learned. Publish your code. A GitHub profile with real commits, real pull requests, and real code reviews tells an employer more than a certificate that expires next March.

The best engineers one has worked with held zero certifications. They held opinions, backed by experience, tested in production. They could explain not just what a tool does but why it was designed that way and when not to use it. No renewal fee required.

Microsoft, to its credit, now offers free online renewal assessments for Azure certifications. It is the only major vendor that does not charge for recertification. One does note this with a certain respect, and a certain awareness of what it implies about everyone else.

The Pattern

The certification industrial complex operates on three revenue streams: initial fees ($100 to $3,500 per exam), recurring renewals ($129 to $995 per person per year), and the invisible cost: vendor lock-in. Once your team is certified in technology X, switching to technology Y means writing off the investment and starting over. The real product is not knowledge. It is switching costs.

When a nonprofit sells its certification business to private equity, the transaction is not a change in strategy. It is a confession. The certifications were always a product. The professionals were always the customers. The knowledge was always incidental.

Pearson VUE delivers 21 million exams per year. CompTIA generates $168 million. Scaled Agile generates $25 million. The industry does not sell knowledge. It sells permission: permission to apply, permission to bid, permission to call yourself qualified.

When private equity acquires the exam, the product is you.

Read the full article on vivianvoss.net →

By Vivian Voss — System Architect & Software Developer. Follow me on LinkedIn for daily technical writing.

Rust Says No

Vivian Voss — Thu, 09 Apr 2026 07:00:16 +0000

By Design — Episode 03

You have written this bug. You have returned null from a function because the real value was not available yet. You have caught an exception three levels up and logged "something went wrong" without knowing what. You have pushed to production, and at 3 AM the on-call phone rang because a pointer pointed to memory that no longer existed.

Every language you have ever used let you compile that code. Every single one said: "Looks fine to me."

In 2006, Graydon Hoare walked up twenty-one flights of stairs to his apartment in Vancouver because his elevator had crashed. The software, written in C, had a memory bug. The elevator ran on code that could access freed memory, dereference null pointers, and corrupt its own state. Hoare lived on the 21st floor. It was not the first time the elevator had crashed. It was, however, the last time he accepted it as normal.

He started writing a programming language that evening. He named it after the rust fungus: an organism, as MIT Technology Review later noted, "over-engineered for survival." Rather fitting.

The Complaint

"Rust is too hard. The borrow checker fights you. There is no garbage collector. No null. No exceptions. No inheritance. Everything is immutable by default. Why does this language say no to everything?"

One does hear this. Usually from someone whose last production outage was caused by one of those features.

The Design: No Garbage Collector

You know the garbage collector pause that spiked your latency? The one you tuned, and tuned, and tuned, and then it spiked again?

Memory in Rust is managed by ownership. Every value has exactly one owner. When the owner goes out of scope, the value is freed. No garbage collector runs in the background. No stop-the-world pauses. No GC tuning. No memory leaked because a reference was held somewhere you forgot about.

Discord documented this in a now-famous blog post. Their Read States service, written in Go, experienced latency spikes every two minutes. The Go garbage collector paused to scan the entire LRU cache. The spikes were not caused by a massive amount of garbage, but by the GC scanning live data to determine what was free. The cache was mostly alive. The scan was mostly pointless. The latency was entirely real.

They rewrote it in Rust. The latency dropped from milliseconds to microseconds. The spikes disappeared. Not because the Rust code was more clever. Because there was no garbage collector to pause. One does rather appreciate a solution that works by removing the problem.

The trade-off is real: you must think about ownership. Every value must have a clear owner. Every borrow must have a clear lifetime. The compiler enforces this at compile time. The cost is paid during development, when your IDE is open and your coffee is warm. The reward is paid in production, when neither is true.

The Design: No Null

You know the null check you forgot last Tuesday? The one that worked fine in testing because the test data always had a value?

In 2009, Tony Hoare (no relation to Graydon) stood before an audience at QCon London and called null his "billion dollar mistake." He had invented it in 1965 for ALGOL W, and he described it as the source of "innumerable errors, vulnerabilities, and system crashes, which have probably caused a billion dollars of pain and damage in the last forty years." One does note that the estimate was conservative even then.

Rust has no null. Instead, it has Option<T>: a value is either Some(value) or None. The compiler forces you to handle both cases. You cannot call a method on a value that might not exist without first checking whether it exists. The billion-dollar mistake is prevented by the type system, at compile time, with zero runtime cost.

This is not convenience. It is the elimination of a defect class. Every NullPointerException your Java code has ever thrown, every segfault from dereferencing null in C, every "undefined is not a function" in JavaScript: Rust's type system makes them structurally impossible. Not unlikely. Not caught by tests. Impossible.

The Design: No Exceptions

You know the try-catch that silently swallowed an error? The one where the catch block logged a message nobody read, and the application continued in an undefined state for forty minutes before someone noticed?

Functions that can fail in Rust return Result<T, E>: either Ok(value) or Err(error). The failure path is visible in the function's type signature. Every caller knows that the function can fail. Every caller must handle the failure or explicitly propagate it with the ? operator.

There are no hidden throws. No catch blocks that swallow errors silently. No stack unwinding that crosses function boundaries invisibly. No "everything is fine" until a try-catch three levels up catches something it does not understand and logs "An error occurred." One does wonder how many production incidents have begun with that exact log message.

The ? operator makes error propagation ergonomic:

fn read_config(path: &str) -> Result<Config, Error> {
    let contents = fs::read_to_string(path)?;
    let config: Config = toml::from_str(&contents)?;
    Ok(config)
}

Each ? either unwraps the success or returns the error to the caller. The code reads linearly. The error handling is explicit. The types tell the truth. Quite the novelty.

The Design: No Inheritance

You know the six-level class hierarchy where nobody remembers what the grandparent overrides? The one where changing the parent class broke seventeen subclasses in ways that only appeared in integration testing?

Rust has no class inheritance. No extends. No class Dog extends Animal. No hierarchy where the behaviour of your object depends on decisions made four levels above by someone who left the company in 2019.

Instead, Rust uses traits: shared behaviour defined as interfaces, implemented by types. A struct can implement as many traits as it needs. Traits compose. They do not cascade.

There is no diamond problem. No fragile base class. No "I changed the parent class and now everything behaves differently." The relationship between types is explicit: this type implements this behaviour. Full stop. One does rather miss problems one never has.

This is composition over inheritance, enforced by the language. Not a design pattern you choose to follow when you remember. A constraint the compiler imposes whether you remember or not.

The Design: No Implicit Mutability

In Rust, all variables are immutable by default. To make something mutable, you write let mut. The act of changing state becomes a conscious, visible decision in the code. Not a default. A declaration.

Combined with the borrow checker's rule that you cannot have a mutable reference and an immutable reference to the same data at the same time, this eliminates data races at compile time. Not by detecting them at runtime. Not by crashing when they occur. By making them structurally impossible to express.

Every concurrent bug you have ever debugged, every "it works on my machine" that vanished under load, every race condition that appeared once in ten thousand runs: Rust's type system prevents them by refusing to compile code that could produce them. The compiler does not trust you. It is, one must concede, entirely justified.

The Trade-Off

Let us be honest. The learning curve is real and well-documented.

The borrow checker will reject code that compiles without complaint in every other language you know. It will reject code that is, in fact, correct. It will reject code that has no bug. It will reject code because it cannot prove the code has no bug, and Rust has decided that "cannot prove safe" is the same as "unsafe."

This is frustrating. It feels adversarial. It feels like the compiler is wrong and you are right and the code is fine and why will it not just compile.

It is not wrong. It is conservative. And in the gap between "probably correct" and "provably correct" lie the bugs you would have shipped to production, discovered at 3 AM, and spent three days debugging whilst questioning your career choices.

The Rust community is honest about this cost. The first three months are painful. The compiler's error messages are unusually good (it tells you what went wrong and often suggests the fix), but the frequency of those messages during learning is high. You will argue with the compiler. You will lose. You will, eventually, realise that losing to the compiler is considerably cheaper than losing to production.

The reward: once it compiles, it works. Not always. But with a frequency and reliability that other systems languages do not match. The category of bugs that Rust eliminates (use-after-free, null dereference, data races, buffer overflows) accounts for approximately 70% of all security vulnerabilities in C and C++ codebases, according to research from Microsoft and Google's Project Zero. Seventy percent. One does find that number rather difficult to ignore.

The Proof

The Linux kernel accepted Rust as a supported language in December 2025. It is no longer experimental. Dave Airlie, maintainer of the DRM subsystem, stated that the DRM project was approximately one year away from requiring Rust and disallowing C for new drivers. The kernel contains 34 million lines of C and 25 thousand lines of Rust. The transition has begun. One does note the ratio with a certain quiet patience.

Microsoft has rewritten 188,000 lines of Windows kernel and DirectWrite code in Rust, with a stated ambition to eliminate C and C++ from its entire codebase by 2030. When Microsoft, a company not traditionally associated with radical architectural courage, decides to rewrite its kernel in a new language, one does pay attention.

Discord's Go-to-Rust migration eliminated garbage collector latency spikes and reduced response time from milliseconds to microseconds. The blog post has become required reading in systems engineering circles. Quite deservedly.

Cloudflare built Infire, a custom LLM inference engine, in Rust, achieving 7% faster inference than vLLM. AWS, Google, and Meta all run Rust in production at significant scale.

Android 16 ships with Rust-built components in the kernel (ashmem memory allocator). Millions of devices run Rust in production without their owners knowing or caring. Which is, of course, exactly how infrastructure should work.

45% of enterprises now run Rust in non-trivial production workloads. The Stack Overflow Developer Survey has named Rust the most admired language for nine consecutive years, with an 83% admiration rate in 2024. Not because it is easy. Not because the learning curve is gentle. Because the elevator stops crashing.

The Principle

Every feature a language grants is a failure mode it accepts. Every convenience added is a bug category normalised. Every "yes" comes with a cost that compounds over decades, paid not by the language designer but by every developer who inherits the codebase.

The courage to say no is the rarest engineering virtue. Rust proves that a language can refuse convenience, refuse familiar patterns, refuse the features that every other language considers mandatory, and win adoption not despite the refusal but because of it.

The borrow checker is not a barrier. It is a boundary. And boundaries, applied with discipline, are what separate systems that survive from systems that merely ship.

Graydon Hoare's elevator still works. One rather suspects it is no longer written in C.

Graydon Hoare stepped down from Rust in 2013. The language he started in frustration on a stairwell is now in the Linux kernel, the Windows kernel, and the infrastructure of every major cloud provider. The fungus, it turns out, was indeed over-engineered for survival.

Read the full article on vivianvoss.net →

By Vivian Voss — System Architect & Software Developer. Follow me on LinkedIn for daily technical writing.

The Page Transition You Never Had to Build

Vivian Voss — Wed, 08 Apr 2026 08:10:36 +0000

Stack Patterns — Episode 11

"We need a single-page application because users expect smooth transitions between pages."

One has heard this argument rather a lot over the past decade. It justified React. It justified Vue Router. It justified Framer Motion (32KB minified). It justified Barba.js. It justified entire application architectures built around the premise that clicking a link should not, under any circumstances, feel like clicking a link.

400KB of JavaScript so that a heading could fade whilst the URL changed. Marvellous.

The browser does it now. In CSS. Three lines.

The Pattern

Both pages include this CSS:

@view-transition {
  navigation: auto;
}

That is the entire opt-in. Click a link that navigates to another page on the same origin. The browser takes a screenshot of the old page, loads the new page, takes a screenshot of the new page, and cross-fades between them. No JavaScript. No framework. No hydration. No virtual DOM diffing the entire document tree so that a title can slide thirty pixels to the left.

The default transition is a cross-fade. It works immediately. For many sites, this is sufficient, and one does find sufficiency rather underrated.

Named Transitions

Want a specific element to animate from its old position to its new one? One CSS property:

h1 { view-transition-name: heading; }

The same view-transition-name on the corresponding element on both pages. The browser snapshots the old position, snapshots the new position, and animates between them. The element does not need to be the same DOM node. It does not need to be the same component. It does not need to exist in the same JavaScript context. It merely needs the same name.

This is the key insight: the browser matches elements across pages by name, not by identity. Two entirely separate HTML documents, served by any backend in any language, connected only by a CSS property. The elegance is rather difficult to overstate.

You can name as many elements as you wish:

h1       { view-transition-name: heading; }
.hero    { view-transition-name: hero; }
nav      { view-transition-name: navigation; }
.sidebar { view-transition-name: sidebar; }

Each named element transitions independently. Unnamed elements participate in the default cross-fade.

Custom Animations

The API exposes CSS pseudo-elements for both the old and new states:

::view-transition-old(heading) {
  animation: 0.3s ease-out slide-out;
}
::view-transition-new(heading) {
  animation: 0.3s ease-in slide-in;
}

The old state and the new state are separate snapshots rendered as replaced elements. You animate them independently. The browser composites them. The result is buttery smooth because it happens in the compositor thread, not in JavaScript.

You can slide, scale, rotate, clip, or apply any CSS animation you would normally use. The full power of CSS animations and keyframes is available. No library API to learn. No React hooks to chain. Just CSS.

The Cost We Paid

Let us be honest about what the industry traded for smooth page transitions.

We abandoned server-rendered HTML. We shipped 300KB JavaScript runtimes to the client. We broke the back button and spent engineering hours rebuilding it in JavaScript. We reinvented routing in userland because the browser's native navigation was "not smooth enough." We lost native browser caching. We invented hydration to fix the problem we created by removing HTML in the first place. We built entire state management libraries (Redux, Zustand, Jotai, Pinia, the list grows quarterly) so the client could remember what the server already knew.

We broke accessibility. Screen readers that worked perfectly with server-rendered pages now had to contend with JavaScript-mutated DOMs, focus management nightmares, and route changes that announced nothing. We broke SEO. Google had to build a headless Chrome renderer just to index SPA content, and for years, sites that relied on client-side rendering ranked lower simply because the crawler could not see the content.

We broke the browser's loading indicator. Users no longer knew whether a page was loading or frozen. So we built skeleton screens to simulate the loading indicator we had removed. Marvellous.

All so a heading could slide.

What It Replaces

The View Transition API does not replace everything. Framer Motion and GSAP remain valuable for complex interactive animations: drag-and-drop, spring physics, gesture-driven sequences. Those are runtime interactions, not page transitions. The distinction matters.

But for page-to-page navigation transitions? The use case that justified SPAs for millions of websites that are, fundamentally, collections of pages linked together with anchor tags?

Solution	Size	JavaScript Required	Works Without JS
Barba.js	7.5KB min	Yes	No
Framer Motion	32KB min	Yes (React)	No
SPA router	Entire framework	Yes	No
View Transition API	0KB	No	Graceful fallback

Zero kilobytes. Because it is the browser. The runtime you already shipped.

Browser Support

Here is where honesty matters, and where many articles about this API get rather conveniently vague.

Full cross-document support:

Chrome 126+ (desktop and Android)
Edge 126+
Safari 18.2+ (including iOS Safari)
Opera 112+
Samsung Internet 29+

Partial or no cross-document support:

Firefox 146+: supports same-document view transitions (Level 1), but cross-document transitions remain behind a flag (dom.viewTransitions.enabled in Nightly only as of April 2026)
Opera Mini, KaiOS Browser, UC Browser: no support

Global coverage: 87.82% of users (CanIUse, April 2026). That is high, but it is not universal. Firefox's absence from the cross-document specification is notable, and for teams whose audience includes a significant Firefox share, this is a legitimate consideration.

Why it does not matter as much as you think:

For browsers that do not support the API: nothing breaks. The user sees a normal page load. No error. No fallback code. No polyfill. No broken layout. No JavaScript exception. The page loads exactly as it would have loaded before the API existed.

This is progressive enhancement in its purest form. The enhanced experience is literally free for supporting browsers (no additional bytes shipped), and the baseline experience is what every user had before. You are not degrading the experience for unsupported browsers. You are enhancing it for supported ones. The risk is zero. The cost is three lines of CSS.

One does rather appreciate an API that fails by doing nothing wrong.

Compare this with the SPA approach: if your JavaScript bundle fails to load (network timeout, CDN outage, ad blocker, corporate proxy), your users see a blank white page. The "enhanced" experience degrades to nothing. The View Transition API degrades to a normal page load. One does note the asymmetry.

The Constraints

Same-origin only. Cross-document view transitions work for same-origin navigations only. You cannot animate a transition from your site to an external URL. This is a security constraint, not a limitation: the browser needs access to both documents to snapshot them. If your site links to external domains, those navigations simply behave normally.

Unique names per page. Each view-transition-name must be unique within a single document. Two elements on the same page cannot share a name. This is rarely a problem in practice, but it means you cannot, for example, animate all list items with the same name. Each needs its own.

Performance consideration. The browser captures raster screenshots of named elements. Naming dozens of elements on a complex page could impact transition performance. In practice, naming three to five key elements (header, hero image, navigation, main content area) produces smooth results. Naming everything is rather missing the point of selective enhancement.

No cross-origin. Worth stating twice. If your architecture relies on navigating between subdomains (app.example.com to docs.example.com), these are cross-origin navigations and will not trigger view transitions. The same-origin policy applies strictly.

Same-Document Transitions

For single-page applications, the same-document View Transition API (available since Chrome 111, Safari 18+, Firefox 133+) provides equivalent functionality using document.startViewTransition(). This variant has broader browser support because it landed earlier.

But the cross-document variant is the interesting one, because it means multi-page applications, the architecture the web was built for, can now match the perceived smoothness of SPAs. Without being SPAs. Without the JavaScript. Without the complexity. Without breaking the back button, the cache, the loading indicator, accessibility, SEO, or the fundamental contract between browser and server.

The Point

For a decade, "smooth page transitions" was the argument that justified client-side routing, framework adoption, and the entire SPA architecture for websites that were, at their core, pages linked together with anchor tags. The browser lacked the capability, so we built it in JavaScript. Reasonably enough.

The browser has the capability now. Three lines of CSS. Zero JavaScript. Progressive enhancement built in. 87.82% global support and climbing. Firefox is the last holdout for cross-document, and the same-document variant already works there.

The argument no longer applies. The question is no longer "should we use an SPA for transitions?" The question is: "what were the other reasons?" One does suspect the list is rather shorter than expected.

One does wonder how many SPAs will be reconsidered. One does rather suspect the answer is: not enough.

Read the full article on vivianvoss.net →

By Vivian Voss — System Architect & Software Developer. Follow me on LinkedIn for daily technical writing.

DTrace

Vivian Voss — Tue, 07 Apr 2026 05:48:27 +0000

Technical Beauty — Episode 30

In 2001, Bryan Cantrill was debugging a production system at Sun Microsystems. The system was misbehaving. The logs said nothing useful. The metrics showed nothing abnormal. By every observable measure, the system was fine. Except it was not fine. It was doing something wrong, and nobody could see what.

Cantrill had helped build an entirely synthetic system: every instruction, every data structure, every byte was placed there by human beings. And yet the system could not answer the most basic question: what are you doing right now?

That bothered him rather a lot.

The Problem

Debugging production systems has always offered two options, both terrible.

Option one: add logging statements, rebuild, redeploy, wait for the issue to recur. This works in development. In production, it means restarting a database serving ten thousand connections to add a printf. The cure is worse than the disease. And if you guessed wrong about where to log, you restart again.

Option two: attach a debugger. Stop the process, inspect memory, step through code. This works beautifully on a developer's laptop. On a production trading system processing four million transactions per hour, stopping the process is not debugging. It is an outage.

Unix has had syscall tracing since the 1990s. truss on Solaris and FreeBSD. strace on Linux, written by Paul Kranenburg for SunOS in 1991, ported to Linux by Branko Lankester. Both trace system calls by intercepting them via ptrace. Both work by stopping the traced process for every single syscall, switching context to the tracer, recording the call, and resuming. The overhead is brutal. Running strace on a production web server is rather like performing surgery whilst repeatedly switching off the lights.

strace traces one process. It cannot follow what happens inside the kernel. It cannot correlate across services. It tells you what happened, but not why. And it makes everything slower in the process of telling you.

The Solution

DTrace does not stop anything.

Bryan Cantrill, Mike Shapiro, and Adam Leventhal designed and built DTrace at Sun Microsystems. The original ideation began in the late 1990s. The implementation was first integrated into Solaris in September 2003. It shipped with Solaris 10 in January 2005. Development took approximately four years of focused engineering.

The tagline: "Concise answers to arbitrary questions about the system."

DTrace works by compiling probe scripts, written in a purpose-built language called D, into safe bytecode that is injected directly into the running kernel. When a probe fires, the bytecode executes in kernel context: no context switch, no process stop, no overhead beyond the probe itself.

When a probe is not enabled, the overhead is zero. Not low. Not negligible. Zero. The original machine instruction runs unmodified. The probe point does not exist until you enable it. This is not an optimisation. It is the architecture.

The Language

The D language is deliberately constrained. It cannot allocate memory. It cannot loop infinitely. It cannot dereference invalid pointers. It cannot modify kernel state. It cannot crash the system.

These are not limitations. They are the reason DTrace is safe for production. The language was designed so that any valid D program is guaranteed not to harm the system it observes. Safety is not a runtime check. It is a compile-time invariant.

A DTrace one-liner to count system calls by process on a running FreeBSD server:

dtrace -n 'syscall:::entry { @[execname] = count(); }'

No recompilation. No restart. No risk. The answer appears whilst the system continues serving traffic.

The Proof

In 2006, DTrace won the Wall Street Journal's Technology Innovation Award, Gold. Not for a consumer product. Not for an app. For a kernel instrumentation framework that lets you ask a running operating system what it is doing. One does find that rather encouraging about the state of technology journalism, at least in 2006.

FreeBSD integrated DTrace in 2008. It ships in base. No packages to install, no modules to compile. macOS has included DTrace since Leopard (2007): every Mac ships with a kernel-level dynamic tracing framework that most users will never know exists. illumos, the open-source continuation of OpenSolaris, inherited DTrace from its birthplace.

On Linux, DTrace's influence is unmistakable. eBPF (extended Berkeley Packet Filter) and its higher-level interface bpftrace provide similar capabilities: safe bytecode execution in the kernel, dynamic instrumentation, production-safe tracing. Brendan Gregg, who wrote the definitive DTrace book, now writes the definitive eBPF material. The ideas travelled. The architecture was validated by imitation.

The Philosophy

Cantrill articulated the core insight clearly: "We had created an entirely synthetic system, yet we could not ask ourselves what the software was doing. There was a real lack of observability in the system."

Observability is not logging. Logging records what the developer anticipated might go wrong. Observability answers questions the developer never thought to ask. The difference is the difference between a searchlight and the sun.

DTrace does not require you to predict your questions in advance. You do not instrument your code for DTrace. You do not add tracing libraries. You do not configure exporters. The probes exist at every function boundary, every syscall, every I/O operation. You simply ask.

This is what makes DTrace beautiful: it treats the running system as something that should be fully transparent to the operator. Not partially visible through pre-configured dashboards. Not indirectly observable through aggregated metrics. Directly, completely, safely observable. In production. Under load. Right now.

The Point

Twenty-three years after its first integration, DTrace remains the standard against which production tracing tools are measured. Its core design has not changed because it did not need to. Zero overhead when disabled. Safe by construction. Concise answers to arbitrary questions.

Bryan Cantrill is now CEO of Oxide Computer Company, building rack-scale computers with the same philosophy: the system should be fully observable, fully debuggable, and fully understood. The principle survived the company that created it. One does find that rather beautiful.

DTrace. Ask the system. It will answer.

Read the full article on vivianvoss.net →

By Vivian Voss — System Architect & Software Developer. Follow me on LinkedIn for daily technical writing.

The Philosophy

Vivian Voss — Mon, 06 Apr 2026 06:42:04 +0000

The Unix Way — Episode 11

In 1978, Doug McIlroy, head of the Bell Labs Computing Sciences Research Centre and inventor of the Unix pipe, wrote four sentences that have outlasted every framework, every methodology, and every architectural manifesto published since:

"Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new 'features'."

"Expect the output of every program to become the input to another, as yet unknown, program."

"Design and build software, even operating systems, to be tried early, ideally within weeks."

"Use tools in preference to unskilled help to lighten a programming task."

Eleven episodes into this series, and one does feel it is rather time to explain why it is called what it is called.

The Principles

Do one thing and do it well. grep searches. sort sorts. awk transforms. cut extracts. None of them does the other's job. None of them needs to. Composition replaces integration: small tools, connected by text streams, solving problems their authors never anticipated. Rob Pike and Brian Kernighan wrote in 1984: "The power of a system comes more from the relationships among programs than from the programs themselves." The pipe is not a feature. It is the architecture.

Text as the universal interface. Not JSON. Not Protocol Buffers. Not YAML with its implicit type coercion. Plain text, readable by humans and machines alike, piped from one process to the next. Peter Salus summarised McIlroy's philosophy in three lines: "Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface." A format that has not required a version number in fifty years.

Silence is golden. A successful command returns nothing. Only failure speaks. The exit code carries the meaning: zero for success, non-zero for failure. In a world of verbose logging frameworks, notification services, and dashboard metrics for metrics, one does rather miss the quiet confidence of $? = 0.

The filesystem as organisational principle. Directories for structure, files for content, paths for relationships, permissions for access control. No schema registry. No configuration server. The hierarchy was always there. chmod predates your IAM provider by four decades. Last week's By Design episode on CSV made the case: the filesystem handles hierarchy so the format does not need to.

Fail noisily. Eric Raymond's Rule of Repair: when you must fail, fail in a way that is easy to diagnose. A Unix tool that encounters an error writes to stderr and exits with a non-zero code. It does not return HTTP 200 with an error buried in a JSON body. It does not silently swallow exceptions. It fails, clearly, and lets the operator decide what to do. Quite the contrast to modern systems where "everything is fine" is the default response to catastrophe.

The Proof

FreeBSD embodies these principles in its architecture more faithfully than perhaps any other operating system in active development. The base system is one coherent unit: kernel, userland, and documentation, all developed together in a single source tree. Third-party software lives strictly in /usr/local/, with its own /usr/local/etc/ for configuration. The separation is clean and inviolable. Each service is a small shell script in rc.d, using shared functions from rc.subr, inspectable and composable. One package system: pkg. One ports tree. One source of truth.

FreeBSD's man pages are written in mdoc(7) semantic markup and are widely regarded as among the most comprehensive and well-maintained documentation in the industry. Documentation is not an afterthought. It is a first-class artefact.

macOS has been certified UNIX by The Open Group since October 2007 (Mac OS X 10.5 Leopard). Every Mac ships with a POSIX-compliant userland: sh, grep, awk, sed, make, and the rest. macOS 26 Tahoe holds current certification for both Intel and Apple Silicon. The tools are there. Most users never open Terminal. One does wonder what they think the machine is for.

OpenBSD took "do one thing well" to its logical extreme: build the most secure general-purpose operating system ever built, because it refused to do anything else first. Two remote holes in the default install in nearly thirty years. That is not a feature list. That is a philosophy applied with discipline.

NetBSD applies the same principles to portability: one codebase running on over sixty hardware architectures, from mainframes to toasters. The code is clean enough to port because the abstractions are honest enough to adapt.

illumos, the open-source continuation of Sun's OpenSolaris, carries the SVR4 Unix heritage forward. ZFS, DTrace, and Zones were born here: filesystem integrity, dynamic tracing, and operating system virtualisation, each doing one thing, each doing it well.

The Departure

Linux is not UNIX. It is Unix-like. The Linux kernel itself is remarkably well-disciplined: Linus Torvalds' rule, "We don't break userspace," has held for decades. The kernel syscall interface is stable, predictable, and trustworthy.

But above the kernel, many distributions are quietly walking away from the principles that made Unix what it is.

systemd has absorbed init, logging (journald), DNS resolution (resolved), NTP (timesyncd), network configuration (networkd), device management (udevd), and an EFI boot manager into a single project. Its creator, Lennart Poettering, openly states: "systemd's sources do not contain a single line of code originating from original UNIX." He describes this as evolution. Others describe it differently.

Logging moved from plain text in /var/log/ to binary journals. One cannot grep a binary file. One cannot tail it. One cannot pipe it to awk. The universal interface was replaced with a proprietary query tool. Users report 40x slower query performance compared to grep on plain text. One does rather find that instructive.

Package management fragmented: apt, snap, flatpak, pip, npm, all managing overlapping software on the same machine. Compare FreeBSD: one package system, one ports tree, one source of truth.

John Goerzen, a long-time Debian developer, put it plainly: "I used to be able to say Linux was clean, logical, well put-together, and organised. I cannot really say this anymore."

The Devuan project forked Debian entirely to preserve "Init Freedom" without systemd entanglement. They described systemd as "an intrusive monolith." Rather un-Unix, that.

The Point

The most reliable systems in computing follow these principles. SQLite: one file, one job, backwards-compatible since 2004. CSV: one format, no opinion, no runtime. ZFS: one filesystem, complete integrity, snapshots and checksums built in. cron: one scheduler, fifty years, never broke.

The philosophy is not nostalgia. It is engineering. Small, composable, transparent, silent when correct, loud when broken. The systems that follow it outlast the systems that ignore it. Every time.

Rob Pike's Rule 5: "Data dominates. If you have chosen the right data structures and organised things well, the algorithms will almost always be self-evident."

Dennis Ritchie said it best: "UNIX is basically a simple operating system, but you have to be a genius to understand the simplicity."

Eleven episodes in. This is why.

Read the full article on vivianvoss.net →

By Vivian Voss — System Architect & Software Developer. Follow me on LinkedIn for daily technical writing.

Why We Teach Tools Instead of Foundations

Vivian Voss — Sun, 05 Apr 2026 07:03:43 +0000

On Second Thought — Episode 02

In the 1990s, a computer science degree taught you C, operating systems, compiler theory, networking, and formal languages. You graduated understanding how a machine works from transistor to process. The tools were thin, the understanding was deep, and nobody asked whether you had a Kubernetes certification because Kubernetes did not exist and neither did the problem it solves.

In 2026, a computer science degree teaches you React. AWS has embedded its certifications into four-year university programmes at Western Governors University and Purdue University Global. The Kubernetes certification saw 250,000 enrolments last year, growing 49% annually. The coding bootcamp market is worth $2.65 billion and projected to reach $14 billion by 2032. React dominates bootcamp curricula for the fourth consecutive year, with popularity up 116% versus 2023.

Alan Kay said it in 2004: "Most undergraduate degrees in computer science these days are basically Java vocational training." Twenty-two years later, one merely needs to replace "Java" with "React." The observation has aged rather better than the curricula.

The Axiom

Learn the tool, learn the craft. The industry treats proficiency with a framework as evidence of engineering competence. A Kubernetes certification proves you can operate Kubernetes. It does not prove you understand what a container actually is: a process in a namespace with resource limits, running on a kernel that was doing isolation long before Docker wrote a CLI for it.

The certification market is worth $76 billion and growing. Cybersecurity certifications alone: $3.9 billion. The CKA (Certified Kubernetes Administrator) represents 54% of Kubernetes-related job postings. The credential has become the proxy for competence. The proxy, however, measures familiarity with an interface, not understanding of a system.

The Origin

Universities once taught foundations because there were no tools to teach. When the only languages were C and Lisp, you had no choice but to understand memory, pointers, and recursion. The tools were thin. The understanding was mandatory.

Then the abstraction layers arrived. And with them, the vendors. AWS Academy operates in 6,800 institutions across 120 countries, with a $100 million commitment to education equity. Microsoft offers $100 per year in Azure credits per student. Google partners with 317 universities through Internet2. The curriculum follows the sponsorship. Not because universities are corrupt, but because funding shapes focus, and focus shapes graduates.

Operating systems, compiler theory, and networking have moved from required to elective in many programmes. Web development, cloud computing, and AI/ML have taken their place. The shift is not conspiratorial. It is economic. Graduates who know React get hired faster than graduates who know how a compiler works. The market rewards the tool, not the understanding.

One does not blame the universities. One merely notes that it is rather difficult to teach fundamentals when the funding comes from companies selling abstractions.

The Cost

MIT, one of the finest computer science programmes on earth, launched "The Missing Semester" in 2020. A course teaching shell tools, version control, text editors, and command-line automation. The course exists because MIT observed that its own students "lack knowledge of tools available to them" and "often perform repetitive tasks by hand." The standard CS curriculum, they wrote, "is missing critical topics about the computing ecosystem."

If MIT's students cannot use grep, one does wonder about the rest.

The abstraction ceiling is real and well-documented. Developers who use Docker cannot explain namespaces and cgroups. Developers who deploy to Kubernetes cannot configure iptables. Developers who write React cannot explain what the browser does with their code after the build step. Docker, as one engineer put it, "feels like magic until your container gets OOMKilled or you can't reach a port you swore was open. Then you realise you aren't running a mini-virtual machine; you're just running a process in a very fancy cage."

When the abstraction breaks, and it always does, they have no layer beneath to fall back to. The foundation was never taught. It was skipped. And skipping foundations does not save time. It borrows it, at interest.

HackerRank's 2025 report found that only 22% of developers are given time for learning and upskilling. 48% must find the time themselves. Employers are "unsure early-career developers can code without heavy AI assistance." The abstraction ceiling is not just a technical problem. It is becoming a hiring problem.

The Question

Dijkstra wrote: "Computer science is no more about computers than astronomy is about telescopes." He meant that the discipline is about thought, not machinery. One does wonder what he would make of a curriculum built entirely around the telescopes.

He also wrote: "Universities should not be afraid of teaching radical novelties; on the contrary, it is their calling to welcome the opportunity to do so. Their willingness to do so is our main safeguard against dictatorships, be they of the proletariat, of the scientific establishment, or of the corporate elite."

One does note the phrase "corporate elite" with a certain quiet interest.

What if we taught the protocol before the framework? The syscall before the container? The language before the library? What if proficiency meant understanding what happens beneath the abstraction, not merely operating the abstraction itself?

250,000 Kubernetes certifications were issued last year. One does wonder how many of them could explain what a process is.

Read the full article on vivianvoss.net →

By Vivian Voss — System Architect & Software Developer. Follow me on LinkedIn for daily technical writing.

AI Code Generation: The Hallucination Tax

Vivian Voss — Sun, 05 Apr 2026 06:20:50 +0000

Performance-Fresser — Episode 20

"AI will write your code! 55% faster! Ship in half the time!"

METR ran a randomised controlled trial. Sixteen experienced developers, 246 tasks, mature codebases averaging one million lines of code. Result: developers using AI were 19% slower. Not faster. Slower.

The developers themselves believed they were 20% faster. They were not. One does admire the confidence.

The Hallucination

19.6% of AI-recommended packages do not exist. Nearly one in five imports point to packages that were never published. 43% of those hallucinated packages reappear consistently across re-queries. The AI does not guess randomly. It hallucinates with conviction, and it hallucinates the same things repeatedly.

This is not an edge case. Across 756,000 code samples and 16 models, the pattern is remarkably consistent. Attackers have noticed, naturally. "Slopsquatting" registers packages matching AI-hallucinated names on npm and PyPI, turning the model's confidence into a supply chain attack vector. Rather entrepreneurial of them.

40% of GitHub Copilot's generated code contains security vulnerabilities (NYU, 89 scenarios, 1,692 programs). Developers with AI access write significantly less secure code than those without, whilst being considerably more confident that their code is secure. Stanford measured this across 47 developers in Python, JavaScript, and C. The less they questioned the AI, the more vulnerabilities they introduced. One does wonder whether "confidence" and "competence" have always been this loosely coupled.

The Complexity Tax

Here is what the benchmarks reveal but the marketing rather conveniently omits: AI performs measurably worse on complex, abstracted code. Framework-specific conventions, proprietary APIs, deep dependency chains: these are the contexts where hallucination rates climb. The more you ask the model to navigate, the more creatively it invents.

20.41% of code hallucinations stem from incorrect API usage. The more framework-specific the API, the more the model confuses conventions, invents methods that do not exist, and mixes patterns from different versions. Higher Halstead complexity, larger vocabulary, deeper abstraction: all correlate with higher failure rates in LLM-generated code. One might call it poetic justice: the abstractions designed to simplify development are now the abstractions that confuse the tool designed to simplify development.

Vanilla code in the language's standard library produces cleaner AI output. Not because the AI is smarter. Because there is less to hallucinate about. Fewer abstractions, fewer proprietary patterns, fewer opportunities for the model to confidently fabricate something that compiles but does not work.

JavaScript illustrates this rather neatly: 21.3% hallucinated imports versus 15.8% in Python. More packages in the ecosystem means more hallucination surface. The complexity you built for humans to struggle with is now the complexity AI struggles with too. The tax, as one might say, compounds.

The Model

Not all models are equal, and the tooling matters as much as the model behind it.

Copilot autocompletes lines. It predicts the next token based on your current file. An agentic model in a proper development environment reasons about architecture, reads your project structure, navigates across files, and understands context at a system level. The difference is not incremental. It is categorical.

Less than 44% of AI-generated code was accepted in the METR study. The developers spent more time evaluating, adjusting, and discarding suggestions than they would have spent writing the code themselves. The tool that was meant to remove friction became the friction. Quite the achievement.

Choosing the right model for the right task is engineering. Using whatever ships with your IDE is hope. Hope is a marvellous thing. It is not, however, a deployment strategy.

The Code Quality Decline

GitClear analysed 211 million lines of code and measured the impact of AI adoption on code quality:

Refactoring collapsed: from 25% of changed lines (2021) to under 10% (2024)
Code cloning surged: copy-pasted lines rose from 8.3% to 12.3% (a 48% increase)
Code churn doubled: lines reverted or updated within two weeks doubled versus the 2021 baseline

The AI generates code faster. The code gets replaced faster. The net effect on the codebase is not acceleration. It is the accumulation of disposable code that nobody refactors because the AI will cheerfully generate more. One does wonder whether "velocity" was always a euphemism for "volume."

The Pattern

AI code generation is a brilliant instrument. In the hands of someone who understands both the tool and the codebase, it accelerates work that would otherwise be tedious. In the hands of someone working inside a framework they do not fully understand, it amplifies the confusion at machine speed.

The answer is not more AI. It is less complexity. Reduce what the model must carry in context. Write code a human can read in one pass: lean, minimal, close to the language's own idioms. The AI will follow, because there is less to get wrong.

45% of developers say debugging AI code takes longer than writing it themselves. One does suspect this has less to do with the AI and rather more to do with not understanding the framework the AI is writing for. A developer who masters the fundamentals of the language itself reviews AI output in seconds. A developer buried in abstractions cannot review anyone's code, including their own.

61% say AI produces code that "looks correct but is not reliable." The hallucination is not in the model. It is in the expectation that a tool can navigate complexity you have not mastered yourself.

The Trust Erosion

The developer community is noticing. Stack Overflow's 2025 survey (65,000 respondents) tells the story:

84% use or plan to use AI tools (adoption is not the problem)
Only 29% trust AI accuracy (down from 40% the previous year)
Favourability dropped from 72% to 60%
75% still prefer asking a human over trusting AI output (rather telling)
77% say "vibe coding" is not part of their professional work (one does hope)

The industry adopted the tool before it understood the tool. Now the understanding is catching up, and confidence is dropping. Not because AI got worse. Because expectations met reality. One does find that reality has rather poor timing.

The Lever

The solution is not abandoning AI. It is reducing what the AI must navigate.

Write lean code: close to the language's own idioms, minimal abstractions, no framework magic. A function that does one thing, named clearly, understood where it is read. The AI generates better output for this code because there is less to hallucinate about. The human reviews it faster because there is less to misunderstand.

This is not nostalgia. It is architecture for the AI era. The same principles that made code maintainable for humans now make it reviewable when generated by machines.

Write lean. The AI will follow.

Read the full article on vivianvoss.net →

By Vivian Voss — System Architect & Software Developer. Follow me on LinkedIn for daily technical writing.

The Update Treadmill

Vivian Voss — Sat, 04 Apr 2026 07:12:46 +0000

Beta Stories — Episode 06

Your software worked on Friday. You ran npm update on Monday. It no longer works on Monday. Nothing in your code changed. Not a single line. The platform moved underneath you.

One does find it rather charming that the industry calls this "progress."

The Promise

"Ship early, iterate fast, embrace change." Agile taught an entire generation that stability is something to overcome. Sprints replaced milestones. Continuous deployment replaced releases. The word "done" quietly left the building and has not been seen since.

The ecosystem took the hint. Angular ships a new major version every six months. Eighteen major versions in ten years. Each with breaking changes, each with an 18-month support window, each politely informing you that the version you spent three months migrating to is now approaching end of life. You are not building software. You are servicing a subscription to your own framework.

The build tools followed suit. Grunt was the standard in 2013. Gulp replaced it by 2015. Webpack took over by 2017. Now it is Vite, with Webpack satisfaction down to 26%. Four generations of tooling in ten years, each requiring its own configuration language, its own plugin ecosystem, its own mental model. The code you built did not change. The machinery around it was replaced four times.

The Decay

44% of breaking changes in npm packages arrive in minor and patch releases. The versions that semantic versioning promises are safe. They are not. They merely look safe, which is rather worse.

Next.js 15 reversed its caching defaults. Code that relied on cached responses now fetches on every request. Next.js 16 dropped Node.js 18 entirely. One does not recall requesting either of these changes, yet here one is, updating CI pipelines on a Tuesday afternoon.

The migrations that do get announced fare little better. Python 2 to 3 took twelve years. Twelve years for a language migration that was meant to be straightforward. The cost to the industry has been estimated at over $100 billion. AngularJS reached end of life in 2021. Four years later, 1.2 million websites still run it, with 419,000 weekly npm downloads. The migration was not completed. It was abandoned.

Stripe measured the damage in 2018: developers spend 33% of their time on technical debt. 13.5 hours per week per developer on inefficiencies that produce no value. Across the industry, 75% of development time goes to maintenance. Not features. Not innovation. Keeping the treadmill running so that one may continue running on the treadmill. Splendid use of engineering talent.

The Mechanism

Framework authors need adoption. Adoption requires novelty. Novelty requires breaking the previous version. Break, migrate, break, migrate. The cycle is not accidental. It is the product.

Agile provided the philosophy. "Embrace change" became "inflict change." Sprints guarantee a next iteration, never a stable release. The process ensures that software never reaches a finished state, and the vocabulary ensures nobody notices: it is not instability, it is iteration. It is not churn, it is evolution.

And the tooling enforces it. Dependabot opens 200 pull requests per week for a monorepo. Not because your code is broken. Because a dependency four levels deep released a patch for a vulnerability in a function you do not call. The automated pull request arrives, the CI runs, the reviewer spends twenty minutes verifying that nothing changed, and the treadmill advances one step. Multiply by fifty projects and you have a full-time job that produces nothing.

The Signal

SQLite has maintained backwards compatibility since 2004. The developers have committed to preserving the current file format, SQL syntax, and C API until the year 2050. That is not a version policy. That is a promise to every system that depends on it.

POSIX has been stable since 1988. Thirty-eight years of API consistency across operating systems, vendors, and hardware architectures. A shell script written in 1990 still runs.

FreeBSD has maintained source compatibility across major releases for over thirty years. Code written for FreeBSD in the 1990s compiles and runs on FreeBSD 14 today. Not because nobody touched the system. Because every change was made with the understanding that other people's software depends on it.

These are not legacy projects. They are proof that stability is a design decision, not a technical limitation. The treadmill is not inevitable. It is profitable.

Your framework updates every six months. Your database file format has not changed in twenty-two years. One of them understood what "production" means.

"Go into frameworks," they said. They did not mention the frame gets replaced every six months whilst you are still hanging in it.

Read the full article on vivianvoss.net →

By Vivian Voss — System Architect & Software Developer. Follow me on LinkedIn for daily technical writing.