DEV Community: Chris Roy

Why Plugin Systems Fail Quietly: Weak Boundaries Explained

Chris Roy — Sun, 09 Aug 2026 10:56:29 +0000

Most plugin systems continue to work just well enough, making them dangerous.

A typical plugin system starts from a reasonable goal of letting people extend functionality without touching the core. The core stays stable. Optional functionality lives at the edges. Responsibility is distributed cleanly, at least on the whiteboard. In the early stages this holds up fine in practice, not just on paper. Plugins are few. Contracts are narrow enough that a plugin author can read the documented interface once and know the whole shape of what they're allowed to touch. A new contributor can look at the system and reason about it locally, without holding the whole thing in their head. Nobody has had to break a promise to anyone yet, but that equilibrium doesn't last, and it rarely announces its own ending.

The failure builds through accommodation: a plugin needs access to a little more state than the contract allows, or another needs to intercept a request earlier in the lifecycle than anyone originally envisioned. Each request, taken on its own, is defensible. Nobody who approved any single one of them was wrong to do it.

Read any one of those changes in a code review and it looks fine. A reviewer sees a small diff, a plausible justification, a deadline attached to it, and approves it. What a reviewer almost never sees is the fifteenth version of that same request, made by a different plugin, for a different reason, landing on a boundary that's already been quietly widened fourteen times before. Each decision is local. The damage is cumulative, and cumulative damage is exactly the kind that doesn't show up in any single code review.

But the plugin system stops being an extension mechanism somewhere in that process. It becomes a second system running alongside the first, wired to it at dozens of points nobody wrote down.

The erosion of boundaries

A plugin system only works if there's a hard line between what the core owns and what plugins are allowed to touch. Almost every real system compromises that line early, because compromising it is the path of least resistance. Internal data structures get exposed. Lifecycle hooks get added for one plugin's sake. Mutable globals stay reachable because locking them down would mean telling three plugin authors their code is about to break. The justification is always some version of "trust the plugin author to use it responsibly."

Trust doesn't scale the way access does.

Once a plugin depends on an internal detail, that detail stops being internal, whatever the documentation says. It becomes a contract by accident, and accidental contracts are the ones nobody can safely break, because nobody agreed to them on purpose in the first place.

Firefox lived this for over a decade. Its original extension model, built on XUL and XPCOM, gave add-ons direct access to the browser's internal component model, i.e., the same libraries that handled networking, rendering, and security, wide open to any extension that wanted them. It made Firefox's extension ecosystem one of the richest on the web for years. It also meant that by the time Mozilla needed to rebuild Firefox around a multiprocess architecture, in pursuit of the same stability and performance gains Chrome had already shipped, legacy XUL extensions were incompatible with it by construction. They reached into places a multiprocess browser couldn't safely expose anymore. Mozilla's own reasoning was direct: reliance on XUL and XPCOM had become an obstacle to work the browser needed to do, specifically the Electrolysis multiprocess project and the Servo rendering engine, both of which required a much cleaner separation between the browser's internals and whatever code an extension was running. You can't safely split a browser across multiple processes if any extension might be reaching directly into a data structure that now lives in a different process entirely. The fix arrived as a new extension API, WebExtensions, deliberately narrower and modeled closely on Chrome's, paired with a hard cutoff: as of Firefox 57 in November 2017, the old add-ons simply stopped loading. Years of accumulated extensions broke at once, because the contract those extensions depended on had never really been a contract to begin with — just access nobody had gotten around to closing off, and closing it off years later took a browser-wide breaking release to do what a narrower API would have made unnecessary from the start.

This pattern isn't unique to browsers. The break, when it finally comes, doesn't look proportional to what caused it. It looks catastrophic, because years of small, reasonable accommodations were never priced as debt until the bill came due all at once.

Shared responsibility, no ownership

In a plugin system that's still healthy, you can always answer one question cleanly: who owns this invariant. Either the core enforces it, or the invariant doesn't really exist. Once a system starts failing, that question stops having a clean answer. Invariants get enforced collectively, informally, by convention. One plugin assumes another has already run. The core assumes plugins will clean up their own state. Execution order becomes "load-bearing," to borrow the phrase everyone reaches for these days, and nobody wrote it down because nobody designed it on purpose.

At that point behavior isn't really being designed anymore. It's being negotiated, extension by extension, mostly by accident.

Atom's package system got there. Packages ran with full access to the DOM and to Node's runtime, which was also the thing that made Atom feel so hackable. You could install a package, and it could reshape almost anything about how the editor behaved, down to intercepting keystrokes before the editor's own core ever saw them. That same openness meant packages could and did collide in ways nobody upstream could fully predict, each one reasonable in isolation, none of them designed against each other. Two packages might both assume they were the first to touch a given buffer. Neither was wrong on its own. Together, loaded in whatever order the user happened to install them, they'd produce a bug that only existed because of the combination, and no one owned the combination.

GitHub announced Atom's sunset in June 2022, archiving the project that December. The stated reason was that community involvement and feature development had stalled, which is true, but it's also true that the editor Microsoft chose to keep investing in instead, Visual Studio Code, had been built from day one on a fundamentally more restrained model for what an extension was allowed to do. Atom's own performance problems, widely attributed at the time to its Electron foundation, were at least partly a symptom of the same openness. An editor that lets every package touch everything has a much harder time optimizing anything, because it can never be sure what depends on the thing it wants to change.

Debugging a plugin-heavy system is miserable for exactly this reason. Failures don't map to a single component. They emerge from interaction, from two pieces of code that were each individually correct and jointly wrong. Removing one plugin "fixes" the bug, and nobody on the team can explain why beyond a shrug. The system still runs. Nobody quite trusts it anymore.

Power dressed up as flexibility

A lot of plugin systems expose power where they should be exposing interfaces. Instead of a narrow, enforceable contract, they hand out a hook into some broad execution context and call it flexibility. It is flexible. It's also the reason the core can no longer promise anything about what happens after a plugin runs. When a plugin can do anything, the core can guarantee nothing, and that gets mistaken for resilience instead of what it actually is.

ActiveX is the clean historical case. Introduced by Microsoft in 1996, it let a web page embed a control that ran with essentially the same privileges as any other program on the machine, no sandbox, no meaningfully scoped permission model. That an ActiveX control could do anything a desktop application could do, right inside the browser, was the appeal. It was also, as one retrospective later put it bluntly, the web's biggest mistake. A long, ugly stream of vulnerabilities followed, most of them boiling down to the same root cause where a hostile web page could get an ActiveX control to run arbitrary code as the user, no exploit chain required beyond convincing someone to visit the wrong site. Microsoft spent years bolting on warnings and kill bits before finally disabling ActiveX by default across Office and Microsoft 365 in 2024, nearly three decades after it shipped, rather than ever retrofitting real boundaries into a system built without any.

ActiveX wasn't alone in this. The broader family of browser plugin technologies built around NPAPI, the interface that let Flash, Java applets, and Silverlight run inside a browser tab, made a similar bet: give the plugin a wide-open door into the host process in exchange for whatever capability the plugin wanted to offer. Chrome and Firefox spent the 2010s walking that bet back, first with sandboxing, then with click-to-play defaults, then by dropping NPAPI support entirely. Chrome removed it in 2015, Firefox followed in 2017. The technology wasn't uniquely badly built; it ran on the same premise as ActiveX, that broad access is a reasonable price for broad capability. Browsers eventually decided it wasn't, and every plugin depending on that access had to be rebuilt or abandoned to match.

The pattern repeats in less dramatic form constantly, inside systems that never make headlines for it. Defensive code creeps into the core because nobody trusts what a plugin might have already touched. Validation gets duplicated at every boundary because there's no boundary anyone's confident is real. The system gets harder to reason about, whether or not the underlying problem actually calls for it, because the extension mechanism made that complexity permissible instead of impossible.

This never arrives as a single moment. There's no line the system visibly crosses. It just gets slowly harder to reason about, harder to test, harder to change, until the honest answer to "where does this system end" is that nobody's sure anymore.

The systems that age better, and why

Some plugin systems avoid all three failures for a long time, and they don't do it by being smarter about accommodation. They do it by refusing more of it.

Eclipse is one of the oldest working examples. Its plugin model, formalized on top of OSGi, is built around what Eclipse's own architecture documentation calls the plug-in contract, a small, explicit set of extension points, each one declared up front, each one owned by whichever plugin defines it. A plugin can't reach into another plugin's internals just because both happen to be loaded in the same process. It has to go through a declared extension point or it doesn't get in at all. OSGi's bundle model reinforces this at the runtime level, with each bundle getting its own classloader and explicit version boundaries between dependencies. That discipline is a large part of why OSGi has stayed a working foundation for enterprise software for close to two decades, hot-swapping components in running systems without the whole platform destabilizing every time somebody ships an update.

Visual Studio Code took a related but distinct approach to the same discipline. Instead of trusting extensions with the same process as the editor, it doesn't give them the choice. Every extension runs in a separate process, the extension host, talking to the main application only through a defined API surface and an RPC layer. An extension can be slow, buggy, even hostile, and the editor's own responsiveness is largely insulated from it, because the extension was never given the kind of direct access that would let it take the rest of the application down with it.

Aside: I don't like VSCode personally. I like what Zed is doing and it is a major personal goal to one day create the best and greatest IDE. For now though, that seems a few years away.

That's the same problem Atom's package model ran into, solved by refusing the access in the first place instead of hoping plugin authors would behave. Where Eclipse enforces its boundary through a declared contract that extensions have to go through, VS Code enforces a version of the same idea through a process boundary that extensions can't get around even if they wanted to. Different mechanism, same underlying refusal.

Eclipse and VS Code both made their refusal at the architecture level, before any extension author ever showed up wanting more. Chrome's Manifest V3 shows what the same refusal looks like made in public, years into an established ecosystem, against developers who had built real businesses on the access being taken away. Chrome's older extension API let extensions intercept and modify every network request a browser made, through the webRequest API, which is exactly the kind of broad, ambient power that makes an extension platform expressive and also exactly the kind of power that makes it impossible to reason about what any given extension might be doing to your traffic. Google's replacement, declarativeNetRequest, asks extensions to declare their filtering rules up front instead of intercepting requests live, which is a real reduction in what an extension can express. Ad blocker developers noticed immediately. Raymond Hill, who maintains uBlock Origin, warned as early as January 2019 that the change threatened to break his tool's core functionality, and the backlash from extension developers has continued through every stage of the rollout since. Google shipped the restriction anyway. The ecosystem got smaller and louder about it in roughly equal measure, and Chrome's extension platform got more predictable in exchange, which was the actual trade being made.

Every one of these systems allows fewer extension points than the ones that failed: narrower contracts, a core that stays authoritative over its own invariants no matter how reasonable the request to bend them sounds. When something can't be expressed cleanly through the declared contract, it gets rejected, or it gets folded into the core through real design work instead of a workaround. These systems get criticized constantly for being inflexible, for having smaller ecosystems, for saying no to things a more permissive competitor would say yes to. But it's also why, years later, you can still point to who's responsible when something breaks, and why a change to the system is something someone decided rather than something that just happened.

What actually separates these systems is their tolerance for ambiguity, held consistently even when holding it costs adoption.

A plugin system that's built to last is conservative on purpose. It doesn't try to be extensible for every use case someone can imagine. It picks a small number of extension points and defends them, and it would rather refuse a plugin author's request than accommodate one more exception it can't fully account for, even when the refusal shrinks the ecosystem. Every one of the systems above paid a real, visible cost for that discipline: Eclipse's plugin authors write against a stricter model than a looser one would demand, VS Code extensions can't reach into the editor's internals the way an Atom package once could, and Chrome's ad blockers lost capability they'd relied on for years. None of that cost showed up as a headline the way a launch does. It showed up as a smaller, slower-growing ecosystem that people complained about constantly and kept building on anyway, because it kept working the way it was supposed to.

The alternative is rarely dramatic. It's a system that keeps technically functioning while getting harder to change and harder to trust, until the day someone tries to fix that and discovers the ecosystem depending on the mess is now too large to break without taking the whole thing down with it. By then it's usually too late to close the boundary without breaking everything built on top.

The system keeps running. It just stops moving, one deferred refusal at a time, until moving it again costs more than anyone's willing to pay.

This is the ordinary, predictable outcome of extension without boundaries, arrived at one reasonable accommodation at a time, by people who were right about every single decision except the sum of all of them.

Sources

"Deprecating XUL for WebExtensions," LWN.net.
"Helping You Find Compatible Extensions," Mozilla Add-ons Community Blog, October 2017.
"GitHub's Atom Text Editor Will Be Retired in December," gHacks Tech News, June 2022.
"Remembering ActiveX Controls, the Web's Biggest Mistake," How-To Geek.
"Microsoft Internet Explorer ActiveX Vulnerability," CISA, 2006.
"Microsoft Office 2024 to disable ActiveX controls by default," BleepingComputer, 2024.
"Notes on the Eclipse Plug-in Architecture," Eclipse Foundation.
"What Is OSGi?," OSGi Alliance / Eclipse Foundation.
"Migrating VS Code to Process Sandboxing," Visual Studio Code Blog, November 2022.
"Chrome Extension Manifest V3 May Break uBlock Origin Content Blocker," BleepingComputer, January 2019.
"Mozilla to Drop Support for All NPAPI Plugins in Firefox 52, Except Flash," BleepingComputer, 2017.

Systems Thinking: How Restraint Prevents Complexity

Chris Roy — Sun, 09 Aug 2026 10:55:03 +0000

Most system failures trace back to a constraint nobody wrote down. By the time something breaks, the architecture is usually still fine. What gave way is an assumption nobody agreed to keep.

Early on, most systems are tractable. The architecture is coherent, the abstractions are few, and the number of invariants in play is small enough that one person can hold all of them in their head at once. At this stage the system rewards local reasoning. You can change a component and predict the effect of that change without opening five other files to check what else might notice.

Formally, the system's behavior can still be approximated as:

S \approx i = 1 \sum n C_{i}

where each component $C_{i}$ contributes independently, and interactions between components stay limited and explicit. Read a function, understand its inputs and outputs, and you understand what it does to the system. Nothing else needs checking.

That property doesn't survive growth, and it rarely gets killed off by recklessness. It erodes through accommodation. A new requirement adds a conditional path to a function that used to have one job. An exception bypasses an invariant because enforcing it that week would have blown a deadline. A shortcut defers a constraint under pressure, with every intention of coming back to fix it later, and the fix never gets scheduled because nothing visibly broke. Each of these changes looks locally defensible at the moment it's made. The system quietly stops being additive, one reasonable decision at a time, without anyone ever sitting down and deciding to compromise the architecture.

Interaction terms accumulate:

S = i = 1 \sum n C_{i} + i \neq = j \sum I_{ij}

Once interactions grow faster than components, local reasoning stops working. The second term takes over. Behavior starts to look emergent instead of designed, and the instinct is usually to blame the people writing the code, but that's rarely where the fault lies. Coupled systems behave this way as a structural matter. It happens on well-run teams with disciplined engineers just as often as it happens anywhere else, because the mathematics doesn't care how careful anyone was being.

Here's what that looks like on the ground. A validation function gets written for one form, on one page. Six months later it's being called from four other places, because it already does most of what those callers need and writing a second function feels wasteful. None of the four callers needs quite the same validation, so each one adds a flag, a special case, an early return for its own use. The function now has to know about four different contexts to do its job in any of them. A helper function turns into a load-bearing wall a decision at a time, because reuse looked free every single time someone reached for it, long before anyone consciously chose to build it that way.

The four callers didn't need a shared function. They needed four small functions that happened to overlap in places, with the overlap left alone rather than extracted. Writing four functions feels like waste in the moment, measured against the immediate task. Each of those four could change independently, break independently, and get deleted independently when its caller goes away. The shared version can do none of those things without a meeting first, and the meeting rarely happens until the function is already too tangled to split back apart.

Reuse is one of the few instincts in software engineering that's taught early and rewarded in code review, usually before anyone's had the chance to watch it fail. Writing the same ten lines twice doesn't get anyone promoted, but spotting the duplication and collapsing it into one place does. The complimented version is the one that, three years later, four teams are afraid to touch, and by then nobody remembers who to ask about why it's shaped the way it is.

Restraint, in this context, is what keeps that growth in check: refusing to add an abstraction before there's evidence it earns its keep, refusing to generalize a function before two separate call sites need it to be general, refusing to promise behavior the system can't keep enforcing five years from now, after everyone who understood the original tradeoff has moved to a different team.

Boundaries are how restraint gets enforced day to day, rather than argued about in a design review once and then forgotten. A boundary defines a contract. It names the interactions that are allowed and excludes everything else by default. It enforces an invariant at the interface, at one point, instead of relying on the discipline of every caller who happens to touch that code later, most of whom will never read the comment explaining why the invariant exists. When a boundary holds, interaction terms stay contained near it. When it doesn't, complexity diffuses outward to wherever the boundary used to be, and it rarely announces that it's doing so.

Of the two, a porous interface is the one that actually hurts you. A module with no interface at least advertises its own recklessness; everyone touching it knows they're on their own. A module with a leaky one looks safe from the outside while failing, without any noise, to protect anything. Once internal details leak across a boundary, that boundary is a boundary in name only. The invariant it was supposed to protect is no longer enforced in one place. It's now distributed, implicitly, across every caller that has come to depend on the leak, usually without anyone on the team keeping a list of who's depending on what.

Picture an object that exposes its internal list instead of a controlled accessor, because returning the list directly was three lines shorter than writing a proper method. For a while, nothing goes wrong. Then a caller three modules away starts mutating that list directly, because it's right there and it works. A year later, someone tries to change how that internal list is stored, maybe to fix a performance problem, and discovers the change breaks code in a part of the system they've never had reason to open. The boundary failed silently, letting something invisible grow behind it.

At that point the cost of a small change stops being small. A one-line fix requires tracing every caller that might be relying on the old behavior, half of which were never documented as dependents in the first place. Understanding gives way to fear. Teams stop touching that part of the code not because they're lazy, but because nobody can say with confidence what else is wired to it, and finding out is expensive enough that it's easier to build around the problem than through it.

Fred Brooks made a version of this argument decades ago in The Mythical Man-Month: conceptual integrity erodes gradually, one reasonable addition at a time, not in a single catastrophic decision that anyone could have vetoed. John Ousterhout makes a related point in A Philosophy of Software Design. A shallow module, one whose interface is nearly as complicated as its implementation, doesn't reduce complexity. It relocates it onto whoever has to call it, and hides the relocation behind the appearance of a clean function signature.

Ousterhout's counterpart to the shallow module is the deep one: a simple interface sitting in front of substantial internal machinery, where the caller gets a narrow surface to learn and the module absorbs the complexity rather than exporting it. A file system is the example he reaches for. Opening, reading, and writing a file is a handful of operations to learn, and underneath that handful sits allocation, caching, and recovery from partial writes. None of it is the caller's problem. Most interfaces people call "clean" are shallow in this sense, and most people can't tell the difference until they're the one stuck maintaining what's on the other side of it.

People sometimes describe systems thinking as the ability to see everything at once, to hold the whole system in view like a diagram pinned to a wall. In practice it usually looks like the opposite: arranging things so that most components need to know as little as possible about the rest of the system. Friction, under this view, isn't always a defect to be engineered away. Sometimes it's load-bearing. Duplicating ten lines of logic across two modules can be cheaper, over the life of the system, than sharing one function that quietly couples them together for years.

There's an old, unglamorous illustration of this in how Unix pipes were designed: small programs, each doing one narrow thing, connected through a plain text interface that none of them needed to understand beyond "read a stream, write a stream." No program in that chain needs to know how the others are implemented. The boundary between them is the text stream itself, and it's inconvenient in exactly the ways that keep it honest: no shared memory, no shared state, no shortcut for one program to reach into another's internals because it's under a deadline.

Not everything from that era aged the same way. The shells wrapping those pipes have accumulated more flags than most people using them could name, and plenty of small Unix tools grew their own tangle of special cases the moment someone needed one badly enough. The pipe itself mostly avoided that fate, though it's worth admitting that some of the credit belongs to the text stream being too dumb to extend rather than to anyone's restraint.

Formally, this is about preserving locality. A system that preserves locality lets reasoning scale roughly linearly with size. Doubling the codebase doubles the amount you need to understand to make a safe change, not the odds that the change ripples somewhere unrelated and unrelated-looking. A system that loses locality forces reasoning to become global well before it doubles in size, and past that point it stops scaling in any useful sense, no matter how much hardware or headcount gets thrown at the problem afterward. More engineers on a system with no locality doesn't parallelize the work. It just adds more people who each need the same global context before they can safely touch anything.

Ben Moseley and Peter Marks make a version of this point precisely in Out of the Tar Pit. Their argument is that essential complexity, the complexity a system can't avoid because it's inherent to the problem being solved, is usually smaller than it looks from inside a struggling codebase. Most of what makes large systems hard to work with is accidental complexity, generated by uncontrolled state and interaction rather than by the actual size of the problem. Unconstrained interaction, not raw scale, is what teams end up fighting for years without naming it that way.

Restraint applies to power as much as it applies to structure. A system that grants unrestricted access in the name of flexibility has quietly given up the guarantees that flexibility was supposed to make possible. If every operation is technically reachable from anywhere, no particular behavior can be relied on anywhere else in the system. Consider a shared configuration object that any part of an application can read and write at any time, because passing it around explicitly felt like extra ceremony. Early on this looks like convenience. Later, a bug shows up where a setting changes mid-request for reasons nobody can trace, because six different subsystems all had write access and none of them coordinated with the others. The flexibility that made the object convenient to build with is the same flexibility that makes it impossible to debug.

Fixing that particular problem rarely means adding more logging or more code review. It means taking write access away from five of the six subsystems and forcing them to ask the sixth one for a change instead of making it themselves, which is a smaller ask than it sounds like until you're the one on the subsystem that lost its shortcut.

A component can only be given real freedom to change its own internals once the rest of the system knows exactly what it will and won't do at its boundary. Freedom without a known boundary reads as flexibility right up until someone needs to rely on it, at which point it's just risk nobody priced in.

Trust, inside a codebase, tends to come from limits rather than from capability. A component that's explicit about what it will do, what it refuses to do, and the conditions under which it fails loudly instead of silently gives engineers something solid to build on top of. It lets a team revisit an old decision months later without treating the whole surrounding area as radioactive, the way people learn to avoid the one file nobody fully understands anymore, the one everyone routes around instead of opening.

The same pattern shows up outside software entirely. Charles Perrow's Normal Accidents looks at failures across nuclear plants, chemical refineries, and aviation, and finds that catastrophic failure in tightly coupled systems is usually a property of how densely the parts interact, not a property of any single part failing on its own. In systems that are already tightly coupled, adding more flexibility, more cross-cutting paths, more ways for one part to reach into another under unusual conditions, tends to increase risk rather than reduce it. This holds even when every individual addition looked, at the time, like it was making the system more capable or more resilient to some specific failure someone had just seen happen.

Time tests a design more honestly than any review ever will. Systems built to anticipate every future requirement tend to accumulate complexity they never use, carrying the weight of speculative features on the chance someone eventually asks for them. Manny Lehman's Laws of Software Evolution get at part of why: a system that keeps growing without a disciplined structure to contain that growth gets progressively harder to modify, on a schedule that tracks the system rather than whoever happens to be maintaining it that quarter. Good engineers can slow the decline, but none of them seem to stop it outright.

This isn't a case for minimalism as a style choice — for stripping a system down because a spare interface looks better in a diagram someone will show at a conference. Past a certain size, discipline at the boundary stops being optional, whether or not anyone on the team decided to call it that.

The systems worth building don't try to absorb every kind of complexity a future requirement might eventually throw at them. They constrain what they'll take on, and they say no to the rest, explicitly and early, before saying no becomes politically impossible because too much has already been promised on top of it. Whatever lets them survive past that point has less to do with cleverness in the implementation, and more to do with how honestly the boundaries were scoped in the first place, and whether anyone kept enforcing that scope after the person who drew it had already left the room.

References

Programming Language Design: Why Decisions Get Forgotten

Chris Roy — Sun, 09 Aug 2026 10:55:02 +0000

Most programming languages are remembered for their syntax. Almost none are remembered for the decisions that produced it.

Every keyword exists because someone decided one approach was better than another, usually after weeks of debate. Writing it took an afternoon. The reasoning behind it lasted about as long as it took everyone to move on.

I've been building BigTalk from the ground up: memory model, effects, regions, verification. I expected the hard part to be parsing and semantics. Instead I kept running into the same quieter problem.

Version control gives us perfect recall, but imperfect memory. Any change can be inspected, compared, blamed. You can find the line that introduced a bug. What git will not tell you is why that line was written that way instead of three other ways that were discussed and discarded, or never written down because they seemed obvious at the time.

Those conversations live in chat threads, calls, whiteboard photos nobody keeps. They are surprisingly hard to recover exactly when you need them most.

Designing a language makes this hard to ignore.

A language is little more than decisions wearing syntax.

Adding a keyword is easy, and rewriting a parser is easy enough. Choosing the keyword is the part that stays difficult after the merge.

Should this feature exist at all? Does explicitness win over convenience here? An inconsistency might have to stay, too, if fixing it breaks something that was never supposed to depend on it.

Those questions don't have single correct answers. They have a context, and context is usually the first thing a project loses. Without it, the next argument is often the old argument again, held by people who do not know it already happened.

There's a habit of treating architecture as structure, as if the diagram were the thing. In practice, architecture is also memory. A system's quality depends not only on the decisions inside it, but on whether it can still explain those decisions to whoever finds them later.

This matters more the longer a project is meant to live. The people deciding today are unlikely to be the people maintaining it in ten years. Even if they are the same people, memory is a poor dependency.

An undocumented decision eventually becomes indistinguishable from an arbitrary one. Nobody trusts an arbitrary decision, so they change it. Then the problem it was quietly solving comes back under a different name.

RFCs get treated as a large-project affectation. Something Rust does. Something a standards body does. Overkill for anything smaller. The value has never had much to do with size.

An RFC captures intent before that intent gets flattened into a diff. A useful one records the problem, the constraints, the paths not taken, and why those paths lost. Implementation detail can live in the code. The logic the implementation is downstream of usually cannot.

Writing RFCs for BigTalk was never the friction. Managing a growing pile of them consistently was: assigning numbers, tracking status, keeping the fortieth document under the same conventions as the first. That kind of upkeep never rivaled the difficulty of a type checker, but it made the collection worse every time it slipped, until the workflow itself became the thing I was fighting.

So I built a small CLI to stop fighting it: rfcman. It shipped under the BigTalk name ahead of the compiler. It does not do anything clever. It manages RFC lifecycles and keeps their relationships stable through UUIDs, so a document can change shape without losing the reasoning attached to it.

It exists to remove friction, the same reason a build system exists. Nothing about it was meant to be the interesting part of the project. It just turned out to be the part that let the interesting parts stay honest.

There's a common belief that systems rot because the code gets more complicated. Usually the code stays readable just fine. What disappears is the reasoning that used to hold it together.

Someone finds an odd abstraction, decides it is unnecessary, and removes it. Months later the problem it was preventing comes back, not because anyone was careless, but because the system had forgotten why the abstraction was there.

Software gets evaluated on correctness and performance. Continuity rarely gets the same weight, though it should. A system worth keeping has to preserve why its behavior looks the way it does, not merely what that behavior is.

Source alone doesn't carry that, and neither does documentation once it drifts. What has to survive is the link between decision and implementation: checkable, not merely remembered.

Building a language taught me something I didn't expect: deciding how it should work turned out to be the easy part. What's hard is making sure that in ten years, someone can still find out why.

Software Longevity: How to Design Systems That Age Well

Chris Roy — Sun, 09 Aug 2026 10:43:55 +0000

Correctness is a low bar. Most systems that reach production clear it. They compile. They pass tests. They satisfy the requirements as they were understood at the time. Plenty of them still turn difficult to touch within a few years, because nobody treated time as a design constraint while there was still time to.

Longevity is the compounding effect of conservative choices made early and upheld consistently — what a system promises, and what it deliberately refuses to promise, with every trade-off pushed into the open instead of left implicit.

Every system eventually gets read more than it gets written, by engineers who weren't in the room for the original decisions, who don't share the same assumptions, and who are working under different constraints entirely. That future reader is the software's real audience. Code that leans on shared context, implicit knowledge, or historical memory asks that reader for a trust it hasn't earned.

Many systems start failing right here. They lean on explanations that live outside the code, and on invariants and abstractions that were never given a boundary anyone wrote down. Such a system can run for years and still never age gracefully. Every change turns into a negotiation with its own past.

Embarrassment is a useful metric for catching this early. A system becomes embarrassing to revisit once its behavior can only be explained by apologizing for it — urgency, or context nobody wrote down — until avoiding the code feels safer than touching it.

This kind of embarrassment almost always traces back to overreach: promises stretched too wide, abstractions built well ahead of any real need for them, defaults that assumed more certainty than the problem actually had.

Software built to last stays conservative about what it claims to do. It doesn't try to anticipate every future use case, and it won't generalize a function before two real call sites demand it. Some repetition is cheaper than premature unification, and some problems are better left unsolved until they're unavoidable.

People read this as timidity, when really it just means preferring decisions you can still undo.

A decision that can be undone is safer than one that must be defended indefinitely.

Defaults are underrated as a design lever. A default that stays safe under partial understanding gets used correctly far more often than a powerful one that demands real expertise to avoid misusing. Strong assumptions baked into a default quietly expand what the whole system has promised, and every one of those promises eventually comes due.

Interfaces work the same way. Exposing internal detail invites coupling even from well-intentioned callers, and that coupling hardens into dependency soon enough. What was once an implementation choice becomes a contract by accident, and changing it later starts to feel like betrayal.

The systems that deserve to last say plainly what they support, what's incidental, and what's deliberately out of scope. Naming those limits is what keeps the system legible once other people start changing it.

Elegance and popularity don't hold up well over years, and neither does theoretical completeness for its own sake. What holds up is a mix of clarity, restraint, and honesty. The systems that survive tend to be the ones that chose, early and often, what not to build.

That's also why longevity resists retrofitting. Once speculative complexity, implicit contracts, and fragile assumptions have piled up, no refactor fully restores a system's original tractability. Understanding gets expensive enough that progress just slows to a crawl.

Building software that lasts means treating that as a constraint from day one: designing as much for future scrutiny as for present correctness, and writing code a stranger can read without defensiveness, modify without fear, and justify without embarrassment.

Every other review a system gets happens once, before it ships. Time keeps reviewing it for years afterward, and it never grades on a curve.

Greenfield Rewrites: Why You Lose Institutional Knowledge

Chris Roy — Sun, 09 Aug 2026 10:43:53 +0000

Every system eventually invites a rewrite.

Almost never because it broke. Usually because nobody left on the team can change it without holding their breath.

The structure feels rigid. The decisions feel outdated. The system resists modification in ways nobody can quite explain, and nobody particularly wants to be the one to justify. A new feature means touching six files that shouldn't be related to it. A fix in one corner breaks something two teams over, for reasons that take a week to trace. At some point somebody says the quiet part out loud in a planning meeting, and a clean slate starts to look reasonable. Sensible, even. Overdue.

Greenfield work makes a good pitch. No history to work around. Every decision made on purpose this time, by people who finally understand the problem the old system was only ever groping toward. I've made this pitch myself, in rooms where everyone was nodding before I finished the sentence. What it delivers, most of the time, is amnesia with a deadline attached.

A rewrite never starts from zero, whatever the kickoff deck says. It starts from assumptions about what the old system did and which of those things mattered, made by people standing outside the system looking in. That's exactly the vantage point from which those assumptions are hardest to check.

What a mature system is made of

A mature system is more than its code. It's accumulated truth about an environment that wasn't the most amenable.

A workaround here, a strange boundary check there, a comment that just says "don't touch this, see the incident from three years ago." Each one is a lesson the system learned the hard way, usually at somebody's expense. Some of that knowledge is written down somewhere findable. Most of it isn't. It exists as shape: the specific, ugly contour of the code, not something you could hand to a new hire and call documentation.

When a system gets rewritten, that shape doesn't transfer. It gets rediscovered, piece by piece, the hard way: in production, by whoever's on call the week some corner case that used to be handled quietly shows up unhandled instead. The new system looks simpler on launch day because it's forgotten what made the old one complicated. Nobody's told it yet.

In April 2000, Joel Spolsky wrote up what happened after Netscape decided its browser codebase had become too messy to keep extending, and rewrote it from scratch instead. Version 4.0 shipped in 1997. The rewrite, which became Netscape 6, didn't ship until late 2000: three years with no serious competitive release, in a market where Internet Explorer was being bundled into every copy of Windows for free. Spolsky's point wasn't really about elegance. The ugliest parts of a mature codebase are usually ugly because they encode a fix for a real bug, a workaround for a browser inconsistency nobody remembers reporting, sometimes just a piece of knowledge paid for with a support ticket or an outage. Thrown out in a single motion, in the name of starting clean, because greenfield feels like progress in a way that patching an old system quietly never does. Netscape never recovered the ground it lost in that gap: three years didn't buy the team clarity, it bought Microsoft a market.

The cost of amnesia

By the time the FBI's Virtual Case File project collapsed, the cost of amnesia had an actual dollar figure attached to it, and a congressional report to go with it.

In 2001, the FBI contracted with Science Applications International Corporation to replace its paper-heavy, mainframe-backed case-management process with a modern application that would let agents and analysts share investigative information electronically across field offices. On paper, this was overdue and obviously correct. The bureau was still moving physical case files by hand in some offices, years after most private-sector organizations had digitized equivalent workflows. But the system being replaced held decades of procedural knowledge about how casework moved, how highly nuanced evidence chains needed documenting to be held up in court, and how different units needed different, sometimes conflicting views into the same case. Some of that compartmentalization between field offices existed for reasons that had nothing to do with software and everything to do with the nature of investigations. None of it lived in a specification. It lived in how people used the old system and worked around its limits, and it had to be rediscovered by a team building the replacement from outside, lacking any contact with that experience. The FBI's own understanding of what it needed kept shifting mid-build, because nobody had fully externalized what "what it needed" meant.

By April 2005, the bureau scrapped the project outright. SAIC had delivered roughly 700,000 lines of code, and most of it turned out to be unusable: an estimated $105 million of the total $170 million spent produced nothing the FBI could put into service. IEEE Spectrum's postmortem is worth reading in full for the granular version. It speaks of continuously shifting requirements, no strong technical leadership on the customer's side of the table, and a build that tried to leap directly to a finished system instead of growing into one through anything resembling incremental delivery.

The FBI eventually did get a working case-management system, from a follow-up effort called Sentinel, delivered in 2012 and built in phases with far tighter scope discipline and closer government oversight of the contractor relationship. The lesson wasn't that government can't build software.

Nobody, inside government or out, can rewrite a system whose real specification exists nowhere but inside the system being replaced.

When greenfield discards the product, not just the code

Digg relaunched as a total rebuild on August 25, 2010, losing the reason people used the product along with the technical knowledge behind it. It had a new design, a new codebase, and new infrastructure. The entire greenfield landed in one release, top to bottom, presented as a clean and overdue modernization. In the process, the team dropped or buried features that had nothing to do with performance and everything to do with what made Digg feel like Digg to the people who used it daily: the "bury" button that let the community collectively downvote content, the "upcoming" page where unproven stories could rise on their own before reaching the front page, the personal history and friend-tracking features longtime users had built habits around. None of them registered as a necessity, or even a consideration, during the rewrite. It also changed who got distribution, giving major publishers a mechanism to auto-submit content directly to the site, which quietly undercut the community-driven ranking that had made Digg worth visiting over any other aggregator. Within about a month, Digg had lost close to 30% of its global traffic, with US traffic down by 26% and UK traffic down by 34% according to some contemporary estimates. A large share of that departing audience went straight to a much younger competitor called Reddit, which had spent years copying the parts of Digg's original mechanic that the rewrite had just thrown away. Digg laid off about 40% of its staff in the aftermath, and Kevin Rose, the founder, resigned not long after. Search Engine Land's contemporary account of the relaunch reads, in hindsight, like a checklist of nearly everything a community product should never do to itself in a single release.

The Digg team mistook the product's implementation for the product itself, and underestimated the migration because of it. They rewrote the former with real technical skill, and discovered six brutal weeks later that the latter had been quietly living inside the parts they'd thrown away.

Longhorn, the internal codename for what eventually shipped as Windows Vista, played out the same failure in slow motion, without a public collapse but with a comparable cost in time. It accumulated features and ambitions for years without anyone drawing a hard boundary around scope, until by mid-2004 it was so far behind that Microsoft reset the project outright, discarding most of what had been built and restarting from the more conservative Windows Server 2003 codebase with a narrower, more defensible target. Vista finally shipped in November 2006, more than five years after Windows XP, at the time the longest gap between major Windows releases in the company's history. The reset was closer to a correction than a rewrite in the Netscape sense. Nobody at Microsoft was disgusted with the old code, exactly; it just arrived years too late, with the company paying twice: once for the original effort, once to partially undo it before building the real thing.

Reversibility is the missing piece

Reversibility is what actually matters here: a property built into a system deliberately, long before anyone needs it, not something a young codebase gets for free.

A reversible system absorbs change instead of avoiding it. A decision can be revisited without the whole structure coming down around it. A constraint can be loosened without triggering a reconstruction project that eats a year of the roadmap. The system keeps its history instead of severing it every few years out of frustration.

That takes restraint, and restraint is unglamorous, because it rarely produces a demo anyone gets excited about. Reversible decisions are frequently less expressive than the alternative. They avoid locking in an abstraction before anyone's sure it's the right one, and they deliberately leave gaps where certainty doesn't exist yet. From the outside, that can look like indecision, or a team that can't commit to an architecture. More often it's just a refusal to encode assumptions the team can't defend today, and almost certainly won't be able to defend in two years either.

Twitter's original backend, written in Ruby on Rails, became infamous for outages severe enough that the fail whale error page turned into a cultural mascot for the whole Web 2.0 era. The company's Ruby-based message queue "hit a wall" as early as 2008, in the words of one Twitter developer at the time, unable to keep up with long-running, memory-intensive work at scale, under exactly the kind of pressure that tends to produce a Netscape-style decision. The obvious, tempting move was the Netscape move: declare Rails the problem, freeze feature work, rewrite the whole backend from scratch. Twitter didn't do that. Engineers migrated pieces of the backend onto a new stack built on the JVM, largely in Scala, component by component, over several years, eventually reconfiguring the system so that all mobile traffic bypassed the Ruby stack entirely. The proof came on election night in November 2012, historically one of Twitter's highest-traffic events. Users posted at a sustained average of nearly 10,000 tweets per second for a full hour, peaking at 874,560 tweets in a single minute. There was no clean launch event, no dramatic relaunch post. The service just held, without the fail whale making an appearance even once, the entire time it was being rebuilt underneath its own users.

Stripe's internal implementation changes constantly. What doesn't move is the promise made to everyone who's already integrated against an older version of the API: engineers write against the current internal version only, and a compatibility layer transforms each outgoing response into whatever version a given client originally requested, sometimes years out of date.

Stripe has described this as treating the API itself as infrastructure, something that has to keep working for people who built on it years ago no matter how much has changed underneath since. The public API hasn't needed a breaking rewrite in more than a decade.

Around 2015, Segment split a core piece of its product into more than 140 separate services, following the fairly common belief at the time that this was simply how serious companies build at scale. The promised velocity never showed up. Instead, three full-time engineers ended up spending most of their working hours just keeping the constellation of services alive, and a single change to a shared library meant coordinating redeployment across all 140-plus of them. In 2018 the team wrote publicly about collapsing that architecture back into a single service. Deploys that used to require coordinating dozens of independently versioned services now took one engineer a few minutes, alone. Nobody had designed reversibility into the original split. Segment simply treated the decision itself as something that could still be undone once it was clearly a mistake, and corrected it once enough people were willing to say so plainly.

Shopify never made the split in the first place. Facing the same pressure every sufficiently large monolith eventually faces (in Shopify's case, a Ruby on Rails codebase worked on by more than a thousand engineers across more than a decade), the company seriously considered breaking the monolith into microservices and concluded the operational cost wasn't worth what it would buy them. Instead, it built an internal tool called Packwerk to enforce hard boundaries between modules living inside the same codebase, giving individual teams the isolation benefits normally associated with separate services: clear ownership, enforced interfaces, the ability to reason about one module without loading the entire system into your head at once, without paying for separate deploys, separate on-call rotations, or the failure modes that come standard with a distributed system. Shopify's own engineering team has described this as ongoing, unglamorous maintenance rather than a finished migration with a launch date; their account of the effort reads less like a triumphant rewrite story and more like a maintenance log that never quite ends, which is closer to how large systems stay healthy in practice. Nobody had to freeze feature work for a year to make any of it happen.

This is close to what Martin Fowler formalized back in 2004 as the strangler fig pattern. It's named for a vine that germinates in the canopy of a host tree and gradually grows down and around it, eventually taking the host's place without the tree ever being felled outright in one motion. Route traffic to new code piece by piece. Let the old system keep running underneath while it's dismantled from the edges inward, module by module, service by service. Every step stays small enough to reverse if it turns out to be the wrong step, and the system never once stops being live while the work happens around it.

When a rewrite is earned

Starting over isn't always the wrong call, either. Greenfield becomes necessary the moment reversibility is already gone. Once decisions can't be undone without triggering cascading failure, or constraints have grown so entangled that no single one can be pulled loose without dragging the others with it, a greenfield rewrite is earned. It's an admission that the system stopped being changeable some time ago, and the rewrite is just the moment everyone in the room finally says so out loud instead of stalling for one more quarter.

The Netscapes and the Diggs and the Virtual Case Files of the world had already lost the ability to change safely, long before anyone rewrote anything. The rewrite just finished the job of discarding what still worked along with what didn't, because from inside a system that's stopped being legible, it's hard to tell which is which until it's too late.

In the end, a greenfield rewrite shouldn't be a choice made on impulse. It should follow the swan song of a system that held onto its ability to change for as long as it could.

The systems that deserve to last are the ones that treated rebuilding as a last resort instead of a first instinct, and that's a property you have to design for, deliberately, long before the day you'd need it.

Sources

Joel Spolsky, "Things You Should Never Do, Part I," Joel on Software, April 2000.
"It's official: FBI scraps $170M Virtual Case File project," Computerworld.
Harry Goldstein, "Who Killed the Virtual Case File?," IEEE Spectrum.
"What Happened to Digg? The Rise, Fall, and Surprising Comeback Story," Startup Booted.
"Digg v4: How To Successfully Kill A Community," Search Engine Land, 2010.
"Looking back at Windows Longhorn," The Register, 2025.
"Twitter survives election after Ruby-to-Java move," The Register, November 2012.
"APIs as infrastructure: future-proofing Stripe with versioning," Stripe.
"Goodbye Microservices," Twilio/Segment Engineering.
"Deconstructing the Monolith: Designing Software that Maximizes Developer Productivity," Shopify Engineering.
"Under Deconstruction: The State of Shopify's Monolith," Shopify Engineering, 2020.
Martin Fowler, "StranglerFigApplication," 2004.