DEV Community: Chris Roy

Why Verification Asks You to Leave the Language You Ship

Chris Roy — Sun, 06 Sep 2026 12:53:58 +0000

A test lives in the file you already have open. You write it in the language you ship, you run it, and when it fails you get an input you can look at. The first week with a verifier doesn't work like that. You pick a function you already understand, you write down what it should guarantee, and you wait. What comes back is often unknown, a timeout, or an error pointing at a formula you never typed. The function was ten lines. Getting the tool to accept those ten lines is the rest of the week.

The proof is asking about every run that meets the spec, and that claim needs more than an example. Bertrand Meyer's version of this, in Design by Contract, is the one that still makes sense to people who ship code: the caller promises the preconditions, the callee promises the postconditions if those promises are kept. You already do a sloppy version of that in your head. The tool wants it written down, in a form it can check, and it won't take "obviously" for an answer.

Preconditions, postconditions, framing, permissions, ownership: none of those are hard on their own, and none of them are what most people were hired to think about. You learn them while you learn the verifier, while you learn which of its errors mean the program is wrong and which mean the solver got bored. That's a lot to hold at once. It would be a lot even if the spec lived in the same language as the code. In the systems we've actually built, it usually doesn't.

Proof obligations and disappointment

The loop is where this stops being a vocabulary problem. Five lines of code. The invariant that makes those five lines provable can take days, because it has to hold on entry, survive every iteration, and imply the postcondition when the loop exits. Dijkstra knew that, and so did the people who designed the tools. You find out by watching a verifier reject a program you're sure is correct.

When the proof fails, three things could be wrong: the code, the specification, or the tool's ability to see why the specification holds. A unit test fails with a counterexample you can run. A verifier often fails with a shrug. Rustan Leino has spent years making that interaction less miserable, in what he and Michał Moskal named auto-active verification: you stay in the editor, the tool tries to prove what it can, and you add assertions, lemmas, and ghost code until the solver goes through. That's better than a thousand-line Coq script, and it's still a second job. The ghost code is a program that exists only to convince a solver, written in a dialect whose diagnostics are about Z3's search that afternoon.

In 2009, Gerwin Klein and colleagues published the functional correctness proof of the seL4 microkernel. The kernel was about 8,700 lines of C and 600 lines of assembly. The proof, in Isabelle/HOL, was about 200,000 lines of script. They weren't padding. A machine-checked claim that a kernel always matches its spec is a large claim, and large claims produce large artifacts. By the time the same group wrote up nearly a decade of keeping that proof alive as the kernel grew, the Isabelle corpus was around 480,000 lines. Xavier Leroy's CompCert is the same shape on the compiler side: a Coq development that lets you believe a property of the C still holds of the assembly. A team with a 200-line service and a sprint deadline doesn't start there. The proof is a second codebase, in a second language, maintained by people who can read both.

Two programs, one product

The other thing you notice, once you've picked a tool, is that you've also picked a language you have to keep. A Dafny spec doesn't become a JML spec, and a Coq development doesn't become a SPARK annotation. You don't get to change your mind in year three without rewriting the argument.

The field tried to share the painful part. Boogie and WhyML exist so a frontend can lower "real" code into a smaller logic and hand the rest to an SMT solver. Jean-Christophe Filliâtre and Andrei Paskevich's Why3 will send the same problem toward Alt-Ergo, CVC, Z3, or Coq, depending on what's installed. Microsoft's Boogie sits under Dafny. Verifier authors get to avoid rewriting SMT encodings every time. Someone trying to verify a program they already have now has three languages in play: the one they ship, the one they wrote the spec in, and the one the error actually happened in. Debugging a failed proof means debugging an encoding.

AWS paid that bill in public. Cedar is the authorization language behind Amazon Verified Permissions. The team wanted to prove things about the authorizer that you actually care about in access control: a request is denied unless a permit says otherwise, and a forbid still wins if both apply. They proved those properties against a model written in Dafny. The thing they shipped was Rust. To believe the Rust, they ran differential random testing: generate policies, data, and requests, feed them to both programs, and see if they agree. Mike Hicks's writeup from May 2023 is worth reading for the texture. They used cargo fuzz. They wrote extra generators because pure random policies almost never mention the same groups as the random requests, so you spend all your time in error-handling code and never touch the authorizer. By the time he published, a nightly run was on the order of a hundred million tests. That process found real bugs, including a bad IP-address parse in a Rust package they depended on, and mismatches in how missing application data got handled. Hicks also noted that the Dafny authorizer was about a sixth the size of the production one, which is why they could stand to read it.

Hicks walks through a version of isAuthorized that looks right on a first read: if any forbid policy applies, deny, otherwise allow. That gets "forbids override permits" for free. It also allows a request when no policy applied at all, which is the opposite of the default-deny rule they thought they had. Dafny refused the lemma. The fix is checking that a permit actually applied, the kind of thing a code review can miss because the function still "looks like" access control. They found more of those while proving the validator sound: policies that type-checked and then blew up at evaluation in ways the design hadn't named yet.

The language you can prove things about and the language you can ship were different languages, so a lot of the assurance budget went to showing they still meant the same thing. Most teams don't have that budget. They pick one side of the split and hope. The people who can keep a Dafny model and a Rust implementation aligned through a year of feature work are a research group. Testing never needed that group. JUnit, pytest, and cargo test live in the language the code is already in.

Who the room is for

Some of what's left is social, and pretending otherwise doesn't help.

Formal methods grew up in universities and national labs. The currency is papers. The prestige objects are complete proofs of kernels and compilers. Those proofs matter. They also set the temperature of the field. A newcomer who asks for a tutorial that starts from a web handler is left with the impression they showed up to the wrong conference. Diagnostics assume you already know the logic. Examples are sorted lists and binary search, which is honest about the state of automation and dishonest about the programs people actually ship. If the price of entry is sequent calculus, the room stays small, and the smallness gets misread as seriousness. I don't think most of the researchers involved intend that. I do think the tools encode it. A system that requires a new language, a new IDE, and a new way of reading error messages will be used by people who already wanted to be there.

The industry work that exists, Cedar included, is still specialist teams around components whose failure is expensive enough to justify the second language. They don't make the ordinary case cheaper.

Testing won the ordinary case because it lived where the code already was. The closest thing verification has to that story is property-based testing, and the interesting part is how little language it needed.

Koen Claessen and John Hughes published QuickCheck at ICFP in 2000. The trick wasn't a new logic. Properties were Haskell functions. Generators were Haskell functions. You imported a module and you wrote prop_ names in the same files as the code. When a property failed, you got a counterexample, often a small one, because the library would shrink the input. You could do this on a lunch break, without asking anyone for a license or a seminar. Hypothesis later did the same thing for Python, and the idea spread into languages that will never run Isabelle, because you didn't have to leave. "Write a property, get a counterexample" became a normal skill, even though a property isn't a proof. The adoption still tells you where the notation has to sit: in the file that's already open, in the language the compiler already understands.

When the spec isn't the program

Most verification systems ask you to leave the language you're shipping. Dafny is the clean version of that bet: you write the program in Dafny, specify it in Dafny, and compile to C# or Java after the proof goes through. The translation between that language and the one you ship is now part of the trusted computing base. If the verifier's idea of what a loop means and the compiler's idea of what a loop means ever drift, you can prove a property of a program you didn't ship. CompCert exists partly because that drift is real for C. Cedar's Dafny-plus-Rust split is the same crack, accepted as an engineering process. Joshua Cohen's foundationally verified Why3 exists because the community already knows those mappings are large, trusted, and mostly unchecked.

Specs rot the way comments rot, only with more confidence attached. A specification that can't be type-checked, refactored, and reviewed with the same tools as the program will be maintained like a comment: carefully at first, then not at all. Two syntaxes means two formatters, two jump-to-definitions, two code reviews. In practice the spec loses, because the compiler is the one that ships.

A separate specification language also has to have a meaning, and that meaning is almost never the meaning of the host language. It's a logic chosen to sit well with SMT: restricted quantification, a particular heap model, a particular story about frames and permissions. You have to learn that logic and the mapping from your language into it. The mapping is where the unsoundness bugs live.

I don't think this split was a mistake in 2005. Boogie and Why3 made it possible to build verifiers without rebuilding SMT encodings every time. An intermediate language is a good backend, the way LLVM is a good backend. LLVM isn't a language you ask application developers to think in. WhyML and Boogie have become that, even when the README says they're intermediate. Error messages leak into the mental model. Eventually you're debugging the encoding.

I don't want anyone thinking in WhyML in order to ship a Rust program.

Verification as a library

I'd put the contracts in a module you can import: preconditions as ordinary expressions, ghost state as data the compiler erases, lemmas as functions whose bodies are proofs, checked and then discarded. The verifier is a checker for that library. The solver stays internal.

QuickCheck already showed that a spec in a library gets used. The next step is to keep that shape and strengthen the checker: same notation, a static proof where the library can see enough, tests where it can't.

In 2023, Verus showed you can write the specs and the proofs in Rust. There's a mode system that distinguishes executable code, spec, and proof, and ghost permissions go through Rust's own borrow checker, which is the whole reason you'd bother: the type system you already fight with is also the type system the proof can use. Ghost code is erased. The SMT solver is still there. The programmer-facing language is Rust. That's as close as I've seen to something a systems team might actually open on a Tuesday. It still isn't a crate you add to Cargo.toml and forget about. Modes and ghost annotations are language extensions. The fragment the solver can handle is baked into the tool, not into an API you could version.

I'd go further. Make even those ordinary: types and attributes the compiler already has, plus a checker that understands a documented fragment. SMT solvers like restricted logics. A full modern language is a bad input to Z3. The library can draw that line. It exports the predicates, ghost types, and proof combinators the checker knows how to handle, and it refuses the rest at the import boundary. That's a boundary in the ordinary engineering sense: named interactions, everything else excluded by default. The restriction lives in an API, which can grow.

Ghost code has to erase, so proofs don't ship. Spec-only functions can't be called from executable code by accident. Error messages have to point at the source you wrote, not at a Boogie file. And the expression in a requires clause has to mean what it means in an if. If the checker and the compiler ever disagree about that expression, that's a compiler bug.

You inherit the host language's mess. If the language has undefined behavior, a library can't wish it away. SPARK's answer was a subset of Ada, encoded as types and banned constructs, without a new parser. Automation will be worse, at first, than in a language designed around the solver. Some properties will stay out of reach until the library grows a lemma collection the way Lean grew mathlib: slowly, in public, by people solving their own problems and leaving the lemma behind. A lemma about maps, written against a verification library, can be depended on by the next program in the same language. A lemma about maps, written in WhyML, helps the next WhyML program. There aren't many of those.

I've been turning this over while working on BigTalk, which is supposed to have verification without asking people to leave the language to get it. Once you're already deciding what a type is, what an effect is, and what a region is, adding a second meaning for "the same" program starts to look like the kind of forgotten decision you'll spend years apologizing for. The library is the decision I'd rather defend: verification as code, in the program, with a checker that shares the compiler's meaning of that code.

References

Bertrand Meyer, Applying Design by Contract, IEEE Computer, 1992.
K. Rustan M. Leino, "Dafny: An Automatic Program Verifier for Functional Correctness," LPAR, 2010. See also Using Dafny, an Automatic Program Verifier.
K. Rustan M. Leino and Michał Moskal, "Usable Auto-Active Verification," UV, 2010.
Gerwin Klein et al., "seL4: Formal Verification of an OS Kernel," SOSP, 2009.
Gerwin Klein et al., "Comprehensive Formal Verification of an OS Microkernel," ACM Transactions on Computer Systems, 2014.
Xavier Leroy, "Formal Verification of a Realistic Compiler," Communications of the ACM, 2009.
Jean-Christophe Filliâtre and Andrei Paskevich, "Why3: Where Programs Meet Provers," ESOP, 2013. See also Why3.
Mike Barnett, Bor-Yuh Evan Chang, Robert DeLine, Bart Jacobs, and K. Rustan M. Leino, "Boogie: A Modular Reusable Verifier for Object-Oriented Programs," FMCO, 2005.
Mike Hicks, "How We Built Cedar with Automated Reasoning and Differential Testing," Amazon Science, May 2023.
Koen Claessen and John Hughes, "QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs," ICFP, 2000.
Andrea Lattuada et al., "Verus: Verifying Rust Programs using Linear Ghost Types," OOPSLA, 2023.
Joshua M. Cohen, A Foundationally Verified Intermediate Verification Language, PhD thesis, Princeton, 2025.

Why Plugin Systems Keep Running After the Contract Is Gone

Chris Roy — Sun, 09 Aug 2026 10:56:29 +0000

Most plugin systems continue to work just well enough, making them dangerous.

A typical plugin system starts from a reasonable goal of letting people extend functionality without touching the core. The core stays stable. Optional functionality lives at the edges. Responsibility is distributed cleanly, at least on the whiteboard. In the early stages this holds up fine in practice. Plugins are few. Contracts are narrow enough that a plugin author can read the documented interface once and know the whole shape of what they're allowed to touch. A new contributor can look at the system and reason about it locally, without holding the whole thing in their head. Nobody has had to break a promise to anyone yet, but that equilibrium doesn't last, and it rarely announces its own ending.

The failure builds through accommodation: a plugin needs access to a little more state than the contract allows, or another needs to intercept a request earlier in the lifecycle than anyone originally envisioned. Each request, taken on its own, is defensible. Nobody who approved any single one of them was wrong to do it.

Read any one of those changes in a code review and it looks fine. A reviewer sees a small diff, a plausible justification, a deadline attached to it, and approves it. What a reviewer almost never sees is the fifteenth version of that same request, made by a different plugin, for a different reason, landing on a boundary that's already been widened fourteen times before. Each decision is local. The damage is cumulative, and cumulative damage is exactly the kind that doesn't show up in any single code review.

Somewhere in that process it becomes a second system running alongside the first, wired to it at dozens of points nobody wrote down.

The erosion of boundaries

A plugin system only works if there's a hard line between what the core owns and what plugins are allowed to touch. Almost every real system compromises that line early, because compromising it is the path of least resistance. Internal data structures get exposed. Lifecycle hooks get added for one plugin's sake. Mutable globals stay reachable because locking them down would mean telling three plugin authors their code is about to break. The justification is always some version of "trust the plugin author to use it responsibly."

Trust doesn't scale the way access does.

Once a plugin depends on an internal detail, that detail stops being internal, whatever the documentation says. It becomes a contract by accident, and accidental contracts are the ones nobody can safely break, because nobody agreed to them on purpose in the first place.

Firefox lived this for over a decade. Its original extension model, built on XUL and XPCOM, gave add-ons direct access to the browser's internal component model, i.e., the same libraries that handled networking, rendering, and security, wide open to any extension that wanted them. It made Firefox's extension ecosystem one of the richest on the web for years. It also meant that by the time Mozilla needed to rebuild Firefox around a multiprocess architecture, in pursuit of the same stability and performance gains Chrome had already shipped, legacy XUL extensions were incompatible with it by construction. They reached into places a multiprocess browser couldn't safely expose anymore. Mozilla's own reasoning was direct: reliance on XUL and XPCOM had become an obstacle to work the browser needed to do, specifically the Electrolysis multiprocess project and the Servo rendering engine, both of which required a much cleaner separation between the browser's internals and whatever code an extension was running. You can't safely split a browser across multiple processes if any extension might be reaching directly into a data structure that now lives in a different process entirely. The fix arrived as a new extension API, WebExtensions, deliberately narrower and modeled closely on Chrome's, paired with a hard cutoff: as of Firefox 57 in November 2017, the old add-ons simply stopped loading. Years of accumulated extensions broke at once. The access those extensions depended on had never been closed off, and closing it years later took a browser-wide breaking release that a narrower API would have made unnecessary from the start.

The break, when it finally comes, looks catastrophic, because years of small, reasonable accommodations were never priced as debt until the bill came due all at once.

Shared responsibility, no ownership

In a plugin system that's still healthy, you can always answer one question cleanly: who owns this invariant. Either the core enforces it, or the invariant doesn't really exist. Once a system starts failing, that question stops having a clean answer. Invariants get enforced collectively, informally, by convention. One plugin assumes another has already run. The core assumes plugins will clean up their own state. Execution order becomes "load-bearing," to borrow the phrase everyone reaches for these days, and nobody wrote it down because nobody designed it on purpose.

At that point behavior is being negotiated, extension by extension, mostly by accident.

Atom's package system got there. Packages ran with full access to the DOM and to Node's runtime, which was also the thing that made Atom feel so hackable. You could install a package, and it could reshape almost anything about how the editor behaved, down to intercepting keystrokes before the editor's own core ever saw them. That same openness meant packages could and did collide in ways nobody upstream could fully predict, each one reasonable in isolation, none of them designed against each other. Two packages might both assume they were the first to touch a given buffer. Neither was wrong on its own. Together, loaded in whatever order the user happened to install them, they'd produce a bug that only existed because of the combination, and no one owned the combination.

GitHub announced Atom's sunset in June 2022, archiving the project that December. The stated reason was that community involvement and feature development had stalled. The editor Microsoft chose to keep investing in instead, Visual Studio Code, had been built from day one on a much more restrained model for what an extension was allowed to do. Atom's own performance problems, widely attributed at the time to its Electron foundation, were at least partly a symptom of the same openness. An editor that lets every package touch everything has a much harder time optimizing anything, because it can never be sure what depends on the thing it wants to change.

Failures don't map to a single component. They emerge from interaction, from two pieces of code that were each individually correct and jointly wrong. Removing one plugin "fixes" the bug, and nobody on the team can explain why beyond a shrug. The system still runs. Nobody quite trusts it anymore.

Power dressed up as flexibility

A lot of plugin systems expose power where they should be exposing interfaces. They hand out a hook into some broad execution context and call it flexibility. When a plugin can do anything, the core can guarantee nothing.

ActiveX is the clean historical case. Introduced by Microsoft in 1996, it let a web page embed a control that ran with essentially the same privileges as any other program on the machine, no sandbox, no meaningfully scoped permission model. That an ActiveX control could do anything a desktop application could do, right inside the browser, was the appeal. It was also, as one retrospective later put it bluntly, the web's biggest mistake. A long, ugly stream of vulnerabilities followed, most of them boiling down to the same root cause where a hostile web page could get an ActiveX control to run arbitrary code as the user, no exploit chain required beyond convincing someone to visit the wrong site. Microsoft spent years bolting on warnings and kill bits before finally disabling ActiveX by default across Office and Microsoft 365 in 2024, nearly three decades after it shipped, rather than ever retrofitting real boundaries into a system built without any.

ActiveX wasn't alone in this. The broader family of browser plugin technologies built around NPAPI, the interface that let Flash, Java applets, and Silverlight run inside a browser tab, made a similar bet: give the plugin a wide-open door into the host process in exchange for whatever capability the plugin wanted to offer. Chrome and Firefox spent the 2010s walking that bet back, first with sandboxing, then with click-to-play defaults, then by dropping NPAPI support entirely. Chrome removed it in 2015, Firefox followed in 2017. The technology wasn't uniquely badly built; it ran on the same premise as ActiveX, that broad access is a reasonable price for broad capability. Browsers eventually decided it wasn't, and every plugin depending on that access had to be rebuilt or abandoned to match.

Defensive code creeps into the core because nobody trusts what a plugin might have already touched. Validation gets duplicated at every boundary because there's no boundary anyone's confident is real.

This never arrives as a single moment. There's no line the system visibly crosses. It just accumulates until the honest answer to "where does this system end" is that nobody's sure anymore.

The systems that age better

Eclipse is one of the oldest working examples. Its plugin model, formalized on top of OSGi, is built around what Eclipse's own architecture documentation calls the plug-in contract, a small, explicit set of extension points, each one declared up front, each one owned by whichever plugin defines it. A plugin can't reach into another plugin's internals just because both happen to be loaded in the same process. It has to go through a declared extension point or it doesn't get in at all. OSGi's bundle model reinforces this at the runtime level, with each bundle getting its own classloader and explicit version boundaries between dependencies. That discipline is a large part of why OSGi has stayed a working foundation for enterprise software for close to two decades, hot-swapping components in running systems without the whole platform destabilizing every time somebody ships an update.

Visual Studio Code took a related but distinct approach to the same discipline. Instead of trusting extensions with the same process as the editor, it doesn't give them the choice. Every extension runs in a separate process, the extension host, talking to the main application only through a defined API surface and an RPC layer. An extension can be slow, buggy, even hostile, and the editor's own responsiveness is largely insulated from it, because the extension was never given the kind of direct access that would let it take the rest of the application down with it.

That's the same problem Atom's package model ran into, solved by refusing the access in the first place instead of hoping plugin authors would behave. Where Eclipse enforces its boundary through a declared contract that extensions have to go through, VS Code enforces a version of the same idea through a process boundary that extensions can't get around even if they wanted to. Different mechanism, same underlying refusal.

Eclipse and VS Code both made their refusal at the architecture level, before any extension author ever showed up wanting more. Chrome's Manifest V3 shows what the same refusal looks like made in public, years into an established ecosystem, against developers who had built real businesses on the access being taken away. Chrome's older extension API let extensions intercept and modify every network request a browser made, through the webRequest API, the kind of broad, ambient power that makes it impossible to reason about what any given extension might be doing to your traffic. Google's replacement, declarativeNetRequest, asks extensions to declare their filtering rules up front instead of intercepting requests live, which is a real reduction in what an extension can express. Ad blocker developers noticed immediately. Raymond Hill, who maintains uBlock Origin, warned as early as January 2019 that the change threatened to break his tool's core functionality, and the backlash from extension developers has continued through every stage of the rollout since. Google shipped the restriction anyway. The ecosystem got smaller and louder about it in roughly equal measure, and Chrome's extension platform got more predictable in exchange, which was the actual trade being made.

Every one of those systems paid a visible cost for the refusal. Eclipse's plugin authors write against a stricter model than a looser platform would demand. VS Code extensions can't reach into the editor the way an Atom package once could. Chrome's ad blockers lost capability they'd relied on for years. None of it showed up as a headline the way a launch does. It showed up as a smaller ecosystem that people complained about and kept building on anyway.

This is the ordinary, predictable outcome of extension without boundaries, produced by people who were right about every single decision except the sum of all of them.

Aside: I don't like VSCode personally. I like what Zed is doing, and building an IDE of my own is a long-term goal. For now that's a few years away.

References

"Deprecating XUL for WebExtensions," LWN.net.
"Helping You Find Compatible Extensions," Mozilla Add-ons Community Blog, October 2017.
"GitHub's Atom Text Editor Will Be Retired in December," gHacks Tech News, June 2022.
"Remembering ActiveX Controls, the Web's Biggest Mistake," How-To Geek.
"Microsoft Internet Explorer ActiveX Vulnerability," CISA, 2006.
"Microsoft Office 2024 to disable ActiveX controls by default," BleepingComputer, 2024.
"Notes on the Eclipse Plug-in Architecture," Eclipse Foundation.
"What Is OSGi?," OSGi Alliance / Eclipse Foundation.
"Migrating VS Code to Process Sandboxing," Visual Studio Code Blog, November 2022.
"Chrome Extension Manifest V3 May Break uBlock Origin Content Blocker," BleepingComputer, January 2019.
"The Final Countdown for NPAPI," Chromium Blog, November 2014.
"Mozilla to Drop Support for All NPAPI Plugins in Firefox 52, Except Flash," BleepingComputer, 2017.

Why Systems That Last Keep Their Boundaries Honest

Chris Roy — Sun, 09 Aug 2026 10:55:03 +0000

Most system failures trace back to a constraint nobody wrote down. By the time something breaks, the architecture is usually still fine. What gave way is an assumption nobody agreed to keep.

Early on, most systems are tractable. The architecture is coherent, the abstractions are few, and the number of invariants in play is small enough that one person can hold all of them in their head at once. At this stage the system rewards local reasoning. You can change a component and predict the effect of that change without opening five other files to check what else might notice.

The system's behavior can still be approximated as:

S \approx i = 1 \sum n C_{i}

where each component $C_{i}$ contributes independently, and interactions between components stay limited and explicit. Read a function, understand its inputs and outputs, and you understand what it does to the system. Nothing else needs checking.

That property doesn't survive growth, and it rarely gets killed off by recklessness. It erodes through accommodation. A new requirement adds a conditional path to a function that used to have one job. An exception bypasses an invariant because enforcing it that week would have blown a deadline. A shortcut defers a constraint under pressure, with every intention of coming back to fix it later, and the fix never gets scheduled because nothing visibly broke. Each of these changes looks locally defensible at the moment it's made. The system quietly stops being additive, one reasonable decision at a time, without anyone ever sitting down and deciding to compromise the architecture.

Interaction terms accumulate:

S = i = 1 \sum n C_{i} + i \neq = j \sum I_{ij}

Once interactions grow faster than components, local reasoning stops working. The second term takes over. Behavior starts to look emergent instead of designed, and the instinct is usually to blame the people writing the code, but that's rarely where the fault lies. Coupled systems behave this way as a structural matter. It happens on well-run teams with disciplined engineers just as often as it happens anywhere else, because the mathematics doesn't care how careful anyone was being.

A validation function gets written for one form, on one page. Six months later it's being called from four other places, because it already does most of what those callers need and writing a second function feels wasteful. None of the four callers needs quite the same validation, so each one adds a flag, a special case, an early return for its own use. The function now has to know about four different contexts to do its job in any of them. A helper function turns into a load-bearing wall without any single change making it one, because reuse looked free every single time someone reached for it, long before anyone consciously chose to build it that way.

The four callers didn't need a shared function. They needed four small functions that happened to overlap in places. Leave the overlap. Writing four functions feels like waste in the moment, measured against the immediate task. Each of those four could change, break, and get deleted independently when its caller goes away. The shared version can do none of those things without a meeting first, and the meeting rarely happens until the function is already too tangled to split back apart.

Reuse is one of the few instincts in software engineering that's taught early and rewarded in code review, usually before anyone's had the chance to watch it fail. Writing the same ten lines twice doesn't get anyone promoted, but spotting the duplication and collapsing it into one place does. The complimented version is the one that, three years later, four teams are afraid to touch, and by then nobody remembers who to ask about why it's shaped the way it is.

Restraint, in this context, is what keeps that growth in check: not adding an abstraction before there's evidence it earns its keep. The same instinct holds off on generalizing a function until two call sites need it, and refuses to promise behavior the system can't keep enforcing five years from now, after everyone who understood the original trade-off has moved to a different team.

Boundaries are how restraint gets enforced day to day. Naming them in a design review once, then forgetting them, doesn't count. A boundary is a contract that names the allowed interactions, excludes everything else by default, and enforces the invariant at the interface instead of relying on the discipline of every later caller, most of whom will never read the comment explaining why the invariant exists. When a boundary holds, interaction terms stay contained near it. When it doesn't, complexity diffuses outward to wherever the boundary used to be, and it rarely announces that it's doing so.

A module with no interface at least advertises its own recklessness; everyone touching it knows they're on their own. A leaky one looks safe from the outside while failing, without any noise, to protect anything. Once internal details leak across a boundary, that boundary is a boundary in name only. The invariant it was supposed to protect is no longer enforced in one place. It's now distributed, implicitly, across every caller that has come to depend on the leak, usually without anyone on the team keeping a list of who's depending on what.

Someone returned the internal list because it was three lines shorter than writing a proper accessor. For a while, nothing goes wrong. Then a caller three modules away starts mutating that list directly, because it's right there and it works. A year later, someone tries to change how that internal list is stored, maybe to fix a performance problem, and discovers the change breaks code in a part of the system they've never had reason to open. The boundary failed silently, letting something invisible grow behind it.

At that point the cost of a small change stops being small. A one-line fix requires tracing every caller that might be relying on the old behavior, half of which were never documented as dependents in the first place. Understanding gives way to fear. Teams stop touching that part of the code because nobody can say with confidence what else is wired to it, and finding out is expensive enough that it's easier to build around the problem than through it.

In The Mythical Man-Month, Fred Brooks wrote that conceptual integrity erodes gradually, under a steady accumulation of individually reasonable additions. John Ousterhout, in A Philosophy of Software Design, describes the shallow module: an interface nearly as complicated as its implementation. It doesn't reduce complexity. It relocates it onto whoever has to call it, and hides the relocation behind the appearance of a clean function signature.

Ousterhout's counterpart to the shallow module is the deep one: a simple interface sitting in front of substantial internal machinery. The caller gets a narrow surface to learn. The complexity stays in the module. A file system is the example he reaches for. Opening, reading, and writing a file is a handful of operations to learn, and underneath that handful sits allocation, caching, and recovery from partial writes. None of it is the caller's problem. Most interfaces people call "clean" are shallow in this sense, and most people can't tell the difference until they're the one stuck maintaining what's on the other side of it.

Systems thinking, in practice, is arranging things so that most components need to know as little as possible about the rest of the system. Duplicating ten lines of logic across two modules can be cheaper, over the life of the system, than sharing one function that couples them together for years.

There's an old, unglamorous illustration of this in how Unix pipes were designed: small programs, each doing one narrow thing, connected through a plain text interface that none of them needed to understand beyond "read a stream, write a stream." No program in that chain needs to know how the others are implemented. The boundary between them is the text stream itself, and it's inconvenient in exactly the ways that keep it honest: no shared memory, and no shortcut for one program to reach into another's internals because it's under a deadline.

Not everything from that era aged the same way. The shells wrapping those pipes have accumulated more flags than most people using them could name, and plenty of small Unix tools grew their own tangle of special cases the moment someone needed one badly enough. The pipe itself mostly avoided that fate, though it's worth admitting that some of the credit belongs to the text stream being too dumb to extend rather than to anyone's restraint.

A system that preserves locality lets reasoning scale roughly linearly with size. Doubling the codebase doubles the amount you need to understand to make a safe change, not the odds that the change ripples somewhere unrelated and unrelated-looking. A system that loses locality forces reasoning to become global well before it doubles in size, and past that point it stops scaling in any useful sense, no matter how much hardware or headcount gets thrown at the problem afterward. More engineers on a system with no locality doesn't parallelize the work. It just adds more people who each need the same global context before they can safely touch anything.

In Out of the Tar Pit, Ben Moseley and Peter Marks argue that essential complexity, the complexity a system can't avoid because it's inherent to the problem being solved, is usually smaller than it looks from inside a struggling codebase. Most of what makes large systems hard to work with is accidental complexity, generated by uncontrolled state and interaction. That's what teams end up fighting for years without naming it that way.

A system that grants unrestricted access in the name of flexibility has already given up the guarantees that flexibility was supposed to make possible. A setting changes mid-request and nobody can say why. Six subsystems all have write access to the same configuration object, because passing it around explicitly felt like extra ceremony, and none of them coordinated with the others. Early on that looked like convenience. The flexibility that made the object easy to build with is what makes the bug impossible to trace.

Fixing that particular problem rarely means adding more logging or more code review. It means taking write access away from five of the six subsystems and forcing them to ask the sixth one for a change instead of making it themselves, which is a smaller ask than it sounds like until you're the one on the subsystem that lost its shortcut.

A component can only be given real freedom to change its own internals once the rest of the system knows exactly what it will and won't do at its boundary. Freedom without a known boundary reads as flexibility right up until someone needs to rely on it, at which point it's just risk nobody priced in.

Trust, inside a codebase, tends to come from limits. A component that's explicit about what it will do, and that fails loudly when it can't, gives engineers something solid to build on top of. It lets a team revisit an old decision months later without treating the whole surrounding area as radioactive, the way people learn to avoid the one file nobody fully understands anymore, the one everyone routes around instead of opening.

The same pattern shows up outside software entirely. Charles Perrow's Normal Accidents looks at failures across nuclear plants, chemical refineries, and aviation, and finds that catastrophic failure in tightly coupled systems is usually a property of how densely the parts interact, not a property of any single part failing on its own. In systems that are already tightly coupled, adding more cross-cutting paths, more ways for one part to reach into another under unusual conditions, tends to increase risk. This holds even when every individual addition looked, at the time, like it was making the system more capable or more resilient to some specific failure someone had just seen happen.

Time tests a design more honestly than any review ever will. Systems built to anticipate every future requirement tend to accumulate complexity they never use, carrying the weight of speculative features on the chance someone eventually asks for them. Manny Lehman's Laws of Software Evolution get at part of why: a system that keeps growing without a disciplined structure to contain that growth gets progressively harder to modify, on a schedule that belongs to the system no matter who is maintaining it that quarter. Good engineers can slow the decline, but none of them seem to stop it outright.

Past a certain size, discipline at the boundary stops being optional, whether or not anyone on the team decided to call it that. What remains, after the person who drew the boundary has left the room, is whether anyone kept enforcing it.

References

Why the Reasoning Behind a Keyword Disappears

Chris Roy — Sun, 09 Aug 2026 10:55:02 +0000

Most programming languages are remembered for their syntax. Almost none are remembered for the decisions that produced it.

Every keyword exists because someone decided one approach was better than another, usually after weeks of debate. Writing it took an afternoon. The reasoning behind it lasted about as long as it took everyone to move on.

I've been building BigTalk from the ground up: memory model, effects, regions, verification. git can find the line that introduced a bug. What it won't tell you is why that line was written that way instead of three other ways that were discussed and discarded, or never written down because they seemed obvious at the time.

Those conversations live in chat threads, calls, whiteboard photos nobody keeps.

Adding a keyword is easy, and rewriting a parser is easy enough. Choosing the keyword is the part that stays difficult after the merge.

Should this feature exist at all? Does explicitness win over convenience here? An inconsistency might have to stay, too, if fixing it breaks something that was never supposed to depend on it.

Those questions don't have single correct answers. They have a context, and context is usually the first thing a project loses. Without it, the next argument is often the old argument again, held by people who don't know it already happened.

This matters more the longer a project is meant to live. The people deciding today are unlikely to be the people maintaining it in ten years. Even if they are the same people, memory is a poor dependency.

An undocumented decision eventually becomes indistinguishable from an arbitrary one. Nobody trusts an arbitrary decision, so sooner or later someone changes it.

RFCs get treated as a large-project affectation. Something Rust does. Something a standards body does. Overkill for anything smaller. The value has never had much to do with size.

An RFC captures intent before that intent gets flattened into a diff. A useful one records the problem, the constraints, the paths not taken, and why those paths lost. Implementation detail can live in the code. The logic the implementation is downstream of usually can't.

Writing RFCs for BigTalk was never the friction. Managing a growing pile of them consistently was: assigning numbers, tracking status, keeping the fortieth document under the same conventions as the first. That kind of upkeep never rivaled the difficulty of a type checker, but it made the collection worse every time it slipped, until the workflow itself became the thing I was fighting.

So I built a small CLI to stop fighting it: rfcman. It shipped under the BigTalk name ahead of the compiler. It doesn't do anything clever. It manages RFC lifecycles and keeps their relationships stable through UUIDs, so a document can change shape without losing the reasoning attached to it.

It exists to remove friction, the same reason a build system exists. Nothing about it was meant to be the interesting part of the project. It just turned out to be the part that let the interesting parts stay honest.

The code usually stays readable just fine. What disappears is the reasoning that used to hold it together.

Someone finds an odd abstraction, decides it's unnecessary, and removes it. Months later the problem it was preventing comes back. The system had forgotten why the abstraction was there.

Software gets evaluated on correctness and performance. Continuity rarely gets the same weight, though it should. A system worth keeping has to preserve why its behavior looks the way it does.

Source alone doesn't carry that, and neither does documentation once it drifts. What has to survive is a checkable link between decision and implementation, so that in ten years, someone can still find out why.

Why Software That Lasts Is Written for a Future Reader

Chris Roy — Sun, 09 Aug 2026 10:43:55 +0000

Correctness is a low bar. Most systems that reach production clear it. Plenty of them still turn difficult to touch within a few years, because nobody treated time as a design constraint while there was still time to.

A system lasts when conservative choices made early are kept. What it promises has to stay visible, and so does what it will not promise.

Every system eventually gets read more than it gets written, by people who weren't in the room for the original decisions. That future reader is the software's real audience. Code that leans on shared context asks that reader for a trust it hasn't earned.

The code can keep running on explanations that live outside it, and on invariants nobody wrote a boundary for. Every change still turns into a negotiation with its own past.

Someone opened a helper that computes totals, found two lines in the wrong order, swapped them, and swapped them back, because three callers already depended on the wrong order. The tests still pass. Opening it again means becoming the person who owns the totals.

Embarrassment is a useful metric for catching this early. Once a file can only be explained by apologizing for leftover urgency, or for context nobody wrote down, avoiding it starts to feel safer than touching it.

This kind of embarrassment almost always traces back to overreach. Someone stretched a promise too wide, or baked more certainty into a default than the problem actually had.

Software built to last stays conservative about what it claims to do. It doesn't try to anticipate every future use case, and it won't generalize a function before two real call sites demand it. Repetition is often cheaper than unifying too early. A problem can wait until it's unavoidable. Those are decisions you can still undo.

A default that stays safe under partial understanding gets used correctly far more often than a powerful one that demands real expertise to avoid misusing. Strong assumptions baked into a default expand what the whole system has promised without anyone recording the expansion, and every one of those promises eventually comes due.

Exposing internal detail invites coupling even from well-intentioned callers, and that coupling hardens into dependency soon enough. What was once an implementation choice becomes a contract by accident, and changing it later starts to feel like betrayal.

The systems that survive tend to be the ones that chose, early and often, what not to build. Saying plainly what a system supports and what it puts out of scope is what keeps it legible once other people start changing it.

Once implicit contracts have piled up, no refactor fully restores a system's original tractability. Understanding gets expensive enough that progress just slows to a crawl.

Every other review a system gets happens once, before it ships. Time keeps reviewing it for years afterward, and it never grades on a curve.

Greenfield Rewrites: Why You Lose Institutional Knowledge

Chris Roy — Sun, 09 Aug 2026 10:43:53 +0000

Every system eventually invites a rewrite.

Almost never because it broke. Usually because nobody left on the team can change it without holding their breath.

The structure feels rigid. The decisions feel outdated. The system resists modification in ways nobody can quite explain, and nobody particularly wants to be the one to justify. A new feature means touching six files that shouldn't be related to it. A fix in one corner breaks something two teams over, for reasons that take a week to trace. At some point somebody says the quiet part out loud in a planning meeting, and a clean slate starts to look reasonable. Sensible, even. Overdue.

Greenfield work makes a good pitch. No history to work around. Every decision made on purpose this time, by people who finally understand the problem the old system was only ever groping toward. I've made this pitch myself, in rooms where everyone was nodding before I finished the sentence. What it delivers, most of the time, is amnesia with a deadline attached.

A rewrite never starts from zero, whatever the kickoff deck says. It starts from assumptions about what the old system did and which of those things mattered, made by people standing outside the system looking in. That's exactly the vantage point from which those assumptions are hardest to check.

What a mature system is made of

A mature system is more than its code. It's accumulated truth about an environment that never cooperated.

A workaround here, a strange boundary check there, a comment that just says "don't touch this, see the incident from three years ago." Each one is a lesson the system learned the hard way, usually at somebody's expense. Some of that knowledge is written down somewhere findable. Most of it isn't. It exists as shape: the specific, ugly contour of the code, not something you could hand to a new hire and call documentation.

When a system gets rewritten, that shape doesn't transfer. It gets rediscovered, piece by piece, the hard way: in production, by whoever's on call the week some corner case the old system absorbed without comment shows up unhandled instead. The new system looks simpler on launch day because it's forgotten what made the old one complicated. Nobody's told it yet.

In April 2000, Joel Spolsky wrote up what happened after Netscape decided its browser codebase had become too messy to keep extending, and rewrote it from scratch instead. Version 4.0 shipped in 1997. The rewrite, which became Netscape 6, didn't ship until late 2000: three years with no serious competitive release, in a market where Internet Explorer was being bundled into every copy of Windows for free. Spolsky's point wasn't really about elegance. The ugliest parts of a mature codebase are usually ugly because they encode a fix for a real bug, a workaround for a browser inconsistency nobody remembers reporting, sometimes just a piece of knowledge paid for with a support ticket or an outage. Thrown out in a single motion, in the name of starting clean, because greenfield feels like progress in a way that patching an old system never does. Netscape never recovered the ground it lost in that gap: three years didn't buy the team clarity, it bought Microsoft a market.

The cost of amnesia

By the time the FBI's Virtual Case File project collapsed, the cost of amnesia had an actual dollar figure attached to it, and a congressional report to go with it.

In 2001, the FBI contracted with Science Applications International Corporation to replace its paper-heavy, mainframe-backed case-management process with a modern application that would let agents and analysts share investigative information electronically across field offices. On paper, this was overdue and obviously correct. The bureau was still moving physical case files by hand in some offices, years after most private-sector organizations had digitized equivalent workflows.

But the system being replaced held decades of procedural knowledge about how casework moved, how evidence chains had to be documented to hold up in court, and how different units needed different, sometimes conflicting views into the same case. Some of that compartmentalization between field offices existed for reasons that had nothing to do with software and everything to do with the nature of investigations. None of it lived in a specification. It lived in how people used the old system and worked around its limits, and it had to be rediscovered by a team building the replacement from outside, lacking any contact with that experience. The FBI's own understanding of what it needed kept shifting mid-build, because nobody had fully externalized what "what it needed" meant.

By April 2005, the bureau scrapped the project outright. SAIC had delivered roughly 700,000 lines of code, and most of it turned out to be unusable: an estimated $105 million of the total $170 million spent produced nothing the FBI could put into service. IEEE Spectrum's postmortem is worth reading in full for the granular version. It speaks of continuously shifting requirements, no strong technical leadership on the customer's side of the table, and a build that tried to leap directly to a finished system instead of growing into one through anything resembling incremental delivery.

The FBI eventually did get a working case-management system, from a follow-up effort called Sentinel, delivered in 2012 and built in phases with far tighter scope discipline and closer government oversight of the contractor relationship. The lesson wasn't that government can't build software.

Nobody, inside government or out, can rewrite a system whose real specification exists nowhere but inside the system being replaced.

When greenfield discards the product, not just the code

Digg relaunched as a total rebuild on August 25, 2010, losing the reason people used the product along with the technical knowledge behind it. It had a new design, a new codebase, and new infrastructure. The entire greenfield landed in one release, top to bottom, presented as a clean and overdue modernization. In the process, the team dropped or buried features that had nothing to do with performance and everything to do with what made Digg feel like Digg to the people who used it daily: the "bury" button that let the community collectively downvote content, the "upcoming" page where unproven stories could rise on their own before reaching the front page, the personal history and friend-tracking features longtime users had built habits around. None of them registered as a necessity, or even a consideration, during the rewrite. It also changed who got distribution, giving major publishers a mechanism to auto-submit content directly to the site, which undercut the community-driven ranking that had made Digg worth visiting over any other aggregator.

Within about a month, Digg had lost close to 30% of its global traffic, with US traffic down by 26% and UK traffic down by 34% according to some contemporary estimates. A large share of that departing audience went straight to a much younger competitor called Reddit, which had spent years copying the parts of Digg's original mechanic that the rewrite had just thrown away. Digg laid off about 40% of its staff in the aftermath, and Kevin Rose, the founder, resigned not long after. Search Engine Land's contemporary account of the relaunch reads, in hindsight, like a checklist of nearly everything a community product should never do to itself in a single release.

The Digg team mistook the product's implementation for the product itself, and underestimated the migration because of it. They rewrote the former with real technical skill, and discovered six brutal weeks later that the latter had been quietly living inside the parts they'd thrown away.

Longhorn, the internal codename for what eventually shipped as Windows Vista, played out the same failure in slow motion, without a public collapse but with a comparable cost in time. It accumulated features and ambitions for years without anyone drawing a hard boundary around scope, until by mid-2004 it was so far behind that Microsoft reset the project outright, discarding most of what had been built and restarting from the more conservative Windows Server 2003 codebase with a narrower, more defensible target. Vista finally shipped in November 2006, more than five years after Windows XP, at the time the longest gap between major Windows releases in the company's history. The reset was closer to a correction than a rewrite in the Netscape sense. Nobody at Microsoft was disgusted with the old code, exactly; it just arrived years too late, with the company paying twice: once for the original effort, once to partially undo it before building the real thing.

Reversibility is the missing piece

Reversibility is what actually matters here: a property built into a system deliberately, long before anyone needs it, not something a young codebase gets for free.

A reversible system absorbs change instead of avoiding it. A decision can be revisited without the whole structure coming down around it. A constraint can be loosened without triggering a reconstruction project that eats a year of the roadmap. The system keeps its history instead of severing it every few years out of frustration.

That takes restraint, and restraint is unglamorous, because it rarely produces a demo anyone gets excited about. Reversible decisions are frequently less expressive than the alternative. They avoid locking in an abstraction before anyone's sure it's the right one, and they deliberately leave gaps where certainty doesn't exist yet. From the outside, that can look like indecision, or a team that can't commit to an architecture. More often it's just a refusal to encode assumptions the team can't defend today, and almost certainly won't be able to defend in two years either.

Twitter's original backend, written in Ruby on Rails, became infamous for outages severe enough that the fail whale error page turned into a cultural mascot for the whole Web 2.0 era. The company's Ruby-based message queue "hit a wall" as early as 2008, in the words of one Twitter developer at the time, unable to keep up with long-running, memory-intensive work at scale, under exactly the kind of pressure that tends to produce a Netscape-style decision. The obvious, tempting move was the Netscape move: declare Rails the problem, freeze feature work, rewrite the whole backend from scratch. Twitter didn't do that. Engineers migrated pieces of the backend onto a new stack built on the JVM, largely in Scala, component by component, over several years, eventually reconfiguring the system so that all mobile traffic bypassed the Ruby stack entirely. The proof came on election night in November 2012, historically one of Twitter's highest-traffic events. Users posted at a sustained average of nearly 10,000 tweets per second for a full hour, peaking at 874,560 tweets in a single minute. There was no clean launch event, no dramatic relaunch post. The service just held, without the fail whale making an appearance even once, the entire time it was being rebuilt underneath its own users.

Stripe's internal implementation changes constantly. What doesn't move is the promise made to everyone who's already integrated against an older version of the API: engineers write against the current internal version only, and a compatibility layer transforms each outgoing response into whatever version a given client originally requested, sometimes years out of date.

Stripe has described this as treating the API itself as infrastructure, something that has to keep working for people who built on it years ago no matter how much has changed underneath since. The public API hasn't needed a breaking rewrite in more than a decade.

Around 2015, Segment split a core piece of its product into more than 140 separate services, following the fairly common belief at the time that this was simply how serious companies build at scale. The promised velocity never showed up. Instead, three full-time engineers ended up spending most of their working hours just keeping the constellation of services alive, and a single change to a shared library meant coordinating redeployment across all 140-plus of them. In 2018 the team wrote publicly about collapsing that architecture back into a single service. Deploys that used to require coordinating dozens of independently versioned services now took one engineer a few minutes, alone. Nobody had designed reversibility into the original split. Segment simply treated the decision itself as something that could still be undone once it was clearly a mistake, and corrected it once enough people were willing to say so plainly.

Shopify never made the split in the first place. Facing the same pressure every sufficiently large monolith eventually faces (in Shopify's case, a Ruby on Rails codebase worked on by more than a thousand engineers across more than a decade), the company seriously considered breaking the monolith into microservices and concluded the operational cost wasn't worth what it would buy them. Instead, it built an internal tool called Packwerk to enforce hard boundaries between modules living inside the same codebase, giving individual teams the isolation benefits normally associated with separate services: clear ownership, enforced interfaces, the ability to reason about one module without loading the entire system into your head at once, without paying for separate deploys, separate on-call rotations, or the failure modes that come standard with a distributed system. Shopify's own engineering team has described this as ongoing, unglamorous maintenance rather than a finished migration with a launch date; their account of the effort reads less like a triumphant rewrite story and more like a maintenance log that never quite ends, which is closer to how large systems stay healthy in practice. Nobody had to freeze feature work for a year to make any of it happen.

This is close to what Martin Fowler described back in 2004 as the strangler application, and retitled years later as the strangler fig pattern. It's named for a vine that germinates in the canopy of a host tree and gradually grows down and around it, eventually taking the host's place without the tree ever being felled outright in one motion. Route traffic to new code piece by piece. Let the old system keep running underneath while it's dismantled from the edges inward, module by module, service by service. Every step stays small enough to reverse if it turns out to be the wrong step, and the system never once stops being live while the work happens around it.

When a rewrite is earned

Starting over isn't always the wrong call, either. Greenfield becomes necessary the moment reversibility is already gone. Once decisions can't be undone without triggering cascading failure, or constraints have grown so entangled that no single one can be pulled loose without dragging the others with it, a greenfield rewrite is earned. It's an admission that the system stopped being changeable some time ago, and the rewrite is just the moment everyone in the room finally says so out loud instead of stalling for one more quarter.

The Netscapes and the Diggs and the Virtual Case Files of the world had already lost the ability to change safely, long before anyone rewrote anything. The rewrite just finished the job of discarding what still worked along with what didn't, because from inside a system that's stopped being legible, it's hard to tell which is which until it's too late.

In the end, a greenfield rewrite shouldn't be a choice made on impulse. It should follow the swan song of a system that held onto its ability to change for as long as it could.

The systems that deserve to last are the ones that treated rebuilding as a last resort instead of a first instinct, and that's a property you have to design for, deliberately, long before the day you'd need it.

An edited version of this article is on Hackernoon.

References

Joel Spolsky, "Things You Should Never Do, Part I," Joel on Software, April 2000.
"It's official: FBI scraps $170M Virtual Case File project," Computerworld.
Harry Goldstein, "Who Killed the Virtual Case File?," IEEE Spectrum.
"What Happened to Digg? The Rise, Fall, and Surprising Comeback Story," Startup Booted.
"Digg v4: How To Successfully Kill A Community," Search Engine Land, 2010.
"Looking back at Windows Longhorn," The Register, 2025.
"Twitter survives election after Ruby-to-Java move," The Register, November 2012.
"APIs as infrastructure: future-proofing Stripe with versioning," Stripe.
"Goodbye Microservices," Twilio/Segment Engineering.
"Deconstructing the Monolith: Designing Software that Maximizes Developer Productivity," Shopify Engineering.
"Under Deconstruction: The State of Shopify's Monolith," Shopify Engineering, 2020.
Martin Fowler, "StranglerFigApplication," 2004, retitled 2019.