I maintain about a dozen small products that share one homemade framework. Last week I built a conformance linter — a little tool that scans each repo for architectural drift — and I was one command away from wiring it into all of them at once.
I didn't. I ran it in two repos first.
That two-repo dry run is the cheapest safety I've added all year, and it caught two things that would otherwise have gone off in every repo at the same time. One of them was the linter being wrong about my own code.
Why fan-out is tempting, and why it bites
When you have a change that clearly belongs everywhere — a lint rule, a CI step, a dependency bump — the obvious move is to apply it everywhere. One sweep, done.
The problem is that a shared tool doesn't fail in the tool. It fails in the consumers, and consumers differ in ways the tool's own tests never exercise. So "all green in the framework repo" tells you almost nothing about what happens when a dozen repos actually pull it in.
If you fan out first and discover the blocker second, you don't get one failure. You get the same failure a dozen times, concurrently, and now you're triaging a wall of red instead of a single problem.
The pilot: two repos, chosen to disagree
The trick isn't "test it somewhere first." It's picking the right somewhere.
I have two ways a repo consumes the shared framework:
- one pulls it from a package registry at a pinned version,
- the other uses a local path repo — a symlink to a checkout that moves with development.
So the pilot was one of each. Not two similar repos — two that maximize the difference along the axis most likely to break. If a change survives both consumption modes, it'll probably survive the rest. If it breaks, it breaks here, cheaply, where I'm paying attention.
Blocker 1: the local green was a lie
The linter ran fine on my machine in every repo. Then I pushed it to the two pilots' CI and both went red immediately — a hard fatal, exit 255, before the linter did any work.
The cause: the linter loaded the framework's own dependencies to run. On my machine that resolved, because my local setup symlinks to a full checkout of the framework, dependencies and all. But a consumer doesn't have the framework's private vendor/ tree, and CI does a shallow clone without installing it. So the exact same command that was green locally was a guaranteed fatal in every consumer.
"Works on my machine" had a specific, mechanical reason to lie: the symlink was hiding the fact that consumers don't get the tool's dependencies. The fix was to resolve against the consumer's dependency tree instead of the tool's. Small change. But if I'd fanned out first, it would have been a dozen red pipelines at once, all with the same confusing exit 255.
Blocker 2: the linter was wrong about my own fleet
The second one was more humbling.
One of the linter's rules flags hardcoded default secrets — a real thing you want to catch. But my correct, intended pattern for dev secrets routes them through a guarded resolver that refuses to use them in production. The rule saw the literal and screamed, not understanding that this literal was the safe, guarded case.
So the linter's very first real run produced a false positive in roughly eleven of my products at once — every repo that used the guarded pattern, which is to say every repo doing it right.
That's the nightmare version of a fleet-wide rollout: a tool that's confidently, uniformly wrong. If those false alarms had landed in a dozen repos on the same afternoon, the rational response would have been to distrust the linter and turn it off — killing the whole effort. Instead, the pilot showed me the rule needed an exception for the guarded pattern before anyone else saw a single false alarm.
What I'd have lost by skipping the pilot
I want to be honest about this part: the "dozen simultaneous red pipelines" is a thing that didn't happen, so there's no log of it. It's a projection, not an incident.
But it's a well-supported projection. Both blockers came from structure shared by every consumer — the dependency layout and the guarded pattern — so both would have fired everywhere. The pilot didn't get lucky with two repos; it exercised the two conditions that made the failures universal. That's the difference between "I tested it" and "I tested the thing that varies."
The method, generalized
- Unknown blockers live in the consumer, not the tool. A shared tool's own green suite doesn't cover the environments it will run in. Assume the interesting failures are downstream.
- Blast radius = consumers × shared structure. If a failure comes from something all consumers share, fanning out multiplies it. Sequence it instead.
- Pick the pilot to span the variation axis. Two repos that differ where breakage is likely beat ten repos that are all the same. Diversity, not count.
- Leave a stop valve. Every rollout step — human or agent — should be allowed to halt and escalate when a premise looks wrong, instead of pushing through.
There was a bonus, too. Once the linter was correct and did fan out, it immediately surfaced a genuine latent auth gap in one repo that everyone had walked past. The tool I built to prevent drift turned out to be a decent detector of the drift already there. But it only earned that trust because it didn't cry wolf a dozen times on day one.
When you roll a change across many repos, what's your pilot — and how do you choose which repos go first?
── Hideyuki Mori (Ayane International) 🔗 hideyuki-mori.com
Top comments (0)