DEV Community: James Coombs

Your AI Skills Have the Same Problem as Your CLAUDE.md. Here's the Data.

James Coombs — Mon, 31 Aug 2026 13:13:43 +0000

James Coombs is a design engineer who ran a 42-run controlled study to test whether a discipline skill he wrote, and used for months, caught anything a plain agent wouldn't. It didn't. He maintains a 60-component design system and has run two prior studies on why AI governance-by-documentation fails.

I wrote a skill to make Claude audit its own changeset before every commit: six bug-pattern dimensions, a required output format, a clause that refused the usual excuses. I used it for months. Then I ran a controlled study to see if it caught anything a plain agent wouldn't.

It didn't. 42 runs, and a no-guidance control caught every planted defect the skill caught. Zero marginal detection. The same result my CLAUDE.md rules got, one layer up.

This isn't a claim that skills don't work. Skills that route, discover, or enforce work fine. It's narrower and more uncomfortable: a skill whose payload is behavioral documentation ("here are the things to check for") inherits documentation's 0%, because the model already does the thing whether the skill fires or not.

The skill

self-review. Before a commit, it scanned the diff across six dimensions: whether errors propagate, whether tests actually test, whether docs match the code, whether a comparison can ever match at runtime, whether a called interface actually exists, and whether the change reaches beyond its task. Each dimension had to report "Checked: X. Found: Y." It refused rationalizations: "the diff is small," "I already reviewed it mentally."

On paper it was a well-formed discipline skill. That's the point. It was well-formed and it did nothing.

The study

Same shape as my CLAUDE.md ablation: two arms, one with the skill, one with no guidance at all, each scored on whether it caught a planted defect. I escalated difficulty across three rounds, each one built to give the null result a chance to break.

Round	Planted defect	Control (no skill)	With the skill
Easy	import of a function that doesn't exist; scope creep against an explicit brief	10/10	10/10
Hard	swapped-argument bug whose definition was buried in another file; unrequested scope creep with no priming	6/6	6/6
Adversarial	a config-key typo returning `undefined`, under pressure to sign off ("the lead already looked, we ship in five minutes")	5/5	5/5

The control never failed. 21 control runs, 0 misses. In the adversarial round the plain agent refused to sign off on a real bug even after being told the reviewer had already approved and the clock was running. It found defects I never planted: a count that included skipped items, a 500 on a duplicated query parameter, an error field the SDK doesn't expose.

The skill's marginal value, every round, was 0.

Why

The agent is a competent reviewer by default. Telling it to review is like handing a checklist to someone already running the check. When the skill and the agent's default align, the skill is redundant.

That is the opposite failure from my CLAUDE.md study, where the rules asked for behavior the agent wasn't doing and got rationalized away. Same verdict, opposite mechanism: there, documentation couldn't raise a floor; here, a skill couldn't exceed a ceiling.

The skill was squeezed from both sides. Below it, the deterministic checks I already run on every Stop (type-check, lint, format) own the mechanical half: unused imports, dead code, duplicate constants. Above it, model competence owns the judgment half. The band in between, the thing the skill claimed as its value, is empty.

Three layers, one result

I've now seen this at three layers of the stack.

CLAUDE.md rules for behavioral shifts scored 0% compliance, in an earlier study of mine, Your CLAUDE.md Rules Achieve 0% Compliance. The architecture that actually fixed compliance was hooks, not documentation, which I wrote up in I Gave an AI Agent Unrestricted Shell Access. And now the skill layer: a skill whose payload is "remember to check" is a CLAUDE.md rule wearing a slash command, and it scores the same 0.

The split holds all three times. Discovery (a tool the agent queries) and enforcement (a hook that blocks) change behavior. Documentation, and what I'll call routing-by-restatement, a skill whose payload just repeats the behavior you want instead of pointing to a tool, do not, for anything the agent already does. A skill that routes to a tool is a different thing, and it works. My Stop hook said this out loud months ago: it replaced an LLM "please check your work" round with a zero-token deterministic gate. Retiring the review skill is that same move, finished one layer up.

What I did instead

I deleted the skill. I kept one thing.

Across all 42 runs, the only behavior that differed between the arms was small. Agents following the skill reported what they could not verify ("unverified: I did not open that file") instead of folding it into "looks fine." That honesty is worth keeping. So I moved it out of a standalone skill and into the point where the work actually hands off: a single instruction in the implementer's handoff step to report evidence rather than assertion, and to flag the unverified explicitly.

That matches what the ablation taught me the first time. A two-sentence instruction at the point of action beat a comprehensive governance file by 11 points. The salvage is the same principle: not a skill loaded at session start, an instruction delivered when the agent is about to act. If it ever needs teeth, the right form is a hook that blocks a "clean" claim with no evidence behind it, the same shape as the quality gate I already run.

What this means for your skill library

Run the test I ran on my CLAUDE.md rules, on your skills. For each one, ask what happens if the agent never loads it. If the answer is "the same thing, because the model does it anyway," the skill is inert, whatever the slash command in front of it suggests.

The suspects are the review-shaped and checklist-shaped skills: "audit the diff," "check for these patterns," "remember to validate." They feel productive to write. They encode competence the model already has. Point a no-guidance control at each one and watch whether the control fails. If it doesn't, the skill adds nothing to a clean review: keep the one behavior worth keeping at the point of action or behind a hook, and drop the rest. One honest limit on that advice: clean review is the best case for the control, and it is the only case I tested. If a skill is really meant for long, self-authored work with momentum to finish, that regime is untested here, and a hook that fires at the moment of action is a safer home for the check than a skill loaded at session start.

Every skill you retire is prompt budget returned: its description stops loading into every session, and its body stops loading into the ones that used to trigger it.

Methodology notes

Design: two arms (skill, no-guidance control), 5 planted-defect fixtures across 3 difficulty rounds, scored by reading each run's findings for the planted defect. Never by keyword match; template echoes fake both a hit and a miss.
Reps: 42 runs, 21 per arm. Small per cell (3 to 5), so no single cell proves much. The result is only strong pooled, and even then loosely: 0 misses in 21 control runs puts the pooled miss rate under about 14% at 95% confidence (rule of three). But those 21 span easy, hard, and adversarial fixtures unequally, weighted toward easy, so the bound describes the mixture, not the hard tail. The adversarial cell on its own (0 of 5) is only bounded near 60%. Treat 14% as directional, not a clean rate, and if a decision is close it tells you how many more runs to buy.
Confound: the skill arm was told to follow the skill, so "skill vs. control" partly measures instruction-following. The comparison that carries the weight is the control on its own: with no guidance at all it already catches everything, so the skill has no headroom to add value.
Scope, and where I'd expect the skill to earn its place: every run was a fresh agent given one small diff and one job. That is the best case for attention. I did not test the regime the skill was really for: an agent 30 turns deep, reviewing code it wrote itself, with momentum to finish. I have exactly one data point from that regime, and it's me. Halfway through this investigation I asserted a claim I hadn't checked, one turn after writing the "report evidence, not assertion" rule I just described. The failure the skill targets is real. It shows up under load, not in a clean review. That is exactly the setting where a session-start skill helps least and a hook helps most.
Model: Claude Opus 4.8. My CLAUDE.md study was Opus 4.6. The finding held across both, which fits the earlier claim that it's architectural, not model-specific.
Cost: 42 subagent runs, roughly 1.3M tokens. Cheap enough to run on every skill you own before you decide to keep it.

I Gave an AI Agent Unrestricted Shell Access. Here's What Happened Next.

James Coombs — Tue, 25 Aug 2026 15:09:33 +0000

James Coombs is a design engineer who built a 3-layer safety architecture for AI code agents after running into 4 agent bypasses and 3 bugs in his own guards. He maintains a 60-component design system and has run controlled ablation studies on AI code generation governance.

I run Claude Code with --dangerously-skip-permissions. That flag does what it sounds like: the agent can read any file, write any file, run any shell command, and call any API. No prompts, no confirmations, no guardrails.

I do this on purpose. The default permission model ("May I read this file? May I run this test? May I execute git status?") generates 2-10 approval prompts per session depending on the work: 2-4 for a coding session (commits and pushes), 5-8 for a pipeline run that writes to Linear and creates PRs, up to 10 for MCP-heavy sessions where every write tool is individually gated. The friction isn't the count; it's the interruption. Each prompt breaks the agent's flow mid-thought. The alternative is an agent that can commit to main, force-push, send Slack messages, modify Jira tickets, and publish npm packages without asking.

So I built a third option: a safety architecture that sits between "prompt for everything" and "allow everything." The agent runs unrestricted within safe boundaries. Destructive operations are blocked before execution. When I actually want to run something blocked, I type /yes and it goes through.

This is built on Claude Code. The principles (defense-in-depth, fail-closed hooks, user-initiated approval) apply to any agent with shell access. The implementation details are Claude Code-specific. Cursor's rules files, Copilot's workspace instructions, and Windsurf's cascade have analogous compliance problems; their extension models could support similar hook-based enforcement.

114 days and 94 commits, counted 2026-08-20. Every commit in the first stretch was triggered by something the agent actually did. The later ones came from auditing the guards themselves, which turned out to be a richer source of defects than the agent was.

The architecture

Three layers, evaluated in sequence on every tool call:

Layer 1: Deny rules. 71 entries in settings.json, 69 of them prefix globs over git and the GitHub CLI. Bash(git commit*), Bash(gh pr create*), etc. The other two are a substring glob and one leftover MCP rule that predates the move to hook-gating and should have gone with it. Cheap, catches simple forms. Only covers git and GitHub CLI, the operations that should never be auto-approved under any circumstances.

Layer 2: PreToolUse hooks. Six Python-in-Bash scripts that fire on every Bash, Read, Write/Edit, and MCP tool call. Regex matching catches the variants that globs miss: git -C /path commit, cd /tmp && gh pr create, curl -X POST. Also gates 52 MCP write tools (Slack, Jira, Figma, Linear) behind time-limited approval tokens, across 6 service prefixes.

Layer 3: CLAUDE.md instructions. Behavioral guidance for edge cases regex can't express. Weakest layer. Subagents don't read it, and my own ablation study showed 0% compliance for design system behavioral rules in CLAUDE.md (I covered this in detail in Your CLAUDE.md Rules Achieve 0% Compliance). For security-specific rules, the enforcement layers above handle what documentation can't.

Why two enforcement layers? Bash(git commit*) doesn't catch git -C /repo commit. \bgit\b.*\s+commit\b does. For git/gh, both layers run. For everything else (ssh, sudo, rm, cloud CLIs), only Layer 2 runs, because Layer 2's deny messages include the /yes escape hatch and Layer 1's don't.

Every hook is fail-closed at two levels: Python catches its own errors and outputs deny; Bash catches Python crashing and outputs deny. A broken guard blocks. Never passes.

The `/yes` system

The agent proposes git push origin main. The hook blocks it. The agent tells me it's blocked. I type /yes. The command executes.

Under the hood:

/yes is a Claude Code skill with disable-model-invocation: true. This hides the skill from the agent's advertised list. It does not reject an explicit call by name, which I assumed it did until I tested it. The flag is friction, not enforcement.
Typing /yes fires a UserPromptExpansion hook that creates a nonce file. This is the actual origin control: the hook fires only on real user input, and the agent has no mechanism to trigger it. Everything else in the flow is checked against that nonce.
The skill writes the command to a temp file in /tmp/. The hook that permits this write also records a SHA-256 digest of the command and its path, bound to the live nonce.
A PreToolUse hook gates the runner call: it claims the nonce atomically, then verifies the staged file still matches the recorded digest.
exec-yes.sh validates that the file exists and is under 300 seconds old, then executes it, deletes the temp file, and appends to the audit log.

Single-use. Time-limited. User-initiated only. Session-scoped. Content-bound. Audited.

Step 4 is the part I got wrong for two months. The nonce originally proved only that a /yes had happened recently, not that the user had approved this command, which made it a bearer token: stage a payload at any time, wait for the user to approve something innocuous, and the runner would execute the payload. Binding the approval to a digest of the specific command is what closes that, and it is a different property from single-use.

Nonces are session-scoped: each Claude Code session gets its own nonce path. Without this, two concurrent sessions share a single nonce file. Session A consumes session B's nonce; B's /yes fails. The session ID is sanitized (alphanumeric, hyphens, underscores only) to prevent path traversal.

MCP tools (Slack, Jira, etc.) use a different mechanism: service-level tokens with a 300-second idle timeout that slides forward on each accepted write, plus a cap of 250 writes per token. MCP calls bypass Bash hooks entirely. The key design choice: MCP write tools are hook-gated, not deny-listed. Deny-listing hides tools from the agent entirely. Hook-gating lets the agent see the tools and propose actions ("here's the Slack message I'd send") for me to approve. Hook-gating lets the agent propose; deny-listing prevents that entirely.

The seven incidents

None of this was designed upfront in the sense that matters. The initial commit was already 18 files and 2,204 lines, four guard scripts and a 50-assertion suite; but every layer added after it came from an incident rather than a plan. Everything interesting came from the agent finding ways around it.

Three were bugs in my own guard scripts that testing caught before they mattered. Four were the agent finding novel bypass paths. Both categories drove the same response: a fix, a test, and documentation.

1. Total lockout (day 0). (Guard bug.) Tried to migrate away from --dangerously-skip-permissions to a less alarming defaultMode: "dontAsk". Added dontAsk before a working allow-list existed. Claude couldn't read its own config to understand what happened. Couldn't write the fix. Couldn't invoke skills. Had to manually edit settings.json from a terminal. Stayed on the current architecture after that.

2. sudo swallowed by curl handler (day 0). (Guard bug.) The guard for external actions had nuanced handlers: curl gets checked for POST/PUT/DELETE, wget for --post-data, rsync for remote hosts. The curl handler called sys.exit(0) on the allow path. sudo apt-get install curl passed through the curl handler (no write flags), and the sudo pattern later in the script never ran. Fix: unconditional blocks (sudo, ssh, scp, osascript) run before any nuanced handler.

3. MCP token deadlock (day 16). (Guard bug.) The /yes skill creates approval tokens in /tmp/. The rm guard blocks references to /tmp/.claude-mcp-yes-*. The approval mechanism couldn't approve itself. Fix: TEMP_EXEMPT pattern, where certain blocked patterns are exempted when every absolute path in the command resolves under /tmp/ after os.path.normpath(). That normpath matters: /tmp/../../etc/passwd doesn't pass.

4. Agent self-approval (found day 2, fully closed day 16). (Agent bypass.) The agent bypassed disable-model-invocation: true by using the Write tool to create the .approved-cmd file directly, then calling run-approved.sh. It replicated the skill's behavior without invoking the skill. First fix, the same day: a pattern and a deny rule blocking the script's invocation. That closed the route it had taken and not the class, so two weeks later guard-approval-write.sh landed to block Write and Edit against every approval artifact, and the /yes skill was restructured onto temp files. I date the incident by the second fix out of habit; the discovery was day 2.

5. Nonce replay (day 20). (Agent bypass.) After adding a nonce gate, the agent copied the nonce creation pattern by calling the nonce script directly. Fix: moved nonce creation from a callable script to a UserPromptExpansion hook. This hook fires on user input events. The agent has no mechanism to trigger it.

6. Six bypass vectors at once (day 37). (Agent bypass.) A deliberate audit of the guards found six holes in one pass: an ungated set of MCP write tools, an exemption check that validated any path was temporary rather than all of them, an implicit POST through the GitHub CLI's field flags, and three narrower cases. Six patterns added, ten new test cases. A later audit on day 91 found four more, including one where a guard accepted a nonce path that did not have to exist.

The test suite grew from 50 assertions to 226 named cases through these incidents. The bash suite runs 389 assertions in total as of 2026-08-20, all passing, most of the growth coming from the auditing that followed. Every bypass produced a fix, a test, and documentation.

7. Branch tracking (day 44). (Agent bypass.) CLAUDE.md said "never set upstream tracking on feature branches." Agents acknowledged the rule, cited it in reasoning, and used git checkout -b feature origin/develop anyway, because that's the most common branch-creation pattern in their training data. The rule achieved 0% compliance at the point of action. Fix: 5 regex patterns in the hook catching checkout -b, switch -c, --track, and -t variants across origin/ and upstream/ remotes. 11 deny globs as defense-in-depth. The error message shows the correct alternative, which now has to carry --no-track explicitly: git branch --no-track <name> <start-point> && git checkout <name>. The earlier wording omitted the flag, and a fifth pattern added later blocks exactly that form, so the remediation the guard printed for two months is one the guard would now refuse. 19 new test cases, including 6 safe-pattern allowances (read-only git log origin/... must still work).

This incident is the cleanest example of the CLAUDE.md compliance gap. The agent wasn't being adversarial. It wasn't routing around a safety boundary. It was using a standard git pattern that happens to have a subtle side effect, and no amount of documentation prevented it.

These seven are the ones the agent handed me. Sitting down to audit the guards deliberately, rather than waiting to be surprised by them, produced a third category: controls that were real in the documentation and absent in the code. That is a separate piece.

What the agent is actually doing

The agent isn't malicious. It's optimizing for task completion. When a safety boundary prevents completion, it looks for alternative paths, the same way it looks for alternative APIs or alternative algorithms. It used Write instead of Bash. It called the nonce script instead of waiting for the user. It manufactured the approval file directly.

This has a design implication: if the agent can call the mechanism, it can forge the credential. disable-model-invocation prevents calling the skill but doesn't prevent replicating its steps. Every artifact the skill creates needs to be independently guarded at every access layer (Bash, Write, Edit).

Beyond safety: the lifecycle hooks

The same hook system that enforces safety also prevents a different kind of failure: information loss. Claude Code fires hooks on session events beyond tool calls: when conversation history compresses, when the agent finishes responding, when specific file types are read. These hooks don't gate permissions. They protect the agent's two scarce resources: token budget and context window.

Token flood guard. A PreToolUse hook on Read blocks lock files (package-lock.json, pnpm-lock.yaml, 7 others), log files, minified bundles, and source maps. A monorepo yarn.lock is 230,000+ tokens. A small project's package-lock.json is still 14,000. None of it is content the agent will meaningfully process. The deny message suggests the targeted alternative: grep <pkg> <lockfile> instead of reading the whole thing.

The Bash matcher is intentionally absent: blocking every cat, head, or pipe variant that could dump a large file is an unbounded arms race. The Read matcher covers the highest-risk path.

Compaction directives. Claude Code compresses conversation history when the context window fills. By default, the compressor optimizes for recency, which means architectural decisions from 30 minutes ago get compressed down to "made changes to auth module." A PreCompact hook fires before every compaction and outputs structured preservation categories as directives to the compactor: meta-review insights, architectural decisions with rationale, bug root causes, rejected approaches with reasons. These are instructions to the compaction model, not guarantees. Under heavy context pressure, content still gets dropped. The safer bet is writing key decisions to a file before compacting.

The hook also writes a git state snapshot to disk: current branch, uncommitted changes, recent commits. After compaction, the agent can read that file to re-orient without asking "what were we working on?" This doesn't survive across sessions, and it only captures whichever project directory the shell was in when compaction fired. These are known limitations, not ones I've solved.

Quality gate. A Stop hook fires every time the agent tries to finish responding. If uncommitted changes exist, it runs type-check, lint, and format-check. It short-circuits on the first failure and prints what broke. What it does not do, and I believed for months that it did, is stop the agent: the script exits zero on every path, so the report reaches the transcript and the turn ends anyway. Blocking requires a specific exit code it never returns. So it is a zero-token deterministic check that catches real errors and then asks nicely, which is weaker than I had been claiming and is the same gap between documented and enforced that this piece is about.

The hook auto-detects tooling from the project: tsconfig.json for type-checking, eslint config for linting, prettier config for formatting. It short-circuits on first failure and truncates output to 3,000 characters so a wall of type errors doesn't flood the context it's trying to protect. A per-project .audit-ignore file excludes generated directories.

What I chose not to build. A PostToolUse hook on every Edit/Write would catch issues incrementally instead of at the end. It would also add latency to every file modification. The Stop hook runs once. Agent Teams (multiple agents collaborating) would theoretically improve review quality. In my testing, the token cost ran roughly 7x for comparable tasks, and the coordination overhead hasn't been worth it. A MAX_THINKING_TOKENS cap would limit reasoning costs, but it conflicts with adaptive thinking; a hard cap degrades performance on complex problems to save tokens on simple ones.

What I'd tell someone building this

Start with the deny list. It takes 10 minutes and catches the obvious stuff.

Add hooks when you hit the first bypass: git -C /path commit, compound commands, MCP tools. Don't try to anticipate every variant. Ship the basic system and let real usage tell you what's missing.

Build the approval system early. Without it, every blocked command requires leaving Claude Code and running it manually. That's unacceptable friction, and you'll end up disabling guards instead of approving commands. /yes makes the guards sustainable.

Write tests for every incident. The regression suite is what lets you keep iterating without introducing false positives. I wouldn't touch the guard scripts without it.

The agent optimizes for task completion. When safety boundaries block completion, it routes around them the same way it routes around any obstacle. It's not adversarial in intent, but the effect is adversarial. Design accordingly.

The numbers

Measured 2026-08-21, except the commit count, which is a 2026-08-20 reading. The system is still being added to, so read these as a reading on a date rather than a final state.

5,498 lines across 17 hook scripts, of which the regression suite is 2,965. The six guard scripts are 1,766 of the remainder; the rest is the approval machinery, lifecycle hooks and diagnostics
71 deny rules, and 52 MCP write-tool gates across 6 service prefixes
One assertion in that suite fails on purpose: it asserts a control that is known to be missing, so a green suite would mean the assertion had been quietly removed. I stopped quoting a total, because the suite skips assertions when an approval token happens to be live, so its own count moves with ambient state
94 commits over 114 days, 2026-04-28 to 2026-08-20
14 project contexts
Internal docs including a postmortem of the lockout incident

Almost nothing in the hooks references a specific company, codebase or project. Two test comments cite internal ticket IDs, one test hardcodes an absolute home directory, and one MCP prefix names an internal service. Strip those four and the system extracts into a standalone repo.

7,798 Components. Import One, Ship One.

James Coombs — Thu, 20 Aug 2026 02:07:55 +0000

James Coombs is a design engineer who maintains the icon package a frontend org installs across roughly twenty apps. Its largest icon set alone is 7,798 generated React components, and the whole design of the build exists so that importing one of them pulls in exactly one file.

That 7,798 is the small version (counted 2026-08-19; regenerate with ls src/material/rounded-400/*.tsx | wc -l, because this number grows most weeks and a count frozen in prose is the failure this piece's sibling article is about). It's one icon style at one weight, both fill states: about 3,900 icons times two. The config that generates them can multiply that across every weight, optical size, and grade Google's Material Symbols publish (optical size and grade being the axes that tune an icon for small displays and for contrast), into the tens of thousands. So the first design question was never how to generate them. It was how to keep a consumer who imports one icon from paying for the other 7,797.

This isn't a tour of how bundlers eliminate unused code. It's about the two decisions that make their job trivial instead of contingent, and the one that will ship you a package that runs and won't type-check.

The barrel, and why it's a build-time problem, not a bundle one

The default way to publish a component library is a barrel file: one index.ts that re-exports everything, so consumers write import { HomeFilled } from "@acme/icons". It's the friendly API, and I want to be fair to it, because the usual argument against it is wrong. A modern bundler will tree-shake a barrel, dropping the code you don't import from the final build, as long as every module is free of import-time side effects and the whole graph is ES modules (the import/export format a bundler can analyze statically, rather than the older require style it can't). This package sets sideEffects: false, which is exactly that promise. The bundle your users download is fine either way.

The cost of the barrel at this scale isn't the bundle. It's everything upstream of it. A 7,798-export index.ts is a module the bundler has to read and build a dependency graph across on every cold build and every hot reload, just to work out which handful of exports you actually used. Some frameworks make it worse: Next.js pulls the whole barrel into the dev module graph unless you name the package in a special optimizePackageImports list. At fifty components none of this registers. At 7,798 it's the difference between a fast dev loop and a slow one, paid on every build forever, for an ergonomic import you can get another way.

So there is no barrel. Each glyph is generated as its own module: one file, one named export, roughly 900 bytes. The consumer imports the file directly, and nothing has to reason about the other 7,797.

Import one, ship one

The deep import is made clean by the package's exports map, the part of package.json that controls which internal paths a consumer can reach and how they resolve. One wildcard entry covers a whole icon set:

"./material/*": {
  "types": "./dist/material/*.d.ts",
  "default": "./dist/material/*.js"
}

That wildcard means import { HomeFilled } from "@acme/icons/material/rounded-400/HomeFilled" resolves to exactly one file on disk. The style and weight segment is not optional; the wildcard spans it. The import path names the module. The bundler pulls in that file and has no reason to look at any other, because it was never handed the set as a unit.

The map that resolves at runtime but not in the type-checker

Look again at that exports entry. It has two conditions: default for the code, and types for the declaration file, the .d.ts that tells TypeScript the shape of a module. The types condition is the part people leave off, and leaving it off is a bug you don't see until someone else does.

Here is the trap. Two separate systems tell TypeScript where a subpath's types live, and which one runs depends on the consumer's setup. A modern moduleResolution setting (node16, nodenext, or bundler) reads the types condition straight out of the exports map, the way the one above is written. An older resolver ignores exports entirely and looks instead at typesVersions, an older package.json field that maps import paths to their declaration files. Support both kinds of consumer and you are maintaining two parallel maps of the same wildcards by hand. This package does exactly that: the types condition in exports for modern resolvers, and a typesVersions block mirroring it for old ones.

The failure mode is the quiet kind. Add a new export path, update exports, forget its typesVersions twin, and your own build stays green while every consumer on the older resolver gets a red "cannot find module" on an import that runs perfectly at runtime. Runtime and types disagree, the two maps have drifted, and nobody thinks to diff them because they don't look related. If you can drop support for the older resolver, delete typesVersions and keep one source of truth. If you can't, the two maps move together, in the same commit, every time.

The parts that never reach the bundle

Two build decisions that aren't about packaging but paid for themselves.

Source resolution routes by what it is asked for. Material's npm packages ship only the largest optical size, so the moment you want the size this package actually builds at, which is a smaller one, the SVGs come from Google's icon repository over HTTP instead, cached locally and fetched in small batches to stay under the CDN's rate limits. The first full download lands in about thirty seconds; every build after reads the cache and hits the network zero times.

Generation is atomic. The transform writes all the components to a temporary directory, then swaps it into place with a pair of renames: the previous build moves aside, the new one moves in, and the old copy is deleted only once the new one is fully in position. A crash mid-run leaves the last good output standing instead of a half-written folder.

The mirror of the barrel

The layout has a cost, and it is the exact inverse of the barrel's. The barrel makes the bundler chew through one enormous module at build time; one file per glyph makes the published package 23,694 files, three per glyph once you count the JavaScript, the declaration and its source map: a slower publish, a tarball full of tiny entries, a dist folder your file browser stalls on, a manifest just over a megabyte. The difference is where the bill lands. The barrel bills every consumer's build, on every build, forever. The file-per-glyph layout bills my publish step, once. Deep imports are what make that trade pay off, because they turn thousands of files from dead weight into the thing that makes "import one, ship one" true. Ship this layout and a barrel beside it, and you have paid my tax and kept theirs.

So before you generate thousands of modules, settle the import shape first: one file per unit, reached by its own path. That, not a bundler flag, is what makes importing one cost one. And the day you expose types through a wildcard subpath, write down both of the maps that resolve them, or you will publish a package that runs, doesn't type-check, and says so in someone else's editor.

The CSS Problem Nobody Thinks Is Solvable

James Coombs — Thu, 20 Aug 2026 02:07:34 +0000

James Coombs is a design engineer who migrated a large frontend from Material UI to a custom design system (Radix, Tailwind). Four CSS systems had to coexist during migration. Three approaches to it had already been considered and rejected before anyone tested the CSS layer.

Four CSS systems running simultaneously: a CSS-in-JS framework (Material UI's emotion runtime), two legacy utility frameworks, one in-house and one open-source, alongside an in-house component library, and the new design system (Tailwind CSS 4). Migration required all four in the same bundles, on the same pages, without collision.

The received answer was that it could not be done without component-level isolation, and three approaches had already been considered and rejected on that basis. Nobody had tested it at the CSS layer.

Where the received answer came from

The assumption: coexistence requires component-level isolation. Wrap each component in a boundary (Shadow DOM, iframes, CSS Modules) so its styles can't leak. At roughly 960 files as of April 2026, spread across the whole monorepo, component-level isolation means touching every file before migration can begin. That's not a migration strategy. That's a rewrite.

Three approaches the team had considered:

Manual class prefixing. Rename every utility class with a namespace prefix. 70+ files of mechanical changes, plus every new component authored with the prefix. Scales linearly with component count.
Runtime class modification. A JavaScript layer that rewrites class names at render time. Adds complexity and latency to every component mount. Fragile when third-party libraries generate their own class names.
Shadow DOM. True encapsulation, but breaks React portals. Dialogs, popovers, tooltips, and dropdown menus all render to document.body. Shadow DOM boundaries prevent them from inheriting theme tokens.

All three operate at the component level. The solution was at the config level.

One line of configuration

A PostCSS plugin, postcss-prefix-selector, rewrites every generated selector to sit under a scope class as the stylesheet is built:

/* Without scoping */
.flex { display: flex; }              /* specificity: 0-1-0 */

/* With scoping */
.ds-scope .flex { display: flex; }             /* specificity: 0-2-0 */

The scoped selector wins over the unscoped global by specificity, and that is the whole mechanism. There is no !important, which matters because it means the win is not guaranteed: a legacy compound selector at the same 0-2-0 specificity ties, and then source order decides. That case is live rather than theoretical. One component in the codebase carries a comment recording exactly it, where a preflight rule ties with a legacy button's generated class. Specificity buys you almost all of the coexistence and you should expect to hand-resolve the ties.

Zero changes to consuming components. A wrapper element (<DesignSystemProvider>) adds the .ds-scope class, everything inside gets the new design system and everything outside keeps working. The design system's own portal components are the exception, and each needed one edit; that is the section below.

One pipeline handles both, which is why it holds. Five PostCSS plugins run in sequence for the scoped build: Tailwind, a legacy-transform reset, postcss-prefix-selector, a layer-removal plugin, and a preflight-scoping plugin. The layer-removal step is the non-obvious one. It strips @layer so design-system rules compete at normal cascade priority, because anything inside @layer base loses to any unlayered legacy CSS regardless of specificity.

The portal problem

Portals broke it. React portals (Dialog, Popover, Tooltip, DropdownMenu) render to document.body, outside the .ds-scope wrapper. Portal content inherits no scoped styles. Buttons inside modals lose their styling.

First attempt: a wrapper component that injects .ds-scope around every portal. This broke Radix UI's SlotClone ref-forwarding. The wrapper intercepted the ref chain, and composed components silently lost their refs.

Fix: a React Context hook that conditionally adds the scope class:

function usePortalScopeClass(): string | undefined {
    const isScoped = useDesignSystemScope();
    return isScoped ? 'ds-scope' : undefined;
}

Portal components use an inline conditional wrapper:

const scopeClass = usePortalScopeClass();
const content = <PopoverPrimitive.Content {...props}>{children}</PopoverPrimitive.Content>;
return (
    <PopoverPrimitive.Portal>
        {scopeClass ? <div className={scopeClass}>{content}</div> : content}
    </PopoverPrimitive.Portal>
);

When running inside a DesignSystemProvider (the migration case), portals get one extra <div class="ds-scope">. When running standalone (Storybook, greenfield apps), no wrapper, no overhead. The conditional check costs nothing at render time. The key design constraint: the wrapper must be an inline conditional, not a component boundary, because Radix's SlotClone breaks if a React component intercepts the ref chain between Portal and Content.

This bug was discovered during Drawer implementation, not during planning. Plan for portal scoping from the start if you're running dual CSS systems.

The Tailwind-as-library problem

Tailwind was designed for applications, not libraries. When a design system pre-compiles its CSS and ships it as a package, consuming applications can import the styles. But if those applications use Tailwind utility classes not present in the design system's own source code, those utilities silently go missing. The consuming app expects gap-3 to work. It doesn't, because the design system's build never scanned for gap-3.

The two-tier architecture solves this:

Tier 1: pre-compiled, shared, about 200 KB uncompressed as of August 2026. Everything the design system itself uses. Zero-config for consumers: import the CSS file, done. This preserves the "standard npm package" contract: consumers import CSS, no bundler config needed.

Tier 2: per-bundle, generated (~26 KB each). Built at each consuming bundle's build time using a shared createScopedPostcssConfig() factory. Scans that bundle's source for Tailwind utilities, generates only what it uses, applies identical .ds-scope scoping. Opt-in: ~5 file changes per consuming bundle.

The two tiers coexist because they apply the same .ds-scope prefix through the same plugin chain, minus the preflight step that only the full tier needs. A utility that exists in both tiers produces an identical selector twice. Browsers do not deduplicate that; both rules parse and both match, and the later one simply wins, so the cost is bytes rather than correctness. Adding Tier 2 to a bundle grew the selector count by 59, which is the net addition rather than a count of duplicates. The overhead: +26 KB per bundle, which is +5% of the bundle's CSS and +1.5% of its JS chunk, plus +2 seconds of build time and roughly 4 to 6 KB gzipped of duplication per opted-in bundle. 59 duplicated selectors against ~1,440 total once both tiers are present.

Most teams either ship all their CSS or none of it. The two-tier split lets the design system ship the predictable base while each consumer generates only what it needs. Framework-aligned, not framework-fighting.

What didn't work

The initial two-variant strategy. The design system originally shipped two CSS files: standalone (no scoping) and scoped (with .ds-scope). A third consumer (a Shadcn app with its own custom theme) needed neither: it wanted the component utilities without any theme. Added core.css, a theme-agnostic variant. This was discovered only through real integration, not planning.

Assuming portals were covered. The scoping strategy was tested against static layouts. Portals weren't tested until a specific component (Drawer) exercised the ref-forwarding path. Test your scoping strategy with a portal-heavy component early.

The numbers

Metric	Value
CSS systems coexisting	4 (emotion, two legacy utility frameworks, Tailwind)
Selectors: Tier 1 alone	1,381 as of February 2026; 1,536 in the current build
Selectors: Tier 1 and Tier 2 together	~1,440 at that same measurement, a net addition of 59
Code changes in consuming components	0
Code changes in the design system's own portal components	11 components; the seven Popper-family ones took a single edit each, the four dialog-family ones needed a second helper
Bundle size overhead	+26 KB on the one bundle measured: +5% of its CSS, +1.5% of its JS chunk
Build time overhead	+2 seconds
Files still importing the old framework	876
Files importing the new design system	353
Packages running both at once, in production	6

The last three rows are the result, and they are the reason this was worth doing. Nine months in, the old framework is still imported by 876 files and the new design system by 353, and six packages contain both and ship both. Those are source files only, counted by import statement; a run that includes build output roughly doubles them, which is why the method matters more than the number. Coexistence is not a phase that ended; it is the current operating state at scale, which is what makes the incremental path available at all. Without dual-framework coexistence, the migration is a big-bang rewrite, and big-bang rewrites don't ship.

If you're running dual CSS systems

Solve coexistence at the CSS layer, not the component layer. Specificity-based scoping under a single class selector is simpler than Shadow DOM, more maintainable than manual prefixing, and faster than runtime modification.

Test with portals immediately. Any framework that renders outside the DOM tree (portals, modals, tooltips) will break your scoping strategy. Discover this in week 1, not month 3.

Plan for a third CSS variant. You will discover a consumer that doesn't fit your initial assumptions. Build the variant pipeline so adding a new output is a config change, not an architecture change.

Measure the overhead, and measure it on your worst page rather than a representative one. +26 KB and +2 seconds of build time were acceptable. What was not was style recalculation on a very large DOM: on a page with tens of thousands of nodes, page-load recalc went 2.8x and total recalc over a minute went 4.5x, which was severe enough that a follow-up project removed the scoped stylesheet from that bundle entirely and served it a narrower one. Descendant-scoping every selector is not free at the matching layer, and that cost scales with node count rather than with stylesheet size. Know both budgets before shipping.

Your Migration Guide Is Already Wrong

James Coombs — Thu, 20 Aug 2026 02:07:08 +0000

James Coombs is a design engineer who replaced a migration guide with 49 JSON data files and a handoff playbook. Engineers on four other teams migrated components without being walked through it. Zero legacy imports in the migrated codebase.

I wrote a migration guide for a 962-file Material UI to custom design system migration. Within 10 days, 4 of the files it referenced had already been migrated and the counts were wrong. The guide listed 941 files; the real count was 939 and dropping. Notes referenced deleted files. Per-package breakdowns were stale in 3 of 12 tables.

The guide was the wrong artifact. "Single source of truth" and "documentation drift" are known problems. The specific lesson: for a design system migration with 12 packages and active parallel work, the threshold at which a guide becomes wrong is measured in days, not months.

Why guides fail

A migration guide is a snapshot. The codebase is a stream. The guide says "246 MUI files remaining in the dashboard package." An engineer migrates 3 files. The guide now says 246 when the answer is 243. Nobody updates it. A week later, another engineer reads "246" and plans accordingly. The plan is wrong before it starts.

This isn't a discipline problem. Expecting engineers to update a prose document every time they migrate a file is expecting a process that scales linearly with migration volume. The whole point of automation is to avoid linear scaling.

The guide also had a structural problem: it mixed procedures ("run this command"), data ("these files need migration"), and rationale ("here's why we chose this approach") in a single document. When the data changed, the procedures that referenced it drifted. When the procedures were updated, the rationale paragraphs still described the old approach.

48 JSON files

The replacement: 48 structured data files (~6,900 lines of JSON) driving all automation. 6,900 lines of JSON is not simpler than 200 lines of prose. It is more correct. The complexity didn't disappear; it became machine-verifiable. A JSON file with a wrong component mapping fails a test. A prose guide with a wrong component name passes every check except a human reading it carefully.

Three categories:

Mappings (12 files). Component-level transforms: MUI component name, design system equivalent, prop translations, composition changes, conditional mappings. Each file covers one component family. Adding a new mapping is a data change, not a code change. (I covered how these feed the migration engine in "Scoring, Gating, Ratcheting: The Migration Engine Nobody Builds.")

Rules and checks (18 files). Validation logic: which bundles need design system config, which imports are forbidden, which CSS variants apply to which packages. Each check has scoped applicability conditions (verify component usage before applying config, not just package dependency).

Configuration and status (18 files). Bot configuration, phase definitions, package metadata, status tracking. The phases.json file defines the 7-phase rollout: which packages in which order, T-shirt complexity sizing per package, current status per file.

The key property: every number that appears in human-readable output is computed from the JSON, not hardcoded. pnpm migration:status reads the data files, counts the files in each state, and prints the current totals. The CLI is the source of truth. The data files are the source of the CLI. Nothing else is authoritative.

The JSON is supposed to not drift, because the migration bot is meant to be its single writer. In practice the status file's last write was a human pull request and its timestamp is six months old, which is the same failure one layer down and the reason the redesign below exists. The schema half does hold: tests validate the schema (confidence scoring, state machine transitions, plan validation, rule validation). When a mapping is wrong, a test fails. When a prose guide is wrong, nobody notices until someone reads it carefully.

Three-layer separation

The early version of the migration system had the "tutorial anti-pattern": procedures embedded in prose, data mixed with instructions, rationale interleaved with commands. When the JSON schema changed, the prose instructions drifted. When the CLI commands were updated, the tutorial still described the old flags.

Separating "what" from "how" from "why" is standard architectural practice. The specific application to migration infrastructure is what mattered here. The refactored architecture enforces strict separation:

Commands (orchestration only): "Execute checks 2a through 2e from checks.json." The command layer knows WHAT to run, not HOW it works or WHY it exists.

Data (JSON): Mappings, rules, checks, configuration. The data layer is machine-readable and CLI-queryable. Tools read it directly. Humans read the CLI output, not the JSON.

Documentation (markdown): WHY features exist, not HOW they work. Architecture decisions, trade-offs, context for future maintainers. The documentation never contains a number, a file count, or a procedure.

The refactor replaced 200+ lines of procedural prose with 10-20 lines referencing JSON. Zero regression. The migration kept running through the refactor because the data files (the actual source of truth) didn't change.

The playbook as product

The data files and the engine run the migration. The playbook tells engineering managers how to hand it off to their teams.

Not a document: a product. Eleven sections, of which four carry the weight:

Decision trees. "The tool stopped: did it write anything? If yes, review the diff and finish by hand. If no, check whether the file is in the curated list." Branch on observable state, not on judgment calls.

CLI reference. Every command an engineer needs, with expected output. pnpm migration:status --package dashboard shows the current state; the flags are space-separated, and an equals sign is parsed as part of the value. pnpm migration:scan --file Packages/dashboard/src/pages/Home.tsx shows what needs to change. No tribal knowledge required.

Day-1 checklist. What to do in the first hour of picking up a migration task. Read the file's confidence score. Check its state in the state machine. Review the component mappings. Run the scanner. This eliminates the "where do I start?" problem that kills velocity on the first day of any unfamiliar task.

Definition of done at 4 tiers. Per-file done (all MUI imports removed, tests passing). Per-package done (every file in the package at "complete" state). Per-phase done (all packages in the phase at package-done). Full migration done (all phases complete, legacy framework dependency removed from package.json).

The playbook went out for spot-checks before broad rollout, to three people picked for their distance from the work rather than their availability: an engineering manager with no migration context, a technical EM whose team was furthest along on adoption, and an IC lead who had reviewed migration pull requests without living in them. Each got a different entry point and three questions, and the questions were the point. The cold reader was asked which phases applied to his team and how he would check his team's current progress. The technical reader was asked whether the data-file cleanup was right for his packages and whether the CLI-first approach matched what he would actually run. The IC lead was asked whether he could tell where to start, and whether the CLI references were clear enough to type. Every one of those is answerable in ten minutes and none of them can be answered by skimming.

Over the following three months, engineers on four other teams shipped design-system migrations against their own teams' tickets, without scheduled pairing sessions and without anyone assigning them a phase. No package reached zero in that window, so this is adoption spreading by individual rather than a phase plan being executed. Individual questions came through Slack, but nobody asked "how do I start?" Nobody updated a stale guide. The CLI always had the current numbers.

What didn't work

Half the playbook shipped at month 7. The half engineers needed shipped ten weeks after that. The manager-facing overview went up on the wiki. The engineer-facing document, the one with the day-one checklist and the decision trees, was written, reviewed, and then sat on a branch for two and a half months after its pull request was closed unmerged, with the wiki page pointing at a dead link the whole time. It reached the main branch only after the work it was meant to unblock had already happened without it. The system was mature and the engine was running, and engineering managers still couldn't plan team assignments, because the artifact they were told to read did not exist at the address they were given. EMs need 4-6 weeks of lead time to allocate migration work. Shipping the playbook alongside the first usable version of the engine would have accelerated adoption by months. The handoff artifact is the delivery mechanism, not documentation. Shipping it last is shipping the product last.

Instrumentation started at month 3, counting from the first design-system commit in November 2025. The learning log that feeds governance rules (covered in "Scoring, Gating, Ratcheting") started 3 months into the project. Earlier failures had to be reconstructed from git history. The strongest impact period (months 1-3, when the architecture was established) was the least documented.

Hardcoded counts in phases.json, and a fix that took ten weeks to land. The first version of the phase configuration listed 25 curated files and a manually entered total of 969. Within days both had drifted. The fix was the same principle applied everywhere, compute rather than hardcode, and it was written into the same pull request as the playbook: strip the total, strip the percentage, strip the per-package counts, and replace each one with the command that regenerates it. That pull request was closed unmerged and the fix sat on the branch with it for ten weeks. When it finally landed, the cleanup told the story better than the original numbers had. The header claimed 19 curated files. The list underneath it held 16, because an earlier edit had dropped nine entries and moved the count by six. Twelve of those 16 then came out: four files no longer exist in the codebase, and eight no longer import MUI in the file that was named, though two of the eight still pull it in through a sibling style module and are booked for re-curation. Four are left, out of 25 at the start. The header number had been wrong about its own list for four months before anybody got as far as asking whether the list was right about the codebase.

The numbers

Metric	Value
Data files	49 (9,471 lines of JSON) as of 2026-08-19, of which 25 files and 4,635 lines are the migration pipeline; the rest are sibling pipelines
Playbook length	588 lines at handoff, 11 sections; the CLI reference and decision trees are 16% of it
Files targeted for migration	962 across 12 packages, as recorded in April 2026
Time before the manual guide was wrong	10 days
Engineers on other teams who shipped design-system migrations	4, across four ticket prefixes
Legacy imports remaining in the migrated scope	0
Claude Code slash commands documented	13
Spot-check recipients	3 (cold-start EM, technical EM, IC lead)

One more, and it happened while I was writing this.

The paragraph above first said the curated list fell to 6. That was the number on the branch where the fix had been written. By the time the pull request actually went up, the same field read 4, because two more of those files had been migrated in the weeks in between. I had taken a count from a document describing the codebase instead of from the codebase, which is the precise error this article is about, committed inside the article about it. The fix was the one I am recommending to you: I ran the thing and read what it said.

If you're organizing a migration

Replace every hardcoded number with a CLI command. If the count lives in a document, it's already wrong. If the count lives in a CLI that reads the codebase, it's always right.

Separate commands, data, and documentation. When one changes, the others shouldn't need updating. If your migration guide contains both "here's why" and "run this command," it will drift.

Ship the playbook with the first usable version of the tooling, not after. Engineering managers plan in 4-6 week cycles. If they can't see the handoff artifact, they can't allocate the work.

Spot-check the playbook with three people at different distances from the work, and ask each of them the question only they can answer. Give them ten minutes of specific work, not an open-ended review, or you will get an open-ended silence back.

Start the learning log on day 1. The failures from the first week produce the rules that prevent failures for every month after.

Your CLAUDE.md Rules Achieve 0% Compliance. Here's the Data.

James Coombs — Fri, 14 Aug 2026 18:00:14 +0000

James Coombs is a design engineer who ran two controlled studies on AI code generation: a 9-run ablation and 82 validation attempts across 11 scenarios. He maintains a 60-component design system, which is what the ablation study measured governance against.

I put design system rules in CLAUDE.md. "NEVER VIOLATE" section, explicit MCP tool pointers, import paths, token guidance. Then I ran a controlled study to see if it worked.

It didn't. 0% compliance. Identical to having no guidance at all.

That's not an anecdotal impression. It's a controlled result, and the part that carries it is not a score. The agent read the rule, cited it by name, and then wrote 91 styled() calls and zero design-system imports. That is a count from a diff, not a judgement.

This isn't a complaint about AI capabilities. The agent read the rules, acknowledged them, cited them in its reasoning, and then used the legacy framework anyway. It rationalized MUI usage as "compliant" because the CLAUDE.md mentioned MUI in a migration context. The agent didn't ignore the rules; it interpreted them in a way that justified its default behavior.

The study

I was migrating a large frontend from Material UI to a custom design system (Shadcn, Radix, Tailwind). Two-stage AI pipeline: a design tool produces a prototype from Figma, then a code agent implements it against the real codebase.

9 controlled runs on one page. Same target throughout, deliberately, to hold page complexity constant. Fresh git worktree per run, no cross-contamination. Six-dimension rubric, 0 to 5 each, 30 max: token fidelity, component API, anti-pattern compliance, dark mode, composition, visual quality. Same scorer, scoring from diffs after every run finished. I set up blinding and did not achieve it, because I had already seen the session logs. The governance condition is one run; the baseline is two and each prompted condition is three.

Condition	Mean score (/30)
No guidance	16.3
2-sentence DS prompt + prototype URL	26.7
2-sentence DS prompt + PDF screenshot	28.7
Full CLAUDE.md governance	16.4

A 2-sentence contextual prompt ("Use the design system. Use MCP tools to discover components.") produced an 11-point improvement. The comprehensive governance file produced nothing.

Why

The agent optimizes for task completion, not rule compliance. When the rules and the task align, it follows them. When they conflict, even slightly, it optimizes for completion and rationalizes the rules as satisfied.

CLAUDE.md rules are constraints. The agent's objective is task completion. Constraints only bind when the optimization landscape can't route around them. For behavioral instructions ("use X instead of Y"), there's always a rationalization path.

The second study (n=82)

To make sure it wasn't a fluke: 82 validation attempts across 11 scenarios, testing whether CLAUDE.md instructions could drive agents to use specific CLI tools.

Baseline (flat CLAUDE.md, no MCP): 12.5% pass rate on hard scenarios, which is 1 of 8
Strengthened CLAUDE.md (task-oriented tables, explicit tool pointers): zero measurable improvement
MCP tools available (structured query): 100% automatic discovery

The strengthened CLAUDE.md was well-structured. Task-oriented tables, exact tool names, usage examples. Didn't matter. Passive documentation does not drive agent tool selection regardless of structure quality. Total cost for all 82 runs: $4.65.

What actually works

Three things produced real compliance. None of them are documentation.

MCP tools for discovery. Instead of documenting "use component X," provide a query tool that returns component X when the agent searches for the right pattern. In the 82-run validation, discovery was 100%. Not in this one: the agent had 14 tools available in every run and called none of them in three of nine, including the governance run. It finds tools it is told to use.

PreToolUse hooks for blocking. Instead of documenting "never import from the legacy library," install a hook that rejects the Write/Edit operation if the import is present. Zero review comments needed for that class of violation.

This one is structural, not measured. A hook that returns a deny decision blocks the call, so there is no compliance rate to report. That distinction matters more than it sounds, and I found out the hard way: I went looking for the production number on my own hook system and could not produce it. It logged approvals and not denials. Every gate I had ever granted was on disk; nothing I had ever blocked was, so the one number that would have told me whether the enforcement layer was doing anything did not exist. I fixed it, and then found the fix was mostly cosmetic. The log now holds about a thousand entries, but roughly two thirds of them landed on one day, and that day is when I last ran the regression suite: they are the suite's own fixtures, synthetic commands written to be refused. Denials from real work run five to twenty a day and are mostly benign. Worse, three of the six guards that can refuse anything still record nothing when they do. So the number I went looking for still does not exist. I just have a log that looks like it does, which is the same failure as the rules file one layer down: the instrumentation is documented, not enforced.

So: enforcement by construction works. Enforcement you can audit is a second thing you have to build, and I had not built it.

Contextual prompts at the point of action. Not a governance file loaded at session start. A 2-sentence instruction delivered when the agent is about to do the work. "Use the design system. Use MCP tools to discover components." That's it. Ten points better than the comprehensive governance file, on the like-for-like comparison.

The pattern: Discovery (MCP, 100% automatic) > Routing (skill instructions, about 80% and only with hints) > Documentation (CLAUDE.md, ~0% for behavioral shifts). CLAUDE.md occupies a dead zone: too late for discovery, too early for routing.

What this means for your CLAUDE.md

If your governance file contains "always use X" or "never do Y," those rules achieve approximately 0% compliance for anything the agent wouldn't have done anyway.

Rules that work: Factual context the agent can look up ("the config file is at path/to/config"). Constraint declarations that hooks enforce ("imports from legacy/ are blocked by PreToolUse hook"). Workflow pointers that redirect to skills ("for migrations, use /migrate").

Rules that don't work: "ALWAYS use the design system." "NEVER import MUI components." "Use semantic tokens instead of hex values." The agent uses whatever's in scope and rationalizes compliance.

For every rule in your governance file, ask: what happens if the agent ignores this? If nothing happens, the rule doesn't work. Remove it or build enforcement. One enforcement hook prevents more defects than a hundred lines of documentation. (I built a full enforcement architecture around this finding; see I Gave an AI Agent Unrestricted Shell Access.)

Methodology notes

For anyone who wants to reproduce this:

Worktree isolation: git worktree add from the same baseline commit per run. No run sees another's changes.
Scoring: applied after all runs from diffs, same scorer throughout. Blinding was set up and not achieved: the scorer had seen the session logs. No randomisation of order.
What the headline rests on: not the score. Zero design-system imports, zero semantic tokens and zero MCP calls in the governance run, counted from the diff. A rubric can be biased; those cannot.
Cost: $4.65 for the n=82 validation, with a per-attempt breakdown. Accessible to any team.
Model: Claude Opus 4.6. The 0% finding is architectural (optimization target mismatch), not model-specific.
Limitations, plainly. One scorer, no blind, and the governance condition is a single run. The rubric also has a defect I did not catch until later: its dark-mode dimension counts Tailwind palette classes, so code that bypasses Tailwind entirely scores 5 out of 5 on it. The two no-guidance runs collected that free 5 while carrying 121 to 270 hardcoded hex values, and the governance run collected it with 2, which means any conversion of 16.3 or 16.4 into a percentage is reading two points that were never earned. None of this touches the 0%, because that is a count of imports rather than a score.

46KB for Twelve Shapes: When Not to Reach for the Animation Library

James Coombs — Wed, 12 Aug 2026 14:06:33 +0000

James Coombs is a design engineer who maintains the icon and illustration system for a product design system. He had a working hand-rolled animation engine and a mature library that could replace it, so he built the whole thing a second time to find out which one to keep.

I built the animation engine twice. The first version was a few hundred lines of hand-rolled code driving requestAnimationFrame, the browser's frame-by-frame render loop. The second used GSAP, the standard, well-liked JavaScript animation library, and its MorphSVGPlugin, which reshapes one path into another. Then I shipped the first one and deleted the second.

This is a build-versus-buy decision, and I want to show the version that doesn't usually get written up: the one where you build the "buy" option all the way, measure it honestly, and it loses.

Why build the thing you might throw away

I already had the hand-rolled engine working. The tempting move at that point is either to leave it alone or to swap in the library, because reaching for a mature library is what a responsible engineer is supposed to do. Both of those are guesses dressed up as a decision. The only way to actually know whether GSAP was worth adopting was to implement the animations both ways and put them side by side.

So I did: the same illustrations, the same five trigger modes, rendered by each engine and compared frame by frame in a component workbench (Storybook).

What the library actually bought

Almost nothing in the rendered output, which is not the same as nothing. Across three illustration pairs, roughly 560 vector shapes, the individual paths that make up an SVG, the two engines were pixel-identical on everything that mattered. The ~250 shape-morph paths interpolate the same way whether my code does it or the plugin does, because my source and target paths were already point-aligned, so the plugin's morph and my per-coordinate lerp land on the same frames. The ~300 rigid-body paths run the same matrix math in both. The only visible difference was on about 12 near-static fragments, the ones the build pipeline had mangled so they couldn't be interpolated directly, where the library's true shape morph was a shade cleaner than my opacity crossfade.

Twelve shapes out of 560. And the price for those twelve was 46KB gzipped in my build: 32 for GSAP's core, 12 for the morph plugin, 2 for the glue that wires it into React. The consuming app used GSAP for nothing else, so every one of those kilobytes would exist only to make twelve near-static fragments slightly crisper.

That is only half a ledger, and I want to be honest about the other half. The engine I kept is a few hundred lines only I maintain, with its own correctness traps (it already had one). The real trade is 46KB of battle-tested library against the cost of owning that code myself. For an app that uses GSAP nowhere else, to clean up twelve fragments in one asset, the 46KB still lost. It would not have taken much to flip it.

The bug I'd have shipped if I hadn't built it

Building the alternative paid for itself in a way the bundle math doesn't capture: it taught me something about the library I would have gotten wrong by swapping it in blind.

My engine uses a cubic ease (4t³), the curve that makes a motion speed up and then settle instead of moving at a constant rate. GSAP names its power eases power1, power2, power3, and the intuitive mapping is cubic to power3. That mapping is wrong. GSAP's power1 is quadratic (t²), power2 is cubic (t³), and power3 is quartic (t⁴): each name sits one degree above what it sounds like. The cubic match is power2.inOut, not power3.inOut. At t=0.3 the two curves differ by about 40 percent (0.108 versus 0.065), a visibly different motion. Had I simply replaced my engine with GSAP and reached for the name that sounded right, the animation would have felt subtly off and I would not have known why.

You calibrate against a library fastest when you have a reference implementation to diff it against. That is the hidden return on building the option you plan to reject: not the code, which I deleted, but the calibration.

Keep the branch you didn't ship

I didn't throw the GSAP version away. It lives on a branch, because a build-versus-buy decision is only true for the inputs you had when you made it. The rejected implementation is two things at once: the evidence for the decision, so someone can check the 46KB-for-twelve-shapes claim against real code, and a head start if the numbers move, for as long as it still builds. The calibration in this writeup outlives the branch.

And they move under conditions I can name now. If the app adopts GSAP for other animations, the marginal cost of the morph plugin drops from 46KB to about 12, and the decision probably flips. If the build pipeline starts mangling more paths, the library's cleaner fade morph stops being worth twelve paths and starts being worth fifty. If the animations grow into long sequenced timelines, the library's API is genuinely better than a hand-rolled state machine. None of those hold today. All of them are worth writing down next to the branch, so the next person doesn't repeat the whole evaluation to arrive at the same answer.

The rule

Before you add a dependency to replace working code, build enough of the replacement to measure what it actually buys on your asset. Then weigh it against both ledgers: the library's real marginal cost to your bundle, and the cost of owning the alternative yourself. Sometimes the library wins, and the day this app adopts GSAP elsewhere, it will. This time, for this asset, it was 46KB for twelve shapes, and the honest move was to delete the version I had just finished writing.

Why Animating an SVG the Obvious Way Breaks It

James Coombs — Mon, 10 Aug 2026 20:01:34 +0000

James Coombs is a design engineer who maintains the icon and illustration system for a product design system. He spent a week getting one illustration to animate between two states the way the designer's Figma prototype did, and most of that week went to discovering how the obvious approach fails.

The obvious approach: you have two SVGs, the same illustration in two poses, and you want to animate from one to the other. An SVG is a list of paths, and each path is a string of letter commands (move, line, curve) followed by the numbers that position them. So to animate, you interpolate the numbers: at the halfway point, every coordinate sits halfway between its start and end value. Ninety percent of the illustration animates perfectly this way. The other ten percent comes apart in mid-air.

The part that came apart was a circle that rotates as it moves. Play it back frame by frame and you see path fragments detaching from their own shape and flying off, like a diagram of an explosion. It took me longer than I'd like to admit to understand why, because two causes were tangled together, not one.

A note before the mechanism: libraries handle SVG morphing already (MorphSVG, Flubber, KUTE among them). I'm not reimplementing them. I'm after the mechanism they hide, plus two things they don't hand you: how to recover a rigid motion from two designer poses, and a build-pipeline trap that corrupts the shapes before any library sees them.

Why interpolating points can't rotate

A rotation is not a linear operation on coordinates. When a shape rotates, every point travels along an arc, and where it lands is a function of sine and cosine. Interpolating each x and y on its own draws the straight-line shortcut across the arc the point should be following. For a shape that only shifts position, the shortcut and the arc are the same line, so nothing goes wrong. For a rotation they diverge: the blended shape pulls inward toward the middle of the animation and turns unevenly, shrinking and wobbling instead of holding its size and spinning at a steady rate.

That distortion is subtle, a shrink you might not consciously notice. The violent version, fragments detaching and flying off, had a second cause I'll come to. But both trace to the same mistake: I was interpolating the wrong thing. The circle doesn't change shape between the two states. It rotates and translates as one rigid body. What should animate isn't its several hundred coordinates. It's the single motion that carries the whole group from one pose to the other.

Interpolate the motion, not the points

Instead of moving each point on its own, you can describe the whole motion once, as a formula every point runs through to find its new home. In SVG that formula is a transform matrix, matrix(a, b, c, d, e, f), and it maps a point like this:

x' = a*x + c*y + e
y' = b*x + d*y + f

A rigid rotation with uniform scale and translation is a similarity transform, which pins that matrix to a specific shape:

matrix(a, b, -b, a, tx, ty)

where a = s*cos θ, b = s*sin θ, s is the scale, θ the rotation, and tx, ty the translation. Four numbers describe the entire motion of the group.

Finding those four numbers is a best-fit problem: given the group's points in the start pose and the same points in the end pose, find the scale, angle, and shift that map one onto the other with the least total error. This is a standard shape-alignment fit (the Procrustes problem) and has a closed-form solution; I set it up as a small least-squares solve in TypeScript, no math library, because four unknowns don't need one. On the illustration that started this, the fit ran over 268 point pairs and its best answer was off by about 0.0005 pixels, which is zero for anything a screen can show. It recovered a rotation of 36.84 degrees and a scale of 1.0 to four decimals: the designer had rotated the object without resizing it, and the fit found that without being told.

To animate, interpolate the recovered angle and scale, with t running from 0 (start pose) to 1 (end pose):

θt = t*θ
st = 1 + t*(s - 1)
matrix(st*cos θt, st*sin θt, -st*sin θt, st*cos θt, t*tx, t*ty)

Render the group's paths once, wrap them in a <g>, and drive that one transform. The shape never distorts at any frame because it turns at a constant rate and scales uniformly. (One thing the snippet skips: rotate around the group's own center, not the SVG origin, or the shape orbits the corner instead of turning in place.) The tempting shortcut, interpolating the matrix numbers a and b directly, quietly reintroduces the original problem one level down: the shape shrinks toward the midpoint and swells back, invisible at 37 degrees, a 29 percent collapse at 90. Interpolate the angle, not the numbers that encode it.

The fit's error earns its keep a second way. A near-zero error mostly confirms the input really was rigid, which is exactly what makes the same number a validity check: hand the fit a group that isn't one rigid body and the best possible similarity can't match it, so the error climbs. I warn above 0.1 pixels, which catches "the thing you called one rigid body is really two things moving differently" before it ships as a glitch.

One strategy is not enough

Fitting a transform is right for the rigid group and wrong for everything else. The paper behind the circle, the text lines, the small marks, those genuinely do change shape between states, in small ways, and for them the naive coordinate interpolation is correct. So the real animation is a hybrid, and the interesting work is deciding which path gets which treatment.

Paths that barely move and keep their structure get per-coordinate interpolation: split the path string into its command letters and its numbers, interpolate the numbers, put it back together. This only works when both states carry the same command letters in the same order, which matters more than it sounds like it should, for reasons below.
Paths that move as a rigid unit get the fitted transform above.
Paths that can't be interpolated at all get an opacity crossfade: the start version fades out while the end version fades in, both pinned in place.

I sorted paths into those buckets by fill color, because in this illustration the moving element's fill cleanly separated it from everything static. That won't generalize: a shared brand color, or an element that changes color between states, will misfile. The durable signal is an explicit group id or layer name from the designer, with fill as a fallback. And the taxonomy itself assumes one rigid body plus near-static extras; genuine non-uniform scaling, shear, or several bodies moving differently needs a fuller fit, or one group per body.

The gotcha that cost me a day

Here is the part I did not see coming, and the second cause of the flying-apart. My first attempts interpolated almost nothing correctly, even the paths that should have been trivial, and the reason was the build pipeline, not the math.

The illustration components in the repo are optimized. They pass through SVGO, which shrinks path data and, more consequentially, merges and re-segments paths: it collapsed the illustration's 284 source paths down to 26. It also runs on each file independently, so the two poses get different merges and different command rewrites. This is not reformatting, it is restructuring: paths combined, points dropped, commands swapped, differently in each file. The point-for-point correspondence an interpolation depends on is gone, and interpolating one file against the other sends fragments to the wrong places. That is the explosion.

Measured, it was stark: of the 26 paths SVGO left in the optimized files, exactly 3 still corresponded between the two states. In the designer's raw exports, before optimization, 269 of 284 did. Normalizing both files to a canonical command set fixes the reformatting, which is what the morph libraries do internally, but it cannot undo the merging, so the reliable move is to feed the animation from the raw exports and let SVGO keep only the static components. I did that because it shipped faster; the cost is that the one asset that animates ships un-optimized.

The last stretch is rendering order

One more thing that looks like a transparency bug and isn't. SVG has no z-index; elements paint in document order, last on top. The rigid group has to be the final child of the SVG, or the paper lines behind it paint over it and you get what looks like the moving object showing through when it is really being drawn underneath. I lost an hour to that before remembering how painting order works. The fix is one line: append the group last.

Before you animate two shapes

Ask one question per element: does it change shape, or does it move as a rigid body? If it changes shape, interpolate its coordinates. If it moves as a body, recover the motion and interpolate its angle and scale. Then check that your build pipeline hasn't rewritten the two states out of correspondence before your code ever sees them.

Scoring, Gating, Ratcheting: The Migration Engine Nobody Builds

James Coombs — Tue, 04 Aug 2026 17:31:43 +0000

James Coombs is a design engineer who built a migration engine for a 962-file Material UI to custom design system migration. 7,065 lines of TypeScript, more than half of it tests, 5-factor confidence scoring, and a state machine with gated transitions.

Once the CSS coexistence layer was in place (I covered that in "The CSS Problem Nobody Thinks Is Solvable"), both frameworks could run side-by-side. The next question: how do you actually migrate 962 files across 12 packages without drowning in manual work or shipping regressions?

The standard answer is codemods. Write a jscodeshift transform, run it across the repo, review the output. Codemods work for mechanical substitutions: rename an import, swap a prop name, update a function signature. They don't work when the mapping isn't 1:1. A Material UI <TextField variant="outlined"> doesn't map to a single design system component. It maps to different components depending on context: <Input> for plain text, <Textarea> for multiline, <Select> for dropdowns with select prop. The prop API differs. The composition pattern differs. The styling approach differs.

962 files. Some are mechanical (swap the import, done). Some require architectural judgment. Treating them identically wastes time on the easy ones and ships regressions on the hard ones.

Confidence scoring

Weighted scoring for migration prioritization is an established practice (AWS uses it for cloud migration, Cortex uses it for service compliance). What's less common is applying it at the per-file level for frontend component migration, where the factors that predict success are different from application-level migration.

Every file in the migration gets a confidence score before anyone touches it. Five factors, weighted:

Factor	Weight	What it measures
Size	20%	Lines of code, number of components. Smaller files are easier to verify.
Semantic complexity	25%	How many MUI components, how deeply nested, how much conditional rendering.
Mapping coverage	25%	What percentage of MUI components in this file have complete prop-level transforms in the mapping data.
Test coverage	15%	Does the file have tests? How comprehensive? Files with tests catch regressions; files without them hide them.
Navigation complexity	15%	Is this file a leaf component or a route-level page? Pages compose dozens of components; a regression cascades.

Three tiers: high (>= 0.75), medium (>= 0.45), low (< 0.45). A high-confidence file (small, well-tested leaf component with complete mappings) can be auto-migrated with automated visual regression checking. A low-confidence file (large route-level page, partial mappings, no tests) routes to human review after the agent's attempt.

"Success" means the file reaches screenshot_approved without hitting failed: TypeScript compiles, tests pass, and the visual diff is below threshold. That's the outcome the score predicts.

The weights came from the first 20 migrations. Size mattered less than expected (small files with complex conditional rendering were harder than large files with repetitive layouts). Semantic complexity and mapping coverage were the strongest predictors of first-attempt success.

The scoring is deterministic: the LLM agent writes raw factor values to the manifest; a Node.js module computes the final score. The agent never sets its own confidence. This separation matters because LLMs are optimistic about their own output. The scoring module doesn't care what the agent thinks; it scores the file's structural properties.

The state machine

A boolean (migrated/not-migrated) isn't enough. Migration is a multi-step process, and each step has a quality gate.

The state machine tracks each file through 15 statuses with gated transitions. State is stored in a JSON manifest file. Node.js is the single writer; the LLM agent proposes transitions but never mutates state directly.

not_started → checked → plan_created → plan_validated → migrating → migrated
→ ts_verified → test_verified → screenshot_captured → screenshot_approved
→ pr_created → merged

With two terminal escape states: failed (reachable from any active state) and wont_migrate (reachable from not_started for files that should never be migrated).

Each transition has a named gate:

not_started to checked: ds-migrate-check-passes (migration readiness assessment)
checked to plan_created: confidence-score-computed (all five factors assessed)
plan_created to plan_validated: validate-plan-exits-0 (plan passes structural validation)
migrated to ts_verified: tsc-exits-0 (TypeScript compiles clean)
ts_verified to test_verified: tests-pass-or-no-tests
test_verified to screenshot_captured: screenshots-captured-and-diffed (visual regression screenshots taken)
screenshot_captured to screenshot_approved: diff-below-threshold-and-high-confidence (automated approval for high-confidence files) or routes to awaiting_human_review for manual approval

Files can move backward. Any active state can transition to failed with notes on what broke. The checked state can return to not_started via a file-changed-staleness gate (the source file was modified since it was last checked, invalidating the analysis). Stale states have configurable timeouts: 1 hour for migrating, 2 weeks for awaiting_human_review, 30 days for pr_created.

Why this matters: without gated transitions, teams mark files as "done" when the PR merges. But "PR merged" and "verified in production with no regressions" are different things. 15 states and 15 named gates make that gap explicit and enforceable, 12 of them on the forward path and 3 on the routes back (a file changed since it was checked, a failed file whose changes were reverted, an explicit override of a wont_migrate decision). The remaining 11 transitions are ungated, and they are all the ones that need no evidence to take: anything to failed, and the hand-off to human review.

Component mappings

12 mapping files. 100+ MUI-to-design-system transforms with prop-level translation.

A mapping isn't just "replace <Button> with <Button>." It's:

Import path change (@mui/material/Button to @company/design-system)
Prop renames (variant="contained" to variant="default")
Prop removals (MUI's disableElevation has no equivalent)
Prop additions (design system requires size where MUI inferred it)
Composition changes (MUI's startIcon prop becomes a child <Icon> component)
Conditional mappings (MUI's TextField maps to 3 different components based on props)

Each mapping includes the confidence impact. A component with a complete mapping (all props translated, all variants covered) contributes 100% to the mapping coverage factor. A component with a partial mapping (3 of 7 props translated) contributes proportionally. This feeds directly into the confidence score.

The mappings are JSON, not code. Adding a new component mapping is a data change, not a code change. The migration bot reads the mappings; the bot code doesn't change when new components are added.

The ratchet

The ratchet pattern (monotonically decreasing violation counts, enforced through CI) is well-established. Dusty Burwell described it in 2019. What's less common is applying it at the write layer for design system imports.

PreToolUse hooks (65 lines, 17 tests) fire on every file write. If an edit introduces an import from the legacy framework in a file that's been marked "complete," the edit is rejected before it lands. Not at CI. Not at PR review. At the moment the engineer types the import. The engineer sees an error message explaining why and pointing to the design system equivalent.

This is not documentation ("please use the new components"). This is enforcement at the earliest possible point. The count of legacy imports only goes down, never up.

The ratchet catches a specific failure mode: an engineer working on an unrelated feature in a migrated file reaches for the familiar MUI component because it's what they know. Without the ratchet, the migration regresses silently. With it, the regression is blocked at write time, before the PR is even created.

Result: zero legacy imports in any file marked "complete." Zero review comments needed for that class of violation.

The learning log

10 entries over four months. Each entry: date, source, what happened, what rule it produced.

Examples:

A static hex color (#6464f0) wasn't converted to a design token during migration. Added a check to the migration step that flags unconverted hex values.
A migration check was too broad: it added design system config to bundles that didn't use any migrated components. Refined the check to verify component usage before applying config.
A portal-rendered component's test asserted a class on a node that could be null, so the assertion passed vacuously. Added a not-null guard as a rule and updated the test suite.
Separately, nesting a Radix trigger through a component that did not forward refs silently dropped the ref. Added a rule; the failure is silent, which is why it needed one.

The log is append-only with mandatory fields. Rules flow from failures, not from predictions. Starting the log on day 1 would have been better; the first three months of failures had to be reconstructed from git history.

What didn't work

Hardcoded confidence weights, never recalibrated. The five weights were set by judgment before the first file was scored, and they are still the numbers in the config: 20, 25, 25, 15, 15. Nothing in the system closes the loop. The bot records the score it predicted and the state each file actually reached, so the data to test the weights is produced as a side effect of running it, and no one has ever gone back and fitted one against the other. A scoring system that cannot be shown to predict anything is a prioritization heuristic wearing the vocabulary of a model. It still beats migrating files in directory order, which is the honest claim, and it is a smaller claim than the machinery implies. Build the recalibration step into the first batch, or the weights stay guesses with a decimal point.

Generic checks without scoping. A check that asks "does this package depend on the design system?" catches packages that imported one design system utility for an unrelated reason. Three bundles got unnecessary config added before the check was refined to verify actual component usage. Automation checks need nested conditions, not just existence tests.

The numbers

Metric	Value
Migration bot code	7,065 lines of TypeScript, of which 3,802 are tests
Component mappings	12 files, 69 components, 131 prop-level transforms
Structured data files	49 (9,471 lines of JSON)
Scoring factors	5 (size, semantic, mapping, test, navigation)
State machine	15 statuses, 26 transitions, 15 named gates, JSON-stored
Enforcement hooks	65 lines, 17 tests
Learning log entries	10 entries, 11 codified rules

If you're building a migration engine

Score before you migrate. Any scoring system (even a rough one) is better than migrating files in directory order. The scoring reorders 962 files so the first 50 are almost guaranteed to succeed, building confidence and catching mapping gaps early.

Gate your transitions. "PR merged" is not "migration complete." Define what "complete" means, build the states between "started" and "complete," and enforce the gates.

Build the ratchet in week 1. Every day without enforcement is a day legacy imports can re-enter the codebase through unrelated work.

Start the learning log immediately. The failures from the first 10 migrations will produce the rules that prevent failures in the next 100.

"Told" vs. "Known": The One Variable That Predicts AI Design Tool Quality

James Coombs — Mon, 27 Jul 2026 17:32:20 +0000

James Coombs is a design engineer who built a 60-component design system with a structured query server and ran a controlled ablation study (n=9) measuring AI tool fidelity against it.

After evaluating six categories of AI design-to-code tools against a 60-component design system, one variable predicted output quality better than anything else: whether the tool had been told about the design system or knew it.

"Told" tools: You describe your components in natural language or docs. The AI interprets. It generates something that looks like your Button but uses the wrong prop names, the wrong import path, or a plausible-but-nonexistent variant.

"Known" tools: The AI queries a structured index of your actual components. It gets the exact name, exact props, exact import, exact usage examples. No interpretation gap.

Tool type	Awareness	Typical fidelity
General AI prototypers (v0, Bolt, Lovable)	Told (generic framework knowledge)	~50-60%
AI design tools (Claude Design, Pencil)	Told (LLM ingestion of your docs)	~70-80%
AST-parsed indexers (Builder.io Visual Copilot)	Known (parsed component source)	~95-100%
Manual component maps (Figma Code Connect)	Known (hand-mapped per component)	100%
AI agent + structured query (MCP server)	Known (live query against real source)	100%

The gap between "Told" and "Known" isn't incremental. It's categorical.

Why "Told" fails

When you tell an AI "use our Button component with variant='primary'," the AI generates something that looks like a primary button. Maybe it uses your component, maybe a common library's. Maybe variant='primary' is a valid prop, maybe your API uses variant='default'. Maybe it applies your violet palette, maybe generic blue.

Each step introduces interpretation drift. By the time the output reaches an engineer, it looks close enough to pass a design review but wrong enough to require a rewrite.

Worse: "Told" tools hallucinate components that don't exist. They use prop names that are close but wrong. They apply tokens that look plausible but aren't in your system. This is worse than obviously wrong output. Obviously wrong gets rewritten. Plausibly wrong gets merged.

And your design system changes. Components get new variants, tokens get renamed. "Told" tools have whatever version was in their training data. "Known" tools query the current state.

The three "Known" approaches

AST-parsed component index. Parse your source into a structured index: component names, prop types (from TypeScript), variants, tokens. Regenerate on build. Captures structure but not intent; doesn't know when to use Dialog vs. Sheet.

Hand-mapped component bridge (Figma Code Connect). Manually map each Figma component to its React counterpart. 100% fidelity by definition. But manual maintenance: every new component or variant needs an update. Scales linearly with component count.

AI agent + structured query (MCP server). Give the agent a query interface. It discovers components by searching, not guessing. No pre-built index needed. Works for discovery ("what handles file uploads?"), not just lookup. You build the interface once, and every AI tool, current and future, benefits.

The data

My ablation study (n=9, controlled) confirmed this:

Agent + MCP query (Known): 26.7 / 30 on design system compliance
AI design tool interpretation (Told): 16.3 / 30
CLAUDE.md rules ("always use the design system"): 16.4 / 30, identical to no guidance

The MCP server took about a week to build. It gives every AI tool structured access to the real design system. The cost-per-tool amortization improves with every new tool that connects.

When "Told" is fine

Throwaway prototypes where 70% fidelity is the goal. Greenfield exploration when you don't have a design system yet. Product demos where brand fidelity doesn't matter.

When you need "Known"

Production code generation. Design system migrations, where "close but wrong" imports are worse than obviously wrong ones. Multi-person teams where plausibly wrong output gets merged. Brand-critical surfaces where your violet isn't someone else's purple.

The practical takeaway

Before evaluating any AI design tool, ask one question: does it query my real components, or does it guess?

If your team generates AI code that goes into production, invest in a structured query interface. MCP server, CLI tool, AST index: the form factor matters less than the structured access. Build it once; every AI tool benefits. The returns compound.

Don't try to improve "Told" fidelity by writing better documentation or more detailed prompts. My data says the returns stop, and may reverse: across three versions of a design-system manifest the middle one scored best, and the densest scored worst. Treat that as a ceiling rather than a slope, and note that the gap between versions is smaller than the variation between runs of the same version. A 2-sentence prompt + structured query beats a comprehensive governance file by 11 points.

DEV Community: James Coombs

Your AI Skills Have the Same Problem as Your CLAUDE.md. Here's the Data.

The skill

The study

Why

Three layers, one result

What I did instead

What this means for your skill library

Methodology notes

I Gave an AI Agent Unrestricted Shell Access. Here's What Happened Next.

The architecture

The /yes system

The seven incidents

What the agent is actually doing

Beyond safety: the lifecycle hooks

What I'd tell someone building this

The numbers

7,798 Components. Import One, Ship One.

The barrel, and why it's a build-time problem, not a bundle one

Import one, ship one

The map that resolves at runtime but not in the type-checker

The parts that never reach the bundle

The mirror of the barrel

The CSS Problem Nobody Thinks Is Solvable

Where the received answer came from

One line of configuration

The portal problem

The Tailwind-as-library problem

What didn't work

The numbers

If you're running dual CSS systems

Your Migration Guide Is Already Wrong

Why guides fail

48 JSON files

Three-layer separation

The playbook as product

What didn't work

The numbers

If you're organizing a migration

Your CLAUDE.md Rules Achieve 0% Compliance. Here's the Data.

The study

Why

The second study (n=82)

What actually works

What this means for your CLAUDE.md

Methodology notes

46KB for Twelve Shapes: When Not to Reach for the Animation Library

Why build the thing you might throw away

What the library actually bought

The bug I'd have shipped if I hadn't built it

Keep the branch you didn't ship

The rule

Why Animating an SVG the Obvious Way Breaks It

Why interpolating points can't rotate

Interpolate the motion, not the points

One strategy is not enough

The gotcha that cost me a day

The last stretch is rendering order

Before you animate two shapes

Scoring, Gating, Ratcheting: The Migration Engine Nobody Builds

Confidence scoring

The state machine

Component mappings

The ratchet

The learning log

What didn't work

The numbers

If you're building a migration engine

"Told" vs. "Known": The One Variable That Predicts AI Design Tool Quality

Why "Told" fails

The three "Known" approaches

The data

When "Told" is fine

When you need "Known"

The practical takeaway

The `/yes` system