DEV Community: James Coombs

Your CLAUDE.md Rules Achieve 0% Compliance. Here's the Data.

James Coombs — Fri, 14 Aug 2026 18:00:14 +0000

James Coombs is a design engineer who ran 91 controlled experiments on AI code generation: 9 ablation runs with blinded scoring and 82 validation attempts across 11 scenarios. He maintains a 60-component design system, which is what the ablation study measured governance against.

I put design system rules in CLAUDE.md. "NEVER VIOLATE" section, explicit MCP tool pointers, import paths, token guidance. Then I ran a controlled study to see if it worked.

It didn't. 0% compliance. Identical to having no guidance at all.

That's not an anecdotal impression. It's a controlled result: 9 runs, fresh worktree per run, blinded scoring, randomized evaluation order.

This isn't a complaint about AI capabilities. The agent read the rules, acknowledged them, cited them in its reasoning, and then used the legacy framework anyway. It rationalized MUI usage as "compliant" because the CLAUDE.md mentioned MUI in a migration context. The agent didn't ignore the rules; it interpreted them in a way that justified its default behavior.

The study

I was migrating a large frontend from Material UI to a custom design system (Shadcn, Radix, Tailwind). Two-stage AI pipeline: a design tool produces a prototype from Figma, then a code agent implements it against the real codebase.

9 controlled runs. Same target page (complex UI, 15+ components). Fresh git worktree per run, no cross-contamination. Blinded scoring with a 6-dimension rubric (0-5 each, 30 points max): component usage, token usage, composition patterns, accessibility, visual fidelity, code quality.

Condition	Mean score (/30)
No guidance	16.3
2-sentence DS prompt + prototype URL	27.5
2-sentence DS prompt + PDF screenshot	28.7
Full CLAUDE.md governance	16.1

A 2-sentence contextual prompt ("Use the design system. Use MCP tools to discover components.") produced an 11-point improvement. The comprehensive governance file produced nothing.

Why

The agent optimizes for task completion, not rule compliance. When the rules and the task align, it follows them. When they conflict, even slightly, it optimizes for completion and rationalizes the rules as satisfied.

CLAUDE.md rules are constraints. The agent's objective is task completion. Constraints only bind when the optimization landscape can't route around them. For behavioral instructions ("use X instead of Y"), there's always a rationalization path.

The second study (n=82)

To make sure it wasn't a fluke: 82 validation attempts across 11 scenarios, testing whether CLAUDE.md instructions could drive agents to use specific CLI tools.

Baseline (flat CLAUDE.md, no MCP): 12.5% pass rate on hard scenarios
Strengthened CLAUDE.md (task-oriented tables, explicit tool pointers): zero measurable improvement
MCP tools available (structured query): 100% automatic discovery

The strengthened CLAUDE.md was well-structured. Task-oriented tables, exact tool names, usage examples. Didn't matter. Passive documentation does not drive agent tool selection regardless of structure quality. Total cost for all 82 runs: $4.65.

What actually works

Three things produced real compliance. None of them are documentation.

MCP tools for discovery. Instead of documenting "use component X," provide a query tool that returns component X when the agent searches for the right pattern. 100% discovery rate in both studies. The agent always finds tools it can query.

PreToolUse hooks for blocking. Instead of documenting "never import from the legacy library," install a hook that rejects the Write/Edit operation if the import is present. Zero review comments needed for that class of violation.

This one is structural, not measured. A hook that returns a deny decision blocks the call, so there is no compliance rate to report. That distinction matters more than it sounds, and I found out the hard way: I went looking for the production number on my own hook system and could not produce it. It logs approvals and not denials. Every gate I had ever granted was on disk; nothing I had ever blocked was. What I could reconstruct from session transcripts, for one sampled week, was four attempts by an agent to route around the approval gate, all four stopped, and twelve blocks that fired only because the guard's own interpreter crashed and it failed closed. One more was a false positive, a read-only command classified as a write.

So: enforcement by construction works. Enforcement you can audit is a second thing you have to build, and I had not built it.

Contextual prompts at the point of action. Not a governance file loaded at session start. A 2-sentence instruction delivered when the agent is about to do the work. "Use the design system. Use MCP tools to discover components." That's it. 11 points better than the comprehensive governance file.

The pattern: Discovery (MCP, 100% automatic) > Routing (skill instructions, ~80%) > Documentation (CLAUDE.md, ~0% for behavioral shifts). CLAUDE.md occupies a dead zone: too late for discovery, too early for routing.

The density inversion

One more thing. More constraints made output worse.

Variant	What the agent got	Mean score
Fewer constraints	DS prompt only	21.3 / 30
More constraints	DS prompt + detailed rules	19.0 / 30
PDF (minimal input)	Screenshot + DS prompt	28.7 / 30
URL (full source)	Full prototype + source code	26.7 / 30

Less input produced more focused output. The agent optimizes for the most specific constraint and ignores broader quality. Dense constraint sets create conflicting optimization targets.

My adage with AI tooling is "measure 40 times, cut once." This is the data behind it. Find the minimal effective prompt. A 2-sentence unconditional instruction outperformed the full governance file by 11 points.

What this means for your CLAUDE.md

If your governance file contains "always use X" or "never do Y," those rules achieve approximately 0% compliance for anything the agent wouldn't have done anyway.

Rules that work: Factual context the agent can look up ("the config file is at path/to/config"). Constraint declarations that hooks enforce ("imports from legacy/ are blocked by PreToolUse hook"). Workflow pointers that redirect to skills ("for migrations, use /migrate").

Rules that don't work: "ALWAYS use the design system." "NEVER import MUI components." "Use semantic tokens instead of hex values." The agent uses whatever's in scope and rationalizes compliance.

For every rule in your governance file, ask: what happens if the agent ignores this? If nothing happens, the rule doesn't work. Remove it or build enforcement. One enforcement hook prevents more defects than a hundred lines of documentation. (I built a full enforcement architecture around this finding; see "I Gave an AI Agent Unrestricted Shell Access. coming soon")

Methodology notes

For anyone who wants to reproduce this:

Worktree isolation: git worktree add from the same baseline commit per run. No run sees another's changes.
Blinded scoring: Rubric applied after all runs, same scorer, randomized order.
Cost: ~$12 for the n=9 ablation. $4.65 for the n=82 validation. Accessible to any team.
Model: Claude Opus 4.6. The 0% finding is architectural (optimization target mismatch), not model-specific.
Limitation: Single scorer. A multi-rater study would strengthen the findings. The gap is large enough (0% vs. 11-point improvement) that scorer bias is unlikely to flip the conclusion.

46KB for Twelve Shapes: When Not to Reach for the Animation Library

James Coombs — Wed, 12 Aug 2026 14:06:33 +0000

James Coombs is a design engineer who maintains the icon and illustration system for a product design system. He had a working hand-rolled animation engine and a mature library that could replace it, so he built the whole thing a second time to find out which one to keep.

I built the animation engine twice. The first version was a few hundred lines of hand-rolled code driving requestAnimationFrame, the browser's frame-by-frame render loop. The second used GSAP, the standard, well-liked JavaScript animation library, and its MorphSVGPlugin, which reshapes one path into another. Then I shipped the first one and deleted the second.

This is a build-versus-buy decision, and I want to show the version that doesn't usually get written up: the one where you build the "buy" option all the way, measure it honestly, and it loses.

Why build the thing you might throw away

I already had the hand-rolled engine working. The tempting move at that point is either to leave it alone or to swap in the library, because reaching for a mature library is what a responsible engineer is supposed to do. Both of those are guesses dressed up as a decision. The only way to actually know whether GSAP was worth adopting was to implement the animations both ways and put them side by side.

So I did: the same illustrations, the same five trigger modes, rendered by each engine and compared frame by frame in a component workbench (Storybook).

What the library actually bought

Almost nothing in the rendered output, which is not the same as nothing. Across three illustration pairs, roughly 560 vector shapes, the individual paths that make up an SVG, the two engines were pixel-identical on everything that mattered. The ~250 shape-morph paths interpolate the same way whether my code does it or the plugin does, because my source and target paths were already point-aligned, so the plugin's morph and my per-coordinate lerp land on the same frames. The ~300 rigid-body paths run the same matrix math in both. The only visible difference was on about 12 near-static fragments, the ones the build pipeline had mangled so they couldn't be interpolated directly, where the library's true shape morph was a shade cleaner than my opacity crossfade.

Twelve shapes out of 560. And the price for those twelve was 46KB gzipped in my build: 32 for GSAP's core, 12 for the morph plugin, 2 for the glue that wires it into React. The consuming app used GSAP for nothing else, so every one of those kilobytes would exist only to make twelve near-static fragments slightly crisper.

That is only half a ledger, and I want to be honest about the other half. The engine I kept is a few hundred lines only I maintain, with its own correctness traps (it already had one). The real trade is 46KB of battle-tested library against the cost of owning that code myself. For an app that uses GSAP nowhere else, to clean up twelve fragments in one asset, the 46KB still lost. It would not have taken much to flip it.

The bug I'd have shipped if I hadn't built it

Building the alternative paid for itself in a way the bundle math doesn't capture: it taught me something about the library I would have gotten wrong by swapping it in blind.

My engine uses a cubic ease (4t³), the curve that makes a motion speed up and then settle instead of moving at a constant rate. GSAP names its power eases power1, power2, power3, and the intuitive mapping is cubic to power3. That mapping is wrong. GSAP's power1 is quadratic (t²), power2 is cubic (t³), and power3 is quartic (t⁴): each name sits one degree above what it sounds like. The cubic match is power2.inOut, not power3.inOut. At t=0.3 the two curves differ by about 40 percent (0.108 versus 0.065), a visibly different motion. Had I simply replaced my engine with GSAP and reached for the name that sounded right, the animation would have felt subtly off and I would not have known why.

You calibrate against a library fastest when you have a reference implementation to diff it against. That is the hidden return on building the option you plan to reject: not the code, which I deleted, but the calibration.

Keep the branch you didn't ship

I didn't throw the GSAP version away. It lives on a branch, because a build-versus-buy decision is only true for the inputs you had when you made it. The rejected implementation is two things at once: the evidence for the decision, so someone can check the 46KB-for-twelve-shapes claim against real code, and a head start if the numbers move, for as long as it still builds. The calibration in this writeup outlives the branch.

And they move under conditions I can name now. If the app adopts GSAP for other animations, the marginal cost of the morph plugin drops from 46KB to about 12, and the decision probably flips. If the build pipeline starts mangling more paths, the library's cleaner fade morph stops being worth twelve paths and starts being worth fifty. If the animations grow into long sequenced timelines, the library's API is genuinely better than a hand-rolled state machine. None of those hold today. All of them are worth writing down next to the branch, so the next person doesn't repeat the whole evaluation to arrive at the same answer.

The rule

Before you add a dependency to replace working code, build enough of the replacement to measure what it actually buys on your asset. Then weigh it against both ledgers: the library's real marginal cost to your bundle, and the cost of owning the alternative yourself. Sometimes the library wins, and the day this app adopts GSAP elsewhere, it will. This time, for this asset, it was 46KB for twelve shapes, and the honest move was to delete the version I had just finished writing.

Why Animating an SVG the Obvious Way Breaks It

James Coombs — Mon, 10 Aug 2026 20:01:34 +0000

James Coombs is a design engineer who maintains the icon and illustration system for a product design system. He spent a week getting one illustration to animate between two states the way the designer's Figma prototype did, and most of that week went to discovering how the obvious approach fails.

The obvious approach: you have two SVGs, the same illustration in two poses, and you want to animate from one to the other. An SVG is a list of paths, and each path is a string of letter commands (move, line, curve) followed by the numbers that position them. So to animate, you interpolate the numbers: at the halfway point, every coordinate sits halfway between its start and end value. Ninety percent of the illustration animates perfectly this way. The other ten percent comes apart in mid-air.

The part that came apart was a circle that rotates as it moves. Play it back frame by frame and you see path fragments detaching from their own shape and flying off, like a diagram of an explosion. It took me longer than I'd like to admit to understand why, because two causes were tangled together, not one.

A note before the mechanism: libraries handle SVG morphing already (MorphSVG, Flubber, KUTE among them). I'm not reimplementing them. I'm after the mechanism they hide, plus two things they don't hand you: how to recover a rigid motion from two designer poses, and a build-pipeline trap that corrupts the shapes before any library sees them.

Why interpolating points can't rotate

A rotation is not a linear operation on coordinates. When a shape rotates, every point travels along an arc, and where it lands is a function of sine and cosine. Interpolating each x and y on its own draws the straight-line shortcut across the arc the point should be following. For a shape that only shifts position, the shortcut and the arc are the same line, so nothing goes wrong. For a rotation they diverge: the blended shape pulls inward toward the middle of the animation and turns unevenly, shrinking and wobbling instead of holding its size and spinning at a steady rate.

That distortion is subtle, a shrink you might not consciously notice. The violent version, fragments detaching and flying off, had a second cause I'll come to. But both trace to the same mistake: I was interpolating the wrong thing. The circle doesn't change shape between the two states. It rotates and translates as one rigid body. What should animate isn't its several hundred coordinates. It's the single motion that carries the whole group from one pose to the other.

Interpolate the motion, not the points

Instead of moving each point on its own, you can describe the whole motion once, as a formula every point runs through to find its new home. In SVG that formula is a transform matrix, matrix(a, b, c, d, e, f), and it maps a point like this:

x' = a*x + c*y + e
y' = b*x + d*y + f

A rigid rotation with uniform scale and translation is a similarity transform, which pins that matrix to a specific shape:

matrix(a, b, -b, a, tx, ty)

where a = s*cos θ, b = s*sin θ, s is the scale, θ the rotation, and tx, ty the translation. Four numbers describe the entire motion of the group.

Finding those four numbers is a best-fit problem: given the group's points in the start pose and the same points in the end pose, find the scale, angle, and shift that map one onto the other with the least total error. This is a standard shape-alignment fit (the Procrustes problem) and has a closed-form solution; I set it up as a small least-squares solve in TypeScript, no math library, because four unknowns don't need one. On the illustration that started this, the fit ran over 268 point pairs and its best answer was off by about 0.0005 pixels, which is zero for anything a screen can show. It recovered a rotation of 36.84 degrees and a scale of 1.0 to four decimals: the designer had rotated the object without resizing it, and the fit found that without being told.

To animate, interpolate the recovered angle and scale, with t running from 0 (start pose) to 1 (end pose):

θt = t*θ
st = 1 + t*(s - 1)
matrix(st*cos θt, st*sin θt, -st*sin θt, st*cos θt, t*tx, t*ty)

Render the group's paths once, wrap them in a <g>, and drive that one transform. The shape never distorts at any frame because it turns at a constant rate and scales uniformly. (One thing the snippet skips: rotate around the group's own center, not the SVG origin, or the shape orbits the corner instead of turning in place.) The tempting shortcut, interpolating the matrix numbers a and b directly, quietly reintroduces the original problem one level down: the shape shrinks toward the midpoint and swells back, invisible at 37 degrees, a 29 percent collapse at 90. Interpolate the angle, not the numbers that encode it.

The fit's error earns its keep a second way. A near-zero error mostly confirms the input really was rigid, which is exactly what makes the same number a validity check: hand the fit a group that isn't one rigid body and the best possible similarity can't match it, so the error climbs. I warn above 0.1 pixels, which catches "the thing you called one rigid body is really two things moving differently" before it ships as a glitch.

One strategy is not enough

Fitting a transform is right for the rigid group and wrong for everything else. The paper behind the circle, the text lines, the small marks, those genuinely do change shape between states, in small ways, and for them the naive coordinate interpolation is correct. So the real animation is a hybrid, and the interesting work is deciding which path gets which treatment.

Paths that barely move and keep their structure get per-coordinate interpolation: split the path string into its command letters and its numbers, interpolate the numbers, put it back together. This only works when both states carry the same command letters in the same order, which matters more than it sounds like it should, for reasons below.
Paths that move as a rigid unit get the fitted transform above.
Paths that can't be interpolated at all get an opacity crossfade: the start version fades out while the end version fades in, both pinned in place.

I sorted paths into those buckets by fill color, because in this illustration the moving element's fill cleanly separated it from everything static. That won't generalize: a shared brand color, or an element that changes color between states, will misfile. The durable signal is an explicit group id or layer name from the designer, with fill as a fallback. And the taxonomy itself assumes one rigid body plus near-static extras; genuine non-uniform scaling, shear, or several bodies moving differently needs a fuller fit, or one group per body.

The gotcha that cost me a day

Here is the part I did not see coming, and the second cause of the flying-apart. My first attempts interpolated almost nothing correctly, even the paths that should have been trivial, and the reason was the build pipeline, not the math.

The illustration components in the repo are optimized. They pass through SVGO, which shrinks path data and, more consequentially, merges and re-segments paths: it collapsed the illustration's 284 source paths down to 26. It also runs on each file independently, so the two poses get different merges and different command rewrites. This is not reformatting, it is restructuring: paths combined, points dropped, commands swapped, differently in each file. The point-for-point correspondence an interpolation depends on is gone, and interpolating one file against the other sends fragments to the wrong places. That is the explosion.

Measured, it was stark: of the 26 paths SVGO left in the optimized files, exactly 3 still corresponded between the two states. In the designer's raw exports, before optimization, 269 of 284 did. Normalizing both files to a canonical command set fixes the reformatting, which is what the morph libraries do internally, but it cannot undo the merging, so the reliable move is to feed the animation from the raw exports and let SVGO keep only the static components. I did that because it shipped faster; the cost is that the one asset that animates ships un-optimized.

The last stretch is rendering order

One more thing that looks like a transparency bug and isn't. SVG has no z-index; elements paint in document order, last on top. The rigid group has to be the final child of the SVG, or the paper lines behind it paint over it and you get what looks like the moving object showing through when it is really being drawn underneath. I lost an hour to that before remembering how painting order works. The fix is one line: append the group last.

Before you animate two shapes

Ask one question per element: does it change shape, or does it move as a rigid body? If it changes shape, interpolate its coordinates. If it moves as a body, recover the motion and interpolate its angle and scale. Then check that your build pipeline hasn't rewritten the two states out of correspondence before your code ever sees them.

Scoring, Gating, Ratcheting: The Migration Engine Nobody Builds

James Coombs — Tue, 04 Aug 2026 17:31:43 +0000

James Coombs is a design engineer who built a migration engine for a 962-file Material UI to custom design system migration. 7,065 lines of bot code, 5-factor confidence scoring, and a state machine with gated transitions.

Once the CSS coexistence layer was in place (I covered that in "The CSS Problem Nobody Thinks Is Solvable"), both frameworks could run side-by-side. The next question: how do you actually migrate 962 files across 12 packages without drowning in manual work or shipping regressions?

The standard answer is codemods. Write a jscodeshift transform, run it across the repo, review the output. Codemods work for mechanical substitutions: rename an import, swap a prop name, update a function signature. They don't work when the mapping isn't 1:1. A Material UI <TextField variant="outlined"> doesn't map to a single design system component. It maps to different components depending on context: <Input> for plain text, <Textarea> for multiline, <Select> for dropdowns with select prop. The prop API differs. The composition pattern differs. The styling approach differs.

962 files. Some are mechanical (swap the import, done). Some require architectural judgment. Treating them identically wastes time on the easy ones and ships regressions on the hard ones.

Confidence scoring

Weighted scoring for migration prioritization is an established practice (AWS uses it for cloud migration, Cortex uses it for service compliance). What's less common is applying it at the per-file level for frontend component migration, where the factors that predict success are different from application-level migration.

Every file in the migration gets a confidence score before anyone touches it. Five factors, weighted:

Factor	Weight	What it measures
Size	20%	Lines of code, number of components. Smaller files are easier to verify.
Semantic complexity	25%	How many MUI components, how deeply nested, how much conditional rendering.
Mapping coverage	25%	What percentage of MUI components in this file have complete prop-level transforms in the mapping data.
Test coverage	15%	Does the file have tests? How comprehensive? Files with tests catch regressions; files without them hide them.
Navigation complexity	15%	Is this file a leaf component or a route-level page? Pages compose dozens of components; a regression cascades.

Three tiers: high (>= 0.75), medium (>= 0.45), low (< 0.45). A high-confidence file (small, well-tested leaf component with complete mappings) can be auto-migrated with automated visual regression checking. A low-confidence file (large route-level page, partial mappings, no tests) routes to human review after the agent's attempt.

"Success" means the file reaches screenshot_approved without hitting failed: TypeScript compiles, tests pass, and the visual diff is below threshold. That's the outcome the score predicts.

The weights came from the first 20 migrations. Size mattered less than expected (small files with complex conditional rendering were harder than large files with repetitive layouts). Semantic complexity and mapping coverage were the strongest predictors of first-attempt success.

The scoring is deterministic: the LLM agent writes raw factor values to the manifest; a Node.js module computes the final score. The agent never sets its own confidence. This separation matters because LLMs are optimistic about their own output. The scoring module doesn't care what the agent thinks; it scores the file's structural properties.

The state machine

A boolean (migrated/not-migrated) isn't enough. Migration is a multi-step process, and each step has a quality gate.

The state machine tracks each file through 15 statuses with gated transitions. State is stored in a JSON manifest file. Node.js is the single writer; the LLM agent proposes transitions but never mutates state directly.

not_started → checked → plan_created → plan_validated → migrating → migrated
→ ts_verified → test_verified → screenshot_captured → screenshot_approved
→ pr_created → merged

With two terminal escape states: failed (reachable from any active state) and wont_migrate (reachable from not_started for files that should never be migrated).

Each transition has a named gate:

not_started to checked: ds-migrate-check-passes (migration readiness assessment)
checked to plan_created: confidence-score-computed (all five factors assessed)
plan_created to plan_validated: validate-plan-exits-0 (plan passes structural validation)
migrated to ts_verified: tsc-exits-0 (TypeScript compiles clean)
ts_verified to test_verified: tests-pass-or-no-tests
test_verified to screenshot_captured: screenshots-captured-and-diffed (visual regression screenshots taken)
screenshot_captured to screenshot_approved: diff-below-threshold-and-high-confidence (automated approval for high-confidence files) or routes to awaiting_human_review for manual approval

Files can move backward. Any active state can transition to failed with notes on what broke. The checked state can return to not_started via a file-changed-staleness gate (the source file was modified since it was last checked, invalidating the analysis). Stale states have configurable timeouts: 1 hour for migrating, 2 weeks for awaiting_human_review, 30 days for pr_created.

Why this matters: without gated transitions, teams mark files as "done" when the PR merges. But "PR merged" and "verified in production with no regressions" are different things. 15 states and 12 gates make that gap explicit and enforceable.

Component mappings

12 mapping files. 100+ MUI-to-design-system transforms with prop-level translation.

A mapping isn't just "replace <Button> with <Button>." It's:

Import path change (@mui/material/Button to @company/design-system)
Prop renames (variant="contained" to variant="default")
Prop removals (MUI's disableElevation has no equivalent)
Prop additions (design system requires size where MUI inferred it)
Composition changes (MUI's startIcon prop becomes a child <Icon> component)
Conditional mappings (MUI's TextField maps to 3 different components based on props)

Each mapping includes the confidence impact. A component with a complete mapping (all props translated, all variants covered) contributes 100% to the mapping coverage factor. A component with a partial mapping (3 of 7 props translated) contributes proportionally. This feeds directly into the confidence score.

The mappings are JSON, not code. Adding a new component mapping is a data change, not a code change. The migration bot reads the mappings; the bot code doesn't change when new components are added.

The ratchet

The ratchet pattern (monotonically decreasing violation counts, enforced through CI) is well-established. Dusty Burwell described it in 2019. What's less common is applying it at the write layer for design system imports.

PreToolUse hooks (66 lines, 17 tests) fire on every file write. If an edit introduces an import from the legacy framework in a file that's been marked "complete," the edit is rejected before it lands. Not at CI. Not at PR review. At the moment the engineer types the import. The engineer sees an error message explaining why and pointing to the design system equivalent.

This is not documentation ("please use the new components"). This is enforcement at the earliest possible point. The count of legacy imports only goes down, never up.

The ratchet catches a specific failure mode: an engineer working on an unrelated feature in a migrated file reaches for the familiar MUI component because it's what they know. Without the ratchet, the migration regresses silently. With it, the regression is blocked at write time, before the PR is even created.

Result: zero legacy imports in any file marked "complete." Zero review comments needed for that class of violation.

The learning log

11 entries over 3 months. Each entry: date, source, what happened, what rule it produced.

Examples:

A static hex color (#6464f0) wasn't converted to a design token during migration. Added a check to the migration step that flags unconverted hex values.
A migration check was too broad: it added design system config to bundles that didn't use any migrated components. Refined the check to verify component usage before applying config.
A Drawer component's portal rendered outside the scope wrapper, breaking ref-forwarding. Added a rule for portal scoping and updated the component's test suite.

The log is append-only with mandatory fields. Rules flow from failures, not from predictions. Starting the log on day 1 would have been better; the first three months of failures had to be reconstructed from git history.

What didn't work

Hardcoded confidence weights. The initial weights were guesses. Size was weighted too high (30%), semantic complexity too low (15%). After 20 real migrations, the data showed which factors actually predicted success. Recalibrating the weights changed the priority order for ~40% of the remaining files. Start with any weights; recalibrate from real data after the first batch.

Generic checks without scoping. A check that asks "does this package depend on the design system?" catches packages that imported one design system utility for an unrelated reason. Three bundles got unnecessary config added before the check was refined to verify actual component usage. Automation checks need nested conditions, not just existence tests.

The numbers

Metric	Value
Migration bot code	7,065 lines + 2,363 lines of tests
Component mappings	12 files, 100+ prop-level transforms
Structured data files	48 (~6,900 lines of JSON)
Scoring factors	5 (size, semantic, mapping, test, navigation)
State machine	15 statuses, 12 named gates, JSON-stored
Enforcement hooks	66 lines, 17 tests
Learning log entries	11 entries, 15+ rule refinements

If you're building a migration engine

Score before you migrate. Any scoring system (even a rough one) is better than migrating files in directory order. The scoring reorders 962 files so the first 50 are almost guaranteed to succeed, building confidence and catching mapping gaps early.

Gate your transitions. "PR merged" is not "migration complete." Define what "complete" means, build the states between "started" and "complete," and enforce the gates.

Build the ratchet in week 1. Every day without enforcement is a day legacy imports can re-enter the codebase through unrelated work.

Start the learning log immediately. The failures from the first 10 migrations will produce the rules that prevent failures in the next 100.

"Told" vs. "Known": The One Variable That Predicts AI Design Tool Quality

James Coombs — Mon, 27 Jul 2026 17:32:20 +0000

James Coombs is a design engineer who built a 60-component design system with a structured query server and ran a controlled ablation study (n=9) measuring AI tool fidelity against it.

After evaluating six categories of AI design-to-code tools against a 60-component design system, one variable predicted output quality better than anything else: whether the tool had been told about the design system or knew it.

"Told" tools: You describe your components in natural language or docs. The AI interprets. It generates something that looks like your Button but uses the wrong prop names, the wrong import path, or a plausible-but-nonexistent variant.

"Known" tools: The AI queries a structured index of your actual components. It gets the exact name, exact props, exact import, exact usage examples. No interpretation gap.

Tool type	Awareness	Typical fidelity
General AI prototypers (v0, Bolt, Lovable)	Told (generic framework knowledge)	~50-60%
AI design tools (Claude Design, Pencil)	Told (LLM ingestion of your docs)	~70-80%
AST-parsed indexers (Builder.io Visual Copilot)	Known (parsed component source)	~95-100%
Manual component maps (Figma Code Connect)	Known (hand-mapped per component)	100%
AI agent + structured query (MCP server)	Known (live query against real source)	100%

The gap between "Told" and "Known" isn't incremental. It's categorical.

Why "Told" fails

When you tell an AI "use our Button component with variant='primary'," the AI generates something that looks like a primary button. Maybe it uses your component, maybe a common library's. Maybe variant='primary' is a valid prop, maybe your API uses variant='default'. Maybe it applies your violet palette, maybe generic blue.

Each step introduces interpretation drift. By the time the output reaches an engineer, it looks close enough to pass a design review but wrong enough to require a rewrite.

Worse: "Told" tools hallucinate components that don't exist. They use prop names that are close but wrong. They apply tokens that look plausible but aren't in your system. This is worse than obviously wrong output. Obviously wrong gets rewritten. Plausibly wrong gets merged.

And your design system changes. Components get new variants, tokens get renamed. "Told" tools have whatever version was in their training data. "Known" tools query the current state.

The three "Known" approaches

AST-parsed component index. Parse your source into a structured index: component names, prop types (from TypeScript), variants, tokens. Regenerate on build. Captures structure but not intent; doesn't know when to use Dialog vs. Sheet.

Hand-mapped component bridge (Figma Code Connect). Manually map each Figma component to its React counterpart. 100% fidelity by definition. But manual maintenance: every new component or variant needs an update. Scales linearly with component count.

AI agent + structured query (MCP server). Give the agent a query interface. It discovers components by searching, not guessing. No pre-built index needed. Works for discovery ("what handles file uploads?"), not just lookup. You build the interface once, and every AI tool, current and future, benefits.

The data

My ablation study (n=9, controlled) confirmed this:

Agent + MCP query (Known): 27.5 / 30 on design system compliance
AI design tool interpretation (Told): 16.3 / 30
CLAUDE.md rules ("always use the design system"): 16.1 / 30, identical to no guidance

The MCP server took about a week to build. It gives every AI tool structured access to the real design system. The cost-per-tool amortization improves with every new tool that connects.

When "Told" is fine

Throwaway prototypes where 70% fidelity is the goal. Greenfield exploration when you don't have a design system yet. Product demos where brand fidelity doesn't matter.

When you need "Known"

Production code generation. Design system migrations, where "close but wrong" imports are worse than obviously wrong ones. Multi-person teams where plausibly wrong output gets merged. Brand-critical surfaces where your violet isn't someone else's purple.

The practical takeaway

Before evaluating any AI design tool, ask one question: does it query my real components, or does it guess?

If your team generates AI code that goes into production, invest in a structured query interface. MCP server, CLI tool, AST index: the form factor matters less than the structured access. Build it once; every AI tool benefits. The returns compound.

Don't try to improve "Told" fidelity by writing better documentation or more detailed prompts. My data says more constraints actually degrade output quality (density inversion; see "Your CLAUDE.md Rules Achieve 0% Compliance" for the full methodology). A 2-sentence prompt + structured query beats a comprehensive governance file by 11 points.