If you've ever handed an AI coding assistant a beautiful Figma design and gotten back a UI that's "close but not quite," you already know the frustration. The spacing is off. The button is the wrong shade of red. The layout technically works but doesn't feel right.
And the first instinct is always the same: "I need a better model."
So you switch. You try Claude Opus. You try Fable. You try whatever the newest, sharpest model on the leaderboard is. And the result is... still off. Maybe slightly better. Still not pixel-perfect.
Here's the thing I've learned after a lot of trial and error building UI with AI agents: the model was never the bottleneck. Your workflow is.
The gap between what you meant and what the model built is bigger than it looks — until you put them side by side.
The real problem: you're describing the UI, not showing it
Think about how most people work with AI on frontend tasks. You look at a design — a Figma file, a screenshot, a reference site — and you translate what you see into words. "There's a box in the top left. It has a red button. The text is bold and centered." Then you hand that description to the model, or to an agent, and ask it to build the thing.
This feels reasonable. It's exactly how you'd brief a junior developer, right?
Except imagine you actually did this with a human. Imagine you're a manager who is excellent at describing UI. You look at the design and you explain it, element by element, in perfect detail. But you never actually show the developer the design itself — just your words about it.
Even with a flawless description, one of two things goes wrong:
- Your description, however good, is never 100% of the actual visual information. Spacing, alignment, subtle shadows, exact hierarchy — some of it always gets lost in translation to text.
- Even if your description were perfect, there's no guarantee the person (or agent) executing it interprets your words the same way you intended.
That's the gap. And it's not a "the AI isn't smart enough" gap — it's an information-loss gap. You're compressing a rich visual artifact into text, and text is a lossy format for visual information.
Why "just write a good plan first" doesn't save you
There's a popular pattern in the AI-coding world right now: plan first, then execute. And for most engineering tasks, it genuinely works — planning before execution produces more reliable results than diving straight into code.
But for UI work specifically, this same pattern quietly breaks down, for a very specific reason: context decay.
Here's what actually happens. You look at the design. You write a detailed plan describing it — box here, button there, this color, that spacing. Writing that plan burns a lot of context. By the time the model actually gets to executing the plan, several things have happened:
- The original design is no longer in front of the model — only your text description of it is.
- That text description is now sitting further back in the context window, competing with everything else that's been generated since.
- The model starts filling gaps with plausible-sounding assumptions instead of facts. That's hallucination, and UI is one of the easiest places for it to happen invisibly — the output still looks like a UI, just not your UI.
So even a great developer (human or AI) working strictly from their own written plan, without the source design still visible, will drift. Not because they're bad at their job. Because the further you get from the source of truth, the more your brain (or the model's context) starts reconstructing instead of referencing.
This is true regardless of which model you use. Opus, Fable, GPT-4, whatever comes next — if the workflow separates the design from the execution step, you'll get drift. The model isn't the variable. The distance between "seeing the design" and "writing the code" is.
The same problem shows up with agents — arguably worse
If you're using sub-agents or a front-end-specific agent to delegate UI work, this problem doesn't go away. It gets amplified.
Here's why: an agent starts with a fresh context. It doesn't know anything except what you explicitly hand it. If your main orchestrating chat only passes along a text description of the UI — "build a card component with a red CTA button, left-aligned title" — the agent has no way to close the gap you couldn't close yourself. It's working purely off secondhand description, filtered through your monolithic chat's interpretation of the design in the first place.
Garbage in, garbage out — except it's not garbage, it's just lossy, which is worse, because it looks plausible right up until you compare it side-by-side with the original.
The fix: treat your plan and your agents like real employees, not API calls
The mental shift that fixed this for me was simple: stop treating the planning step and the agent-delegation step as purely textual handoffs. Start treating them the way you'd manage an actual team.
If you hired a contractor to build a UI from a Figma file, you wouldn't just describe the design to them over the phone and expect pixel-perfect results. You'd hand them the actual file. You'd expect them to reference it constantly, not just once at the start. And you'd have them compare their output against the original before calling it done.
Do the same thing with your AI workflow:
1. Always attach the actual design artifact — not just a description of it.
Whether that's an HTML export from Figma, a screenshot, or a reference URL, make sure the literal design file travels with the plan, not just your summary of it. Your plan document shouldn't just say "build a card with a red button" — it should explicitly reference and include the design file itself, so it's retrievable at execution time, not just at planning time.
2. If you're using agents, hand the design file directly to the agent — every time.
Don't let the design get filtered through your main chat's paraphrase of it. When you delegate frontend work to a sub-agent, explicitly pass along the actual HTML/design file in that handoff. The agent has zero prior context, so whatever you don't give it, it will never know existed.
3. Close the loop — don't just generate once and move on.
This is the part most people skip. Set up a verification loop using /goal where the agent builds the UI, then checks its own output against the original design, and iterates until it actually matches — not until it's "probably close enough." This kind of self-checking loop is honestly underused right now; a lot of people don't build it into their workflow at all, but it's the difference between "AI-generated approximation" and "pixel-perfect implementation." The agent builds, compares against the source design, and if it doesn't meet the bar, it goes again — automatically — until it does.
The takeaway
If your AI-generated UI keeps coming out "almost right," don't reach for a bigger model first. Reach for a better handoff.
- Keep the actual design artifact in the loop at every stage, not just at the start.
- Don't let your plan become the only source of truth — the plan should point back to the design, not replace it.
- If you're using agents, give them the real file, not your paraphrase of it.
- Build in a compare-and-correct loop instead of a single-shot generation.
The model was never the bottleneck. The workflow was. Fix the handoff, and pixel-perfect UI stops being a lucky outcome and starts being the default.
Written by Muneeb Khan — senior full-stack engineer and architect working on AI-assisted development workflows. Connect on LinkedIn.
Top comments (0)