Engineering an AI Design System

Why do AI agents struggle with frontend design?

AI agents can build working interfaces remarkably quickly. Yet the result can still feel generic, visually flat or unlike the experience I had in mind. A single screen might look acceptable in isolation while making no sense within the broader application. Asking for a “better design” often produces another variation without revealing where the first attempt went wrong.

I encounter this problem from a specific position: I am not a frontend designer. I do not have the skill to open Figma and create a visually stunning interface from a blank canvas.

When a design isn't working, I can tell you that it's not right. But I can't tell you how to fix it. So I need to make AI-generated design a more reliable and useful process.

So I can have less of this:

And more of this:

Can frontend design be engineered?

I’m engineering the system around AI coding. Frontend design is one process within that larger investigation. If I can break the work into observable stages with explicit inputs, outputs and feedback loops, I can begin improving the process rather than treating AI-generated frontend design as a black-box lucky dip.

My hypothesis is that separating AI-generated design into explicit stages, handoffs and feedback loops will make AI a more reliable tool for producing frontends that are visually strong and coherent with the wider product.

I want AI-generated design to become more predictable, reliable and globally coherent. New features should preserve the product intent, interaction rules and visual language instead of locally optimising one screen at the expense of the broader experience.

This does not mean eliminating taste or agent creativity. The goal is controlled variation: different creative outcomes produced within a process that reduces unexplained drift and makes intervention possible.

What is an agent actually doing when it designs?

I know my prompting isn't 100% perfect, but I thought I was doing a pretty good job in my prompts.

"Recreate the mission room incorporating react flow, semantic zoom, object-graph mapping adapter, highly interactive using spatial layout techniques, immersive feel as a user with stage inspector, and use a vertical top-down node direction with wires connecting. Nodes to be clickable to expand left to right."

Apparently not.

To understand how the prototype was made, I reviewed the agent transcript. It turns out that the agent had not simply received my prompt and “designed a screen.” Between my request and the working prototype, it had to:

1. Interpret my request using the accumulated product context.
2. Translate a broad ambition into a specific design problem.
3. Decide what a successful experience should achieve.
4. Inspect the existing application, repository and visual references.
5. Identify the legitimate product objects.
6. Define the relationships between those objects.
7. Define their important attributes and states.
8. Create realistic data and operational scenarios.
9. Decide the information hierarchy.
10. Decide what should appear at different levels of depth.
11. Identify the components the interface required.
12. Define their anatomy, variants and states.
13. Compose the overall screen and spatial layout.
14. Define node, edge, camera and semantic-zoom behaviour.
15. Define selection, navigation, focus and inspector behaviour.
16. Preserve identity and spatial context as the user navigated.
17. Generate several structurally different design directions.
18. Choose the typography, colour, material, composition and motion language.
19. Choose the technical approach, prototype scope and supporting scaffolding.
20. Construct, exercise, compare, diagnose, revise and verify the prototypes.

Generating UI designs is not one task.

Those twenty steps contain a vast number of assumptions and decisions that the agent quietly makes for me. They were not a formal sequence. I grouped them into six stages so that each decision had an owner and an explicit handoff.

Why didn't better prompts solve the problem?

I had treated AI-generated design as two stages: build something and evaluate it. In this model, those are stages five and six. The first four stages were still happening, but hidden inside the agent's execution.

Incorrect product assumptions, weak experience structure and generic creative direction could therefore compound before I saw the interface. A larger prompt added context, but it did not make the decisions or handoffs visible. I needed explicit boundaries—not simply more words.

The six-stage framework

My current model contains six core stages and one supporting lane:

These are working boundaries, not an established design standard.

1. Product Intent

Product Intent defines what is being built, for whom and why: the problem, scope, desired experience, constraints and non-goals. It supports intent preservation by allowing later agents to distinguish an intentional decision from an attractive invention.

2. Experience Architecture

Experience Architecture defines what exists, how it is organised and how it behaves: objects, relationships, information hierarchy, layout, states and interactions. These rules provide constraint preservation, helping new screens and features remain globally coherent as the product grows.

3. Creative Direction

Creative Direction defines how a candidate should look and feel. It turns references and preferences into a visual thesis covering composition, typography, colour, material, density and motion. Several approaches can then explore controlled variation while preserving the product's broader visual language and design continuity.

4. Implementation Blueprint

The Implementation Blueprint translates the experience and creative direction into a construction plan: data, technical choices, components, scope and verification criteria. It makes the Definition of Done concrete enough to build and inspect.

5. Prototype Construction

Construction turns those decisions into a runnable experience: components, screens, state, interactions, navigation and styling. Builders retain local judgment without silently redefining upstream product, interaction or visual decisions.

6. Prototype Evaluation

A successful build is not automatically a successful prototype.

Evaluation closes the loop. A navigation failure may belong to Experience Architecture; a generic result may belong to Creative Direction; a broken click path may belong to Construction. Evaluation must also consider the broader product, preventing local optimisation of one screen at the expense of the overall experience. The aim is to correct the earliest wrong decision rather than repeatedly patching the final screen.

Supporting lane: Build Scaffolding

Repositories, starter applications, worktrees, shared schemas and conventions are Build Scaffolding. They support design continuity and robustness across iterations and agent handoffs, but are not another creative stage each candidate should repeat.

The framework moves control upstream: from correcting final pixels to shaping the decisions that produce them. A stage may produce a diagram, table, reference or a few explicit decisions. Its value is whether it removes an important ambiguity from the next stage.

Can this work for someone who isn't a designer?

I want agents to propose experience structures and genuinely different creative directions that I can compare against stable product intent. The goal is not deterministic templates. It is enough structure to explore strong ideas without each candidate silently reinventing the product.

Does it actually work?

A series of experiments and tests over the coming weeks will be the judge.