Every "AI generates UI" tool ends the same way. You get a picture, or you get a pile of React you did not ask for.
A picture cannot be edited by a designer. Generated React is a diff nobody reviewed. Neither one is a design.
I wanted the middle. A real, editable design file, produced by agents, in a tool a human can keep working in. So I built Canvas Crew: four agents that argue over a single Penpot canvas until the design passes. Everything runs on my machine, and the project has zero dependencies.
This post is the architecture, the code, and the four bugs that only showed up when I pointed it at a real canvas.
Why Penpot
Penpot is the interesting middle ground, for three reasons.
It is open source, so you can self host it and keep the data. It stores designs as real web structures, so the file is legible to a program. And since 2.18 it ships an MCP server, which means an agent can read and write the canvas directly.
The catch is that Penpot's MCP surface is essentially one tool: execute_code. The model writes raw Penpot Plugin API JavaScript, and that code runs against your actual design file. There is no typed create_frame for it to call. Codegen quality is the entire ballgame.
That constraint shaped everything else.
The one decision that makes this work
Most agentic design tools iterate on generated code. That is why they fall over. A mid-tier model asked to patch imperative canvas code will quietly corrupt the file, and you have no way to see what changed.
Canvas Crew iterates on a specification instead.
brief ──► ScreenSpec ──► Plugin API JS ──► live canvas ──► PNG ──► critique
▲ │
└────────────────── revise (findings + computed facts) ──┘
The spec is a small JSON document: a palette, a type scale, and an ordered list of sections with their copy and colours.
{
"title": "Inkwell Pricing",
"board": { "width": 1440, "height": 900, "background": "#0B0B0F" },
"palette": [
{ "name": "bg", "hex": "#0B0B0F" },
{ "name": "text", "hex": "#FAFAFA" },
{ "name": "accent", "hex": "#4F46E5" }
],
"type_scale": [
{ "name": "display", "size": 48, "weight": 700 },
{ "name": "body", "size": 18, "weight": 400 }
],
"sections": [
{
"id": "hero",
"role": "hero",
"height": 200,
"background": "bg",
"text": [
{ "role": "headline", "copy": "Your notes, your disk.", "size": 48, "weight": 700, "color": "text" }
]
}
]
}
Three things fall out of this that do not fall out of code iteration:
-
A spec is diffable. Every round writes
spec-diff.json, so you can see exactly what the critic changed. No guessing about what the agent did. - A mid-tier model can revise JSON. JSON in, JSON out, is far more reliable than surgical edits to imperative canvas code.
- The spec is where the accessibility maths lives, which means the gate has something deterministic to check.
The canvas is a projection of the spec. Each round re-renders it from scratch, so a bad round can never compound into a broken file.
The crew
Four roles, each with a different job and a different capability requirement.
| Role | Job | Needs |
|---|---|---|
| Planner | brief to spec, then revise the spec from findings | JSON mode |
| Builder | spec to Plugin API JavaScript, executed on the canvas | tool calling |
| Critic | looks at the exported PNG and judges the pixels | vision |
| Arbiter | decides whether another round is worth spending | JSON mode |
The loop in src/crew.js is small:
const planned = await planner.plan(brief);
spec = planned.spec;
for (let round = 1; round <= config.maxRounds; round++) {
const rendered = await builder.render(spec); // spec -> canvas
const exported = await canvas.exportShape(rendered.boardId, 'png'); // canvas -> PNG
const audit = auditSpec(spec); // deterministic
const critique = await critic.critique({ brief, spec, audit, round, imageUrl });
const verdict = await arbiter.decide({ history, round, audit });
if (verdict.decision === DECISIONS.ACCEPT || verdict.decision === DECISIONS.HALT) break;
spec = (await planner.revise({ spec, critique, audit, history })).spec;
}
The exported PNG is the oracle. The critic judges pixels, not intentions.
The part that is deliberately not a model
Contrast is arithmetic. It does not belong in a language model.
src/a11y.js implements the WCAG 2.x relative luminance definition directly:
export function relativeLuminance(hex) {
const rgb = parseHex(hex);
if (!rgb) return null;
const [r, g, b] = rgb.map((v) => {
const c = v / 255;
return c <= 0.03928 ? c / 12.92 : Math.pow((c + 0.055) / 1.055, 2.4);
});
return 0.2126 * r + 0.7152 * g + 0.0722 * b;
}
export function contrastRatio(a, b) {
const la = relativeLuminance(a);
const lb = relativeLuminance(b);
if (la === null || lb === null) return null;
const light = Math.max(la, lb);
const dark = Math.min(la, lb);
return (light + 0.05) / (dark + 0.05);
}
auditSpec() walks every text block in the spec, resolves its colour against the surface behind it, and returns findings with the measured ratio and the required threshold.
Those findings are then merged into the critic's findings and forced to at least major:
for (const f of audit?.findings ?? []) {
const already = findings.some((x) => x.area === f.area && x.issue.slice(0, 40) === f.issue.slice(0, 40));
if (already) continue;
findings.push({
severity: f.severity === 'blocker' ? 'blocker' : 'major',
area: f.area,
issue: f.issue,
fix: f.fix,
});
}
A model can be argued with. A ratio cannot. If there is a computed blocker, the verdict is revise no matter what the critic claimed, and the arbiter will not accept the design.
This turned out to be the highest-value part of the project. On a real run the gate caught two call-to-action buttons whose labels were rendering at 1.00:1 contrast, which is to say invisible, and drove the spec to fix them.
Why the arbiter is rules-first
An LLM that decides when the design is "good enough" is how you get a loop that either never stops, or declares victory on a broken screen.
So the decision is a pure function in src/roles/arbiter.js, and the model is consulted only for the reason text:
export function applyRules({ history, round, maxRounds, passScore }) {
const latest = history[history.length - 1];
// 1. Budget. Always respected, whatever the score says.
if (round >= maxRounds) return { decision: DECISIONS.HALT, rule: 'budget' };
// 2. Computed blockers must be fixed before anything is accepted.
if (latest.blockers > 0) return { decision: DECISIONS.REVISE, rule: 'blockers' };
// 3. Good enough and clean.
if (latest.score >= passScore) return { decision: DECISIONS.ACCEPT, rule: 'pass-score' };
// 4. Oscillation guard: three flat rounds means another round goes in circles.
if (history.length >= 3) {
const [a, b, c] = history.slice(-3).map((h) => h.score);
if (c <= b && b <= a) return { decision: DECISIONS.HALT, rule: 'no-improvement' };
}
// ...plus a regression guard for a score that falls sharply
}
Every branch is unit tested. That is the whole point. You cannot unit test a model's opinion, so the opinion does not get to decide.
Four things that broke, and what they taught me
These are the reason I am writing this post. None of them showed up in tests against mocks. All four showed up the moment I pointed it at a real canvas and a real model endpoint.
1. Penpot reports tool failures without setting isError
This is the dangerous one.
Penpot returns a failed tool call as ordinary text content with the isError flag absent. My client did Boolean(result.isError), so every failed call looked like a success.
The consequence was silent and nasty. The plugin connection died between the build step and the export step. The build reported "executed cleanly". export_shape returned no image. The critic quietly fell back to reviewing the spec. The run looked healthy and the report looked fine, while the critic had never seen a single pixel.
The fix is to detect failure from the content:
const TOOL_FAILURE_PATTERNS = [
/^Tool execution failed/i,
/^Error:/i,
/No Penpot instance connected/i,
/frozen by the browser/i,
/No userToken found in session context/i,
];
export function looksLikeToolFailure(text) {
const t = String(text ?? '').trim();
if (!t) return false;
return TOOL_FAILURE_PATTERNS.some((re) => re.test(t));
}
The patterns are anchored on purpose. A successful execute_code can legitimately return data containing the word "Error", so an unanchored match would poison good results.
Lesson: silent degradation is worse than a crash. If a step cannot be done, say so loudly.
2. A vision call without a system prompt returns prose
The critic needs the render. So I sent it an image.
What I did not send was its system prompt, which is where the "Return ONLY a JSON object" contract lives. With no contract, the model replied in markdown. The JSON parse failed. And then the fallback did something worse than failing.
3. A parse failure must never become a default score
When the critique could not be parsed, my code substituted a default 5/10.
That is worse than no score at all. The arbiter then wrote a paragraph of reasoning about a number nobody had produced. The transcript said "Critic scored 5/10" with total confidence. The loop was optimising against fiction.
The fix:
// No parsed response means no score. Never substitute a default.
const score = parsed ? clamp(Number(src.score), 1, 10, null) : null;
An unparsed critique now carries score: null, is marked parsed: false, and halts the run with rule critique-unparsed.
Lesson: if a model's judgement is missing, the honest output is a missing judgement. A plausible default is indistinguishable from a real value downstream, which makes it the worst possible failure mode.
4. Reasoning tokens eat your max_tokens budget
These endpoints serve reasoning models, and they emit a reasoning_content field. Reasoning tokens are drawn from the same max_tokens budget.
With a small budget, the model can spend everything on reasoning and return an empty content string.
This bit my own capability probe. A probe at max_tokens: 40 reported vision=no for a model that reads images perfectly well. The probe was measuring my budget, not the model.
The fix is to treat both empty content and truncation as retryable rather than as a capability signal:
const empty = !contentOf(parsed) && !msg?.tool_calls?.length;
const truncated = choice?.finish_reason === 'length';
if (retryOnEmpty && (empty || truncated) && attempt < retries) {
if (typeof body.max_tokens === 'number') {
body = { ...body, max_tokens: Math.min(body.max_tokens * 3, 16_000) };
}
continue;
}
Lesson: an empty response is not evidence of a missing capability. It is evidence that you under-budgeted.
What a real run looks like
The first live run, against a self-hosted Penpot 2.18, three rounds, on the brief "a landing page for DailyBuild.xyz":
Round 1 builder wrote 253 lines, executed on the canvas | gate: 11 blockers | critic 5/10
Round 2 builder wrote 267 lines, executed on the canvas | gate: 2 blockers | critic 4/10
Round 3 builder wrote 193 lines, executed on the canvas | gate: 0 failures | critic 7/10
Arbiter: HALT (budget) scores 5 -> 4 -> 7
The builder wrote and executed real Plugin API JavaScript against a real canvas three times. The planner revised the spec twice, adding a deeper accent colour to fix two buttons whose labels were invisible. The gate drove the blockers from 11 to 0.
That run also had the export bug from lesson 1, which is how I found it.
Every round is written to disk: the spec, the generated canvas code, the exported PNG, the critique, the computed audit, and the spec diff. Plus a report.html with the rounds side by side and the score trajectory. Nothing is hidden behind a spinner.
Run it yourself
No dependencies, so there is nothing to install. You need Node 20 or newer, a self-hosted Penpot 2.18+ with enable-mcp in PENPOT_FLAGS, and any OpenAI-compatible inference endpoint (Particle.ai, LM Studio, Ollama, Gemini).
cp .env.example .env # Penpot MCP URL with your token, plus an API key
npm run check # probes the MCP and measures what your models can do
npm run crew -- --brief "a pricing page for a local-first note taking app"
npm run check is not decoration. It measures each model's real tool-calling, vision and JSON support with live calls, and refuses to start a run whose roles cannot be served. My two available models turned out to be genuinely different: one was consistently good at vision, the other was intermittent. A hardcoded capability table would have got that wrong.
There are 47 tests, and they run with no Penpot and no API keys, because the suite drives a mock Penpot server and a mock inference server. Two of those mocks exist purely to reproduce the bugs above: one reports failures without the isError flag, and one has the critic reply in prose.
One honest caveat: the plugin lives in a browser tab, so the connection drops when that tab loses focus. Running the crew from a terminal on the same machine hides the browser and causes exactly that. Keep the Penpot window visible beside your terminal. A headless bridge that pins a browser and keeps the plugin session alive is the obvious next step, and it would unlock cron and CI.
What is next
The interesting open lanes, roughly in order of value:
- The headless bridge. Remove the focused-tab requirement. This is the single biggest constraint on the project.
- Screenshot to spec. The inverse direction: give it an image of any UI and get an editable Penpot board back, not a picture.
- Multi-screen flows. Generate linked boards for onboarding, empty and error states, then export a clickable prototype.
-
A
.penpotreader and writer for Node. The community has asked for this repeatedly. It would let agents work on the declarative file format directly, with no browser in the loop at all.
The code, the full trap list and the fork guide are in the repository. If you build something on it, I would like to see it.
Code & more: https://www.dailybuild.xyz/project/282-penpot-canvas-crew
Top comments (0)