Every image API returns the same thing: a URL. A PNG. And the moment you have that PNG, the design is finished — not "done", finished. You cannot ask it to make the headline smaller, or to swap the accent colour because the client's brand changed last week, or to hide the background so a designer can check the type in isolation. There is no "the headline". There's a rectangle of pixels where a headline happens to be.
That is fine for a mockup and fatal for a production pipeline. Clients revise. Revisions are cheap combinatorially and expensive pixel-wise: each one costs another generation, another roll of the dice, and the dice are loaded against the one thing you need — the rest of the image staying the same.
I spent a while driving the ArtCraft creative suite (PhotoCraft, VectorCraft, FilmCraft, WordCraft, PdfCraft — open source, Apache-2.0) from TypeScript, and the whole thing rests on refusing that collapse. This post is the smallest useful version of the idea: one Next.js route, one LLM call, one MCP batch, and a .pcraft document instead of a PNG.
The loss is information, not aesthetics
An 8-bit RGB image of W × H is a function I: {0..W-1} × {0..H-1} → {0..255}³. That's the entire content of the file. Everything else — what a region is, which operation produced its current value, what should happen if a client asks for it warmer — is destroyed when a computation history is collapsed into a final measurement.
It's the same loss as evaluating an expression. (x + 3) · sin(y) has structure: a tree, parameters, a defined sensitivity to each input. Its value at x=2, y=0.5 is one number, and from that number you cannot recover the expression. You can't ask it which term dominates y; you can't differentiate it. You can only measure it and move on.
Flat generation amplifies that over a whole pipeline. Compositional intent, colour science, relational facts, counterfactual potential — all gone the instant the model emits its final tensor. The file looks right, and it has no handles. Professional work is nothing but a system for grabbing handles.
The document as a reactive state graph
A modern creative document is not a picture. It's a directed acyclic graph of operations over separable sources: every layer is a raster, an adjustment node, a transform, a mask, a composite operator, or a procedural generator. Open a .pcraft file and you're opening a JSON tree.
Rendered pixels are the output of evaluating that graph bottom-up: read the leaves, transform adjustments, multiply masks into alphas, fold the stack. But evaluation is not the document. Evaluation produces a view; the document is the plan that produces the view.
This is React, one level down. A component is a function from state to UI; the DOM is not the source of truth, it's a render of it. Change state, and the framework reconciles the minimum. A layered document works identically: change a property node, and the app recomputes that node, propagates it through the transform group, and re-folds the stack. The undo stack stores the before state of the property, not the before pixels.
That's what makes "make the subhead a bit lighter" a one-line operation instead of a new image.
Why screen-scraping automation never worked
Creative automation used to mean AppleScript and ExtendScript driving a GUI. Which is screen-scraping with extra steps: it depends on the app being open, focused, at the right zoom, with the right panel visible and no modal in the way. Brittle by construction.
MCP inverts the direction. The engine's command registry — 500+ commands — is exposed as a flat namespace of typed operations with schemas, mediated by a handful of meta-tools:
| tool | what it does |
|---|---|
command_run { id, params, wait? } |
execute one registry command |
command_batch { steps, stop_on_error? } |
many commands in one IPC call |
command_list |
discover commands and their schemas |
doc_new, doc_open, doc_save, doc_export, doc_inspect
|
document lifecycle |
doc_render_preview { id, max_side } |
flattened PNG, for vision models only |
One word of warning about command_batch: it is not transactional. It's one round-trip instead of N — which is where the latency win comes from — but if step 4 fails, steps 1–3 stay applied. Write your retries accordingly.
The server models the engine, not the GUI. Ask for a Gaussian blur with a given radius and the filter goes straight onto the active layer's pixels. The menu item in the desktop app dispatches the same command.
The consequence for agents is bigger than it sounds: an agent does not need vision to drive this pipeline. It needs schema discovery and command execution. It can enumerate layer operations, read parameter docs, and compose a workflow without ever looking at a pixel — and reach for a preview render only when the decision is aesthetic.
The route
A product brief in, a layered document out. The LLM produces a plan, Zod validates it, and the plan is executed as engine commands. The only image-adjacent thing in the file is the absence of an image.
// ---------------------------------------------------------------------------
// app/api/studio/layered-render/route.ts
//
// "Hello World" of the layered-generation pattern:
// 1. A free-text brief → 2. a structured plan (Zod-validated JSON)
// 3. that plan → 4. a MCP command_batch against PhotoCraft
// 5. the .pcraft file → 6. a save, then an inspect for downstream agents
//
// The output is NOT a PNG. It is a .pcraft document whose layers,
// masks, text runs, and adjustment parameters are all still addressable.
// ---------------------------------------------------------------------------
import { NextResponse } from 'next/server';
import { generateObject } from 'ai';
import { createOpenAI } from '@ai-sdk/openai';
import { z } from 'zod';
import { Client } from '@modelcontextprotocol/sdk/client/index.js';
import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js';
// ---------------------------------------------------------------------------
// 1. THE CONTRACT
//
// The shape of what the LLM is allowed to say. Every field the model emits
// is a parameter we can hand to a discrete MCP command. Anything that
// cannot be expressed as a PhotoCraft command does not belong in the plan.
// ---------------------------------------------------------------------------
const hex = z
.string()
.regex(/^#[0-9a-fA-F]{6}$/, 'must be a #rrggbb hex colour');
const CreativePlan = z.object({
canvas: z.object({
width: z.number().int().min(320).max(4096),
height: z.number().int().min(320).max(4096),
}),
palette: z.object({
background: hex,
foreground: hex,
accent: hex,
}),
headline: z.string().min(1).max(60),
subhead: z.string().max(160),
exposure: z.number().min(-1.5).max(1.5), // stops
contrast: z.number().min(-50).max(50), // levels
});
type CreativePlan = z.infer<typeof CreativePlan>;
// ---------------------------------------------------------------------------
// 2. BRIEF → PLAN
//
// We ask the model for structured data only, never for prose and never for
// a picture. `generateObject` validates the shape on the way out, so a
// hallucinated JSON blob is rejected before it can poison the tool batch.
// ---------------------------------------------------------------------------
async function planFromBrief(brief: string): Promise<CreativePlan> {
// Any OpenAI-compatible endpoint: the three OPENAI_COMPATIBLE_* variables.
const model = createOpenAI({
apiKey: process.env.OPENAI_COMPATIBLE_API_KEY,
baseURL: process.env.OPENAI_COMPATIBLE_BASE_URL,
})(process.env.OPENAI_COMPATIBLE_MODEL!);
const { object } = await generateObject({
model,
schema: CreativePlan,
system: [
'You are the art-director module of a design-tool SaaS.',
'Return ONLY a structured brief. No prose. No markdown.',
'Pick a restrained palette; keep the headline punchy.',
'Exposure is in stops (roughly -1.5 to 1.5).',
'Contrast is in levels (roughly -50 to 50).',
].join(' '),
prompt: `Product brief: ${brief}`,
});
return object;
}
// ---------------------------------------------------------------------------
// 3. THE MCP BRIDGE
//
// One helper that owns the lifecycle of a stdio MCP session against
// photocraft-cli. The engine is headless: no window, no GPU, no user.
// Read/write roots are pinned to the working directory so the automation
// sandbox cannot wander outside the SaaS job's scratch space.
// ---------------------------------------------------------------------------
async function withPhotoCraft<T>(
fn: (mcp: Client) => Promise<T>,
): Promise<T> {
const transport = new StdioClientTransport({
command: process.env.PHOTOCRAFT_BIN ?? 'photocraft-cli',
args: [
'mcp',
'--automation-read-root', process.cwd(),
'--automation-write-root', process.cwd(),
],
});
const mcp = new Client(
{ name: 'studio-layered-render', version: '1.0.0' },
{ capabilities: {} },
);
await mcp.connect(transport);
try {
return await fn(mcp);
} finally {
await mcp.close();
}
}
// ---------------------------------------------------------------------------
// 4. THE ROUTE
// ---------------------------------------------------------------------------
export async function POST(req: Request) {
const { brief, outputName = 'studio-render' } =
(await req.json()) as { brief: string; outputName?: string };
// 4a. Brief → plan. This is the only LLM call in the whole request.
const plan = await planFromBrief(brief);
// 4b. Plan → layered document, executed inside a single MCP session.
const result = await withPhotoCraft(async (mcp) => {
// Fresh RGB 8-bit canvas.
await mcp.callTool({
name: 'doc_new',
arguments: {
width: plan.canvas.width,
height: plan.canvas.height,
mode: 'rgb',
depth: 8,
background: plan.palette.background,
name: outputName,
},
});
// Every visual decision becomes a discrete, re-openable layer.
await mcp.callTool({
name: 'command_batch',
arguments: {
steps: [
// Background as a *fill layer*, not baked pixels.
{
id: 'layer.newFillLayer.solidColor',
params: { color: plan.palette.background },
},
// Accent block as a live, editable shape layer.
{
id: 'shape.create',
params: {
kind: 'roundedRect',
rect: [
Math.round(plan.canvas.width * 0.07),
Math.round(plan.canvas.height * 0.12),
Math.round(plan.canvas.width * 0.86),
Math.round(plan.canvas.height * 0.76),
],
radii: 48,
fill: plan.palette.accent,
stroke: null,
name: 'Accent Panel',
},
},
// Headline as live type — the string stays editable.
{
id: 'type.create',
params: {
x: Math.round(plan.canvas.width * 0.11),
y: Math.round(plan.canvas.height * 0.35),
text: plan.headline,
font: 'Inter',
size: Math.round(plan.canvas.height * 0.09),
color: plan.palette.foreground,
},
},
// Sub-headline as its own type layer.
{
id: 'type.create',
params: {
x: Math.round(plan.canvas.width * 0.11),
y: Math.round(plan.canvas.height * 0.51),
text: plan.subhead,
font: 'Inter',
size: Math.round(plan.canvas.height * 0.032),
color: plan.palette.foreground,
},
},
// Exposure + contrast as an adjustment layer, not a raster pass.
{
id: 'layer.newAdjustmentLayer.brightnessContrast',
params: {
brightness: Math.round(plan.exposure * 100),
contrast: Math.round(plan.contrast),
},
},
],
},
});
// Persist as a *native* document, not a screenshot.
const saved = await mcp.callTool({
name: 'doc_save',
arguments: { path: `${outputName}.pcraft` },
});
// Hand the agent tree back to the caller. The next revision cycle
// can address any node by id without re-running the LLM.
const inspection = await mcp.callTool({
name: 'doc_inspect',
arguments: {},
});
return { saved, inspection };
});
return NextResponse.json({
ok: true,
plan,
document: `${outputName}.pcraft`,
inspection: result.inspection,
});
}
Five steps in the batch, and each one is a promise about the future:
- the background is a fill layer, not the canvas colour, so a later agent can hide it and the canvas colour shows through;
- the accent block is a shape layer with geometry, not a flattened rectangle;
- headline and subhead are type runs — the strings stay editable;
- exposure and contrast are an adjustment layer, so changing your mind is a parameter change, not a re-render.
The response is { ok, plan, document, inspection }. No image. That's deliberate: a PNG response would tempt downstream code to treat the output as finished, while a .pcraft handle forces it to treat it as revisable structure. inspection is the whole layer tree, so the next service — a brand-compliance check, a colour audit, a revision — can address the Accent Panel by layer id and change it without calling the model at all.
Multimodal, as a translator rather than a generator
"Multimodal" usually gets read as "the model sees an image and draws a replacement". In a production studio it means something narrower and much more useful: the agent inspects a render, forms a critique, and translates it back into symbol-space operations.
Say an agent applies develop settings and wants to know if the shadows clipped. It calls doc_render_preview { id, max_side: 1024 }, sends that PNG to a vision model with a tightly scoped question, and gets a scalar back. Then it applies that scalar to the adjustment node.
The model never draws. It labels and evaluates; the mutation is one parameter in a closed, symbolic space. That's the difference between measure, decide, adjust and prompt, sample, hope.
The pipeline as a compiler
It helps to call the layered document a source language and flat pixels machine code. The engine is the toolchain: parser, optimiser, platform backends. Revisions are recompiles from source with different inputs. Skip the source and every revision is a manual binary patch.
Which is why the file extension matters more than it sounds. .pcraft is the source. .png is the build artifact, and you should be able to produce it again at any resolution, in any colour space, without asking a model to guess what you meant.
Run it
The repository is MIT and it's here: github.com/edgarmilvus/artcraft-book. The harnesses re-execute these commands against the real engine binaries. Honest about the cost first: the engines are about 4 GB of CLI builds, and a fresh clone needs them installed once.
git clone https://github.com/edgarmilvus/artcraft-book
cd artcraft-book
npm ci # one node_modules, linked into every project
./tools/install_craft_clis.sh # the ten engine binaries, checksummed
./chapter1/run_all.sh # 11 harnesses + a verifier
No API key is needed for any of that: the harnesses never call a model, they replay the engine commands and assert the files that come out.
Twenty chapters, 144 projects: layered .pcraft documents, vector booleans with editable paths, integer-timebase video, signed PDFs and DOCX, spreadsheet formulas. Each project ships the harness that produced its output, so a red line means the engine moved, not that you misread the prose.
run_ex2 20 assertions passed
And when something does break, it says which file to open:
run_ex2 FAILED — see .../output/run_ex2.log
If you want the reasoning behind each decision — the failures, the shapes that didn't work, the parts where the obvious answer is wrong — there's an ebook that follows this code chapter by chapter.
- github.com/edgarmilvus/artcraft-book — MIT, 20 chapters, 144 projects, every harness runnable
- ArtCraft: The Agentic Creative Studio in TypeScript — the ebook
The repo stands on its own; the book is the prose around it. Corrections welcome — if something here misdescribes how the engines behave, tell me and I'll fix it.
Top comments (0)