Vincent Tran

Posted on Aug 26 Originally published at 0xgosu.dev on Aug 26

Teaching a Language Model to Paint with Editable Code

#ai #productivity #software

Most image generators give us one narrow way to participate: describe an image, wait for pixels, and ask the model to try again. The result may be beautiful, but it is also flattened. We cannot select the loop that drew one petal, change the color rule behind a wash, or make the composition denser by editing the artifact itself.

Surya Narreddi and collaborators explored a different medium. They trained a language model to write complete JavaScript sketches using p5.brush , rendered those programs into watercolor-like images, and improved the model with reinforcement learning. The deliverable is not only a PNG. It is a program that remains available for inspection and revision.

That apparently small change—from generating pixels to generating the procedure behind the pixels—reshapes the whole system. Rendering becomes a tool call. Compilation becomes a hard correctness check. Aesthetic taste must become a reward. Prompt design becomes part of the action space. And because there is no objective answer to “is this a good watercolor?”, the hardest engineering work moves into the evaluator.

The project is therefore about more than flowers. It is a compact case study in how to train models for subjective work without reducing taste to a vague score.

Code is a more participatory image format

A raster image stores a finished grid of colors. A creative-coding sketch stores decisions: brush selection, pigment, stroke order, geometry, randomness, density, and layering. Rendering turns those decisions into pixels, but the decisions remain editable.

That creates several useful properties:

Local changes are possible. A person can adjust the stem, petal count, palette, or background without asking the model to regenerate everything.
Behavior is inspectable. The source reveals whether the image came from careful structure, accidental complexity, or an invented API.
Results can be reproduced. With fixed code, dependencies, canvas dimensions, and random seeds, the image can be rendered again.
The medium can be extended. The same sketch can become an animation, print, parameterized series, or interactive tool.
Failure is legible. A blank canvas can be traced to syntax, unsupported methods, bad coordinates, or a poor visual idea.

The trade-off is equally important. Code does not magically make image creation easier. It adds a runtime, an API, a sandbox, and a much longer path from intent to picture. Direct image models are faster and have absorbed far more visual knowledge. The value here is agency over the artifact, not raw convenience.

The chosen drawing layer, p5.brush, adds natural-media tools to p5.js: pencils, charcoal, markers, watercolor fills, hatching, and vector fields that bend strokes. Instead of asking a language model to emit millions of pixels, the system asks it to compose a relatively small vocabulary of visual operations.

The training system closes a tool-use loop

Each training example begins with a visual request such as a peach hibiscus painted in watercolor. The language model produces a complete p5.brush program. Puppeteer runs the sketch in a browser environment and captures the canvas as a PNG. A separate model compares the result with curated reference images, and that preference is converted into the reward used to update the code-generating model.

The model acts in code, but the learning signal comes from the rendered image.

The loop connects two different spaces:

Token space , where the policy chooses JavaScript tokens.
Execution space , where the sketch either runs or fails.
Image space , where pigment, composition, depth, and resemblance become visible.
Preference space , where a judge decides which image better represents the target taste.

This separation is powerful. Syntax and API use can be checked exactly. The final visual result can be judged holistically. A program that compiles but paints a dull icon is valid code and poor art; a beautiful-looking snippet that calls imaginary functions never reaches the canvas. The reward has to recognize both cases.

It also creates an obvious security boundary. Model-written JavaScript is untrusted input. A production renderer needs process isolation, time and memory limits, restricted networking and filesystem access, deterministic dependencies, and an actual browser sandbox. Puppeteer’s own troubleshooting guidance strongly discourages running Chromium without its sandbox. Creative output does not make generated code safe.

Why the first reward function failed

The first training rubric looked comprehensive. It contained nine signals:

whether the sketch compiled;
whether it actually used p5.brush rather than plain p5.js;
a code-length ramp aimed around 3,000 tokens;
HPSv3, an image preference model;
prompt adherence from a council of judge models;
recognizability;
aesthetics;
technique;
depth.

The run improved until its reward reached roughly 0.65, then stopped making meaningful progress. Outputs converged on the same safe solution: a flat flower with five rounded petals. The score rose, but the capability did not.

The problem was not a lack of feedback. It was too much duplicated feedback.

The quality judges and prompt-adherence judge were correlated between 0.85 and 0.95. Five labels appeared to describe five qualities, but they were largely measuring the same underlying impression. Their combined weight counted one opinion several times. Meanwhile, the code-length reward saturated early and stopped producing a useful gradient. HPSv3 was the signal with visible variation, yet it carried only 10% of the reward.

This is a common failure mode in composite evaluation. A long rubric feels safer because it names every concern. If the signals move together, however, adding them does not add information. It only amplifies a hidden preference. Reward dashboards can look richly instrumented while the optimizer receives the equivalent of one repeated instruction.

There was another trap: a metric can remain numerically present after it has stopped teaching. Once nearly every rollout satisfies a binary gate or length target, that component contributes no useful distinction inside the batch. A saturated reward is bookkeeping, not learning signal.

The diagnosis required inspecting each sub-reward separately: its variance over time, its correlation with other signals, and whether changes in it corresponded to visible capability changes. The total score alone concealed the collapse.

Pairwise judgment gives taste a usable scale

The original visual judges assigned absolute scores. In principle, “rate this from zero to ten” gives a clean scalar. In practice, model judges often compress their answers into a narrow region or apply the scale inconsistently. Is a competent synthetic watercolor a four, a six, or an eight? The number has no stable anchor.

Pairwise comparison asks an easier question: which image better matches the target watercolor?

For every rollout, the system samples two reference paintings and asks a judge to choose. The rollout’s reward is the fraction of comparisons it wins. The task is still subjective, but the decision has context. Instead of inventing the meaning of seven out of ten, the judge distinguishes one concrete artifact from another.

This changes the geometry of the feedback:

The reference set anchors the meaning of “good.”
Close comparisons expose small improvements that an absolute scale may hide.
Win rates spread results across a wider dynamic range.
The team can audit disagreements by looking at the exact images shown to the judge.

Pairwise preference is also a natural fit for HPSv3. The HPSv3 paper trains its preference model from pairwise annotations and uses an uncertainty-aware ranking loss. That does not make it a universal taste oracle. It makes the model useful as one broad prior about human image preference, which can then be balanced with project-specific references.

A reference pool is taste made operational

The team generated 1,664 candidate paintings and rated each one into three buckets: love , okay , or nope. The 117 love-tier examples became the initial taste anchor. Additional acceptable images and supplemental generations expanded coverage, producing a pool of 581 references.

Those numbers reveal how much human work sits behind a supposedly automated reward. The system did not discover the target aesthetic from the words “good watercolor.” Someone selected examples, rejected others, noticed gaps in color coverage, and decided which variations belonged together.

The reference pool performs several jobs at once:

It turns personal taste into examples rather than an abstract instruction.
It gives the judge a consistent comparison context.
It reveals missing regions, such as colors with too few strong examples.
It can be revised without relabeling every future rollout.
It creates an auditable record of what the reward is trying to preserve.

It also creates bias. Every reference was model-generated because suitable human-made p5.brush work was difficult to source. The policy is therefore learning to outperform a synthetic neighborhood selected by one curator, not learning an independent definition of watercolor quality. If the pool overrepresents centered flowers or clean backgrounds, the model may learn those conventions as taste.

The next natural step would be to train a smaller reward model directly on the ratings. That could generalize the curator’s preferences beyond repeated comparisons with a fixed pool. But it would not remove the design problem. It would encode the same choices into weights, making dataset balance, calibration, and held-out evaluation even more important.

The revised reward is smaller and more informative

After the diagnosis, nine components became four:

compilation and verified p5.brush use: 5% ;
code-length check: 5% ;
HPSv3: 30% ;
pairwise judgment against the curated pool: 60%.

Hard gates keep outputs runnable; most of the gradient now comes from visual preference.

With the same base model and training data, the revised run reached the earlier plateau about three times faster and continued improving beyond it. Generated programs also shrank from roughly 13,500 tokens to fewer than 2,000. Once verbose code stopped being rewarded heavily, the model found that good compositions could be expressed more directly.

The result is not an argument that every creative reward should use these exact weights. It demonstrates a method for finding useful weights:

Keep correctness checks as gates with enough weight to prevent invalid work.
Measure the variance of every soft reward inside actual rollout groups.
Compute correlations so duplicated judges do not impersonate independent evidence.
Inspect examples at low, middle, and high scores.
Remove or reduce signals that saturate before capability improves.
Re-run with one major change at a time so the cause of improvement remains legible.

A compact reward is easier to reason about. Every component has a distinct responsibility: executability, economy, broad human preference, and project-specific taste.

GRPO learns from relative outcomes within a group

The optimizer used Group Relative Policy Optimization, or GRPO. Introduced in DeepSeekMath, GRPO samples several outputs for the same input and estimates each output’s advantage relative to the group rather than training a separate value model.

In this setting, one prompt produces several candidate sketches. They are rendered and rewarded. A sketch that performs better than its siblings receives positive relative advantage; one that performs worse receives negative advantage. The policy update then makes the better token trajectories more likely while limiting how far the new policy moves from the previous one.

That relative structure pairs well with the project’s judging strategy. Both stages avoid pretending there is an absolute aesthetic truth:

the image judge compares a rollout with concrete references;
GRPO compares a rollout’s reward with other rollouts from the same prompt.

The distinction matters. Pairwise judging defines the scalar reward. GRPO decides how that scalar changes the language model. It does not solve reward design. If the judge favors flat clip art, GRPO will efficiently train the policy to make flat clip art.

Group-relative learning also needs diversity. If every candidate earns the same reward, the group provides little advantage signal. If one mechanical constraint dominates every comparison, the model learns that shortcut. Good rollout sampling and informative reward variance are part of the algorithm’s practical input.

The system prompt improved when documentation disappeared

Reward design was only half the problem. The model also needed to know how to use a niche graphics library.

An early system prompt included about 400 lines of p5.brush API documentation. Frontier models responded with polished JavaScript that confidently called methods the library did not have. More reference material increased the surface area for plausible invention.

The team used GEPA, an evolutionary prompt optimizer, to search for a better instruction. GEPA evaluates candidate prompts, reflects on execution traces and failures, proposes textual mutations, and preserves candidates that perform well across the evaluation set. After 200 iterations against a taste-anchored seven-example judge, the successful prompt was much smaller: a strict allowlist of eight brush methods, with no long API reference and no examples.

The first version that reliably produced visible hibiscus forms arrived after most of the documentation was removed.

This does not mean documentation is generally harmful. It means a generative model does not use an API manual like a compiler. Long context can mix authoritative names with nearby patterns and invite completion by analogy. For a constrained tool task, a small executable vocabulary may be more reliable than a broad descriptive reference.

A practical tool prompt should therefore distinguish:

allowed operations , listed exactly;
forbidden fallbacks , such as drawing with native p5.js when p5.brush is required;
runtime invariants , including canvas setup and completion behavior;
output contract , such as one complete sketch and no prose;
recovery behavior , for example simplifying after an execution failure.

Everything else can be retrieved only when needed or exposed through typed wrappers. The goal is not to teach the model the whole library. It is to make the valid action space obvious.

What can generalize beyond watercolor

The experiment suggests a reusable design for subjective, tool-mediated work:

1. Keep the artifact editable

Generate a structured source whenever the source is itself useful: HTML and CSS for layouts, vector instructions for diagrams, CAD operations for objects, MIDI for music, shader code for visuals, or node graphs for compositing. A rendered preview remains essential, but it should not erase the underlying decisions.

2. Separate validity from quality

Use deterministic checks for properties that genuinely are deterministic. The program compiles, calls allowed methods, finishes within limits, and produces a non-empty render. Do not ask an aesthetic model to guess those facts.

3. Prefer comparisons for ambiguous qualities

When an absolute scale is unstable, compare concrete artifacts. References make the intended standard visible and reviewable. Pairwise evaluation is slower, but it often creates a cleaner signal.

4. Treat curation as model design

The examples in a reference set shape the learned behavior as directly as the optimizer does. Track who selected them, which regions are overrepresented, and which failure modes are missing. Version the pool alongside the training configuration.

5. Audit reward information, not just reward count

More metrics are not automatically more supervision. Measure variance, correlation, saturation, and agreement with human review. Delete signals that merely repeat stronger ones.

6. Evaluate the entire pipeline

The policy, prompt, tool API, renderer, judge, references, and optimizer form one system. A blank image may be a policy failure, an API hallucination, a browser crash, or a capture bug. End-to-end traces are more useful than a single final score.

The real creative medium is the feedback system

The most interesting artifact in this project is not one hibiscus. It is the machinery that decides which hibiscus teaches the model.

Creative reinforcement learning cannot avoid taste by adding more judges. Someone still chooses the rubric, the references, the comparisons, and the failure boundaries. The engineering achievement is to make those choices explicit enough to test: hard gates for execution, a broad preference prior, a curated local standard, and relative updates across alternative programs.

Painting with code is slower than asking a direct image model for pixels. It is also more open to human intervention. The user can edit the source, the researcher can inspect the action, and the team can revise the meaning of “better” without pretending it was objective all along.

That is the larger lesson. For subjective model behavior, the reward is not a neutral measurement attached after the creative work. Designing the reward is part of the creative work.

DEV Community