Graphite: making two LLMs draw their own comparison

How to build an app where two models get the same CSV, write matplotlib code, get sandboxed and executed, and a third model judges the actual rendered output — measured, not vibes.

---

The problem with AI comparisons

Ask two models the same question and you get two confident answers. Which is better? Nobody knows, because nothing was built. Prompt battles compare vibes; leaderboards compare other people's votes.

There's a class of task where comparison can be brutally objective: write code that produces an artifact. Either the code runs or it doesn't. Either the chart has axis labels or it doesn't. Either the PNG reveals the step change hidden in the data or it shows noise.

So I built Graphite: two model slots (an older model as "A", a newer one as "B") both receive the same CSV and the same instruction —

"Write Python code using pandas and matplotlib that produces the single most insightful chart of this data. Output only code."

— and a sandbox executes both snippets. The UI shows the two rendered PNGs side by side, the code that produced them, verbatim tracebacks when they crash, measured properties of each figure, and a judge scorecard picking which chart to ship.

The viral line writes itself: the models are drawing their own comparison.

---

Architecture: three processes, three jobs

┌────────────────────────────────────────────────────────────────────────────┐
│  Browser — http://localhost:5173                                           │
│  Vite + React + TS (plain DOM + <img> for the PNGs)                        │
└─────────────┬──────────────────────────────────────────────────────────────┘
              │ /api/* proxied by Vite
              ▼
┌────────────────────────────────────────────────────────────────────────────┐
│  Node gateway :3001 — the ONLY place model calls happen                    │
│  prompt + nonce · concurrent slot calls · reasoning_content stripped       │
│  empty-content retry · judge call at T=0 · /models proxy                   │
└──────┬──────────────────────────────┬──────────────────────────────────────┘
       │ POST /execute                │  plain fetch (no SDK)
       ▼                              ▼
┌───────────────────────────────┐   ┌──────────────────────────────────────────┐
│  FastAPI executor :8000       │   │  Providers                               │
│  temp dir with only data.csv  │   │  Particle.ai / Ollama / LM Studio /      │
│  AST allowlist → 20s kill →   │   │  OpenRouter / anything OpenAI-compatible │
│  512MB cap → no network       │   │  /v1/chat/completions                    │
│  → PNG + measured metadata    │   └──────────────────────────────────────────┘
└───────────────────────────────┘

Why three processes instead of one monolith?

1. The gateway exists because of CORS and keys. Local providers (Ollama on :11434, LM Studio on :1234) don't send CORS headers, so a browser can't call them directly. And API keys should never ship to the client. One thin Node process solves both: the browser talks to localhost:3001, the gateway talks to whoever you configured.
2. The executor exists because of isolation. Model-written Python must run somewhere that isn't your app. A separate FastAPI process with subprocess isolation means a crashing snippet kills a throwaway temp process, not your UI.
3. The frontend stays dumb. Plain DOM, an <img> tag for each PNG, a fetch wrapper. The PNGs are the star; no chart library, no three.js.

The run, end to end

idle → asking → executing → judging → finished

POST /api/graphite builds one prompt and fires both slots concurrently with Promise.all. The prompt contains a preview of the CSV (header + 25 rows), the note "the full file is at ./data.csv", the question — and a fresh random nonce:

function buildUserPrompt(question: string): string {
  const { preview, rowCount } = buildDataPreview();
  const nonce = crypto.randomBytes(8).toString("hex");
  return [
    `Here is data.csv (${rowCount} data rows). The first rows look like this:`,
    "", preview, "", "...", "The full file is at ./data.csv.", "",
    `Task: ${question}`, "",
    "Write Python code using pandas and matplotlib that produces the single most insightful chart of this data.",
    "Save the figure to stdout as PNG bytes (fig.savefig(sys.stdout.buffer, format=\"png\")) or to a file like chart.png.",
    "Output only code. No explanations, no markdown fences.", "",
    `(run nonce: ${nonce})`,
  ].join("\n");
}

The nonce isn't decoration. Providers cache responses; if you re-run the "same" experiment to compare two models across sessions, a cache can hand you yesterday's answer and fake determinism. So the gateway keeps a Set of prompt hashes and throws on reuse:

function assertFreshPrompt(prompt: string): void {
  const h = crypto.createHash("sha256").update(prompt).digest("hex");
  if (seenPrompts.has(h)) {
    throw new Error("internal: prompt reuse detected — nonces must make every prompt unique");
  }
  seenPrompts.add(h);
}

---

Capability detection, never assumptions

This app must work with a $20M frontier model and a 4-bit quantized model running on a laptop with no API key. Those two worlds disagree about everything, so nothing is assumed — everything is detected:

Reasoning tokens. Some providers report usage.completion_tokens_details.reasoning_tokens. Ollama and LM Studio usually don't. The rule: read it from usage, or show n/a — never print 0, never invent a number:

const reasoningTokens =
  typeof usage?.completion_tokens_details?.reasoning_tokens === "number"
    ? usage.completion_tokens_details.reasoning_tokens
    : null;   // null → UI shows "n/a" and hides the thinking toggle for that slot

The thinking switch. Exactly one provider+model family wants chat_template_kwargs {"enable_thinking": false}; other providers ignore or reject unknown fields, so it's sent only when the slot is Particle.ai and the model starts with deepseek-.

Empty content on HTTP 200. Reasoning models can burn the entire token budget on hidden chain-of-thought and return an empty string. That's not a refusal — it's a budget problem. The gateway retries once with a doubled budget, capped at 4000:

if (!content.trim() && attempt === 0 && !opts.isJudge) {
  maxTokens = Math.min(maxTokens * 2, 4000);
  retried = true;
  continue;
}

reasoning_content — the CoT itself. Some providers return the model's hidden thinking in message.reasoning_content. Graphite strips it at the gateway boundary. It is never logged, never displayed, never persisted — only the token count survives. (You don't want your screen recording to leak a chain-of-thought you're not licensed to show.)

Dead local servers. If Ollama isn't running, the UI doesn't say "Something went wrong". It says:

Cannot reach http://127.0.0.1:11434 — is the local provider running? (fetch failed)

Model lists. Pickers are populated live from GET {base_url}/models, but you can always type a name by hand — deepseek-v4-flash-0731 doesn't appear in Particle's /models list and still responds. A dead /models never gates a run.

---

The sandbox: best-effort, loudly

Model-written code is untrusted code. Graphite's executor (backend/executor.py) stacks four defenses:

1. Static rejection (AST allowlist). Before a single byte executes, every Import, ImportFrom, Call and Attribute node is checked:

BLOCKED_MODULES = {"os", "subprocess", "socket", "shutil", "requests",
                   "urllib", "pathlib", "ctypes", "importlib", ...}
DANGEROUS_BUILTINS = {"eval", "exec", "compile", "open", "input", "__import__", ...}
DANGEROUS_ATTRS = {"system", "popen", "environ", "connect", "rmtree", ...}

>>> analyze("import socket")
[{'kind': 'import', 'detail': "import of blocked module 'socket'", 'line': 1}]

Rejected code never runs; the UI shows a BLOCKED badge with the exact rule. The model's mistake becomes part of the story instead of a mystery.

2. Process isolation. The snippet runs in a fresh temp dir containing only data.csv, the code, and the harness runner — with a scrubbed environment (HOME=tmp, TMPDIR=tmp, MPLBACKEND=Agg), a 512 MB memory cap via resource.setrlimit in a preexec_fn, and a 20-second hard kill:

proc = subprocess.run(cmd, cwd=tmp, env=env, capture_output=True,
                      timeout=TIMEOUT_S, preexec_fn=_limit_memory)

3. OS sandbox when available. On macOS the child is wrapped in sandbox-exec with (deny network*). If the host refuses to nest sandboxes — containers, CI — the executor detects sandbox-exec: in stderr and falls back to the remaining layers, reporting seatbelt: false so nobody is lied to about the protection level.

4. Honesty about the limits. The UI and README both state it plainly: best-effort sandboxing is not a security boundary — use a container if the code is untrusted. A static allowlist is not a VM. Saying so is a feature.

Capturing the chart (the fiddly part)

Models save figures in three different ways: fig.savefig(sys.stdout.buffer), fig.savefig("chart.png"), or just plt.show() and hope. The harness runner handles all three:

# 1) PNGs written to stdout were captured by the parent already
# 2) PNG files the model wrote to the temp dir:
new_files = sorted(set(glob.glob("*.png")) - _saved_files_before)
for name in new_files:
    _emit_png(open(name, "rb").read())
# 3) figures that were never saved anywhere: render them ourselves
if _exit == 0 and not new_files and _stdout_writes[0] == 0:
    for fig in [plt.figure(n) for n in plt.get_fignums()]:
        _emit_fig(fig)

(The _stdout_writes counter exists because the first version double-rendered: if the model already saved to stdout, step 3 must not fire.)

One bug worth confessing: the first version of the runner used contextlib.redirect_stdout to keep model print()s out of the PNG stream — which promptly broke savefig(sys.stdout.buffer), because the redirected stdout had no .buffer. The fix was to let prints through (text before binary is harmless when you scan for the PNG signature) and count buffer writes instead.

---

Honest measurement: the code can lie, the figure can't

This is the part I care about most. "Did it label its axes" is only a real comparison if the answer comes from the rendered artifact, not from parsing the model's code or trusting its commentary.

So the harness measures two things:

Live matplotlib objects. The runner patches Figure.__init__ to track every figure, then walks each figure's axes:

metas.append({
    "figsize_in": [fig.get_figwidth(), fig.get_figheight()],
    "dpi": fig.dpi,
    "axes": len(fig.axes),
    "lines": sum(len(ax.lines) for ax in fig.axes),
    "patches": sum(len(ax.patches) for ax in fig.axes),
    "titles": sum(1 for ax in fig.axes if (ax.get_title() or "").strip()),
    "xlabels": sum(1 for ax in fig.axes if (ax.get_xlabel() or "").strip()),
    "ylabels": sum(1 for ax in fig.axes if (ax.get_ylabel() or "").strip()),
    "legends": sum(1 for ax in fig.axes if ax.get_legend() is not None),
    "distinctColors": sorted(colors),
})

The PNG itself. Dimensions, bit depth and colour type are decoded from the IHDR chunk of the captured bytes — not from any matplotlib API.

The proof test: feed the executor a snippet whose strings and comments claim "9 axes, 42 colours, legend present":

NINE_AXES = 9
FORTY_TWO_COLOURS = 42
LEGEND_PRESENT = True
fig, ax = plt.subplots(figsize=(6, 4))
ax.plot([1, 3, 2], color="tab:green")
ax.set_title("honest title")
# this chart has 9 axes, 42 distinct colours and a legend, honest

Measured result: 1 axis, 1 colour, no legend. The measurement read the figure, not the fiction.

These facts are then what the judge sees — never pixels, never the model's self-description:

Chart A (deepseek-v4.1-flash) — measured properties of the rendered figure:
figure size 10x5.5 in at 110.0 DPI
1 axes
3 line objects
3 distinct colours #2563eb #9aa0a6 #e05c4f
title present: yes
x-axis label present: yes
y-axis label present: yes
legend present: yes

---

The judge

A third slot, called at temperature 0, max_tokens 1200, system prompt "You are a strict JSON API. Output only valid JSON.", asked to score both charts 1–10 on insight and clarity, give a one-sentence verdict each, and name which chart it would ship. The raw JSON is shown in a disclosure in the UI — the verdict is inspectable, not a black box.

Judging on measured properties rather than pixels keeps the prompt small, the comparison grounded, and the whole thing honest: a chart with no axis labels can't argue its way to a good clarity score.

---

The repair pass

When a model's code crashes, the panel shows the verbatim traceback and a button: "Send the traceback back and retry." One repair pass, same slot:

const repairPrompt =
  basePrompt +
  "\n\nYour previous attempt was:\n```

python\n" + code + "\n

```\n" +
  "It crashed with this Python traceback:\n\n" + traceback +
  "\n\nReturn corrected Python code only. Output only code, no markdown fences.";

The outcome is deliberately part of the story. Watching a model read its own stack trace and fix a KeyError is more persuasive than any benchmark table — and when it can't fix itself, that's data too.

---

What the dataset secretly contains

The shipped CSV is 215 working days of a build log: date, weekday, what shipped, the stack, tools, build seconds, tests run/failed, lines changed, and a cache flag. Two real signals are planted in it:

1. A step change — from 2025-01-15 the team enabled a remote build cache and every stack's build time dropped.
2. A persistent outlier — Swift builds run 2–4× slower than everything else, all the way through.

A chart that aggregates (rolling mean of daily build time, with the cutover marked) answers the question. A chart that plots 215 raw rows as a bar chart buries it. The judge's job is to say which is which — and the measured properties make its verdict auditable.

---

Verification, not vibes

Everything in the repo was exercised, not assumed:

• a working snippet rendered a real 1100×605 PNG; pixel-decoding confirmed the expected colours;
• a crashing snippet returned its verbatim KeyError traceback;
• an infinite loop was killed at exactly 20.0 s;
• the liar-snippet measured 1 axis / 1 colour / no legend;
• reasoning tokens came back as 87 from a reporting provider and n/a from one that doesn't;
• reasoning_content never appeared in any payload;
• the repair pass turned a crash into a render;
• empty-content responses triggered exactly one budget-doubling retry;
• import socket was blocked pre-execution with the violation named;
• two identical questions produced different prompt hashes (nonce working);
• the same-model control run produced identical code on both sides — and A-vs-B diverged.

For development without any API key, scripts/mock-provider.mjs is a tiny OpenAI-compatible fake with four personas (works, crashes, eats its budget, tries to import socket) — point a Custom slot at http://127.0.0.1:8100/v1 and the entire pipeline runs offline.

---

Stack summary

Layer	Choice
Frontend	Vite 7 + React 19 + TypeScript, plain DOM, hand-rolled CSS
Gateway	Node + TypeScript, plain fetch, zero runtime deps
Executor	Python + FastAPI + uvicorn + resource.setrlimit + Seatbelt
Data plane	pandas + matplotlib (Agg) inside the sandbox
Providers	Particle.ai, Ollama, LM Studio, OpenRouter, any /v1

The repo is a clean-clone runnable: npm install, a venv with four pip packages, npm run dev, paste a key. No database, no auth, no notebooks, no chart-generation APIs — the models write the matplotlib themselves, and the sandbox keeps them honest.

Code & more: https://www.dailybuild.xyz/project/264-graphite