DEV Community: oubakiou

Which model should your agent outsource to? Benchmarking 18 CLI child models on the same Godot game task

oubakiou — Sat, 18 Jul 2026 14:27:53 +0000

This article is a follow-up to my previous post: "Don't make the expensive model do everything — 'delegate-skills', a casual multi-model setup built on standard features (skills)". That post introduced delegate-skills, a mechanism for a parent agent to outsource tasks to other vendors' CLI child models.

godot-llm-gamebench is a benchmark that measures the entire "outsourcing" workflow: when a parent agent (such as Fable on Claude Code) delegates an implementation task to a child agent, which model is the best one to hand it to? The exact same Godot game implementation task is delegated from the parent agent (Claude Code) through the delegate-implement skill introduced in the previous post to 18 child models behind various vendor CLIs, and each run is measured along two axes — quality and efficiency — via auto-grading against hidden tests plus measured metrics.

The task: "Conveyor Courier" — an original, contamination-free spec

If the task were a well-known game like Tetris, we could not rule out the possibility that a model has "seen the answer" in its training data. So this bench uses an original puzzle called Conveyor Courier.

Colored packages flow along conveyor belts on an 8×8 grid
The player builds routes by placing and rotating belts, delivering red packages to the red exit and blue packages to the blue exit
The whole game is tick-driven: packages spawn every 3 ticks over a 120-tick run
Movement is a deterministic two-phase process (exit resolution → iterate movement passes until nothing moves), and splitters toggle left/right only when a package actually leaves them — the spec is packed with details you cannot possibly guess without reading it precisely

The implementation is Godot 4.x + typed GDScript, and only the BoardModel API that the hidden tests call (setup / step_tick / belt placement & rotation / spawn_item, etc.) is fixed as the public contract. The task prompt is frozen at benchmarks/tasks/conveyor-courier/prompt.md, and the same byte-identical text is handed to every model in every repetition.

Caveats you must keep in mind when reading this benchmark

Before we get to the results table, let me spell out what this measurement does and does not measure. If you want a more accurate picture of each model's ability, I recommend cross-referencing multiple benchmarks with different characteristics:

ReactBench blog

Artificial Analysis

What is directly measured is "the ability to implement a puzzle game in Godot." Even for the same implementation task, the ranking could shift if it were, say, a React UI implementation. Some low-scoring models here may simply lack Godot training while being strong implementers in general
Even today's top models could drop in rank on harder material. This bench mainly tests the ability to read a spec precisely and turn it into an implementation; it does not measure relative strength on tasks demanding deeper problem-solving
The whole benchmark is directed by Fable, so there may be a bias favoring Anthropic models
This measures "model + execution-harness CLI" as a set, not model weights in isolation. The same model may perform differently on a different harness. For example, gpt-5.5 driven from Codex and gpt-5.5 driven from Cursor could produce different results, but no such cross-comparison was done
Each model is n=3. Run-to-run variance can easily override any interpretation of "model characteristics." In fact, a model that looked like it had a "systematic habit of failing the splitter in every rep" in one round scored perfect in 2 of 3 runs when re-measured. Read the per-model "quirk" descriptions as hypotheses at best
Auto-grading only checks what can be inspected headlessly. In early rounds, unplayable implementations — "the board is pitch black," "clicks don't register" — slipped through with perfect scores. After humans caught these by actually playing, 3 view-behavior tests were added and every run was re-graded; even so, rendering defects like "an opaque background covering the board" cannot be detected headlessly and remain. The guarantee of "visually playable" is still partial
Cost-accounting precision differs per model, and none of it is "what you would actually pay." The methods mix CLI-measured figures (Claude family), token-price conversion (Codex / Cursor families), and estimates from published prices (Devin family). Actual out-of-pocket cost also depends heavily on your contract (flat-rate plans, free previews). The USD figures here are "a common yardstick for cross-model comparison assuming pay-as-you-go API pricing," not an estimate of your bill
Code quality is LLM-judged. It uses a two-judge scheme — primary judge (claude-sonnet-5) plus an independent-vendor cross-check (gpt-5.6-sol) — and the middle band (3–4.5) carries an inter-judge error of ±0.5–1.0
This is a snapshot from early July 2026. CLI versions were pinned during measurement; results may change as models, CLIs, and prices update

Results

Column definitions and accounting rules are in the notes below the table. For more detailed numbers see impressions.md (Japanese), and for more detailed per-model comments see the "benchmarked models" section of impressions.md.

Model (harness)¹	Play	Auto-graded score (max 300)²	Code quality (1–5)³	Total cost (median)	Wall clock (median)	Note
claude-sonnet-5 (Claude)	Play	290.00	3.8	$2.27 (measured)	8.6 min	Quality leader, incident-free. Two zero-warning runs
swe-1.7 (Devin)	Play	280.06	4.1	$1.41 (bundled)	12.5 min	+57.00 over swe-1.6. Its former 100-point run fell to 98.06 on the view click check
gpt-5.5 (Codex)	Play	280.00	4.0	$3.48 (converted)	5.8 min	Perfect functional score in every rep, incident-free. Zero warnings in 1/3
cursor-grok-4.5 (Cursor)	Play	276.00	3.3	$1.38 (converted)	4.8 min	Fastest tier, fewest tokens. No view deductions — rose to 4th in the re-grade
devin-deepseek-v4-pro (Devin)	Play	270.06	3.4	$3.58 (estimated)	11.1 min	Model-layer tests perfect in every rep, but slowest tier and pricey
claude-opus-4-8 (Claude)	Play	270.00	3.5	$2.35 (measured)	8.4 min	Child aces functional tests every time. All failures were parent/harness side
gpt-5.6-sol (Codex)	Play	268.06	4.2	$3.10 (converted)	7.1 min	A stable 88–90 in every rep. Roughly ties terra with the same quality at twice the unit price
cursor-gemini-3.1-pro (Cursor)	Play	268.06	2.5	$2.13 (converted)	8.4 min	Warning-count swings and a stall habit. Highest token use among Cursor models
cursor-kimi-k2.7-code (Cursor)	Play	268.06	4.5	$1.60 (converted)	10.6 min	Splitter quirk did not reproduce. Type discipline stays top-tier
gpt-5.4-mini (Codex)	Play	266.12 (rep run: board not rendered)	3.3	$2.04 (converted)	11.1 min	Fewest type warnings but slow, with a determinism drop and stalls
gpt-5.6-terra (Codex)	Play	266.12 (rep run: tofu arrows)	4.2	$2.38 (converted)	6.3 min	Matches sol at roughly half the cost and faster — the efficiency pick
devin-glm-5.2 (Devin)	Play	266.12 (rep run: no belt arrows, mouse dead)	3.8	$2.28 (estimated)	5.9 min	Fast and functionally accurate. Pre-completion stalls are the one flaw
composer-2.5 (Cursor)	Play	264.28 (mouse dead in all reps)	3.8	$1.29 (converted)	6.5 min	Still incident-free. Measured child cost is the lowest of all models
composer-2.5-fast (Cursor)	Play	264.28 (mouse dead in all reps)	3.7	$1.76 (converted)	7.8 min	Rose to a tie with the base model, but slower than it after re-measurement
gpt-5.3-codex-spark (Codex)	Play	264.17	3.6	≥ $1.47 (child unknown)⁴	4.5 min	Fast but unstable on details. 3–5× the input tokens of gpt-5.5
gpt-5.6-luna (Codex)	Play	250.55	3.3	$1.95 (converted)	7.3 min	Failed the splitter exit-toggle test in every rep (follow-up 1 traced it to reasoning effort; fixed at xhigh)
swe-1.6 (Devin)	Play	223.06 (rep run: tofu arrows)	3.8	$0.95 (bundled)	3.9 min	Cheapest (bundled pricing) but lowest quality on both axes
claude-haiku-4-5 (Claude)	Play	86.11 (n=1) (rep run: mouse dead)	2.7 (n=1)	$2.53 (measured)	9.3 min	Completed 1/7. Frequent stalls, least reliable

Note: swe-1.7's child cost is $0 under bundled pricing during its free preview period (until 2026-08-08). claude-haiku-4-5 completed only once (out of 7 attempts), so its figures are n=1 reference values. Parenthesized notes in the score column are view defects discovered by humans playing the finished game ("rep run" = the median-score run of the 3 adopted reps).

Baseline condition (does not go through the delegation protocol; reference only — do not compare directly with the table above):

Condition	Play	Auto-graded score (max 300)²	Code quality (1–5)³	Total cost (median)	Wall clock (median)	Note
fable-direct (no delegation)	Play	300.00	4.6	$2.62 (measured)	5.7 min	Baseline: the parent (Fable) implements directly

¹ Each model's effort level is fixed to its single default. Bold marks models that passed the hidden tests (the 70-point axis) in all 3 repetitions.

² The sum over 3 repetitions (max 300) of a 100-point auto-graded rubric: hidden tests never shown to the delegatee (70), determinism (10), type quality (10), and project health (10).

³ A 5-point qualitative rating (average of 3 runs) of design aspects such as thoroughness of typing, Model/View separation, readability, and use of constants. Primary judge (claude-sonnet-5) layered with a cross-check judge (gpt-5.6-sol); on divergence the operator (Fable) adjudicated the representative value.

⁴ The child process's token usage could not be measured, so the child cost is unknown; this is a lower bound of parent cost only (the actual total is higher). Excluded from the cost scatter plot.

Scatter plot: code quality × auto-graded score

Scatter plot: cost × code quality

Highlights

Per-token price and per-task cost do not necessarily agree. Even with a cheap unit price, a model can end up costing more if it consumes more tokens or takes longer (inflating the parent-side cost). In this bench, devin-deepseek-v4-pro (input $1.74 / output $3.48) — priced at 1/3 to 1/9 of gpt-5.5 (input $5 / output $30) — consumed roughly twice the input tokens and ended up above gpt-5.5 in median total cost, $3.58 vs $3.48. Price sheets are only a starting point; what matters is the bill on real tasks.

Passing the functional tests in every rep is not exclusive to top-tier models. Four models passed the hidden tests (the 70-point axis) in all 3 repetitions: claude-sonnet-5 / gpt-5.5 / cursor-grok-4.5 / claude-opus-4-8 (bold in the table). But only the fable-direct baseline reached a summed 300, and what spread the delegated models apart was the type-quality axis.

Even with equal functional-test results, code quality diverges. Among those same 4 all-pass models, gpt-5.5 scored 4.0 and claude-sonnet-5 scored 3.8, while claude-opus-4-8 stayed at 3.5 and cursor-grok-4.5 at 3.3. That is the vertical spread in the chart. If you want code that is maintained, not just tests that pass, this axis is worth including in your choice.

The default is claude-sonnet-5; deviate only for a clear advantage. With the top delegated auto-graded score (290.00) and an incident-free record, claude-sonnet-5 is the delegation target most users will find easiest to use from Fable for implementation tasks. Choosing anything else requires some advantage. If code quality matters and you need something cheaper, swe-1.7 (based on Kimi K2.7) or cursor-kimi-k2.7-code are candidates; if speed matters and you need something cheaper, cursor-grok-4.5 and the like.

Delegation is not free. The fable-direct baseline (no delegation) took 5.7 min at $2.62 and was the only condition in the main round with perfect scores in every run. Compared with the delegated runs on the same Claude harness (8.4–8.6 min, $2.27–$2.35), the cost difference on a single implementation task is small, and a naive mechanism like delegate-skills adds roughly 3 minutes of delegation overhead. If you only ever use Fable and the Claude family, you may not need delegate-skills much. Simply instructing the system prompt to proactively launch a specified model via the subagent tool for particular tasks may work well enough.

Follow-up 1: Can reasoning effort buy quality? — gpt-5.6-luna medium vs xhigh

After the main round, I also ran a reasoning-effort A/B on the same bench with the same frozen prompt. The subject is gpt-5.6-luna, which the main round described as "systematically failing the splitter spec." Both arms (medium / xhigh) were run 3 reps each, interleaved on the same day, as fresh measurements.

Model	Play	Auto-graded score (max 300)	Code quality (1–5)	Cost USD (median)	Wall clock (median)
gpt-5.6-luna@medium	Play	252.50	3.7	2.30	8.0 min
gpt-5.6-luna@xhigh	Play	282.23 (+12%)	3.8 (+3%)	3.09 (+34%)	13.0 min (+63%)

Per-run quality was 86.11 / 80.28 / 86.11 for medium and 88.06 / 100 / 94.17 for xhigh. Every xhigh deduction was on the view axis (tofu arrows, and forfeiting the view tests via a branch that detects headless execution and self-terminates); the Model-layer tests passed fully in all 3 reps.

What looked like a "systematic weakness" vanished with effort. Medium again failed splitter-family tests in 2 of 3 reps (the remaining rep failed a different category), reproducing the main-round observation. Xhigh passed every Model-layer test in all 3 reps. That cleanly separates the cause: the missed spec details were not a "model quirk" but a "shortage of thinking budget."

Type warnings disappeared, but design-level code quality could not be bought. Medium scored 0/10 on type quality in all 3 runs (same as the main round), while xhigh took 10/10 in 2 runs and even produced one perfect 100-point run. The summed 282.23 sits alongside the main round's second-place delegated group (swe-1.7 / gpt-5.5, around 280).

Yet qualitative code quality did not improve: medium 3.7 / xhigh 3.5 (both judged by sonnet-5). Looking inside, the 10/10-warning xhigh run had abandoned the enum cell-representation type internally in favor of Array[PackedInt32Array] and untyped nested arrays — exactly the pattern of "type everything as Array[int] and you get zero warnings, but throwing away the enum type is a design deduction."

What effort buys is precision of spec reading and disciplined zero-warning operation — not good design taste.

The price is paid mostly in time. Child input tokens balloon by about 2.4×, but because luna's unit price is cheap the cost increase stays at +34%, and most of the increment shows up as wall-clock time (+63%). With the usual n=3 caveat, this round's answer is: "run luna at xhigh and you can buy swe-1.7 / gpt-5.5-class quality for $3.09 / 13 min."

It may be a rescue lever for "cheap × high code quality × low auto-graded score" models. Effort lifted only the precision of spec reading (the auto-graded axis) and left design-level code quality unchanged. Flip that around: a model that is cheap and already strong on code quality but sinking on the auto-graded axis — in the main table, a profile like cursor-kimi-k2.7-code (code quality 4.5, $1.60, score 268.06) — might cover exactly its weak side just by raising an effort-equivalent parameter. Whether the same phenomenon actually occurs outside gpt-5.6-luna is unverified.

Follow-up 2: Can a skill raise the quality floor?

The qualitative observations in this bench had shown that the sharpest between-model differentiator is "how far the model carries type annotations into its internal implementation (thoroughness of Typed GDScript)." So I built the gdscript-quality skill — conventions for type annotations, typed collections, eliminating dict-as-struct, zero-warning operation, plus a headless verification script — and measured its effect in a with/without-skill A/B.

Model	Play	Auto-graded score (max 300)	Code quality (1–5)	Cost USD (median)	Wall clock (median)
claude-sonnet-5@noskill	Play	282.00	4.2	2.64	9.9 min
claude-sonnet-5@gq	Play	298.06 (+6%)	5.0 (+19%)	3.46 (+31%)	12.4 min (+25%)
claude-opus-4-8@noskill	Play	270.00	3.0	2.84	9.0 min
claude-opus-4-8@gq	Play	300.00 (+11%)	4.0 (+33%)	3.25 (+14%)	10.5 min (+17%)

The first summed 300 from a delegated run. claude-opus-4-8@gq scored a perfect 100 in all 3 reps — recall that in the main round only the non-delegated fable-direct managed a summed 300. sonnet@gq also hit perfect scores in 2 of 3 runs (5 of 6 runs across the skill arms). The skill worked not by "extending the upside" but by "guaranteeing a quality floor for any model."

The "design quality" that effort couldn't buy, the skill could. In follow-up 1, raising effort left qualitative code quality essentially flat (3.7 → 3.8 representative). The skill moved the design axis substantially: sonnet 4.2 → 5.0 (all 3 runs carrying typed Dictionary throughout — top marks), opus 3.0 → 4.0 (representative). Moreover, opus@gq's 3 runs converged on the same-shaped 4.0 ("only _grid: Array[int] remains untyped, everything else typed"), so the floor-leveling showed up in score variance too. Effort buys zero-warning operation; the skill buys design quality as well — the two follow-ups pair up neatly.

Overhead is +14–31% on the total. Child tokens were sonnet +20% / opus +47%, totals +14–31%, time +17–25%. "For +14–31% extra cost, delegated runs line up at perfect scores" seems like a thoroughly realistic option.

Follow-up 3: Does the skill work across harnesses? — re-testing with composer-2.5 on Cursor

Follow-up 2 left two open questions. First, both sonnet and opus already aced the functional tests unassisted, so we couldn't see how the skill behaves on a model with functional weaknesses.

Second, follow-up 2 was only verified on the Claude harness, leaving it unconfirmed whether the skill's effect generalizes across execution harnesses (CLIs).

So I ran the same A/B on Cursor CLI's composer-2.5, which carries both type warnings (3–9 of them) and a view defect (mouse dead) (2026-07-14, same skill commit as follow-up 2).

Model	Play	Auto-graded score (max 300)	Code quality (1–5)	Cost USD (median)	Wall clock (median)
composer-2.5@noskill	Play	268.18	3.4	1.23	5.0 min
composer-2.5@gq	Play	292.23 (+9%)	4.2 (+24%)	1.18 (−4%)	6.2 min (+24%)

Note: code quality uses the same two-judge scheme as the main round (sonnet-5 / sol = noskill 3.3/3.5, gq 4.2/4.3; all rows diverged by ≤ ±0.5, split rows took the floor of the average as the representative value). Calibration anchors were the 3 adopted composer-2.5 runs from the main measurement.

The skill's floor-raising reproduced on the Cursor harness. Type warnings went from 6 / 9 / 3 → 0 / 0 / 0. Qualitative code quality also improved 3.4 → 4.2, the same magnitude as opus in follow-up 2 (3.0 → 4.0). Inside, the effect concentrates on consistent adoption of Godot 4.4's typed Dictionary syntax (Dictionary[K, V]): the _items / _splitter_toggles that bare composer left untyped in every run were typed in all 3 skill runs. Meanwhile the untyped nesting of _grid: Array[Array] remained in 2/3 even with the skill — outlining the skill's coverage: "effective for typing Dictionaries, not quite enough for typing nested Arrays."

And functional weaknesses, as expected, are not fixed by the skill. The view's dead mouse (a ColorRect whose mouse_filter swallows clicks) remained in 2 of 3 runs even with the skill, and the sum fell short of 300 (with the mouse deduction in place the ceiling is about 294). The interesting one is the remaining run: it chose Polygon2D instead of ColorRect for cell rendering, so the mouse works. Composer's dead mouse is thus not an "always-on defect" but a probabilistic habit depending on the rendering-node choice, and with n=3 no causal claim about the skill can be made. The skill buys zero-warning operation and design quality; a "playable view" cannot be bought by either effort or skill — that is the summary across all three follow-ups.

One by-product. Despite child tokens increasing +48%, composer's child unit price is so low ($0.04–0.06/run) that the total actually stayed flat ($1.23 → $1.18); the real price is time only (+24%). "Cheapest-class child + skill buys a type-quality floor" is an attractive combination.

Running it yourself

Point your local coding agent at https://github.com/oubakiou/godot-llm-gamebench/tree/main/docs and ask it questions.

Wrap-up

Based on these latest benchmark results, I'm currently trying the following routing in real-world use. If anything comes up in practice, I may append it to this article.

// .claude/settings.local.json
{
  "env": {
    "DELEGATE_EXPLORE_MODEL": "composer-2.5",
    "DELEGATE_IMPLEMENT_MODEL": "swe-1.7",
    "DELEGATE_CHORE_MODEL": "composer-2.5",
    "DELEGATE_REVIEW_MODEL": "gpt-5.6-sol"
  }
}

All design details are public in docs/design/bench_common_design.md (model roster, measurement, fairness limits, anti-cheating design) and docs/design/delegate_implement_bench_design.md (task spec and the full rubric). The license is MIT. In the agent era, "which model do I outsource to?" is a question worth answering with numbers — take a look.

Don't Make the Expensive Model Do Everything — delegate-skills, a Casual Multi-Model Setup Built on Nothing but Standard Skills

oubakiou — Sat, 11 Jul 2026 03:29:10 +0000

Putting a top-tier model like Claude Fable 5 in charge of your main agent raises the quality of its judgment — but it also means paying flagship prices for "labor": reading piles of code, applying mechanical fixes, staring at diffs. Fable 5 costs $10 input / $50 output (per 1M tokens). Claude Haiku 4.5, meanwhile, is $1 / $5.

Do you really need to pay 10× the rate for grunt work?

As it happens, Anthropic itself has answered this question. According to a post from @ClaudeDevs, a Claude Managed Agents configuration that splits Fable 5 into the planner/integrator role (orchestrator) and Sonnet 5 into the read-and-execute role (worker) achieved 96% of standalone Fable 5's performance at 46% of the price on the BrowseComp web-research benchmark. This "plan with a big model, execute with small models" pattern is covered in detail in the official cookbook Plan big, execute small.

delegate-skills is a skill set that brings this same idea to the world of CLI agents like Claude Code — using nothing but the standard skill mechanism. Your main agent stays on the expensive model, while exploration, implementation, review, and chores are delegated to child processes running cheaper models (or models from other vendors), compressing your token costs.

What you get

1. Token-cost compression

Routine read/write-heavy work is offloaded to cheap models, so an expensive model like Fable spends its tokens on decision-making and final responsibility. Delegate code exploration to Haiku via the delegate-explore skill, for example, and the token unit price of that work drops to 1/10.

2. Context isolation

Delegation isn't only about cost. Bulk file reads and trial-and-error logs stay confined to the child process; the main agent reads the result markdown progressively — index first, then only the sections it needs. Because the main conversation context never bloats with investigation logs, the quality of its judgment holds up even in long sessions.

3. Crossing vendor walls

Delegation targets are not limited to the Claude family. Just specifying a model name selects the execution backend.

Model name	Backend	Launched via
`sonnet` / `haiku` / `opus` / `fable`	Claude subprocess	`claude -p`
`gpt-*`	Codex subprocess	`codex exec`
`swe-` / `devin-`	Devin CLI subprocess	`devin -p`
`composer-` / `cursor-`	Cursor agent CLI subprocess	`agent -p`

Throwing an implementation at GPT from Claude Code, or an exploration at Haiku from Codex — cross-vendor delegation works through a single mechanism. The requesting side (the main agent) can also be any of Claude Code / Codex / Devin CLI / Cursor CLI.

The skill lineup

Skill	Purpose	Tool permissions	Default model
delegate-explore	Read-only code & documentation exploration	read-only	haiku
delegate-implement	Code implementation & fixes (one commit)	Edit/Write/Bash (no push)	sonnet
delegate-chore	Fallback miscellaneous work	Edit/Write/Bash (no push)	haiku
delegate-review	Code / document review (surfacing findings on a diff)	read-only	opus

Each skill's default model can be changed via an environment variable, DELEGATE_<TYPE>_MODEL (e.g. DELEGATE_IMPLEMENT_MODEL=gpt-5.5).

How it differs from similar efforts

Attempts to combine multiple models are multiplying right now: LLM routers that pick a response model per query, dedicated orchestration frameworks, and the first-party multi-agent features that each agent CLI is starting to ship. delegate-skills sits in this same current, but its position is distinct.

Standard skills only — and casual about it

The biggest differentiator is that everything runs on the agent's standard skill mechanism alone. No dedicated server, no resident process, no SQLite, no orchestration framework to adopt. Since the substance is just shell scripts and a file-based protocol, it works the moment you gh skill install it — and when you no longer need it, you just uninstall. Getting started isn't the major undertaking of "building a multi-model architecture"; it's as casual as adding one more skill.

vs. Claude Code's built-in multi-agent sessions

Claude Code ships with a built-in multi-agent session mechanism. delegate-skills is not a replacement for it — the aims differ.

Token-cost efficiency is the top priority — where most general-purpose multi-agent mechanisms center on parallelism or role separation, delegate-skills is designed first and foremost to "offload to cheap models and shrink the bill." The cheap-model defaults, progressive reading via md2idx, and measured costs in the observe JSON all serve this goal
The only topology is a tree from a single parent — agents don't talk to each other as peers; delegation always flows one way, parent → child, and a recursion guard prevents runaway chains. By deliberately limiting expressiveness, both accountability (final responsibility for the deliverable always rests with the root agent) and the flow of context stay simple
Multi-vendor by design — it isn't confined to division of labor within the Claude family: a model name alone delegates to Codex (gpt-*), Devin CLI, or Cursor agent CLI. The requesting side isn't limited to Claude Code either

Children never communicate horizontally, and results always return to the parent through files. Who owns the final deliverable is structurally unambiguous at all times.

So which model should you actually delegate to?

That's the next question, naturally. To answer it, I run a benchmark called godot-llm-gamebench: the implementation of an original-spec puzzle in Godot 4.x + typed GDScript (not a known game, to avoid training-data contamination) is delegated from a parent agent (Claude Fable 5) to each vendor's models via delegate-implement, and auto-graded with a 100-point rubric built on hidden test code the delegatee never sees.

Each model runs 3 repetitions, with quality and efficiency (time, tokens, cost) measured on separate axes. The benchmark repository will be published later and covered in detail in a separate article; here is an excerpt of the latest adopted values from measurements taken 2026-07-05 through 07-10.

Model (harness)¹	Auto-graded score (max 300)²	Code quality (1–5)³	Total cost (median)	Wall clock (median)	Note
claude-sonnet-5 (Claude)	285.00	3.8	$2.27 (measured)	8.6 min	Top auto-graded score, incident-free. Two zero-warning runs
swe-1.7 (Devin)	273.00	4.1	$1.41 (bundled)	12.5 min	Big improvement over swe-1.6, with a 100-point run. But slowest tier
gpt-5.5 (Codex)	270.00	4.0	$3.48 (converted)	5.8 min	Perfect functional score in every rep, incident-free
cursor-grok-4.5 (Cursor)	264.00	3.3	$1.38 (converted)	4.8 min	Perfect functional score, fastest tier, fewest tokens
gpt-5.4-mini (Codex)	258.18	3.3	$2.04 (converted)	11.1 min	Fewest type warnings but slow, with a determinism drop and stalls
devin-deepseek-v4-pro (Devin)	258.00	3.4	$3.58 (estimated)	11.1 min	Perfect functional score in every rep, but slowest tier and pricey
composer-2.5 (Cursor)	257.36	3.8	$1.29 (converted)	6.5 min	Child cost of $0.04 is the lowest of all models
composer-2.5-fast (Cursor)	257.36	3.7	$1.76 (converted)	7.8 min	Wall clock swings widely with measurement timing
gpt-5.6-sol (Codex)	255.00	4.2	$3.10 (converted)	7.1 min	A stable 85 in every rep. Same quality as terra at twice the unit price
gpt-5.6-terra (Codex)	255.00	4.2	$2.38 (converted)	6.3 min	Matches sol at roughly half the cost and faster — the efficiency pick
claude-opus-4-8 (Claude)	255.00	3.5	$2.35 (measured)	8.4 min	Child aces functional tests every time (failures were parent/harness side)
devin-glm-5.2 (Devin)	253.18	3.8	$2.28 (estimated)	5.9 min	Fast and functionally accurate. Pre-completion stalls are the one flaw
cursor-gemini-3.1-pro (Cursor)	253.18	2.5	$2.13 (converted)	8.4 min	Warning-count swings and a stall habit. Highest token use among Cursor models
cursor-kimi-k2.7-code (Cursor)	253.18	4.5	$1.60 (converted)	10.6 min	Top-tier type discipline, but slowest of the Cursor models
gpt-5.3-codex-spark (Codex)	251.36	3.6	≥ $1.47 (child unknown)⁴	4.5 min	Fast but unstable on details
gpt-5.6-luna (Codex)	244.09	3.3	$1.95 (converted)	7.3 min	Cheap, but systematically misses spec details
swe-1.6 (Devin)	212.28	3.8	$0.95 (bundled)	3.9 min	Cheapest (bundled pricing) but lowest quality on both axes
claude-haiku-4-5 (Claude)	83.18 (n=1)	2.7 (n=1)	$2.53 (measured)	9.3 min	Completed 1/7. Frequent stalls, least reliable

Note: swe-1.7's child cost is $0 under bundled pricing during its free preview period (until 2026-08-08).

Baseline condition (does not go through the delegation protocol; reference only — not directly comparable with the table above):

Condition	Auto-graded score (max 300)²	Code quality (1–5)³	Total cost (median)	Wall clock (median)	Note
fable-direct (no delegation)	300.00	4.6	$2.62 (measured)	5.7 min	The parent (Fable) implements directly

¹ Each model's effort level is fixed to its single default. Bold marks models that passed the hidden tests (the 60-point axis) in all 3 repetitions.

² The sum over 3 repetitions (max 300) of a 100-point auto-graded rubric: hidden tests never shown to the delegatee (60), determinism (10), type quality (15), and project health (15).

³ A 5-point qualitative rating (average of 3 runs) of design aspects such as thoroughness of typing, Model/View separation, readability, and use of constants. Primary judge (sonnet) layered with a cross-check judge (gpt-5.6-sol); on divergence the operator (Fable) adjudicated the representative value.

⁴ The child process's token usage could not be measured, so the child cost is unknown; this is a lower bound of parent cost only (the actual total is higher). Shown as a triangle (▶) in the scatter plot.

There are plenty of caveats to reading these results (grading-method details, differing cost-accounting precision, sample size, confounding parent-side factors, and so on), but I'll leave those to the future article introducing the benchmark itself and skip them here.

Per-token price and per-task cost do not necessarily agree. Even with a cheap unit price, a model can end up costing more if it consumes more tokens or takes longer (inflating the parent-side cost). In this bench, devin-deepseek-v4-pro (input $1.74 / output $3.48) — priced at 1/3 to 1/9 of gpt-5.5 (input $5 / output $30) — consumed roughly twice the input tokens and ended up above gpt-5.5 in median total cost, $3.58 vs $3.48. Price sheets are only a starting point; what matters is the bill on real tasks.

Also, if you're on a vendor's subscription plan, your effective per-token cost is substantially lower than the published unit price, and the discount varies by company — but this article uses published token prices as a uniform yardstick.

Since this skill set puts token-cost efficiency first, let's compare the 16 models with an auto-graded score of 240 or higher on the two axes of cost and code quality.

Here's what the data suggests:

Passing the functional tests in every rep is not exclusive to top-tier models — eight models passed the hidden tests (the 60-point axis) in all 3 repetitions: claude-sonnet-5 / swe-1.7 / gpt-5.5 / cursor-grok-4.5 / devin-deepseek-v4-pro / gpt-5.6-sol / gpt-5.6-terra / claude-opus-4-8. But only the fable-direct baseline reached a summed 300, and what spread the delegated models apart was the type-quality axis
Even with equal functional-test results, code quality diverges — among those same all-pass models, gpt-5.6-sol / gpt-5.6-terra scored 4.2, gpt-5.5 4.0, and claude-sonnet-5 3.8, while claude-opus-4-8 stayed at 3.5, devin-deepseek-v4-pro at 3.4, and cursor-grok-4.5 at 3.3. That is the vertical spread in the chart. If you want code that is maintained, not just tests that pass, this axis is worth including in your choice
The default is claude-sonnet-5; deviate only for a clear advantage — with the top delegated auto-graded score (285.00) and an incident-free record, claude-sonnet-5 is the delegation target most users will find easiest to use from Fable for implementation tasks. Choosing anything else requires some advantage. If code quality matters and you need something cheaper, swe-1.7 (based on Kimi K2.7) or cursor-kimi-k2.7-code are candidates; if speed matters and you need something cheaper, cursor-grok-4.5 and the like
Delegation is not free — the fable-direct baseline (no delegation) took 5.7 min at $2.62 and was the only one of all 19 conditions with perfect scores in every run. Compared with the delegated runs on the same Claude harness (8.4–8.6 min, $2.27–$2.35), the cost difference on a single implementation task is small, and roughly 3 minutes of delegation overhead is added. If you only ever use Fable and the Claude family, you may not need delegate-skills much. Simply instructing the system prompt to proactively launch a specified model via the subagent tool for particular tasks may work well enough

To repeat: this is a small-scale measurement — a single task, 3 repetitions each. Please take it as nothing more than "an initial default for choosing a delegation target."

Finally, here's my own non-work configuration (.claude/settings.local.json) as an example:

{
  "env": {
    "DELEGATE_EXPLORE_MODEL": "composer-2.5",
    "DELEGATE_IMPLEMENT_MODEL": "composer-2.5",
    "DELEGATE_CHORE_MODEL": "composer-2.5",
    "DELEGATE_REVIEW_MODEL": "gpt-5.6-sol"
  }
}

I may publish a separate benchmark for delegate-review rather than delegate-implement. (Or I may not.)

Try it

Prerequisites

Node.js and md2idx (installing beforehand isn't strictly required, but it's used for parent–child communication, so npm install -g md2idx is recommended)
jq
The CLI matching your delegation target (claude / codex / devin / Cursor's agent, each logged in)

Install

It installs via GitHub CLI (v2.90.0+) gh skill:

# Install individually (for Claude Code)
gh skill install oubakiou/delegate-skills delegate-explore --agent claude-code --scope project

# Or everything at once
for skill in delegate-explore delegate-implement delegate-chore delegate-review; do
  gh skill install oubakiou/delegate-skills "$skill" --agent claude-code --scope project
done

You can also install with the skills CLI from vercel-labs/skills:

npx skills add oubakiou/delegate-skills

Run

If you're fine with the default models, no extra configuration is needed. Just ask your main agent as usual, and delegation happens automatically based on each skill's description.

Find out where authentication is implemented in this repository

→ The main agent triggers delegate-explore, and a haiku child process does the investigating. The main agent reads the result file's index, then only the sections it needs.

You can also delegate explicitly by naming a skill:

Review the diff on this branch with delegate-review

And if you want it to delegate more aggressively, add a single line to your project's CLAUDE.md / AGENTS.md:

- To save tokens, actively delegate tasks to subagents using the delegate-\* skills

How it works under the hood

The main agent and the child talk through a file-based request/response protocol. Both files are written in md2idx format (index + sections), so they can be read progressively, saving tokens.

Each skill's execution flow:

prepare.sh — prerequisite checks → model resolution (DELEGATE_<TYPE>_MODEL env → skill default) → recursion guard against multi-hop delegation → request generation
dispatch.sh — deterministically selects the backend from the model-name prefix and launches the child process via a shell wrapper
read-response.sh auto — reads the response (progressively if large) → validation

Fail-safe design

Guardrails are built in so that "fire and forget to a cheap model" doesn't end in accidents:

Recursion guard — if the same kind of delegation occurs twice in a multi-hop chain, it stops with exit 4
Fail-closed — if a prerequisite (like md2idx) is missing, it fails instead of silently continuing
Restricted tool permissions — delegatees can't push; explore / review are read-only

Files generated per delegation

Each delegation (run) generates the following under DELEGATE_WORK_DIR (when unset, TMPDIR per mktemp's defaults, else /tmp):

request file — what the parent asks of the child. Alongside the md2idx-format JSON the agent reads (*_req.json), a same-content markdown (*_req.md) is generated for human debugging
response file — the child's report of its results. Also a pair: agent-facing JSON (*_res.json) and human-debugging markdown (*_res.md)
observe JSON — an observation record of execution state, heartbeats, token usage, and more
per-run scratch directory — temporary work area for the child process

Since the default /tmp location can be wiped across days, if you want to investigate failures or monitor externally, pinning it to an ignored directory inside your repo — e.g. DELEGATE_WORK_DIR=.temp/delegate/work — is recommended.

Resumable sessions for review-fix loops

Normal delegations are throwaway, but for larger delegate-implement / delegate-chore work where a review → fix loop is expected, the main agent can explicitly start a resumable initial run. The observe JSON records a resume handle and lineage, and subsequent follow-ups resume the same backend session. This is fail-closed too: if validation fails, it does not silently fall back to a new session.

Delegation-cost observability

The child process's token usage is recorded in the observe JSON. All four backends — Claude / Codex / Devin / Cursor — can report measured values (measured); when unavailable, it falls back to a chars/4 estimate (estimated), and the two are clearly distinguished. For backends that report measured tokens but no cost (like Codex), a cost_usd_estimated converted from a bundled price table is attached. In other words, you can verify after the fact whether delegating actually made things cheaper.

Who it's for

You use Claude Code / Codex / Devin / Cursor as your main agent, and:

your token bill is on your mind
in long sessions, your main agent's context fills up with investigation logs and you can feel the quality of its judgment degrade
you have CLIs from multiple vendors and want to use the right model for the right task

Wrap-up

At the core of delegate-skills is a simple division of labor: "the expensive model's job is judgment and responsibility; the cheap model's job is reading and writing." It realizes this with nothing but the skill mechanism — no dedicated infrastructure — spanning four backends with a single model name, with reliability underwritten by a file-based protocol and fail-safe design.

Installation is one gh skill install. Start with delegate-explore, and experience investigation tasks getting done at Haiku prices.

Repository: https://github.com/oubakiou/delegate-skills
License: MIT

Rendering Charts Beyond Mermaid's Reach with Vega-Lite and Embedding Them in Markdown (via LLM Skills)

oubakiou — Tue, 30 Jun 2026 04:34:02 +0000

AI agents like Claude Code and Codex can write code, polish documentation, and automate complex tasks. But ask them to "make a scatter plot of this data" or "show the trend as a heatmap," and they hit a wall. Mermaid tops out at basic bar charts and pie charts — statistical visualization is outside its scope.

dataviz-svg is a skill built to fill that gap. It generates SVG charts from Vega-Lite declarative JSON specs and embeds them directly into Markdown documents.

Where Mermaid Falls Short

Mermaid is a great tool, but it excels in a different domain.

Mermaid's strengths	dataviz-svg's strengths
Flowcharts, sequence diagrams, ER diagrams	Scatter plots, heatmaps, histograms
State diagrams, Gantt charts	Box plots, area charts, bubble charts
Simple bar charts with a few data points	Charts with 20+ data points
Structural diagrams in general	Multi-axis encoding (x, y, color, size)

In short: Mermaid is for structural diagrams, dataviz-svg is for data-driven statistical visualization. When an agent can use both, the expressiveness of its documentation output improves dramatically.

Supported Chart Types

An example actually generated from the skill

dataviz-svg inherits the full expressiveness of Vega-Lite, so the range of supported charts is broad. Check out the official Vega-Lite gallery to see what's possible at a glance.

Bar charts — vertical, horizontal, grouped, stacked (including 100% stacked)
Line charts — single series, multi-series (with strokeDash for print-friendly output)
Scatter plots — basic scatter, bubble charts (size + color + log scale)
Area charts — stacked area, streamgraph
Heatmaps — 2D grid with overlaid text layers
Histograms — distribution via binning
Box plots — quartile display
Pie / Donut charts — donut variant via innerRadius
Layered charts — line + point + text label overlays
Facet / Concatenation — small multiples, horizontal/vertical multi-chart layouts

How It Works

Tech Stack

The core of dataviz-svg is simple. Vega-Lite and Vega are bundled into a single ESM file (~1.6 MB) with Vite, so it runs on Node.js alone.

Vega-Lite JSON spec
    ↓ compile()
Vega spec
    ↓ Vega runtime
SVG string
    ↓ writeFileSync
output.svg

No native dependencies like canvas or puppeteer. No additional npm install needed. The bundle ships with the skill, so it works right after installation.

Execution Flow from the Agent's Perspective

When an agent uses this skill, it follows 5 steps:

Understand data and visualization requirements — identify the input data (inline values / CSV / JSON), chart type, and axis/color mappings
Create a Vega-Lite JSON spec — write the spec as a JSON file, referencing the bundled pattern collection
Render the SVG — run render-svg.sh to generate the SVG
Embed in Markdown — reference with ![Chart Title](./assets/chart.svg)
Review and adjust — inspect the SVG, tweak label angles or color scales, and re-render

From the user's perspective, you just say "turn this data into a heatmap" and the agent handles everything — spec creation, rendering, and document embedding — end to end.

Example: Visualizing Sales Data

For example, to create a bar chart of quarterly sales data, the agent generates a Vega-Lite spec like this:

{
  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
  "width": 400,
  "height": 300,
  "background": "white",
  "data": {
    "values": [
      { "quarter": "Q1", "sales": 120 },
      { "quarter": "Q2", "sales": 185 },
      { "quarter": "Q3", "sales": 210 },
      { "quarter": "Q4", "sales": 165 }
    ]
  },
  "mark": "bar",
  "encoding": {
    "x": { "field": "quarter", "type": "ordinal" },
    "y": { "field": "sales", "type": "quantitative" },
    "color": { "field": "quarter", "type": "nominal" }
  }
}

Pass this to render-svg.sh and an SVG is generated — no browser required.

Design Trade-offs

Practicality comes first. Here are the deliberate trade-offs:

SVG only: No PNG/PDF support. Eliminating native dependencies (canvas / sharp) means zero setup friction
Static output only: Vega-Lite's interactive features (hover, zoom, selection) don't carry over to SVG
No font embedding: Relies on system fonts. sans-serif is the safest choice
No canvas package: Text bounding box calculations are approximate, which may cause slight misalignment with long CJK (e.g., Japanese) labels

These are intentional. The top priority is "works immediately in any environment, with no extra installation."

Installation

# For Claude Code
gh skill install oubakiou/skills dataviz-svg --agent claude-code --scope project

# For Codex
gh skill install oubakiou/skills dataviz-svg --agent codex --scope project

The only prerequisite is Node.js 18 or later. Vega and Vega-Lite are bundled, so no separate installation is needed.

Summary

dataviz-svg is laser-focused on one thing: giving AI agents the ability to create data visualizations.

Generates statistical charts that Mermaid can't handle, powered by Vega-Lite
No browser, no extra dependencies — just Node.js
The agent handles the entire pipeline: spec creation, rendering, and document embedding

"Turn this data into a chart." — With that single request, the agent picks the right chart type, generates it, and embeds it in your document. Naturally integrating data visualization into the AI agent workflow — that's what dataviz-svg is for.

Your LLM reads the whole file. It doesn't have to.

oubakiou — Fri, 19 Jun 2026 12:05:00 +0000

Coding agents read specs, design docs, and long READMEs every day. Most of the time, they only need a few sections. Yet they load the entire file into context.

The hidden cost of "just read the file"

Here's a scenario that plays out constantly. You ask your agent to check the error handling section of a 5,000-line API spec. The agent opens the file, reads all 5,000 lines into its context window, finds the 80 lines it needs, and answers your question.

The result is correct. But the agent also consumed a large number of tokens on the 4,920 lines it didn't need. Repeat this for every file read in a session, and the waste compounds fast.

The cost isn't just tokens. A context window stuffed with irrelevant content makes the agent's answers worse.

What humans do differently

When a human picks up a 300-page technical book, they don't read cover to cover to find the chapter on authentication. They flip to the table of contents, scan the chapter titles, and jump to page 47. LLMs can do the same thing.

Markdown is already structured

Markdown documents have a built-in structure: headings. A # Title followed by ## Section A followed by ### Subsection A.1 creates a hierarchy that mirrors a book's table of contents.

Split a Markdown file at heading boundaries, and you get a natural "table of contents + sections" structure. Each heading starts a new section, the heading text becomes the index entry, and the section number becomes the address.

This is the idea behind md2idx, a CLI tool.

How to use md2idx

md2idx converts a Markdown file into JSON with two fields:

index: a numbered table of contents where each line is <# markers for depth> <serial>. <heading text>
sections: a flat array of raw Markdown strings, one per heading

$ npx md2idx spec.md | jq -r '.index'
# 0. API Specification
## 1. Authentication
## 2. Endpoints
### 3. GET /users
### 4. POST /users
## 5. Error Handling
## 6. Rate Limiting

The serial numbers match the array indices. To read the Error Handling section:

$ npx md2idx spec.md | jq -r '.sections[5]'
## Error Handling

When a request fails, the API returns a JSON error object with...
(just the content of that one section)

To read a heading and all its children together:

$ npx md2idx spec.md | jq -r '.sections[2:5][]'
# Endpoints, GET /users, and POST /users — all 3 sections

For a 5,000-line spec where the agent needs 2 sections, context usage goes from ~5,000 lines to ~100 lines (20-line index + 80 lines of content). Depending on the document and which sections are needed, the reduction is typically 80–98%.

The output is designed to work with jq. One-line JSON by default (pipe-friendly), --pretty for formatted output. Reads from a file argument or stdin.

Why not just grep for headings?

grep -nE '#{1,6} ' spec.md gives you a list of headings. For simple cases, that works. But md2idx covers problems that grep can't solve:

Section body retrieval: grep only returns heading lines. To get the body, you need to calculate the line range and use Read with offset/limit. With md2idx, jq '.sections[N]' is all it takes
Setext headings (=== / ---): invisible to grep's # pattern
# inside code fences: grep returns false positives. md2idx skips fenced blocks
Inline markup: grep includes [link](url) etc. as-is. md2idx strips markup in the index while preserving it in section content

Automate with a Claude Code skill

Skill details (SKILL.md)

With the md2idx-read skill, the agent autonomously handles everything from fetching the index to selecting sections.

The agent checks the file size and, if large, calls md2idx to fetch the index
It reads the index and identifies which sections are relevant to the current task
It retrieves only those sections via jq slicing
If more sections are needed, it goes back to step 3 (the index is already in context)

# install the skill
gh skill install oubakiou/md2idx md2idx-read --agent claude-code --scope project

# or with npx
npx skills add oubakiou/md2idx --skill md2idx-read --agent claude-code --yes

Once installed, the agent uses the skill proactively whenever it encounters a large Markdown file. No manual invocation needed — it reads the index first, picks sections, and skips the rest.

A fallback is included. If md2idx isn't available (network-restricted environments, permission issues), it falls back to grep for headings and Read tool with offset/limit. Less accurate, but functional.

Try it now

# read the index of any Markdown file
npx md2idx README.md | jq -r '.index'

# grab a specific section by number
npx md2idx README.md | jq -r '.sections[2]'

# search within a section
npx md2idx data.md | jq -r '.sections[4]' | grep Tokyo

# global install
npm install -g md2idx

md2idx has zero external dependencies — a self-contained line scanner, not a Markdown AST parser. It handles ATX headings (# style), setext headings (=== / --- underlines), code fence skipping, and inline markup stripping.

md2idx is MIT-licensed and fully open source. If your LLM agents are reading entire large Markdown files, give it a try:

GitHub repo →
npm → npx md2idx your-file.md | jq -r '.index'
Claude Code skill → gh skill install oubakiou/md2idx md2idx-read

If you've tried it in your agent workflow, I'd love to hear how it went — drop a comment below or open an issue on GitHub.

The next bottleneck after AI writes your code is reviewing the docs it writes

oubakiou — Wed, 17 Jun 2026 03:43:15 +0000

Coding agents draft specs, architecture docs, changelogs, and README updates in seconds — but a human still has to judge the quality of all that output.

The bottleneck shift

A year ago, the typical workflow was: you write a spec, you get comments, you revise, then you implement and get code review. Humans did most of the writing and coding. Now, agents produce first drafts of design docs, API references, runbooks, and onboarding guides — and they do it in seconds.

Code implementation and code review can now be handled by agents, so those are no longer the bottleneck. What surfaced instead is the step right before: document review. A human has to read 2,000 lines of generated markdown and decide what's wrong. The writing part got dramatically faster. LLMs can assist with document review too, but compared to code implementation and code review, the human judgment required is still larger.

This asymmetry compounds fast. Every agent-assisted project now has a stack of "needs human review" documents growing in a shared folder. If you're running multiple agent loops in parallel — one for the spec, one for the implementation plan, one for the test strategy — review becomes a pipeline stall. GitHub PRs remain the right tool when you need third-party review. But the step before that — the fast local self-review loop where you and your agent iterate on a draft — doesn't belong in a PR. Branching, diffing, and assigning reviewers is a lot of process for a first draft the agent wrote in seconds.

Why prose feedback is lossy

The most common workaround today is to have the agent read the document and then fix things based on natural-language feedback:

"The error handling in section 3.2 is too vague — be specific about what happens on timeout."

This looks reasonable. The agent reads it, searches for something about error handling, and makes a change. But several things go wrong:

Position is ambiguous. If section 3.2 has three paragraphs about error handling, which one did the reviewer mean? The agent guesses, and sometimes guesses wrong.

Context is lost. The reviewer was looking at a rendered document — they saw the Mermaid diagram, the table layout, the code blocks with syntax highlighting. The prose feedback carries none of that visual context.

Round-trips are vague. The agent applies a fix. Did it address the right spot? The reviewer pastes the updated draft back into the chat. No diff. No anchoring. Another round of guessing.

The fundamental problem: prose feedback loses the structure the reviewer had in mind. The reviewer knows which text range they're commenting on — but that spatial information is discarded the moment they express it in natural language.

Feedback as a machine-readable artifact

What if the reviewer's comments came back as structured JSON, with every comment anchored to a precise location in the source markdown?

That's what MDXG Redline (online demo) produces. The reviewer selects any text range in a rendered markdown document, leaves an inline comment, and on export the tool writes a JSON file like this:

{
  "document": "api-spec.md",
  "docHash": "a1b2c3d4e5f6a7b8",
  "exportedAt": "2026-06-15T10:30:00.000Z",
  "comments": [
    {
      "id": "f3a1c8b2",
      "quote": "The service retries up to 3 times",
      "comment": "Specify the backoff strategy — is it exponential? Fixed interval? This matters for downstream timeout config.",
      "headingPath": ["## 3. Error Handling", "### 3.2 Retry Policy"],
      "sourceLine": 142,
      "created": "2026-06-15T10:28:11.000Z"
    }
  ]
}

Every comment carries:

quote — the exact text the reviewer selected (human-readable reference)
comment — what the reviewer wrote
headingPath — the heading ancestry, from shallowest to deepest (["## 3. Error Handling", "### 3.2 Retry Policy"]), so the agent can navigate the document hierarchy
sourceLine — the 1-origin line number in the source markdown, so the agent can jump straight to the right spot

(The actual export also includes blockId, startOffset, and endOffset — internal anchoring fields used for round-trip re-import into the next review round. The four fields above are the ones your agent needs.)

No guessing, no ambiguity. The agent reads the JSON, opens the source file, goes to line 142, and applies the feedback directly. The feedback JSON is a plain file on disk — any agent that can read JSON and edit files can consume it, regardless of framework or provider.

The loop in practice

MDXG Redline is designed around a review loop where an agent and a human pass a document back and forth:

The agent uses the CLI to generate a <name>-<hash>-review.html file from the markdown — a self-contained HTML file that renders the document with syntax highlighting, diagrams, and math.
The CLI auto-launches the reviewer's browser.
The reviewer reads the rendered document, selects text ranges, and leaves inline comments. One click writes <name>-<hash>-feedback.json to the same folder.
The agent monitors in the background, automatically picks up the JSON, and applies the feedback.

Each revision gets a unique hash in the filename, so review/feedback pairs for different revisions never collide.

For Claude Code and Codex users, there's a md-review skill that automates the entire loop — HTML generation, waiting for feedback, and applying revisions:

SKILL.md

# install the skill
gh skill install oubakiou/mdxg-redline md-review --agent claude-code --scope project

# then just tell the agent:
# "please request a review for spec.md"
# or invoke the slash command directly:
# /md-review README.md

The skill is a convenience wrapper. Since the feedback JSON is a plain file, the loop works with any agent — you can wire it into a custom LangChain pipeline or a simple shell script that polls for the JSON.

What it renders

MDXG Redline isn't just a review overlay — it's a markdown rendering engine packed into a single standalone HTML file with no backend and no dependencies.

Smartphone support — On mobile screens, the UI switches to a layout with a footer bar for TOC, Comments, and Search. You can review and comment from your phone — no need to wait until you're back at your desk.

Syntax highlighting for ~235 languages — The CLI auto-detects which languages your document uses and injects only those Shiki grammars, keeping the file lean.

Mermaid diagrams and KaTeX math — Fenced `mermaid blocks render as SVGs; $...$ / $$...$$ math renders via KaTeX. Both are auto-detected and injected only when present.

Word-style stacked pages — The document splits at H1/H2 boundaries into paper-like sheets you scroll through vertically, with a sidebar for navigation. Footnotes and left-hand keyboard navigation are also built in.

See the full feature list in the README for details.

Try it in 30 seconds

No install needed — open mkdn.review and load any public GitHub raw URL. Here's a one-click example that loads the project's own README:

Open this README on mkdn.review →

Select any text in the rendered document, leave a comment, and click "Copy as JSON" to see the structured output.

Want to try with your own project? Just replace the URL: mkdn.review/?url=<your-github-raw-url>.

Local review — if you have Node.js:

npx mdxg-redline your-draft.md

This generates a review HTML in the same directory and opens it in your browser. The generated file is fully self-contained — you can email it, drop it in Slack, or put it in a shared folder. Everything stays local; no content or comments are sent anywhere.

Privacy note: The standalone and CLI builds enforce connect-src 'none' via CSP — your document body and comments cannot leave the browser. The online version (mkdn.review) likewise never sends your content or comments anywhere — the only network request is the ?url= fetch, which retrieves public raw content from a short allowlist (GitHub raw, Gist raw).

It's an MDXG implementation

MDXG Redline is a third-party implementation of the Markdown Experience Guidelines (MDXG) by Vercel Labs — the read-only Viewer rendering profile, with a review layer (inline commenting + structured JSON export) added on top. See the compliance table for details.

MDXG Redline is MIT-licensed and fully open source. If reviewing agent-generated documents is a bottleneck in your workflow, give it a try:

Try the online viewer →
GitHub repo →
npm → npx mdxg-redline draft.md
Claude Code skill → gh skill install oubakiou/mdxg-redline md-review (usage details)

If you've tried it with your agent workflow, I'd love to hear how the structured JSON feedback format worked for you — drop a comment below or open an issue on GitHub.