DEV Community: Rodion Kazennov

My Godot level load fell from 34.5 s to 2.7 s, and the best fix removed work nobody needed

Rodion Kazennov — Wed, 11 Nov 2026 12:31:14 +0000

On the morning of 2 November the landing in my Godot 4 game, Planet Wanderer, took 34.5 seconds to load. A week later it takes 2.7. The biggest single step was not threads or caching. A footprint calculation went from 8.3 s to 0.3 s because it was walking models with 50,000 to 400,000 triangles when a copy with a few hundred would do. Below are the steps in the order I took them, with the log line that justified each, and one fix that measured as nothing.

All times are from one Windows PC with 8 worker threads, read from a timing line the game prints for each part of the load. Each number comes from a separate run, so treat anything under about half a second as noise.

What "landing" loads

When the player picks a landing spot and presses "Sit here", the game builds a camp around the rover. It lays out two small camps and a town, places a hundred or so props, places the people who live there, plants grass and flowers, and works out where each thing stands so the grass bends away from it. Props and people are 3D models made from images, so some of them carry 50,000 to 400,000 triangles.

I had been measuring the load as one number. That number only told me it was bad. Splitting it into named parts, each printed with its time, is what made everything below possible. If you do one thing from this post, do that.

Step 1: a frame that holds for 36 seconds

After "Sit here", one function placed the camps in a single frame: 9.3 s. Then one function planted the ground around them in a single frame: 36.7 s. The loading ring on screen stood still for most of that, because nothing could draw while the frame was busy.

The fix was the boring one: make both steps advance a little each frame and let the ring move. Same total work, the screen alive. This did not make the load shorter, but it made it measurable, because the next steps could now be timed one at a time while the game kept rendering.

Step 2: plants as one buffer (36.7 s in the part, 92 ms for the plants)

Most of that 36.7 s was the ground plants. I moved them into a single MultiMesh buffer and wrote the transforms into it directly. On the Windows PC the plants alone then measured 92 ms. After this and the next step, the whole landing load went from 34.5 s to 21.6 s.

Step 3: the footprint walk, moved to workers (13.1 s + 4.0 s to 8.3 s)

To make the grass lie flat around a tent, the game needs the tent's footprint: which ground cells its triangles cover. The code walked every triangle in GDScript, twice, once for the footprint and once for the feet of the people. The cost was not reading the mesh surfaces (that took 0.3 s in total). It was the walking: 13.1 s and 4.0 s.

I changed it to one walk that fills both masks, and ran it on WorkerThreadPool, one task per ground cell:

var gid: int = WorkerThreadPool.add_group_task(func(i: int):
    var jb: Dictionary = jobs[i]
    var wk: Dictionary = _walk(jb["surfs"], jb["pm"] != null)
    jb["mask"] = wk["mask"]
    jb["feet"] = wk["feet"], jobs.size(), -1, true, "camp footprints")
while not WorkerThreadPool.is_group_task_completed(gid):
    await planet.get_tree().process_frame
WorkerThreadPool.wait_for_group_task_completion(gid)

Reading scene nodes and meshes must stay on the main thread, so a first pass on the main thread collects each model's vertices and indices into plain arrays, and the workers only do arithmetic on those. The await on the frame loop keeps the ring moving while they run. Result: 8.3 s on 8 threads, and the frames kept coming.

Step 4: the fix that measured as nothing

The camp build was almost entirely props: the "heart" of the camp took 4.2 s of 4.4. A hundred things were each being repainted (materials adjusted, one by one). My idea: paint the first thing of each model path, and let the rest be copies of it.

I added first_of(path), kept the first painted node, and duplicated it for the rest. The heart went from 4.2 s to 4.6 s. Slower, within noise.

The reason was in the data, not the code. At the first camp every prop is the first of its path, so there were no copies to share. I only saw that because I added a per-model timing line next time instead of re-running the stopwatch around the whole function.

Step 5: it was the disk read, 180 to 300 ms per model

The per-model line said the time was almost all the first read of each model from disk: 180 to 300 ms for each of about 30 models. Painting was 0 ms. Copies and far copies cost 2 to 35 ms.

So the fix was to start reading them earlier. The screen where you choose a landing spot comes before the build, and ResourceLoader can load in the background while it is open:

static func warm(paths: Array) -> void:
    for p in paths:
        if _warm.has(p) or not ResourceLoader.exists(p):
            continue
        _warm[p] = true
        ResourceLoader.load_threaded_request(p, "", true)

The place-selection screen requests all 59 files (the models and their far copies, and the people). When the camps are built the requests are already finished and the "read" is a lookup. Measured: camp near the landing 6,957 ms to 479, town 3,378 to 617, heart props 4,924 to 29. Whole landing load 22.8 s to 13.6 s.

One trap here: warm() keeps the requests, and something has to collect them afterwards or they pile up in memory. A forget_firsts() call frees them once the camps are up.

Step 6: stop walking the big model (8.3 s to 0.3 s)

The footprint walk was still the largest part, 8.3 s on 8 workers. I looked at which triangles it walked and found it went through every mesh under every prop, including the full 50,000 to 400,000 triangle model, and also the model's far copy: the low-triangle stand-in the game shows at a distance.

For a footprint on the ground, the far copy is enough. Now when a thing has a node called far, the walk takes that and stops:

var fc: Node = nd.get_node_or_null("far")
if fc != null:
    todo.append(fc)
    continue

The footprint walk went from 8.3 s to 0.3 s. I checked that the result stayed the same: plants removed 6,026 against 6,041 before, plants trampled 850 against 836. The footprints are nearly the same, because a few hundred triangles at the right silhouette give the same ground cells as 300,000 at the same silhouette. Whole load: 13.6 s to 5.7 s.

This is the step I would have missed if I had stopped at "use more threads". Eight threads took 17 s of work down to 8.3 s. Doing less work took it to 0.3.

Step 7: a release that came too early (1.7 s to 0.3 s)

The locals' camps took 1.7 s in one frame, and the per-model line showed they were reading their people's models from disk again. The cause: forget_firsts() freed the pre-read models right after the camps were built, before the locals, which use the same files. I moved the call to after both locals' camps. Locals' camps 1,762 ms to 321. Whole load 6.3 s to 4.7 s (a different run from the 5.7 above, so the baselines differ).

Step 8: the town's free-ground calculation, and the last models

Two smaller things took the load from 4.7 s to 2.7 s:

The town's locals spent 816 ms in a free-ground calculation per cell, almost all of it on the main thread. I collected the data on the main thread and ran the triangle walk on workers, one cell per worker. 816 to 329 ms, camps 321 to 209. I compared the logged places of the locals before and after: identical to the hundredth of a metre.
The camp and town cell models and the wounded person with the blanket were still read at the moment of use. Reading them early too: camp 617 to 244 ms, town 736 to 97, the wounded figure 348 to 28. Whole load 4.2 s to 2.7 s.

The table

Step	What changed	Whole landing load
start (2 Nov)	one 36.7 s frame	34.5 s
1 to 3	frames split, plants in one buffer, footprint walk on workers	21.6 s
5	models read ahead during place selection	13.6 s
6	walk only the far copy	5.7 s
7	locals' models kept until used	4.7 s
8	town on workers, last models read ahead	2.7 s

What I would do again, and the part I still do not understand

Three habits made this go quickly. Print a named time for every part and keep the line in the log permanently. When a fix does not help (step 4), go to a finer-grained measurement before inventing the next fix. And check that a speed-up did not change the result: I compared plants removed and the locals' positions each time.

What I cannot explain yet is the other machine. In the game's log from my Linux PC, slicing the ground after the load costs 39 s of CPU, against 5.8 s on the Windows PC. The numbers above are all from the Windows PC. Is that gap fewer fast cores free at that moment, a different Godot build, or something in how the workers are scheduled there? I have not measured it, and I would like to know where else a seven-fold difference like that could come from.

Godot 4 animation retargeting with free mocap: five traps that look like bad animation

Rodion Kazennov — Mon, 09 Nov 2026 12:31:06 +0000

I retargeted free CMU motion capture onto a humanoid character in Godot 4 with SkeletonProfileHumanoid. Most of the time I lost was not on the animation itself. It went on results that looked like a bad clip or a bad rig and turned out to be a setting, a facing, or a measurement that read the wrong pose. These are the five that cost the most, each with what I saw, what caused it and what fixed it.

1. Wide legs: the source's rest pose is not its stance

The CMU skeleton rests in a wide straddle, with its ankles 0.818 m apart. In the clip itself the actor stands with the ankles 0.268 m apart. I first read the rest and called it "the source's stance", which made a correct transfer look broken.

The rest also poisons the transfer. SkeletonProfileHumanoid measures every motion against the rest pose. The source's legs moved inward relative to their straddled rest, so the target's legs moved by the same amount the other way and came out wide.

The fix belongs on the source import, not the target: retarget/rest_fixer/fix_silhouette/enable = true on the mocap file. It straightened the source rest while leaving the clip's world pose alone (the ankle gap in the take stayed 0.268 m). On the target the same option did harm: it moved the target's rest 32 cm away from its bind pose.

If you also straighten the legs in your own export script, turn that off. With both on, the stance came out wrong again.

2. Inverted arms: `overwrite_axis` and a rig that faces −Z

My rig was built facing −Z, so its left side is on −X. The humanoid profile and the mocap face +Z with the left on +X. Both rigs were named correctly; the problem is that overwrite_axis forces the profile's bone axes onto the target. On the legs and spine that is harmless, because those bones point along Y. On the arms, the only bones that point along X, the forced axis ran against the geometry, and every arm rotation came out inverted: the hands ended far above the hips in poses where the source has them below.

Turning the character to face +Z removed the conflict. To check facing and side, I put markers in the frame (a sphere at +X, one at +Z and a box on the bone named LeftHand) instead of deducing the side from a render. Two of my three guesses from renders alone were wrong.

3. Measurements that read the rest pose in a headless run

I measured legs and angles from a script run with godot --headless -s script.gd. Skeleton3D.get_bone_global_pose() returned the rest pose on every frame, even after force_update_all_bone_transforms() and with the AnimationPlayer playing. The animation was applied: get_bone_pose_rotation() on the same bone changed frame to frame. Only the global composition was stale. The numbers looked plausible, so I drew several wrong conclusions from them.

Composing the chain by hand gives the right answer:

func chain(n: String) -> Transform3D:
    var t := Transform3D.IDENTITY
    var b := sk.find_bone(n)
    while b >= 0:
        t = sk.get_bone_pose(b) * t
        b = sk.get_bone_parent(b)
    return t

In a run that renders, the API and this function agree to three decimals.

4. 170 degrees of twist on a clip that looks fine

A twist audit built on get_bone_pose_rotation() reported 160 to 180 degrees of twist on almost every bone of clips that looked perfectly normal. That call returns the pose including the bone's rest orientation. Take the delta from the rest instead:

var rest_q := sk.get_bone_rest(bi).basis.get_rotation_quaternion().normalized()
var delta := (rest_q.inverse() * sk.get_bone_pose_rotation(bi).normalized()).normalized()

With that change the same audit read about 55 degrees on the upper arms of a lively take and 8 to 14 in a calm one, which is what the eye sees.

5. A perfect T-pose at rest and one arm straight up in every pose

This one starts in Blender, before the GLB reaches Godot. I rotated the mesh and the armature with bpy.ops.object.transform_apply(rotation=True), one after the other. Only the armature rotated. The skin's bind matrices absorbed the difference, so the exported GLB looked perfect at rest and threw one arm straight up in every animated pose.

Rotating the data instead keeps the two bound together:

body.data.transform(mat)
rig.data.transform(mat)

Rest renders cannot show this, so check it with the same markers as in trap 2, in an animated pose.

Checks that would have saved me the time

Fix Silhouette on the mocap source only, and the source's stance measured inside the take, never from its rest.
Facing and left/right verified with markers in Godot before any retarget setting is tuned.
Bone measurements in headless scripts composed by hand from get_bone_pose(), and rotations compared against the rest.
A measurement that stays exactly the same across a parameter change means the change never reached the export. In my case a flag had switched the bones to quaternion rotation, and a later smoothing step kept editing the now inert euler channels.

Judge clips in the engine, from frames captured in Godot, rather than from Blender's clay renders. Six hand-authored versions were rejected in a row while Blender's workbench renders of them looked passable; the first honest look came from frames captured in Godot. Bone world positions in the two agreed to the millimetre when I checked, so a disagreement between the two views is a misreading, not an import bug.

WebGPU fails silently: buffer limits, a missing COPY_SRC and default requiredLimits

Rodion Kazennov — Wed, 04 Nov 2026 12:32:59 +0000

The run finished at 0.04 ms per step, roughly 500 times faster than the simulation should have run at that size. The health report was clean: no NaN, no infinity, the largest force was zero, and every particle had zero neighbours. It was clean because nothing had been computed.

This happened in protocell-genesis, a WebGPU simulation I built to test whether a coarse-grained "primordial soup" can assemble itself into a closed vesicle, a bilayer membrane with water trapped inside. In this model it can't, and the repository has the measured reasons. The largest run had 483,268 particles. Over the project, six bugs behaved like the one above: no exception, no error on the page, and numbers that looked plausible or even good. Three of them are specific to WebGPU. Here they are, with what each one looked like and what changed.

1. A buffer larger than the device allows

At a box size of 85 σ the Verlet neighbour list needed 5,484,474,528 bytes. The device's maxStorageBufferBindingSize was 4,294,967,292.

WebGPU does not throw when createBuffer fails validation. It prints a warning to the console and returns an invalid buffer, and every later dispatch on a bind group that uses it is dropped. The simulation loop kept stepping and kept reading back zeros: max|F| = 0, 0 non-finite values, 0 neighbours, 0.0405 ms per step.

The fix is a check before allocation. createSoup computes every buffer whose size scales with the particle count and compares it with the device's own limits:

const bindingLimit = device.limits.maxStorageBufferBindingSize
const bufferLimit = device.limits.maxBufferSize
for (const [what, bytes] of candidates) {
  if (bytes > bindingLimit || bytes > bufferLimit) {
    throw new Error(
      `buffer ${what} needs ${bytes} bytes at N=${capacityN}; ` +
        `maxStorageBufferBindingSize=${bindingLimit}, ` +
        `maxBufferSize=${bufferLimit}`,
    )
  }
}

2. Reading back a buffer without COPY_SRC

Reading data off the GPU means copying it into a mappable staging buffer. If the source buffer was created without GPUBufferUsage.COPY_SRC, the copy is a validation error. That invalidates the command buffer, so the submit does nothing. The staging buffer was just created, WebGPU zero-initialises new buffers, and mapAsync resolves normally. You get zeros that look like data.

Two random-number buffers were missing the flag. All 72 checkpoints on disk had saved a random-number state of pure zeros, so every resumed run seeded the thermostat with 0 for every particle. With identical Langevin noise on every particle, the whole system drifts as one body: the mean squared displacement was 62.32 σ² for all seven species alike, and relative diffusion, which is what lets aggregates find each other, was gone. After the fix, the largest aggregate at step 90,000 grew from 268 to 588.

usage is a readable property of a GPUBuffer, so the whole class of bug costs one bitwise test:

if ((src.usage & GPUBufferUsage.COPY_SRC) === 0) {
  throw new Error(
    'readBack: the buffer was created without ' +
      `GPUBufferUsage.COPY_SRC (usage=${src.usage})`,
  )
}

3. requestDevice() without requiredLimits

A device requested without requiredLimits gets the default limits, not the ones the adapter supports. The default allows 8 storage buffers per shader stage. The bond kernels grew past that, pipeline creation with an 'auto' layout failed with a console warning, and every affected pass did nothing. No bonds formed at all.

The size limits fail the same way, at a particle count instead of a code change. The default maxStorageBufferBindingSize is 128 MiB. With 1,000 neighbour slots per particle, the list fit at 28,400 particles (113.6 MB) and silently did nothing at 36,400 (145.6 MB).

The device now asks for the adapter's own maxima:

const { limits } = adapter
const device = await adapter.requestDevice({
  requiredLimits: {
    maxStorageBuffersPerShaderStage:
      limits.maxStorageBuffersPerShaderStage,
    maxStorageBufferBindingSize: limits.maxStorageBufferBindingSize,
    maxBufferSize: limits.maxBufferSize,
  },
})

4. A NaN guard that could never fire

A safety check threw when particles drifted further than half the Verlet skin, using sqrt(maxDriftSq) > skin / 2. Every comparison with NaN is false. Once positions became NaN, the drift was NaN and the check passed on every tick. The guard ran the whole time and could not see the failure it was written for.

The fix scans the IEEE-754 exponent bits of every position on the same tick, before the old comparison. It costs 0.212 ms per chunk of 1,000 steps that takes 331.68 ms, which is 0.064%. On the composition that had been diverging silently, the run now throws at step 1,000 and names 6,957 non-finite components out of 60,750.

5. Forces from before the box changed size

Drying and rehydration change the box size. The code rescaled the coordinates, rebuilt the neighbour grid and the Verlet list, and then took the next step with forces computed before the rescale. The result is a wrong number, which is harder to spot than zeros.

A test now pins it from both sides. With the fix, the force buffer on the GPU matches a fresh calculation to within 1.5e-4 at a force scale of 513. Without it, the buffer equals the old forces bit for bit and is off by 313.7.

It had moved a result I had already written up. At box 30 the largest aggregate came out 2.28 times smaller once the forces were right (three runs on each side, ranges not overlapping), and the effect I had reported for drying cycles shrank from 12.5–14.5× to 8.1–8.4×. A guard that had fired 9 times out of 216 fired 0 times out of 36 after the fix.

6. Gates that recognised their input by name

This one has nothing to do with the GPU. The percolation gates, which turn measurements into pass or fail verdicts, found their input by the campaign label, zfB54, written into the code. The first campaign with a different label flipped the gate to "unproven" without an error. Rows now carry a role field, and the label is only a fallback.

A pipeline that identifies its inputs by name goes quiet when someone runs a new experiment, which is exactly when its verdict matters.

A seventh: the instrument read zeros

The test that checked whether the page was drawing read the WebGPU canvas through drawImage into a 2D canvas. It reported 0 changed pixels out of 30,000, on a page that had just drawn 119 frames with 661 instances. The test now uses the scene's own instance counters and a real page.screenshot().

What I treat as a failure now

Zero forces, zero neighbours and zero bonds are failures until something shows the physics can produce them. So is a speed-up nobody made. The same goes for a guard that has never fired, until I have seen it fire on purpose.

WebGPU reports validation errors to the console and to the device's uncapturederror event, and error scopes (device.pushErrorScope('validation')) let you await them around a block of work. This project uses explicit checks that throw and name the buffer and the byte count. Whichever you pick, the default is a console warning and a result that looks fine.

The code, the gate report and the full verdict are in the repository, and the snapshot gallery renders the real checkpoints in any current browser.

I capped my coding agent's tool output to save tokens. It used 2.65 times as many.

Rodion Kazennov — Mon, 02 Nov 2026 13:01:04 +0000

To save tokens, I capped every response from my coding agent's code-graph tool at 12,000 characters. One answer then took 18 turns instead of 7, and the input tokens it was billed for went from 267k to 708k, 2.65 times as many.

This post explains why a smaller response used more tokens, and what else I found when I rebuilt the measurement on public repositories: 252 headless Claude Code runs (non-interactive, one question each), every token counted. You'll also get the benchmark's exact command line and a script that shows how much of your own sessions' input comes from the prompt cache. The runner and every raw run, answers included, are in claude-code-token-bench.

Why I was cutting tool output

I wanted to know whether a code graph makes Claude Code, Anthropic's coding agent, cheaper than plain search. The graph came from codebase-memory-mcp, an open-source MCP server that builds a graph of a repository's symbols and the calls between them. MCP, the Model Context Protocol, is how Claude Code plugs in outside tools.

The first measurement went badly for the graph. On five questions, the agent with the graph was billed 2.35 times the input of the agent with grep, 4.26M tokens against 1.81M, and it came out cheaper on only one question.

So I looked at what the server sent back. One search hit was 1,342 bytes of JSON, most of it hash vectors, complexity metrics and a list of every identifier in the function's body. The same hit as a grep line was 115 bytes. On top of that, the server's 14 tool definitions rode along in every prompt.

I put a thin MCP server of my own in front of it, graph-ask. This wrapper dropped the fields no code question needs, which brought a hit down to 251 bytes. It also capped every response at 12,000 characters: it halved the longest list until the response fit, and noted how much it had cut.

What the cap did

The tool that returns a repository's structure hit the cap. The model got a shortened list with a note saying how much was missing, and asked again. That answer took 18 turns where it had taken 7, and its billed input rose from 267k tokens to 708k.

A turn is one round trip to the model. Each one is a new API request, and it carries the whole conversation: the system prompt, the tool definitions, the question and every tool result so far. Divide the totals by the turns and the average turn comes to about 38,000 tokens before the cap and 39,000 after. The smaller responses didn't make a turn any cheaper. The turn count went up 2.57 times and the tokens 2.65 times, which came to 441k more tokens in all.

That is also why a turn costs more the later it comes. Each tool result joins the history that every later turn sends again, so billed input grows faster than the number of turns.

One limit on the 2.65: in that round I counted every input token the same, including the ones served from the prompt cache, the unchanged start of the conversation that Anthropic keeps between requests. Anthropic bills those cache reads at a tenth of the input price, so 2.65 is a ratio of tokens, not of dollars. The public benchmark below prices each kind of token at its own rate.

What I changed in the wrapper

I kept the trimming and rebuilt the rest around the number of turns. One tool call now answers a whole structural question, with a function's definition, source, callers and callees together, so the model doesn't assemble the answer over several turns. A script summarizes each repository's layout once into a file, and the agent reads that file instead of querying the graph. Asked for a project's HTTP routes, the tool returns each handler along with the config files that hold the URL prefixes. And the cap went up to 60,000 characters.

Together those changes took that round's questions from 83 turns and 3.46M billed input tokens to 36 turns and 1.62M. Listing a project's HTTP routes alone went from 23 turns to 8.

The current version also checks a name before it queries. The graph matches names exactly, so a near miss returns nothing and costs a turn to recover. When the name doesn't exist, the tool takes the top search hit and says it did.

The wrapper still labels a cut when a response hits the cap. From server.mjs, shortened:

// Bring a response under MAX_CHARS by shortening its longest arrays, halving the
// worst offender at a time, and recording what was dropped next to it. Slicing the
// serialized JSON instead would hand the model invalid syntax and no idea what is
// missing — the truncation has to stay legible, or it reads as a complete answer.
const kept = worst.arr.slice(0, Math.max(1, Math.floor(worst.arr.length / 2)));
parent[key] = kept;
parent[`${key}_truncated`] = `showing ${kept.length} of ${original}`;

The 12,000-character version had the same label, and the model went back for the rest anyway. The label stays so that a cut list never passes for the complete answer.

Bash and subagents hid more turns

The cap wasn't the only thing adding turns in that round. The grep-only setup was allowed Bash and subagents, the helper agents Claude Code can start for a subtask, and the model ran grep | head loop after loop, much of it inside subagents. It was billed 1.8M input tokens. With Bash and subagents denied, the same setup needed 252k.

Two things kept this out of sight. Claude Code's turn count covers only the main loop, so a subagent's turns are billed but missing from the count of the run you're watching. And scoped Bash permissions don't match piped commands: a grant for grep alone doesn't cover the pipelines the model writes, so different setups hit different numbers of permission denials.

A denial can even look like a saving. In headless mode nobody is there to approve a tool, so an MCP server's tools have to be listed in --allowedTools, or every call is refused while their definitions are still paid for. My very first graph run looked six times cheaper than grep. In that run the graph tools had been refused.

For read-only questions the benchmark gave the agent only Read, Grep and Glob, Claude Code's built-in tools for opening a file, searching file contents and finding files by name. Bash was denied, and so was Agent, the tool that starts subagents. For an agent that only has to read code, that's the set I'd start from. The benchmark ran each question like this, from the repository's checkout:

claude -p "Which functions or route files call createPaymentIntent? Ignore tests. Give the file:line of each call site, nothing else." \
  --output-format stream-json --verbose \
  --strict-mcp-config --setting-sources project --no-session-persistence \
  --allowedTools Read Grep Glob \
  --disallowedTools Bash Agent

--strict-mcp-config loads only the MCP servers named on the command line, none here. --setting-sources project skips user-level settings, hooks and plugins, so nothing installed on the machine steers the model. stream-json reports each tool call and the run's token usage, which is what the benchmark recorded.

The earlier result didn't survive public repositories

That earlier round had also found my wrapper 26% cheaper than grep with Claude Haiku and 18% with Claude Sonnet. But it had given a routing rule, a line in the prompt that says when to use the graph and when to use grep, to every setup, including the one with no graph at all. And its repositories were private, so no one else could check it. I rebuilt the benchmark on public repositories, and that result did not reproduce.

The setup: Claude Code 2.1.283 in headless mode with Claude Haiku 4.5 and Claude Sonnet 5. Three small repositories have 1.6k to 5k graph nodes each, where a node is one function, class, file or similar item: FastAPI's full-stack template, ltx-2-mlx (an MLX port of the LTX-2 video model) and avoid-ai-writing (a tool that flags AI writing patterns). One large repository, dub, a link attribution platform, has 25k. The three small repositories share twelve questions, and dub has twelve of its own. In each set, six are about structure, such as who calls a function or the path from an endpoint down to the database write, and six are about exact strings, such as a config value or an error message. Every answer is checked for the facts it must contain. The 252 runs cost $14.69 at API prices.

Tokens are priced by kind, with Anthropic's multipliers: fresh input at 1, a cache write at 1.25 and a cache read at 0.1. The setups run in a different order on each repetition, only the setups with a graph get the routing rule, and none of them has Bash or Agent.

Median priced tokens per run, and correct answers:

repositories, model	graph through my wrapper	graph server directly	Read and Grep only
3 small, Haiku 4.5	30,321 · 28/36	43,846 · 32/36	31,159 · 33/36
dub, Haiku 4.5	18,791 · 22/24	20,687 · 22/24	14,955 · 20/24
dub, Sonnet 5	17,227 · 22/24	23,085 · 22/24	13,348 · 22/24

The graph never beat Read and Grep on tokens. With Sonnet on dub, Read and Grep were cheaper on 11 of the 12 questions at the same accuracy (p = 0.034, a Wilcoxon signed-rank test over per-question medians). My wrapper did beat the bare graph server in all three settings, on 8, 11 and 9 of the 12 questions.

The graph's one clear win was accuracy. On dub, Haiku answered the structural questions correctly 12 times out of 12 with it and 8 out of 12 without, at a median of 23.5k tokens against 16.0k on those questions.

The most expensive setup was a graph the agent mostly ignored

By default, Claude Code 2.1.283 lists an MCP server's tools by name and loads a tool's full definition only when the model asks for it through its ToolSearch tool. In headless mode Haiku mostly didn't ask. On the twelve small-repository questions:

setup	runs that called the graph	correct	median priced tokens
definitions deferred (the default), routing rule in the prompt	2/12	11/12	41,008
all definitions loaded, no rule	4/12	9/12	62,008
all definitions loaded, with the rule	22/36	28/36	30,321

The first two rows are one run per question; the last is the main run, three per question. Setting ENABLE_TOOL_SEARCH=false in the environment makes Claude Code load every definition up front. In six pilot runs with the default, the graph was never called. With that variable set, the same question went through the graph in three turns.

Loaded definitions are paid for on every turn, whether anything calls them or not. That is why the mounted graph with no rule cost the most of any setup here. Even with the rule, Sonnet picked the graph in 10 of its 24 runs on dub and Grep for the rest.

If you keep an MCP server mounted, load its definitions and tell the agent when to use it. The benchmark passed the rule with --append-system-prompt. Day to day it belongs in CLAUDE.md, the instruction file Claude Code reads at the start of a session:

Structure (who calls what, call chains, routes, module layout, a symbol end to end): one graph_ask call.
Exact strings, config values, flags, error text: Grep and Read.

graph_ask is my wrapper's one-call tool; name your own server's tool there. If you can't say when the agent should use a server, unmount it.

A warm cache made later setups look cheaper

Back-to-back runs on the same repository share a prompt cache. When the setups ran in a fixed order, the later ones started warm and got a discount the first one never had, which is why the order now rotates.

You can see how much the cache does in your own sessions. tools/cache-audit.mjs in the repository reads Claude Code's local session logs, reports what share of billed input came from the cache, overall and per project, and lists the worst sessions:

node tools/cache-audit.mjs --days 7

In my earlier benchmark runs, 83 to 88 percent of billed input came from the cache. A much lower share points at something that changes the start of the prompt between turns, like a timestamp in a system prompt or a tool set that shifts.

Where a graph does pay

A second benchmark ran in GitHub Copilot CLI on two much bigger repositories, kubernetes (146k nodes) and vscode (219k). There the graph cut Haiku's priced tokens on structural questions to 0.29 of grep's, and it was cheaper on 15 of those 16 questions. Sonnet's went to 0.74 and Opus 5.5's to 0.97. The details are in Where a code graph pays off. In Claude Code I haven't tried repositories that large, or Opus.

What these runs can't tell me

Every run in this post was a single question in a fresh session. In a long interactive session two things change, and they pull in opposite directions: each extra turn sends a much longer history again, and most of that history is read from the cache at a tenth of the price. So in a long session, does a tight cap on tool output cost more than the 2.65 times I measured here, or less?

Chaining image-to-video clips from the last frame: what 10 links did to a face (MiniMax-H3, measured)

Rodion Kazennov — Thu, 29 Oct 2026 05:07:26 +0000

For a dialogue scene I generated clips with MiniMax-H3, MiniMax's open-weights video model that can start from a given first frame, end on a given last frame, or both. To make one shot run longer than one clip, I started every clip from the last frame of the clip before. After ten clips the woman on screen no longer looked like the woman I had drawn:

Then I measured what each link did. People on forums describe this drift a lot (LTX-2.3, Wan, Seedance), but I didn't find numbers per link, so here are mine.

The setup

Model: MiniMax-H3, the first/last-frame checkpoint (FL2VA), run locally on an Apple Silicon Mac through a fork of h3.c, with an INT8 copy of the transformer and lightx2v's 8-step Turbo LoRA (fl2v_turbo_8step_v1.0_768p).
Every clip: 768x1344, 124 frames (5.17 s at 24 fps), 8 steps, seed 42.
Conditioning: a first frame only, no end frame. Clip 1 starts from a keyframe I drew. Every later clip starts from the previous clip's last frame:

ffmpeg -sseof -0.05 -i clip_n.mp4 -frames:v 1 -update 1 clip_n_last.png
# clip_n_last.png becomes the first frame of clip n+1

Three chains: A, 10 clips of a woman in an armchair; B, 7 clips of a man by a door; C, 9 clips of a woman with a phone. Each prompt gave the person on screen one line of dialogue (the model voices it too), and every prompt said "The camera holds a static shot" and "natural skin texture".

What I measured

On the first frame of each clip, which is the last frame of the clip before, I computed four numbers on the whole frame and divided each by its value on the drawn keyframe:

g = np.asarray(Image.open(path).convert("L"), np.float64)

lap = -4*g[1:-1,1:-1] + g[:-2,1:-1] + g[2:,1:-1] + g[1:-1,:-2] + g[1:-1,2:]
edge_sharpness = lap.var()                      # Laplacian variance

F = np.abs(np.fft.rfft2(g - g.mean()))**2
r = np.hypot(np.fft.rfftfreq(g.shape[1])[None, :], np.fft.fftfreq(g.shape[0])[:, None])
finest_detail = F[r > 0.25].sum() / F[r > 0.01].sum()   # energy above 0.25 cycles/px

contrast = (g / 255).std()
near_white = (g / 255 > 0.92).mean()            # share of pixels

Detail goes first

Finest detail dropped on the first link: 77 % of the keyframe in chain A and 80 % in chain B at the start of clip 2. It bottomed out at clip 4 (61 % and 65 %) and never returned to the keyframe's level: chain A ended at 65 %, chain B at 88 %.

Edge sharpness fell too at first, to 73 % and 82 % at clip 3, and then it overshot. From clip 5 on it was above the keyframe, and it reached 174 % in chain A (clip 9) and 159 % in chain B (clip 7). Detail wasn't coming back: the finest band stayed low. What rose were harder, coarser edges. On the shirts you can see it as a crunchy pattern that the keyframe didn't have.

Then the faces turn oily

Contrast rose with almost every link: by 14 % in chain A (clip 9) and 16 % in chain B (clip 7). Near-white pixels grew faster: from 0.20 % to 0.55 % of the frame in chain A, 2.8 times, and from 0.054 % to 0.124 % in chain B, 2.3 times. On a face this shows up as shine on the forehead, nose and cheeks. Up close:

The freckles from the drawing are gone by clip 4, the skin is glossy although every prompt asked for natural skin texture, and by clip 9 the bone structure has changed. I didn't score identity with a face-embedding model, so that last part is by eye.

One oddity: the first frame of chain A's clip 10 is brighter overall (mean luma up 10 %, saturation down 15 %), which pulls its contrast and near-white numbers back. I haven't looked into why.

Mistakes ride along

Chain C shows a different failure. At the end of clip 2 the model drew a chat bubble with made-up English over the frame, and since clip 3 starts from that frame, the bubble was in the scene from then on:

It stayed through the next seven clips, with the text inside it changing from clip to clip. The model treats whatever is in the conditioning frame as part of the scene. A last-frame chain has no way to drop it.

The rule I use now

When I watched the long chain, I saw smeared text on the phone and faces that looked oiled. Since then I work like this:

At most 3 clips in a row start from a previous clip's last frame. Then a fresh drawn keyframe.
If a new person or object enters, the clip still starts from the last frame, but it ends on a freshly drawn end keyframe: an edit of the start frame with the new detail painted in.
A face close-up or a new location always starts from a freshly drawn keyframe.

The cap of 3 is a judgment call, not a threshold the numbers found. In these two chains, the first three clips lose fine detail (to 66 % and 69 %), while contrast has barely moved (100 % and 106 %) and near-white pixels are up 16 % and 11 %. From clip 5 on, all four metrics move away from the keyframe together. You could argue for 2.

Check your own chain

The measurement needs only the PNGs you already feed the model. Compute the four numbers above for every first frame and divide by the keyframe. A falling finest-detail share with rising contrast is the pattern to watch for. Before you trust a long chain, look at the first, middle and last frame of every clip side by side. A contact sheet per chain would have caught the chat bubble at clip 3. I found it much later, in a rough cut.

These are two clean chains with one model, one seed and one resolution, and the metrics cover the whole frame while people move inside it. If you run the same check on another model, I'd like to compare curves.

Real-CUGAN on Linux vs Windows: same flags, two NVIDIA GPUs, SSIM 0.9977

Rodion Kazennov — Thu, 29 Oct 2026 05:05:53 +0000

I upscale short AI-generated clips to 4K with Real-CUGAN, the ncnn-vulkan build. Two laptops with NVIDIA GPUs can do the work:

	Windows laptop	Linux laptop
CPU, RAM	i7-13700H, 32 GB	i5-9300HF, 23 GB
GPU	RTX 4060 Laptop, 8 GB	RTX 2060, 6 GB
OS	Windows 11 (build 22631)	Arch Linux, kernel 7.2.9
NVIDIA driver	610.88	615.78.08
ffmpeg	8.1.3	9.0.2
realcugan-ncnn-vulkan	20220728 release	20220728 release

Before I split one video's clips between them, I wanted to know whether they give the same picture. If they didn't, a cut between two clips upscaled on different machines could show it.

The test

One clip, 768x1344 at 24 fps, 104 frames, went through the same steps on both machines: crop to an exact 9:16, write PNG frames, Real-CUGAN at 3x with the models-se weights and denoise level 0, Lanczos down to 2160x3840, then lossless x264. The Linux side:

ffmpeg -i "$src" -vf "crop=$cw:$h" -fps_mode passthrough in/f%06d.png
realcugan-ncnn-vulkan -i in -o up -s 3 -n 0 -g 0 -m models-se -f png
ffmpeg -framerate 24 -i up/f%06d.png -i "$src" -map 0:v -map '1:a?' \
  -vf scale=2160:3840:flags=lanczos -c:v libx264 -qp 0 -preset veryfast \
  -pix_fmt yuv420p -c:a copy -frames:v 104 out_4k.mp4

Windows runs the same three commands from Python, with one difference: -g 1. On that laptop GPU 0 is the integrated Intel chip, and -g 0 would run the upscale there.

The result

Both files have 104 frames. I compared them with ffmpeg's own filters:

ffmpeg -i windows_4k.mp4 -i linux_4k.mp4 -lavfi \
  "[0:v]split[a0][a1];[1:v]split[b0][b1];[a0][b0]ssim=stats_file=ssim.log;[a1][b1]psnr=stats_file=psnr.log" \
  -f null -

SSIM came out at 0.9977 (Y 0.9976, U 0.9978, V 0.9980) and PSNR at 54.2 dB. Frame by frame the range was narrow: SSIM 0.99758 to 0.99779, PSNR 54.00 to 54.45 dB. Nothing drifts over the clip.

A single score hides how the differences are spread, so I also counted the absolute difference of every Y, U and V sample, 1.29 billion of them:

75.5 % of the samples are identical, and 99.97 % are within one level out of 255. The largest difference anywhere is 5 levels, and only 5 samples reach it. The mean absolute difference is 0.25 of a level. Side by side at 100 % zoom I couldn't tell the two apart.

The core of the counting script, in case you want to run it on your own pair:

p = lambda f: subprocess.Popen(["ffmpeg", "-v", "error", "-i", f, "-f", "rawvideo",
                                "-pix_fmt", "yuv420p", "-"], stdout=subprocess.PIPE)
a, b = p(win), p(lin)
size = 2160 * 3840 * 3 // 2          # one yuv420p frame
hist = np.zeros(256, np.int64)
while (x := a.stdout.read(size)) and (y := b.stdout.read(size)):
    d = np.abs(np.frombuffer(x, np.uint8).astype(np.int16) - np.frombuffer(y, np.uint8))
    hist += np.bincount(d, minlength=256)

Where the difference comes from: not separated

The two sides differ in GPU, driver, operating system and ffmpeg version, all at once. The candidates are the floating-point math in the Vulkan shaders, on two GPU generations and two drivers, and ffmpeg's crop, Lanczos scaling and RGB to YUV conversion in two versions. The x264 step at -qp 0 is lossless; the only loss in that step is the conversion to yuv420p, which belongs to the ffmpeg candidate.

To split it, compare the PNGs that Real-CUGAN writes before the encode step. My pipeline deletes them after each clip, so I don't have that answer yet.

Speed, from the job log

I run these jobs with stageload, a small open-source tool that runs heavy jobs one at a time and logs each one's run time. Its log gives whole-job numbers: copying the clip over the LAN, extracting frames, upscaling, encoding and copying the result back. Divided by frame count:

Job	Machine	Frames	Run time	Per frame
Real-CUGAN 3x, 4K lossless out	RTX 4060, Windows	506 (one job, 5 clips)	515.5 s	1.02 s
Real-CUGAN 3x, 4K lossless out	RTX 2060, Linux	104 (one clip)	232.7 s	2.24 s
Real-ESRGAN x4plus, 4K lossless out	RTX 4060, Windows	73 each, 11 jobs	383 to 422 s, mean 401.5 s	5.50 s
Real-ESRGAN x4plus, 1080p out	RTX 2060, Linux	73 each, 3 jobs	542 to 558 s, mean 548.8 s	7.52 s

Read these with care. The Real-CUGAN rows are one job each, and the Linux job also ran a 4K lossless x264 encode on a CPU with 8 threads, limited to 4 threads at low priority. The Real-ESRGAN rows come closer to a GPU comparison, though not all the way, because the Linux jobs encoded to 1080p, the lighter encode. There the RTX 2060 laptop took 1.37 times as long. One two-clip Windows job took 3,374 s, far above the rest, and I left it out because I don't know why.

On the same Windows laptop, Real-ESRGAN x4plus (4x) cost about 5.4 times as much per frame as Real-CUGAN at 3x. I still use it for photoreal footage: Real-CUGAN is trained on anime, and on realistic faces it turned skin into painted strokes.

What I do now

Before I mix machines in one video, I upscale one clip on both and run the two checks above. If the frame counts match and the differences look like this histogram, the clips go to whichever laptop is free. The SSIM and PSNR pass took 11 seconds on my Mac for 104 frames at 4K. It catches setting mistakes, such as the wrong -g or a different denoise level, before they reach an edit.

"Resource provisioning failed" on Vercel previews: the Neon 10-branch limit, measured

Rodion Kazennov — Thu, 29 Oct 2026 05:05:46 +0000

My Next.js app runs on Vercel with a Neon Postgres database connected through the Vercel-managed Neon integration. In late October every preview deployment started to fail about a second after it was created. There was no build log at all, and production deployments kept working. Vercel showed one line:

BUILD_FAILED
Resource provisioning failed

The cause was the Neon free plan's limit of 10 branches per project. Below is how I confirmed it, how fast the project filled up again after I emptied it, and what to do before you delete anything.

Where Neon's reason shows up

With this integration, each preview deployment asks Neon for a database branch named preview/<git-branch>, a copy-on-write copy of the production branch. That request runs as an integration step before the build starts (Neon docs). The production branch counts toward the 10, so nine previews fill the project. When the project is full, Neon refuses the new branch and Vercel reports the whole step as "Resource provisioning failed".

The first time I hit this, in September, I found Neon's own reason in the deployment's integration actions, read through the Vercel REST API (GET /v1/deployments/<deployment-id>/integrations). The status text there said Branch limit reached. Counting the branches is faster:

npx neonctl branches list --project-id "$NEON_PROJECT" --output json | jq 'length'

# who made them and when
npx neonctl branches list --project-id "$NEON_PROJECT" --output json \
  | jq -r '.[] | [.created_at, .creation_source, .name] | @tsv'

If the first command prints 10 and most rows say vercel and preview/..., that's your answer. Other people on the Vercel community forum traced the same error to the same limit. Some threads report the error with free slots left, so check the count first. Don't assume it.

How fast ten branches fill

My repository gets a lot of short pull requests. In October, up to the 27th, 484 of the project's 851 Vercel builds were previews. Every new git branch that got a preview deployment also got a Neon branch.

On 28 October I emptied the project at about 05:02 UTC, leaving only production. This is what happened next:

Nine new preview branches appeared between 05:03 and 14:11 UTC, all with creation_source: vercel. The project was full again 9 hours 8 minutes after I emptied it. I deleted nine more at 18:29 to 18:31 UTC, and by 01:45 UTC the next morning five new ones had arrived.

The same cleanup came up every day that week. These are the preview branches I deleted by hand:

Day (UTC)	Preview branches deleted
24 October	9
25 October	6
26 October	3
27 October	5
28 October	9 in the morning, 9 in the evening

That's 41 branches in five days. The ones I looked at in the listing had each kept a compute endpoint active for about 300 seconds (active_time_seconds).

Merging the pull request doesn't free the branch

Every branch I deleted belonged to a pull request that was already merged, and its git branch was usually gone. With the Vercel-managed integration, Neon deletes a preview branch when the branch's Vercel deployments are deleted, and Vercel keeps preview deployments for 180 days by default (Neon docs). A merge doesn't delete deployments, so with that default retention the database branch stays for months.

Skipping the build doesn't skip the branch

My first fix was to stop building previews with Vercel's ignored build step in apps/web/vercel.json. The first version also cancelled a push to main, because VERCEL_ENV didn't read as production in that step. The second version tests the branch name:

// first try: also skipped production
"ignoreCommand": "[ \"$VERCEL_ENV\" != production ]"

// second try: build main, and any deploy whose branch is unknown
"ignoreCommand": "[ -n \"$VERCEL_GIT_COMMIT_REF\" ] && [ \"$VERCEL_GIT_COMMIT_REF\" != main ]"

The builds were skipped, but the database branches kept coming. All 14 branches in the chart were created after this step was in place. The integration step runs before the ignored build step, so a skipped preview still provisions its branch, and when the project is full it fails the same way. The ignored build step saves build minutes. It does nothing for the branch count.

Back up before you delete

A preview branch may hold data that a preview wrote, so I dumped each one before deleting it. One catch: pg_dump refuses to dump a server with a newer major version than its own (the error and the rule). My Neon project runs Postgres 17 and my Mac had pg_dump 16.15, so I installed the 17 client first. A loop like this one does the rest:

brew install postgresql@17
PG_DUMP="$(brew --prefix postgresql@17)/bin/pg_dump"   # 17.11 here

npx neonctl branches list --project-id "$NEON_PROJECT" --output json \
  | jq -r '.[] | select(.name | startswith("preview/")) | "\(.id) \(.name)"' \
  | while read -r id name; do
      uri=$(npx neonctl connection-string "$id" --project-id "$NEON_PROJECT")
      "$PG_DUMP" -Fc -f "backup/${name//\//__}.dump" "$uri"
    done

The preview branches I listed showed about 52 MB of logical size each, the same as production, since it starts as a copy. The nine dumps came to 27.7 MB in all, between 3.03 and 3.08 MB each, and finished 41 to 48 seconds apart. Only after that did I delete the branches, by id:

npx neonctl branches delete "$id" --project-id "$NEON_PROJECT"

The same limit breaks your CI

Before a schema migration merges, I rehearse it on a throwaway branch copied from production, then delete the branch. Reduced to its shape:

const created = neon(['branches', 'create', '--parent', MAIN, '--name', name]);
const id = JSON.parse(created.stdout).branch.id;
try {
  // prisma migrate deploy against the copy, then compare the schema
} finally {
  neon(['branches', 'delete', id]);   // also when a step fails
}

The finally keeps the rehearsal from leaking branches. It can't help when previews have taken every slot: on several days in the table the rehearsal failed at branches create until I cleared space by hand.

What stops the refill

Deleting branches only buys hours. The setting that stops the refill is in the integration: Neon's docs describe a "Create database branch for deployment" choice with Preview and Production boxes when you connect the project. With Preview off, a preview deployment gets no branch of its own, so check which DATABASE_URL your Preview environment uses then. It must not be production. Two other levers: shorten Vercel's retention for preview deployments, so the branches go with them, or delete the branch from CI when the pull request closes.

I haven't measured the fix yet. At 01:45 UTC on 29 October preview branching was still on and the project had 6 of 10 branches. My next step is to switch it off and check that the count stays at 1.

Checklist

1. neonctl branches list | jq length     -> 10 means the limit, not your code
2. pg_dump at the server's major version  -> back up the preview branches
3. delete by id, previews of merged PRs only
4. turn off "Create database branch for deployment" for Preview,
   or shorten preview retention, or delete branches on PR close
5. leave slots for CI throwaway branches

MPS backend out of memory: the "max allowed" number on every Mac size

Rodion Kazennov — Sat, 24 Oct 2026 20:02:36 +0000

The PyTorch error on a Mac carries one number that people search for. This one is from a 16 GB Mac (AudioLDM2 #20):

RuntimeError: MPS backend out of memory (MPS allocated: 18.02 GB, other allocations: 107.10 MB, max allowed: 18.13 GB). Tried to allocate 4.50 MB on private pool.

In my earlier note I measured where it comes from on my 48 GB Mac. "max allowed" is PYTORCH_MPS_HIGH_WATERMARK_RATIO, 1.7 by default, times the GPU working set macOS recommends, which torch.mps.recommended_max_memory() returns. One machine can't tell you what that number is on the others, so I read it off other people's errors.

Where the numbers come from

On 24 October I searched GitHub for issues that quote the error with its "max allowed" value. That gave 250 issues and pull requests, 167 of them with a number in the text. Most values fall into a few groups, and where a reporter named their Mac's memory, the groups line up with the size. Divided by 1.7, each value gives the working set macOS gave the GPU on that machine.

Mac memory	max allowed, GiB	GPU working set, GiB	share of memory	reports	seen
8 GB	9.07	5.33	67 %	27	2023 – 2026
16 GB	18.13	10.67	67 %	18	2023 – 2026
24 GB	27.20	16.00	67 %	9	Jan 2025 – Jan 2026
24 GB	30.19	17.76	74 %	3	Dec 2025 – Sep 2026
32 GB	36.27	21.33	67 %	6	2024 – 2025
36 GB	45.90	27.00	75 %	3	2024 – 2026
48 GB	63.65	37.44	78 %	4, plus mine	Sep – Oct 2026
64 GB	81.60	48.00	75 %	7	Dec 2023 – Apr 2025
64 GB	88.13	51.84	81 %	6	Mar – Oct 2026
128 GB	163.20	96.00	75 %	4	2024 – 2025
128 GB	182.78	107.52	84 %	2	Dec 2025 – Jan 2026

Older PyTorch versions label these numbers "GB" and newer ones "GiB". Both are the binary unit: 1.7 × two thirds × 16 GiB is exactly 18.13.

What changed

Until late 2025 the rule looks simple. macOS gave the GPU two thirds of memory on Macs up to 32 GB and three quarters above that. Reports from late 2025 on show larger shares on the 24 GB, 64 GB and 128 GB Macs. The three 64 GB reports that name their system all say macOS 26. The 8 and 16 GB values haven't moved. My own 48 GB Mac runs macOS 26.5.2 and returns 37.44 GiB, 78 %.

Two limits on what this shows: the newer rows rest on two to six reports each, and I can't tell from them whether macOS or the chip made the difference.

What it means when you see the error

At the default ratio the limit is above physical memory on every row of the table: 1.7 × two thirds is 113 % of memory, and 1.7 × 84 % is 143 %. By the time a job reaches "max allowed", it has been swapping for a while. That matches my own measurement: on my Mac, swap began between 33.5 and 36 GB of footprint, far below the 63.65 GiB limit. The advice at the end of the error, a ratio of 0.0, removes the limit, so the job can swap further.

If you'd rather get a clean error than a swapping Mac, set the limit below your memory, for example a ratio of 1.0, which is the working set itself. Set the low ratio with it: on PyTorch 2.14, a high ratio below 1.4 set on its own fails with "invalid low watermark ratio 1.4".

PYTORCH_MPS_HIGH_WATERMARK_RATIO=1.0 PYTORCH_MPS_LOW_WATERMARK_RATIO=0.8 python your_script.py

To see your own Mac's working set:

import torch
print(torch.mps.recommended_max_memory() / 2**30, "GiB")

The numbers that don't fit

About a third of the reports show other limits, such as 1.70 GiB, 3.40 GiB, 6.77 GiB or 3.00 GiB. Those aren't the Mac's default. An app or a user set its own ratio or cap: one video tool caps MPS at 3 GiB even on a 128 GB Mac. Another 20 reports sit at 7.93, and I couldn't tie that value to one Mac size. If your number is far below the table, look in the tool's settings before you blame the Mac.

Sources

Two reports for each row. The full list of 167 is in the data file.

8 GB, 9.07: noScribe #84, pytorch #105839
16 GB, 18.13: pinokio #941, AudioLDM2 #20
24 GB, 27.20: pytorch #145374, qwen3-optimizer-study #13
24 GB, 30.19: ml-bokeh #3, ComfyUI #17264 (ratio 1.0, so 17.76)
32 GB, 36.27: ComfyUI-LivePortraitKJ #82, instructlab #2568
36 GB, 45.90: ACE-Step-1.5 #282
48 GB, 63.65: trellis2-mlx #1, ask-physics #24
64 GB, 81.60: bobo-fun #8, ComfyUI-CogVideoXWrapper #312
64 GB, 88.13: EtherReel-releases #7, ComfyUI #17309
128 GB, 163.20: ComfyUI #9308, lm-evaluation-harness #1835
128 GB, 182.78: ComfyUI #11584, ComfyUI-WanVideoWrapper #1856

MPS backend out of memory on a Mac: what the numbers mean and how to set the watermark

Rodion Kazennov — Wed, 14 Oct 2026 00:01:59 +0000

When a PyTorch job runs out of memory on a Mac, the error ends with a piece of advice:

RuntimeError: MPS backend out of memory (MPS allocated: 18.50 GiB, other allocations: 448.00 KiB, max allowed: 18.72 GiB). Tried to allocate 512.00 MiB on shared pool. Use PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure).

I wanted to know what following that advice does, so I filled MPS memory in 512 MiB steps on my 48 GB M5 Pro (macOS 26.5.2, PyTorch 2.14.0), once per setting, under a guard that stops the process when swap grows by 2 GB. Two things came out of it. On this Mac the default limit is higher than the RAM, so a job that outgrows memory slides into swap and never gets to this error. And lowering the limit takes two variables, not one.

Where "max allowed" comes from

max allowed is the limit: PYTORCH_MPS_HIGH_WATERMARK_RATIO times what torch.mps.recommended_max_memory() returns. On my Mac that is 37.44 GiB, and 0.5 × 37.44 = 18.72 GiB, the number in the error above. MPS allocated matched torch.mps.current_allocated_memory() in my log, the memory held by tensors. To see your own base number:

import torch
print(torch.mps.recommended_max_memory() / 2**30, "GiB")

Why a 48 GB Mac swaps first

PyTorch's documentation gives the default high ratio as 1.7. On my Mac that puts the limit at about 63.6 GiB, more memory than the machine has. This is what each setting did:

setting	what stopped the fill	tensors held	swap during the run
default (1.7)	the guard	38.5 GiB	6.1 → 12.0 GB
0.0	the guard	36.0 GiB	5.1 → 9.8 GB
1.0, low 1.0	the guard	34.5 GiB	4.6 → 8.7 GB
0.8, low 0.8	PyTorch, max allowed 29.95 GiB	29.5 GiB	no change
0.5, low 0.5	PyTorch, max allowed 18.72 GiB	18.5 GiB	no change

With the default, with 0.0 and with 1.0, swap started to grow once the process passed 33.5 to 36.2 GB, and PyTorch never raised: its limit was further away than the end of the RAM. So 0.0, the advice inside the error, changed nothing I could see here. The limit was already out of reach, and nothing in PyTorch stopped the run.

Setting it: two variables

Setting only the high ratio fails at the first MPS call:

$ PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.8 python fill.py
RuntimeError: invalid low watermark ratio 1.4

The low ratio defaults to 1.4 on Apple Silicon and may not be above the high one (unless the high one is 0.0, which turns the limit off). Set both:

PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.8 PYTORCH_MPS_LOW_WATERMARK_RATIO=0.8 python run.py

From Python, set them before the first tensor goes to mps. PyTorch reads them when MPS starts, not at import torch, and ignores changes made after that:

import os
os.environ["PYTORCH_MPS_HIGH_WATERMARK_RATIO"] = "0.8"
os.environ["PYTORCH_MPS_LOW_WATERMARK_RATIO"] = "0.8"
import torch

The documentation calls the low ratio a soft limit and says that "setting 0.0 disables adaptive commit and garbage collection". I set it equal to the high one and did not measure it on its own.

Picking a value

The limit should stay below the memory your Mac really has free, with your usual apps open. On mine 1.0 (37.44 GiB) was already too much, and 0.8 (29.95 GiB) gave a clean error with no swap. After a clean error you can lower the batch size or the resolution and try again; a run sliding into swap slows down the whole Mac first.

A trap when you test this

My first run filled the tensors with torch.ones. With the default setting it reached the 46 GiB cap I had set, with no error and no swap at all: macOS compresses pages of identical values almost to nothing. Model weights don't compress like that, so a memory test on a Mac needs random data.

When the model really doesn't fit

A clean error only tells you the model doesn't fit. For a pipeline of several models, loading only the models of the current stage does more than any ratio, which is why I wrote stageload. The fill script and the traces of every run above are in its repository, in bench/results/2026-10-13-mps-watermark.

PyTorch on a Mac: why memory looks freed after .to("cpu")

Rodion Kazennov — Tue, 13 Oct 2026 08:43:56 +0000

On a Mac, a tensor moved from mps to the CPU seems to disappear from memory. The process footprint, the number Activity Monitor shows in its Memory column, drops to almost nothing, and so does the MPS driver count. The copy is still there and still takes the same memory. This is the minimal example I put together for a diffusers discussion about offloading, with a 4 GiB tensor standing in for a model's weights (torch 2.14.0, macOS 26.5.2, M5 Pro with 48 GB):

import ctypes, os, time
import torch

# rusage_info_v2 from <libproc.h>; phys_footprint is the Memory column of Activity Monitor
FIELDS = ("user_time", "system_time", "pkg_idle_wkups", "interrupt_wkups", "pageins",
          "wired_size", "resident_size", "phys_footprint")
class RUsageInfoV2(ctypes.Structure):
    _fields_ = [("uuid", ctypes.c_uint8 * 16)] + [(f, ctypes.c_uint64) for f in FIELDS] \
        + [(f"rest{i}", ctypes.c_uint64) for i in range(10)]

libproc = ctypes.CDLL("/usr/lib/libproc.dylib")

def show(step):
    torch.mps.synchronize()
    info = RUsageInfoV2()
    libproc.proc_pid_rusage(os.getpid(), 2, ctypes.byref(info))
    print(f"{step:<14} footprint {info.phys_footprint / 2**30:4.1f} GiB, "
          f"mps driver {torch.mps.driver_allocated_memory() / 2**30:4.1f} GiB")

show("start")
w = torch.randn(2**30, device="mps")      # 4 GiB of float32, standing in for a model's weights
show("on mps")
w = w.to("cpu")                           # what model CPU offload does when a model's turn ends
torch.mps.empty_cache(); time.sleep(2)
show("moved to cpu")
float(w.sum())                            # read every page of the cpu copy
show("cpu copy read")

start          footprint  0.1 GiB, mps driver  0.0 GiB
on mps         footprint  4.2 GiB, mps driver  4.0 GiB
moved to cpu   footprint  0.1 GiB, mps driver  0.0 GiB
cpu copy read  footprint  4.2 GiB, mps driver  0.0 GiB

I ran it three times and got the same output each time.

What the numbers say

Right after the move, both counters are back near zero, but w holds all 4 GiB of its values. Once the CPU reads every page of the copy, the footprint is 4.2 GiB again. So a footprint taken between .to("cpu") and the next time the CPU touches those weights does not show them.

The reason the move frees nothing in the first place is the hardware. On Apple Silicon the CPU and the GPU use one pool of physical memory. On a PC, moving weights from the GPU to the CPU frees VRAM for the next model. On a Mac, the weights "on the CPU" sit in the same RAM the GPU needs.

Why it matters for model CPU offload

diffusers' enable_model_cpu_offload() keeps each model on the CPU and moves it to the GPU only while it runs. I ran Qwen-Image-2.1 at 1024 × 1024 that way on the same Mac (trace and write-up). The process footprint never went above 18.5 GB, which would fit on a 48 GB Mac with room to spare. Swap grew by 3.5 GB during denoising and had grown by 9.6 GB by the VAE decode, and my memory guard stopped the run before it wrote an image.

A chart of the footprint alone would have called that run a fit. The example above shows one way the footprint can stay low while the weights are still in memory. I didn't measure which pages went to swap in the offload run.

How to check your own run

Look at swap, not only at the footprint:

sysctl vm.swapusage             # compare "used" before and after the run
sysctl kern.memorystatus_level  # system-wide free memory, in percent

If swap grows while the job runs, it doesn't fit, whatever the footprint says. stageload (pip install stageload), a small library I wrote for this, records both in a trace every 0.5 s with MemoryMeter, and stageload guard stops a command once swap grows past a budget:

stageload guard --wait-free 40 --swap-budget 8G -- python generate.py

What does free the memory

For a pipeline that runs its models one after another, the fix is to load a model when its stage starts and release it when the stage ends, instead of parking it on the CPU. stageload's release(module) moves every parameter and buffer to PyTorch's meta device, collects garbage and empties the MPS cache; a tensor on meta has no storage, so the memory goes back even while other code still holds a reference to the module. With the text encoder (16.3 GB) and the transformer (13.3 GB) loaded one at a time this way, Qwen-Image-2.1 peaked at 19.0 GB on the same Mac, and swap did not grow in either of two runs. The full setup is in How to run Qwen-Image-2.1 on a 48 GB Mac without swap.

What I did not measure

The free memory of the whole system during the example: other jobs on this Mac move it by several GiB between runs, so I couldn't get a clean number.
Whether del plus torch.mps.empty_cache() gives the memory back as fully as moving to meta, and how quickly.
Other macOS and torch versions, and Macs with more memory.

Godot Movie Maker and screenshots on macOS without --headless

Rodion Kazennov — Mon, 12 Oct 2026 21:22:55 +0000

I wanted Godot to write screenshots and short videos from a script on my Mac, the way it does in CI on Linux. Godot 4.7.1 on macOS 26.5.2 can't do it in --headless, but it can with the real renderer and the window off the screen. One thing I haven't solved: the app still comes to the front while it runs.

Everything below ran on a 640 × 360 test project with one script:

extends Node2D

# Saves the 30th frame to the path given after "--" and quits.
var frames := 0

func _process(_delta: float) -> void:
    $Box.position.x = 80.0 + frames * 4.0
    frames += 1
    var args := OS.get_cmdline_user_args()
    if frames == 30 and args.size() > 0:
        await RenderingServer.frame_post_draw
        get_viewport().get_texture().get_image().save_png(args[0])
        get_tree().quit()

What --headless does with a capture

Godot.app/Contents/MacOS/Godot --headless --path . -- "$PWD/out.png"

The script reached frame 30 and then waited at await RenderingServer.frame_post_draw for good: the process was still running 15 seconds later, and nothing was saved. Reading the texture without the await fails at once:

ERROR: Parameter "t" is null.
   at: texture_2d_get (./servers/rendering/dummy/storage/texture_storage.h:110)

In --headless Godot runs a dummy renderer, so the viewport has no texture to read; the path in the error is the dummy renderer's texture storage. On Linux the usual way round is a virtual X display under a real renderer. Godot's macOS build draws through the macOS window server and Metal, so that trick doesn't carry over.

Render for real, off the screen

Keep the real renderer and keep the window out of the way. Window settings take effect when the window is created, before any script runs, so they go in an override.cfg next to project.godot for the length of the run:

[display]

window/size/no_focus=true
window/size/initial_position_type=0
window/size/initial_position=Vector2i(-6000, -6000)
window/size/always_on_top=false

Then start the app through open:

open -n -W -g -j -a Godot.app --stdout run.log --stderr run.log \
  --args --path "$PWD" -- "$PWD/out.png"
rm override.cfg

-n starts a new instance even if Godot is already open, -W waits until it quits, -g keeps it in the background, -j launches it hidden, and --stdout and --stderr bring its output back to a file. Without them open drops the output, because Godot isn't a child process of your shell. The frame came back at 640 × 360 in the scene's own colours: a mean RGB of 188, 117, 82, the orange background with the blue box on it.

Delete override.cfg when the run ends. Godot reads it on every launch, so a run interrupted before the rm leaves the next normal launch with a window that never gets keyboard focus.

Movie Maker: a PNG sequence, then an MP4

Movie Maker renders every frame at a fixed rate, so the video's timing doesn't depend on how fast the machine renders:

open -n -W -g -j -a Godot.app --stdout run.log --stderr run.log \
  --args --path "$PWD" --write-movie "$PWD/movie/f.png" --fixed-fps 24 --quit-after 48

48 frames at 24 FPS (movie length: 00:00:02:00), recorded in 00:00:01 (200% of real-time speed).

It wrote f00000000.png to f00000047.png, with the audio next to them as f.wav. ffmpeg turns the frames into a video:

ffmpeg -framerate 24 -i movie/f%08d.png -c:v libx264 -pix_fmt yuv420p -crf 18 film.mp4

The result was 48 frames at 24 fps, 2.0 seconds long.

Not solved: focus

no_focus keeps the window from taking keyboard focus, but Godot still makes itself the active app when it opens its window. I polled the frontmost app during two runs. Godot was in front in 1 of 21 samples during the single-frame run and in 4 samples in a row during the Movie Maker run, which took about a second, and the app I was working in came back to the front each time Godot quit. In September a copy of the app bundle with LSUIElement set to true didn't help either. How long Godot stays in front during a longer recording, I haven't measured.

Limits

Godot 4.7.1 on macOS 26.5.2, one Mac, the Forward+ renderer. I didn't try the Compatibility renderer.
The off-screen position was tested with one display.

ulimit doesn't cap memory on macOS: stop a job before it swaps

Rodion Kazennov — Mon, 12 Oct 2026 14:31:35 +0000

On Linux a job's memory can be capped with ulimit -v, systemd-run -p MemoryMax= or a cgroup, and a job that grows too big fails on its own instead of taking the machine down. macOS has no systemd and no cgroups, and ulimit refuses. This is macOS 26.5.2:

$ /bin/bash -c 'ulimit -v 1048576'
/bin/bash: line 0: ulimit: virtual memory: cannot modify limit: Invalid argument
$ /bin/bash -c 'ulimit -m 1048576'
/bin/bash: line 0: ulimit: max memory size: cannot modify limit: Invalid argument

resource.setrlimit(resource.RLIMIT_DATA, ...) in Python raises ValueError, and in the shell where ulimit -m had failed, a script went on to allocate and touch 3 GiB. What macOS does instead is compress memory and write it to swap. A model that doesn't fit doesn't fail: it slows down, and so does everything else on the Mac.

So the questions worth asking change: is there room before a job starts, and is swap growing while it runs?

Check swap while a job runs

sysctl vm.swapusage             # compare "used" before and after a run
sysctl kern.memorystatus_level  # system-wide free memory, in percent

The process footprint, the number Activity Monitor shows, doesn't count pages that are already in swap. In one run of Qwen-Image-2.1 with diffusers' model CPU offload on my 48 GB M5 Pro, the footprint never went above 18.5 GB while swap grew by 9.6 GB. A chart of the footprint alone would have called that run a fit.

Stop a job before it swaps

stageload, a small library I wrote for loading models one stage at a time, ships a guard for this (pip install stageload):

stageload guard --wait-free 40 --swap-budget 8G -- python generate.py

It waits until kern.memorystatus_level is at least 40 %, then starts the command. If swap grows more than 8 GB over what was in use at the start, or free memory stays under 10 % for two samples, it sends SIGTERM to the command's process group and, five seconds later, SIGKILL to whatever is left. It needs no root, and the command and its process group are the only processes it ever stops. Exit code 3 means the guard stopped the command and 4 that it never found room to start it; otherwise you get the command's own exit code.

That is how the eager Qwen-Image run ended: it got through all 20 steps, the VAE decode took the footprint to 43.6 GB, swap grew by 8 GB, and the guard stopped it before the Mac stalled.

Two heavy jobs at once

A guard decides alone. Two guards that find room at the same moment both start their jobs, and each job was sized for a Mac with nothing else running. On 5 October one of my runs started in the gap between two video jobs of another script, and the next video job took swap past its budget within 20 seconds.

stageload run puts the jobs in one queue for the machine. It takes the guard's flags. Here two jobs hold 3 and 2 GiB, started from two terminals:

stageload run --who images -- python3 alloc.py 3 10    # holds 3 GiB for 10 s
stageload run --who upscale -- python3 alloc.py 2 4    # holds 2 GiB for 4 s

$ stageload queue
ram     0 running  images               pid 32488   since 2026-10-12T14:08:56+00:00  python3 alloc.py 3 10
ram     1 waiting  upscale              pid 32499   since 2026-10-12T14:08:58+00:00  python3 alloc.py 2 4

The second job starts when the first one ends, and then the guard's checks run as usual:

14:08:58Z queue: waiting in ram (1 ahead: images)
14:09:08Z queue: start in ram after 10 s, swap 6.0 GB, stop above 14.0 GB or under 10% available twice

$ stageload queue --history 2
done   ram    exit 0    images               waited 0 s, ran 10 s, peak 3.0 GB  python3 alloc.py 3 10
done   ram    exit 0    upscale              waited 10 s, ran 4 s, peak 2.0 GB  python3 alloc.py 2 4

Details that mattered on a shared Mac:

Jobs join a lane, ram by default. Lanes don't wait for each other, so a job that waits on a remote API quota can go in a lane of its own (--lane api --wait-free 0) and not hold up the ones that need memory.
Turns are fair between owners. The owner who started least recently goes next, so a script that queues twenty jobs takes turns with the others instead of holding the lane. The owner is --owner, or the part of --who before its first dash.
Each run writes a memory trace of the command and every process under it, and one line to ~/.stageload/history.jsonl with the wait, the run time, the peak footprint and swap.
Engines started outside the queue go in ~/.stageload/busy.txt, one pattern per line. Every run in the ram lane waits until none of them has been seen for 20 seconds.