DEV Community: Athreya aka Maneshwar

One Rust Unicode Bug, Three Coding Models: A Reproducible Comparison

Athreya aka Maneshwar — Sun, 18 Oct 2026 05:10:27 +0000

Coding-model leaderboards reveal little about how a model handles a real bug. A Rust function can work on ASCII and still panic on valid Unicode because it slices UTF-8 at the wrong byte boundary. The fix must respect character boundaries and preserve the &str return type. This article uses one reproducible bug, one prompt, and the same regression tests to compare three models on the quality of their patches, not the confidence of their explanations.

A one-line Rust slice can panic on valid UTF-8. That makes it a useful coding-model challenge. The fix must preserve the function signature, handle character boundaries, and pass regression tests. This comparison uses the same bug for three models and judges their patches using compiler output rather than leaderboard scores.

The bug

The function below is supposed to return the first n characters of a string. It works on plain ASCII and panics when n cuts through a multibyte UTF-8 character.

pub fn prefix(input: &str, n: usize) -> &str {
    if n >= input.len() {
        return input;
    }
    &input[..n]
}

#[cfg(test)]
mod tests {
    use super::prefix;

    #[test]
    fn ascii() {
        assert_eq!(prefix("rustacean", 4), "rust");
    }

    #[test]
    fn unicode() {
        assert_eq!(prefix("café", 4), "café");
    }

    #[test]
    fn beyond_length() {
        assert_eq!(prefix("é", 10), "é");
    }
}

The original implementation counts bytes, not characters. In UTF-8, the final é in café occupies two bytes. Slicing at byte index four cuts through that character and panics. Save the code as src/lib.rs in a new Cargo library project. Run cargo test to reproduce the failing Unicode case.

Three models, one task

The three candidates are DeepSeek V4 Pro, Claude Fable 5.1, and Grok Build 0.1. These are the target model pages for the comparison. Model availability, exact API identifiers, and supported parameters must be checked against the live catalog before execution.

Model	API identifier	Input	Version
DeepSeek V4 Pro	Confirm from catalog	Same Rust file and prompt	Record API version
Claude Fable 5.1	Confirm from catalog	Same Rust file and prompt	Record API version
Grok Build 0.1	Confirm from catalog	Same Rust file and prompt	Record API version

Use the same request body, system instructions, maximum output allowance, and test environment for every model. Record the exact model ID returned by the API. A zero temperature reduces randomness but does not guarantee identical answers on repeated calls.

The prompt

Use the following prompt unchanged for every model.

Fix the Rust function below. It must return the first n Unicode scalar values without panicking. Keep the return type &str. Explain the bug briefly, return a complete src/lib.rs, and add a regression test for an emoji and a combining character. Do not add external dependencies.

Append the entire original src/lib.rs after that instruction. Do not tell one model about another model's failure. Do not silently repair a response before testing.

What to inspect in each model's patch

DeepSeek V4 Pro

Capture the raw response, its reported input and output tokens, the elapsed request time, and the first cargo test output. A correct patch should iterate over char boundaries rather than slice at an arbitrary byte index. If the returned code does not compile, retain the error and count the next prompt as another attempt. Add a screenshot of the response and terminal output from the actual run.

Claude Fable 5.1

Use the identical source and prompt. Record whether the model preserves the borrowed &str return type and whether it handles a request longer than the character count. Check the combining-character case carefully. Unicode scalar values are not the same as grapheme clusters. A request for one scalar value from an e followed by a combining accent may return only the e. That is correct under this task's definition.

Grok Build 0.1

Repeat the procedure without editing the prompt. Review whether the model changes the API, introduces a dependency, or returns a String instead of &str. These changes matter even when the visible output looks correct. Keep the first attempt and any follow-up attempts as separate records.

What a correct fix looks like

One valid approach uses char_indices to find the byte boundary after the requested number of scalar values.

pub fn prefix(input: &str, n: usize) -> &str {
    let end = input
        .char_indices()
        .nth(n)
        .map(|(index, _)| index)
        .unwrap_or(input.len());
    &input[..end]
}

This reference implementation is a correctness oracle for the exercise, not an attributed output from any model. The test suite should include ASCII, multibyte characters, empty strings, n equal to zero, n greater than the character count, emoji, and decomposed accents. The reference code should be kept out of model prompts.

Comparison scorecard

Model	First attempt passes	Attempts	Wall time	Input tokens	Output tokens	Notes
DeepSeek V4 Pro	Enter from cargo test	Enter from logs	Enter from logs	Enter from API	Enter from API	Compiler and behavior notes
Claude Fable 5.1	Enter from cargo test	Enter from logs	Enter from logs	Enter from API	Enter from API	Compiler and behavior notes
Grok Build 0.1	Enter from cargo test	Enter from logs	Enter from logs	Enter from API	Enter from API	Compiler and behavior notes

The scorecard becomes useful once each row is populated from the first response, Cargo output, and API logs. Until then, it is a repeatable evaluation worksheet rather than a model ranking.

What to pick for each coding task

For short patches, first-pass compilation and correct edge cases matter more than a polished explanation. For larger refactors, examine how often a model changes unrelated code. For code review, test whether it identifies the byte-versus-character distinction without prompting. For terminal automation, compare how predictably each model follows the required output format.

The best LLM for coding is task dependent. One Rust bug cannot establish which model performs best across a whole repository, language, or toolchain. It can reveal whether a model handles a concrete failure mode under controlled conditions.

How to repeat the comparison

Create a Cargo library with cargo new unicode-prefix --lib. Replace src/lib.rs with the buggy example and run cargo test. Send the same prompt and file to each confirmed model identifier. Save the raw JSON response, request timestamp, token usage fields, and model version. Extract the proposed code into a clean working tree for each attempt. Run cargo fmt --check, cargo test, and cargo clippy. Record failures without modifying the first answer.

Keep the repository organized as follows:

baseline/src/lib.rs
oracle/src/lib.rs
prompts/task.txt
runs/<model>/response.json
runs/<model>/cargo-test.txt

Publish a repository link only after the files are uploaded. The reproducible artifact is the code and logs, not a ranking graphic.

A shareable comparison includes the repository, unedited first responses, and Cargo logs. Readers can inspect both the proposed fixes and the failures.

We Rendered a HQ Demo Video on a Free Farm Run by Strangers

Athreya aka Maneshwar — Wed, 14 Oct 2026 11:08:09 +0000

Hello, I'm Maneshwar, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.

Last post, my four-laptop render farm lost a race against my own laptop.

I promised you the ending.

Here it is.

It involves a free farm run by strangers and a Java file called sheepit-autoupdate.jar.

The lesson I was too slow to learn

If you missed it, last time we turned four laptops into a Flamenco render farm.

Athreya aka Maneshwar

Oct 12

We Built a Blender Render Farm Out of Four Laptops

#blender #networking #tailscale #devops

16 min read

It worked.

And then the farm spent almost all of its time downloading a 115 MB .blend over my 5 Mbit/s home upload, while my laptop finished the job alone.

The takeaway was one sentence:

A render farm is a file distribution problem wearing a GPU costume.

So the question for round two was not "where do I find more GPUs".

It was "who can I hand a file to once, and have it spread out without my upload being the highway".

Meet the flock

SheepIt is a free, community-powered render farm for Blender.

The deal is beautifully simple.

You install a small client on your machine.

While your machine is idle, it renders other people's frames.

That earns you points.

You spend the points to have the community render yours.

Everyone lends a machine, everyone gets frames.

I did not build a farm.

I joined a flock.

The client is a single Java jar (sheepit-autoupdate.jar, which updates itself, hence the name), and the upload side is a web page where you drop a zip.

That is the entire infrastructure I had to run.

No Samba, no tailnet, no NAT archaeology.

The file goes to one place, and it is their problem to hand it to a hundred machines.

Points: the currency of the flock

SheepIt runs on points.

You earn them by rendering other people's frames.

You spend them when you order your own.

The core loop needs no money, just render time in and render time out.

I registered on October 10th, the same Saturday the laptop farm gave up, and left the client running.

Here is our profile a few days later:

Let me read that back to you.

Our laptops spent 2 days and 7 hours rendering strangers' frames.

That is 6,981 frames for other people.

And in return I ordered 18,651 frames across 17 projects.

I ordered almost three times as many frames as I rendered, and the balance still says 1,261,589 points.

I cannot give you SheepIt's exact exchange rate, so treat this as an observation, not a formula.

As far as I can tell, a light 360p frame is cheap to order, a heavy 1080p ray traced frame costs more, and a stronger GPU earns faster.

What I can show you is the shape of it:

The points line only goes up.

Four days of render streaks, and I never ran dry.

The frames line flattens after the first couple of days, which is my guess for the laptop getting busier with my own work.

That is the part I did not expect.

I thought I was borrowing a farm.

I was joining a barter economy where the same laptop that edits Miles during the day renders somebody else's sheep at night.

Everyone is the customer and the supplier at the same time.

The catch: strangers cannot run your scripts

There is exactly one thing SheepIt asks of you, and it is the whole engineering problem.

Your .blend has to run on a machine you have never seen, unattended, with none of your project around it.

My scenes were not built for that.

They are built by Python scripts that load a city, retarget animations, key thousands of values and bake the result into a .blend.

On my laptop that file leans on a whole project folder:

Absolute paths to textures, like /home/lovestaco/pers/lr_demo/....

Image sequences for the animated screens in the city (the bugs board, the queue counters).

Sound strips for the voice-over.

Render settings that my own scripts apply at render time.

A stranger's machine has none of that.

It cannot run my scripts, so settings I apply from code simply never happen.

So I wrote a packer, sheepit_pack.py, whose job is to turn a built scene into a hermetic one:

Pack every image into the .blend.

Textures travel inside the file now.

Copy image sequences next to it with relative paths.

Blender cannot pack a sequence, so the folder rides along in the zip and the paths become //assets/....

Drop the sound strips.

The voice-over and sound effects are mixed later with ffmpeg anyway, so the farm never needs them.

Bake the settings into the file.

1920x1080 at 100%, 64 samples, ray tracing at full resolution, motion blur on, PNG output.

Then zip it with the .blend at the root.

My zips came out between 130 and 240 MB.

SheepIt takes up to 2 GB, so that is a comfortable fit.

Trust nothing, verify everything

Here is the thing about sending your scene to strangers.

If it is broken, you find out hours later, when a stranger's machine has already burned your points on a hundred identical black frames.

So every package goes through a gate before it leaves my laptop.

flowchart LR
    A[Built .blend] --> B[Pack<br/>images, relative paths, baked settings]
    B --> C[Zip<br/>blend at the root]
    C --> D[Extract into a clean folder]
    D --> E{Verify<br/>paths, settings, no sound}
    E -->|fail| X[Fix and repack]
    E -->|pass| F[Render 1 or 2 test frames<br/>from the extracted copy]
    F --> G[Upload the zip]
    G --> H[Community renders]
    H --> I[Download the frames zip]
    I --> J{Frame check<br/>count, gaps, size, black frames}
    J -->|fail| Y[Re-queue those frames]
    J -->|pass| K[Assemble with audio and subtitles]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef mine fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef farm fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef bad fill:#ff9a5c,stroke:#c4642a,color:#1a1a1a
    class E,J decision
    class A,B,C,D,F,K mine
    class G,H,I farm
    class X,Y bad

The important part is the extraction.

I do not test the original .blend.

I unzip the package into a fresh temporary folder, the way the farm will see it, and run the checks there.

Every external file must exist and be a relative path.

No sound strips may be left.

The resolution, samples, ray tracing, motion blur and PNG format must all match what I think I baked.

Then I render one or two real frames from that extracted copy, with the baked settings.

If those frames look right, the package is right.

On my GTX 1650 those test frames take between 7 and 35 seconds at 1080p.

Which is a quick way to tell the difference between "this will take a day" and "this will take a week" before anyone else spends a point on it.

I run it as one command per part, and it stops with a loud error at the first problem.

It also never deletes anything.

An old package is moved aside, not overwritten.

The bug where the file was fine and the check was wrong

The first time I ran the new packager for part 1, it died at the zip step with ERROR: street_part_1_final.blend is not at the zip root.

It was at the zip root.

I looked at the zip.

The file was right there, first entry.

The check looked like this, with the fix underneath:

set -euo pipefail

# broken: dies on SIGPIPE even though the file is in the zip
unzip -Z1 "$ZIP" | grep -qx "street_part_1_final.blend" || die "not at the zip root"

# fixed: capture first, then grep
LIST=$(unzip -Z1 "$ZIP")
grep -qx "street_part_1_final.blend" <<<"$LIST" || die "not at the zip root"

grep -q is polite about one thing: the moment it finds a match, it exits.

That leaves unzip writing the rest of the file list into a pipe nobody is reading, so it gets SIGPIPE.

And pipefail says that if any stage of the pipe fails, the whole pipe failed.

So the file was found, and the check reported it missing.

The fix is to stop piping.

If you ever write producer | grep -q under set -o pipefail, remember this.

It works until the producer has more output than the pipe buffer, and then it fails on the one day it matters.

Render only what the film keeps

This part is satisfying.

The film is not one long take.

Parts of the Blender scenes are cut from the final video (we removed a mural scene, the LED ticker and a subway lightbox), and the 2D product boards in part 3 are not rendered in Blender at all.

If you upload a scene and ask for "frame 1 to the end", the community renders all of that for you, for free, and you throw half of it away.

Rude to the sheep.

So each package gets uploaded with exactly the frame ranges the film needs.

Part 0 is one range, 458 frames.

Part 1 is three ranges, because three scenes are cut.

That skips about 535 frames that never reach the final video.

Part 2 is one range, 2,441 frames.

Part 3 is the same zip uploaded twice, frames 1 to 664 and 2150 to 3661.

The 2D boards sitting between them come from a Python script on my laptop in a couple of minutes.

The whole film is 8,229 frames, or 4 minutes 34 seconds.

Last post I said 7,300.

The film grew.

Films do that.

Assembling what comes back

Each part comes home as a zip of PNG frames.

A 1080p part is between 2 and 4 GB.

I do not trust anything that arrives from a farm of strangers.

So before a single frame goes into a video, a check script looks at the zip:

Is the zip intact?

Is the frame count exactly what the build says it should be, numbered from the first frame to the last, with no gaps and no extras?

Does every PNG decode, at exactly the expected size?

Is any frame entirely black?

An all-black frame is the farm's way of telling you that a worker had a bad day.

If everything passes, the frames are hard-linked into a numbered folder (no copies, so a 4 GB set costs nothing extra), ffmpeg encodes the base video, and my assembly script mixes in the voice-over, sound effects, music, subtitles and the frame-number burn-in.

That last one is for the team.

When someone says "the thing at f 1201 is off", I can find it.

Stale frames beat missing frames

My render scripts skip frames that already exist, which is wonderful when a long render gets interrupted.

It is terrible after you change the scene.

Stale frames from the old version sit in the folder, the renderer says "already done", and your new film has the old shot spliced in.

The rule I follow now: when the scene changes, the old frame folder gets moved aside (not deleted, never wiped), and the render starts into a clean one.

A draft at home, the real thing at the farm

Here is the workflow that has been settled for the project, and it is the actual answer to the render bill from last post.

Drafts are local.

360p, EEVEE, no ray tracing, no motion blur.

About 0.7 seconds per frame, so part 2's 2,441 frames take roughly 30 minutes.

The whole film is about two hours.

Make chai, come back, watch Miles.

Finals are on the farm.

1080p, 64 samples, ray traced reflections, motion blur.

A day of laptop time, spread across a flock that does not need me.

And between them sits the thing that makes drafts useful: a review copy the team can actually open.

Did Miles get his 1080p?

Yes.

Two of the four parts have already come home at 1080p from the flock, a 2 GB zip of PNGs for the opening and 3.7 GB for part 2.

Then the team's notes arrived and the film changed under them.

So there is a fresh set of packages sitting verified and ready for the community, each one with a test frame already rendered from the extracted copy.

What I would tell last week's me

Do not build a farm when a community already runs one.

Four laptops taught me a lot about networking.

A flock taught me that I had been solving the wrong problem.

Make your scene hermetic before you make it fast.

If the file only works on your machine, it works on one machine.

Verify the package, not the original.

Extract it into a clean folder and render a frame from that.

Keep a cheap draft loop.

It is the thing that lets a teammate say "this word is wrong" without it costing you a day.

And a thank you to the strangers whose GPUs are going to render the final.

You are not a sheep.

You are the shepherd.

I would love to hear from you in the comments.

Have you ever lent your machine to a community farm, or used a community compute network, for a project?

Follow along, because next time I will show you the final cut.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Risk-Aware AI Code Review for Business-Critical Systems

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by tests…

View on GitHub

Click below to try LiveReview with your codebase:

We Built a Blender Render Farm Out of Four Laptops

Athreya aka Maneshwar — Mon, 12 Oct 2026 17:06:37 +0000

We built a render farm out of four laptops on a Saturday evening.

It had a job queue, a web dashboard, a private network spanning three cities' worth of ISPs, and a shared drive that every machine could see.

It worked.

And then it lost a race against my own laptop, rendering alone.

This is the story of that farm, the networking rabbit hole it dropped us into, and why a 130 MB file taking seven minutes to cross the internet ended the whole experiment.

Where we left Miles

If you have been following along, you know the setup.

In part one I rigged a Spider-Man model in Blender without ever having opened Blender before.

Athreya aka Maneshwar

Oct 3

I Made Spider-Man Swing Without Animating a Single Frame

#blender #mcp #ai #beginners

13 min read

In part two Miles moonwalked off a billboard and a Python test caught it.

Athreya aka Maneshwar

Oct 5

Spider-Man Moonwalked Off a Billboard and Only My Test Suite Noticed

#blender #python #gamedev #debugging

14 min read

Since then, the film has grown into a roughly four minute LiveReview explainer in three parts.

That is about 7,300 frames at 30 fps.

And every one of them has to be rendered by something.

360p is a lie we tell ourselves

For review drafts, I render at 360p with EEVEE, Blender's real-time engine, at 8 samples with ray tracing and motion blur switched off.

On my laptop's GTX 1650 that comes out to about one second per frame.

A single part renders in 20 to 50 minutes.

Go make chai, come back, watch Miles swing.

Totally fine.

Then you want the real thing.

1080p, ray traced reflections on the wet street, proper sample counts.

So I asked Claude to stop guessing and time actual frames from our scene.

The 1080p EEVEE versions we actually wanted sit between 10 and 22 hours for the whole film.

Cycles, the path tracer, came in at 76 to 90 seconds per frame.

That is a week.

Of my laptop doing nothing else.

With 6 GB of disk left.

"Is it not even using the GPU?"

That is the first question anyone asks, and it is a fair one.

So we sampled nvidia-smi during a render.

Blender was on the card, using about 1.5 GB of the 4 GB VRAM, at roughly 44% utilization.

At 360p the GPU finishes its part of the frame quickly and then sits there waiting for the CPU, which is busy posing the characters, evaluating the crowd and writing the JPEG.

So the next obvious move was to run more Blender processes in parallel.

Two workers were only 1.3x faster than one.

Three workers were slower than one, because three Blender processes fighting over eight cores and one GPU is not parallelism, it is a custody battle.

The only free speed-up we found locally was rendering each frame range as a single animation call instead of calling render() once per frame, which skips redoing scene setup every time.

That bought about 22% on one worker.

Nice, but nobody turns a 22 hour render into an overnight one with a 22% discount.

The conclusion was pretty clear.

We did not need a faster script. We needed more GPUs.

And there were three other laptops in the team, mostly idle on a Saturday evening.

Enter Flamenco

Flamenco is Blender's own open source render farm software.

It is a lot smaller than the phrase "render farm software" suggests.

There are exactly two programs:

Flamenco Manager is the brain.

It holds the job queue, serves a web dashboard, and splits each job into tasks.

Flamenco Worker is the muscle.

It runs on each rendering machine, polls the manager for work, launches Blender in the background, and reports progress.

Both are single Go binaries you download, extract and run.

No database server, no Kubernetes, no YAML archaeology.

When you submit a job, say "render frames 2370 to 2429 of this .blend", the manager cuts the range into chunks of N frames.

Each chunk becomes a task.

Every idle worker grabs the next task, renders it, and asks for another, so a fast machine naturally ends up doing more of the work than a slow one.

While a task runs, the worker keeps pinging the manager with a "may I keep running?" check, which is how cancelling a job actually reaches a laptop in another city.

If a task fails, it gets retried on a different worker.

After three failures on the same job, that worker gets blocklisted for the job so it stops eating chunks.

Here is the farm we built:

flowchart LR
    ME[My laptop<br/>Manager + Worker]
    SMB[(spidey share<br/>Samba)]
    TS{{Tailscale<br/>private network}}
    SH[Windows desktop<br/>later WSL]
    LI[pop-os laptop]
    RU[ASUS TUF laptop]

    ME --- SMB
    ME <-->|jobs + status| TS
    TS <--> SH
    TS <--> LI
    TS <--> RU
    SH -.->|reads .blend, writes frames| SMB
    LI -.->|reads .blend, writes frames| SMB
    RU -.->|reads .blend, writes frames| SMB

    classDef host fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef net fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef store fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef peer fill:#ff9a5c,stroke:#c4642a,color:#1a1a1a
    class ME host
    class TS net
    class SMB store
    class SH,LI,RU peer

Notice the dotted lines.

That is the part that would end up mattering.

The one assumption Flamenco makes

Flamenco does not move your files around.

The shared storage docs are upfront about it: the manager and every worker must be able to see the same files, on a NAS or a file server or some other shared drive.

The job just says "open this path", and it is your problem to make sure that path exists everywhere.

On four laptops scattered across home connections, that sounds like a strange requirement.

To be fair to Flamenco, a studio render farm lives on one fast LAN with one NAS, and this is a completely reasonable design there.

Our "LAN" was four laptops on four home connections.

Which brings us to the networking.

Tailscale: a LAN made out of the internet

My first worry was the obvious one: does this mean my machine has to push everything over my internet connection?

It does not.

Tailscale just creates a private network so all our laptops can talk to each other.

Every machine runs the Tailscale client and logs into the same account (a "tailnet").

Each machine gets a stable private address in the 100.x.y.z range, and every pair of machines talks over an encrypted WireGuard tunnel.

Tailscale's servers only do the coordination: who is in the network, what their public keys are, where they might be reachable.

Your actual traffic is supposed to go machine to machine.

"Supposed to" is doing a lot of work in that sentence, and we will get there.

On my laptop, Claude set up a Samba share called spidey and kept it bound to loopback only, so it was never exposed to my Wi-Fi or the internet.

Then Tailscale itself forwarded the SMB port, so only machines on the tailnet could reach it:

# on my laptop: Samba listens on 127.0.0.1 only,
# Tailscale forwards tailnet traffic on 445 to it
tailscale serve --bg --tcp 445 tcp://127.0.0.1:445

# on each Linux worker: mount it at the same path everywhere
sudo mount -t cifs //100.76.212.8/spidey /mnt/spidey \
  -o credentials=/root/.spidey_smb,uid=$(id -u),gid=$(id -g),vers=3.0

The first milestone was gloriously low-tech.

Someone created a file from Windows.

I created one from Linux.

Both files showed up on both machines.

We celebrated a text file like it was a product launch.

The path boss fight

Having a shared folder is step one.

Getting Blender on three operating systems to agree on what is in it is the actual game.

Paths baked into the .blend

Claude inspected street_part2_b.blend and found 31 external asset references.

25 of them were absolute paths starting with /home/lovestaco/pers/lr_demo/....

The Windows desktop has no /home/lovestaco.

Neither does the pop-os laptop.

The fix was to copy the project into the share and convert every reference to a Blender relative path (//assets/...), resolved from wherever the .blend itself lives.

After that, all three street files had zero absolute paths left.

Paths in the job itself

The .blend location and output folder in a Flamenco job are a separate problem, and Flamenco does solve this one with two-way variables.

The manager stores job paths with a placeholder, and each worker swaps in its own platform's value:

# flamenco-manager.yaml
variables:
  spidey:
    is_twoway: true
    values:
      - platform: linux
        value: /mnt/spidey
      - platform: windows
        value: D:\spidey

Two gotchas we learned the hard way.

On Windows, Flamenco wants a drive letter, not a \\server\share UNC path.

And on Linux, every machine has to use the same mount point, because there is only one Linux value.

That is why everyone mounted at /mnt/spidey, and why the wizard rejected ~/spidey outright (it does not expand ~).

Creating D:\spidey with mklink also failed, which turned out to be because mklink is a builtin of the old Command Prompt, not of PowerShell.

cmd /c mklink ... from an admin window fixed it.

Where is Blender?

Then the Windows worker printed this:

There was an error finding Blender on this system.

Blender was on PATH.

The shell had been restarted.

It still could not find it.

The Blender variable docs explain why: on Windows the worker looks up which program is associated with .blend files, rather than trusting PATH.

The fix was gloriously petty.

Create an empty junk .blend file, use it to set the file association, and the worker immediately reported Blender 5.2.2 LTS.

By 8:15 pm we had four workers awake on the dashboard:

image pending: Flamenco web dashboard showing four workers awake and the farm status idle

My laptop, a Windows desktop, a pop-os laptop and an ASUS TUF gaming laptop.

All on Blender 5.2.2.

An actual render farm.

On a Saturday.

The first job

We kept it small on purpose.

60 frames of part 2b at 360p draft quality, split into three chunks of 20.

Three workers picked up a chunk each within seconds.

Frames started landing in the shared folder.

And the farm genuinely worked.

When the job finished, Claude compared frames rendered by three different machines and they were identical: same lighting, same colours, same Miles.

That part is honestly very cool.

Then we looked at how long everything took.

My worker read the 115 MB .blend off its own disk in a few seconds and rendered at about 0.4 s per frame.

The two relayed laptops spent over three minutes just opening the file before rendering a single frame.

A 20 frame chunk at 360p renders in seconds.

So the farm was spending almost all its time downloading and almost none of it rendering.

"Missing DNA block"

The Windows worker did something more dramatic.

It spent 82 seconds reading the file and then Blender died with:

ERROR Failed to read blend file 'D:\spidey...\street_part2_b_360p.blend': Missing DNA block

This error is a great little window into how .blend files work.

A .blend is basically a memory dump.

It is a header followed by a long run of data blocks, each one a chunk of Blender's in-memory structs written straight to disk.

To read those structs back, Blender needs to know their layout, which fields, which types, which offsets, and that description lives in a special block called DNA1 (the "SDNA").

That is how a file saved years ago, by a Blender with differently shaped structs, can still be opened today.

The catch is that DNA1 gets written near the end of the file.

So if a read dies partway through, Blender never reaches the block that explains everything before it, and you get "Missing DNA block".

We checked the obvious thing first.

The copy on Windows had the right size, 114,783,416 bytes, and once its SHA-256 finally finished (that alone took ages over the link), the hash matched mine exactly.

The file was fine.

The read was not.

The chunk size argument

This sparked the best debate of the night.

Claude suggested bigger chunks, so each worker would pay the slow load less often.

The obvious pushback: wouldn't smaller chunks download faster, in more pieces, spread across more machines?

The intuition makes sense if chunks map to pieces of the file, like a torrent.

They do not.

A chunk is a range of frames, but every chunk opens the entire scene.

Smaller chunks mean more full reads, not smaller ones.

The next question was a better one though: once the file has been read, isn't it done? Shouldn't only the first read be slow, since the OS caches the file?

So we ran a second job with 10 frame chunks and watched the ASUS laptop take its second chunk.

The first read took 3 min 02 s.

The second took 31 seconds.

Caching was real.

Not instant, but real.

On Windows, though, the cached retry failed with the same "Missing DNA block", so that SMB client and Blender were clearly not getting along.

Moving that worker into WSL got past the error, and it promptly rendered at about 50 seconds per frame, which smells a lot like EEVEE running without access to the GPU.

A side quest: whose GPU is it anyway

Even after loading, the remote workers were slow per frame.

The pop-os laptop took about 2.3 s per frame against my 0.4 s, and its GPU sampling step alone was over 3x slower than mine.

Its lspci explained it: an NVIDIA GTX 1650 Mobile, and an AMD Radeon Vega built into the Ryzen processor.

Two graphics chips, like most gaming laptops.

And glxinfo said OpenGL was running on the AMD integrated chip by default, because that is what laptops do to save battery.

EEVEE was happily rendering Miles on the weak chip while the GTX 1650 sat there.

The fix is PRIME render offload, starting the worker with __NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia so Blender lands on the NVIDIA card.

Worth knowing if you ever farm out renders to other people's laptops.

Their GPU is not necessarily the GPU Blender gets.

Direct vs DERP (and what on earth is "blr")

Back to the real problem, the reads.

tailscale status on my machine had one telling word at the end of each peer's line.

The WSL worker was connected direct.

The pop-os and ASUS laptops were going through relay "blr".

Here is what those two words actually mean.

Direct

Tailscale's first choice is a direct WireGuard tunnel over UDP between the two machines.

The hard part is that almost nobody's laptop has a public IP.

You are behind your home router's NAT, and quite often behind your ISP's NAT too.

So Tailscale does NAT traversal, also known as hole punching.

Each machine discovers what its public IP and port look like from the outside, the coordination server shares those with the peer, and both sides fire UDP packets at each other at the same time.

Each router sees an outgoing packet, creates a mapping for the reply, and if the timing and ports line up, the two outgoing "holes" meet in the middle.

Tailscale's How NAT traversal works is one of the best networking posts on the internet if you want the full version.

DERP

When hole punching fails, Tailscale falls back to DERP, short for Designated Encrypted Relay for Packets.

DERP servers are relays that Tailscale runs around the world.

Your packets go to the relay over HTTPS and it forwards them to the other machine.

They stay WireGuard-encrypted end to end, so the relay cannot read them, it just shuffles opaque bytes.

DERP regions are named after airport-style city codes.

"blr" is Bengaluru.

So "relay blr" just means "that laptop's packets to me were taking a detour through Tailscale's Bengaluru relay".

Direct is faster for the reasons you would guess.

A relay adds an extra hop, it is a shared server rather than your own path, and it runs over TCP, so a congested relay hurts you in ways a direct UDP path does not.

Pings from the pop-os laptop through blr ranged from 58 ms to 415 ms, and one test run stalled after receiving about 159 MB, with even a plain ls on the share hanging for two minutes.

Why it could not go direct

tailscale netcheck on the pop-os laptop said:

* UDP: true
* IPv6: no, but OS has support
* MappingVariesByDestIP: true
* PortMapping: UPnP
* Nearest DERP: Bengaluru (blr: 48.6ms)

The important line is MappingVariesByDestIP: true.

That is a "hard" NAT.

Its router assigns a different public port for every destination, so the port my laptop learns about is not the port that router will actually accept traffic on from me.

Hole punching needs at least one side to be predictable.

Mine was the easy kind, but my router offered no UPnP or NAT-PMP to open a port automatically, and that side had no IPv6 to sidestep the whole mess.

Here is roughly how Tailscale decides:

flowchart TD
    A[Two peers want to talk] --> B{Either side has<br/>a public, open port?}
    B -->|yes| D[Direct WireGuard over UDP]
    B -->|no| C{Both NATs map ports<br/>predictably?}
    C -->|yes| E{Hole punch succeeds?}
    C -->|no, hard NAT| F{IPv6 on both sides?}
    E -->|yes| D
    E -->|no| G[Relay through nearest DERP]
    F -->|yes| D
    F -->|no| G

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef good fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef bad fill:#ff9a5c,stroke:#c4642a,color:#1a1a1a
    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    class B,C,E,F decision
    class D good
    class G bad
    class A start

The suggested fix was forwarding UDP 41641 on my router so the hard NAT side would have an open door to knock on.

The twist: direct did not save us either

Then we noticed something awkward.

The WSL worker was on a direct connection.

And it still took about six minutes to load the file.

So we measured my connection.

0.65 MB/s upload.

About 5 Mbit/s.

Every worker in this farm was reading the .blend from my laptop.

Which means every byte, direct or relayed, had to squeeze out through my upload.

At 0.65 MB/s, one 115 MB read takes about three minutes.

Two workers reading at once, about six.

That worker took in 136 MB in roughly six minutes, about 0.38 MB/s, which is exactly what half of my upload looks like.

Fixing the hard NAT would have shortened the detour.

It would not have widened the pipe.

I switched to a faster Wi-Fi network for one more run, which bumped my upload to about 2.15 MB/s, and the remote first loads dropped to about a minute and a half.

Better.

But in that same run my laptop alone finished four of the six chunks while the remote workers were still loading.

Why we pulled the plug

For a remote worker to help, two things have to be true.

It has to get the scene fast, and it has to render fast once it has it.

Ours had neither.

The scene came through my home upload, partly via a relay, and the laptops that did load it were either on the wrong GPU, in WSL with no GPU, or tripping over SMB on Windows.

Every fix we found was real, and every one of them was its own project.

Copy the project to every laptop ahead of time and lose the "one shared folder" simplicity that made the setup appealing.

Fight every home router.

Fix every laptop's GPU setup.

Make sure every laptop stays awake, plugged in and on Wi-Fi for ten hours.

So at 9:52 pm we shut it all down.

Flamenco manager and worker stopped, Samba stopped and disabled, the Tailscale forward turned off, Tailscale itself stopped.

Nothing deleted, just switched off.

A few things I would tell past me before this Saturday:

A render farm is a file distribution problem wearing a GPU costume.

The render was the easy part.

Getting the same 130 MB to every machine, reliably, was the whole job.

Measure your upload before you build anything.

Five minutes with a speed test would have predicted most of this evening.

"Direct" is about the route, not the speed.

A perfect peer-to-peer tunnel is still capped by the slowest pipe on either end.

Check which GPU Blender is actually using.

Dual-GPU laptops will quietly hand you the integrated chip.

Flamenco is fine.

It assumes a studio LAN with a NAS, and we gave it four home connections and a laptop pretending to be a NAS.

So did Miles ever get his 1080p?

We really wanted this one to work.

Four laptops, one Saturday, our own little render farm.

It is a great story, it just is not a fast one.

But the film still needs rendering, and we did eventually crack it, by spreading the work across multiple providers instead of multiple laptops.

That is the next post.

Follow along so you do not miss how we finally got Miles rendered.

And I would love to hear from you in the comments.

Have you ever tried to farm out a render, or any big compute job, across machines you did not own?

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Risk-Aware AI Code Review for Business-Critical Systems

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

LiveReview is built for engineering teams that ship often, but also care about keeping production stable for customers. It approves some 40-50% of low-risk changes automatically, and it gives your team the tools to handle the rest:

Understand risky changes faster.

Fix problems in those changes, with human attention directed to the work that needs it.

Enforce your engineering standards.

Learn from production outcomes, such as incidents, rollbacks, and queue wait times.

Improve the review process over time and make it more efficient.

Connect to the major engineering tools that your team already uses.

Auto-Approve: Ship the Safe 50% Without a Human

Auto-Approve Upto 50% of Your Pull Requests, Safely. Risk scoring decides which MRs/PRs are safe enough to ship without a human, so your team only spends review time on risky and critical changes.
demo.mp4
LiveReview auto-approving a low-risk change while…

View on GitHub

Click below to try LiveReview with your codebase:

The Stack I'd Need for Claude to Direct a Whole YouTube Video in Blender

Athreya aka Maneshwar — Fri, 09 Oct 2026 18:10:15 +0000

Two posts ago, I had never opened Blender.

Since then, Miles Morales has swung on a web, yanked a DEV board into frame, and moonwalked 6.5 metres off the edge of a billboard without anyone noticing.

Every one of those shots was me and Claude Code, pair programming a 3D scene one prompt at a time.

It works.

It is also slow, because I am still the one holding the whole movie in my head.

So this post is about the next question, the one I keep circling back to at 1 a.m.:

What would it actually take for me to type one paragraph and get a finished YouTube video out the other end?

Not "Claude helps me in Blender".

Claude directs the thing.

I went digging through every MCP server, Claude Code skill, mocap source and audio model I could find, checked the GitHub stars so we know what is battle tested and what is a weekend project, and wrote down the stack I would build.

Spoiler: Blender is maybe a third of it.

The job description: director, not animator

The single most useful thing I learned in part one was not about rigging.

It was about where the LLM is good and where it is hopeless.

Claude was great at writing bpy code, relinking broken textures, sequencing clips on a timeline, and stitching root motion between them.

Claude was never once asked to invent how a human body lands from a jump.

That came from Mixamo, which came from a real performer in a mocap suit.

And that split is the whole architecture.

An LLM is a brilliant writer of structure and code, a decent critic of a still image, and a terrible source of biomechanics.

So every part of a video should go to whoever is actually good at it:

If it moves like a human, fetch it.

If it is code, write it.

If it is a sound, order it from a model that does nothing but sound.

Here is the whole stack with that rule applied:

flowchart TD
    ME[One paragraph from me] --> DIR[Claude Code<br/>director skill]
    DIR --> SL[Shot list<br/>YAML]
    SL --> BR[Blender bridge<br/>one MCP server]
    SL --> MO[Motion library<br/>Mixamo, CMU, Rokoko]
    SL --> AS[Assets<br/>Poly Haven, Sketchfab, image to 3D]
    SL --> AU[Audio<br/>TTS, SFX, music]
    AU --> LS[Rhubarb<br/>mouth shapes]
    MO --> BR
    AS --> BR
    LS --> BR
    BR --> BL[Blender, headless<br/>EEVEE drafts, frame sequences]
    BL --> QA{Checks pass?<br/>eyes + asserts}
    QA -->|no| DIR
    QA -->|yes| FF[FFmpeg<br/>frames + stems]
    AU --> FF
    FF --> YT[YouTube-ready MP4]

    classDef human fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef brain fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef fetch fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef order fill:#ff9a5c,stroke:#c4602a,color:#1a1a1a
    classDef engine fill:#6ea8ff,stroke:#2f62c4,color:#1a1a1a
    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a

    class ME,YT human
    class DIR,SL brain
    class MO,AS,LS fetch
    class AU order
    class BR,BL,FF engine
    class QA decision

Let's walk it bottom up, starting with the piece everyone argues about.

Layer 1: the Blender bridge (pick exactly one)

This is the MCP server that lets Claude reach into a running Blender and press buttons by writing Python.

There are now a lot of them.

Here are the four I looked at seriously, with stars as of this week:

Bridge	What it is good at
mcp-for-blender ⭐ (~30.2k) (formerly `blender-mcp`)	The one I used in parts 1 and 2. Code execution, viewport screenshots, plus Poly Haven, Sketchfab, Poly Pizza, Hyper3D Rodin, Hunyuan3D and Tripo built in
Official Blender MCP ⭐ by Blender Lab	Understanding a scene: explaining node setups, finding what uses a material, cleanup. Blender's page lists 5.1 or newer
carlosh7/blender-mcp ⭐ (8 stars)	248 tools across 14 categories, 371 commits, 479 passing tests, MIT
claude-blender ⭐ (5 stars)	Undo checkpoints, camera auto-framing, local TripoSR and Shap-E generation

The official one is official in the sense that Blender developers built it, and Anthropic announced it as a Claude connector this year.

Read its own page though, and it is pitched as a way to understand complex setups, not to churn out a film.

It also carries the most honest warning I have seen on any MCP page: the server "will execute LLM generated code in Blender without any guards in place", and Blender suggests a VM or a machine without sensitive data on it.

That applies to every server in the table, by the way.

They all ship some flavour of "run this Python", and that is the tool doing the real work.

Which brings me to the 248 tool one.

It is clearly a lot of careful work, with real tests and CI.

But it made me realise that tool count is not a feature.

Every tool an MCP server exposes comes with a schema: a name, a description, a JSON shape for its arguments.

Many clients put all of those in the context window on every single turn, whether you use them or not.

Some, Claude Code included, can defer them and search on demand, which fixes the space problem.

It does not fix the picking problem.

If there are five slightly different ways to move a cube, the model has to choose one, and every choice is a chance to choose wrong.

One execute_blender_code and a model that writes good bpy already covers all 248, because every button in Blender is a thin skin over bpy anyway.

So my pick is boring: mcp-for-blender as the one bridge, installed with claude mcp add blender uvx mcp-for-blender, and the official one kept around for "why is this scene slow" archaeology.

And I am stealing the best idea from the 5 star repo without installing it.

Checkpoints before risky edits are a great idea.

In part two the .blend file became a build artifact generated by scripts, so my checkpoints are just git commit.

Layer 2: skills, not more tools

If tools are expensive, how do you teach Claude cinematography?

With Claude Code skills.

A skill is a folder in .claude/skills/ with a SKILL.md inside: a one line description that is always visible, and a body of instructions and scripts that only loads when the task needs it.

It is the lazy import of prompting.

blender-skills (~143 stars) is a good example of the idea.

It has camera moves as skills, turntable, slow-zoom, dolly-rotate, crane-shot, perfect-loop, plus image to 3D through Meshy and a toolkit that does Mixamo retargeting.

It is aimed at product shots, so I would not install it wholesale for a superhero short.

I would copy its shape.

The thing I actually want is a small library of film vocabulary, each word backed by tested code:

establishing_wide, push_in, pull_out, orbit, crane_up, tracking, over_the_shoulder, handheld_shake
rainy_night, golden_hour, villain_red, neon_street, soft_closeup_key

Then "start wide, track him to the ledge, push in slowly as he says the line" stops being a creative writing exercise for the model.

It becomes three skill calls with numbers.

Same idea applies to lighting.

A preset that is right once is right forever, and nobody has to rediscover three point lighting at 2 a.m. because Miles's suit rendered as a black blob again.

Layer 3: motion, fetched not generated

This is where most "AI made my animation" demos fall apart, and it is the layer I am most opinionated about.

Do not let the LLM keyframe a human.

It will produce something that technically moves and spiritually does not.

Real motion has weight shifts, overshoot, a little settle at the end of a landing.

Mocap has all of that for free, because a person did it.

The sources I would wire in, in order of effort:

Mixamo. Free, auto-rigs a T-posed character from five markers, thousands of clips. I downloaded 78 of them one click at a time in part one, and I would do it again.

CMU Motion Capture Database. About 2,500 free clips, with a BVH mirror on GitHub (~203 stars). Needs a retargeter, which part one built and part two debugged.

Rokoko. Their Blender add-on is free and does retargeting. Their free plan includes Rokoko Vision, single camera AI mocap. That last one is the exciting bit: the motion Mixamo does not have, I can act out in my living room with a webcam.

And then the controls.

In part one I dragged Miles's hips down and his whole body sank through the floor like a ghost, because the rig had bones but no IK.

Two fixes, depending on budget:

Mixamo Rig is a free, GPL extension that builds a proper IK control rig on top of a Mixamo skeleton and bakes animation in and out of it. Check the version range on the listing before you build on it.

Auto-Rig Pro by Artell is the paid, serious option, about $50, used in shipped games like Manor Lords, with rigging, retargeting between skeletons and facial rigs in one place.

If I spend money anywhere in this stack, it is here.

Rigging is the one job where a mistake quietly poisons every shot that comes after it.

Layer 4: the face, or the close-up problem

Here is a thing I did not appreciate until I tried to frame a close-up.

In a wide shot, nobody looks at the face.

Then the camera pushes in for the line, and the face is the only thing on screen.

A mouth flapping open and closed on a timer reads instantly as fake.

So does a character who never blinks, or whose eyes are locked dead ahead like a mannequin's.

The good news is that most of facial performance can be fetched too, and the most important piece is wonderfully small.

Rhubarb Lip Sync (~2.6k stars) is a command line tool that listens to a voice recording and writes out a timeline of mouth shapes.

It uses nine shapes, A to H plus X for rest, the same small set classic cartoon studios used, and you can hand it the script with --dialogFile so it knows the words instead of guessing.

It says it is built for 2D, but the output is just JSON with timestamps, and a 3D face can use it just as well.

There is a Blender add-on for it (~228 stars), but it has not been touched since 2023.

The glue is short enough that I would rather have Claude own it:

# rhubarb -f json -d line.txt -o line.json line.wav
import bpy, json

cues = json.load(open("line.json"))["mouthCues"]
keys = bpy.data.objects["Miles_Head"].data.shape_keys.key_blocks
fps, start = bpy.context.scene.render.fps, 120   # line begins on frame 120

# Rhubarb's letters -> this model's shape key names
VISEME = {"A": "MBP", "B": "EE", "C": "EH", "D": "AA", "E": "OH",
          "F": "OO", "G": "FV", "H": "L", "X": "rest"}

for cue in cues:
    frame = start + round(cue["start"] * fps)
    for shape in VISEME.values():
        keys[shape].value = 1.0 if shape == VISEME[cue["value"]] else 0.0
        keys[shape].keyframe_insert("value", frame=frame)

That is the whole trick: audio in, a keyframe per mouth change out.

Blender's default Bezier interpolation eases between the keys, so the mouth glides instead of snapping.

If your model drives its mouth with face bones instead of shape keys, like the 143 bone face rig on my Miles, the same loop poses bones instead.

Blinks and eye darts are even cheaper.

A blink every two to six seconds, randomised, and a tiny eye flick toward whatever the character is reacting to, is a dozen lines of procedural keyframing, and it is the difference between a person and a wax figure.

Notice the dependency hiding here, too.

Lip sync needs the audio to exist first.

Which means the voice is not post production.

It is pre production.

Layer 5: assets, mostly fetched

Same rule again: generate only what nobody has made yet.

Poly Haven. Thousands of CC0 HDRIs, textures and models. CC0 means no attribution and no strings. mcp-for-blender searches and imports from it directly, and an HDRI alone fixes half of "this scene looks fake".

Sketchfab. Huge, and also where my Miles came from. Read every license, because they vary model to model.

Image or text to 3D, for the props nobody has made:

Hunyuan3D-2 (~15k stars) and Hunyuan3D-2.1 (~4.1k stars, adds PBR materials), from Tencent, runnable locally if your GPU is braver than my 4 GB GTX 1650
TripoSR (~7k stars), fast single image reconstruction
Hosted options like Hyper3D Rodin and Tripo, already wired into mcp-for-blender

One honest caveat from someone who just learned what skin weights are.

Generated meshes are usually a dense soup of triangles.

That is fine for a mailbox in the background.

It is not fine for anything that has to bend, because rigging wants clean edge loops around elbows and knees, and a triangle soup deforms like wet cardboard.

So: generated props, sourced or hand built heroes.

Layer 6: audio is half the movie

Every engineer I know who tries video makes the same mistake I did.

They spend three days on render settings and zero minutes on sound.

Put a whoosh on a jump, a thwip on a web shot and a low thud on the landing, and the exact same frames suddenly look like they cost ten times more.

The trick is to treat audio as separate stems, never one mixed track:

Dialogue. ElevenLabs for character voices. Generate it first, because the face needs it.

SFX and ambience. ElevenLabs sound effects turns a text description into a sound, up to 30 seconds per generation, with a seamless loop mode for rain and city hum, and 48 kHz WAV for one shots.

Music. Its own model, its own stem. stable-audio-tools (~3.9k stars) is Stability's open code for their audio models if you want local control. Check the license of whatever you pick before it goes on a monetised channel.

Separate stems are what make the edit programmable.

Music too loud under the line? Change one number.

Want the music to dip automatically whenever someone talks? That is FFmpeg's sidechaincompress filter, keyed off the dialogue stem.

None of that is possible once everything is baked into one WAV.

Layer 7: render like a build system, assemble with FFmpeg

Part two already turned the scene into a build artifact, so this layer is mostly "keep doing that, at movie scale".

Three rules:

Render headless. blender -b with no UI, driven by scripts, so a render is a command, not a person clicking.

Render frame sequences, not MP4s. If Blender crashes on frame 1,400 of a movie file, you lose the movie. If it crashes on frame 1,400 of a PNG sequence, you lose one frame and resume. Blender's own docs recommend exactly this.

Draft at 360p, final once. Part two's --fast mode, with ray tracing, motion blur and extra samples switched off, roughly halved frame time. Everything is reviewed in drafts.

Then FFmpeg (~64.9k stars, and it is only a mirror) glues frames and stems into a video:

# render shot 30 as a PNG sequence (flags before -a, Blender reads them in order)
blender -b build/sh030.blend -o //frames/sh030_#### -F PNG -s 1 -e 96 -a

# frames + three stems -> one mp4, music ducked under the dialogue
ffmpeg -framerate 30 -i frames/sh030_%04d.png \
  -i audio/dialogue.wav -i audio/sfx.wav -i audio/music.wav \
  -filter_complex "[1:a]asplit[d][key];[3:a][key]sidechaincompress=threshold=0.05:ratio=8[m];\
[d][2:a][m]amix=inputs=3:normalize=0[a]" \
  -map 0:v -map "[a]" -c:v libx264 -pix_fmt yuv420p -crf 18 -c:a aac sh030.mp4

asplit makes two copies of the dialogue, one to hear and one to act as the key that pushes the music down.

normalize=0 stops amix from quietly turning everything down to make room, which is a fun one to debug by ear.

-pix_fmt yuv420p is there because some players will happily show a black screen without it.

I would still do the final colour pass and titles in DaVinci Resolve for anything going on my channel, but a machine should be able to produce the rough cut alone.

Layer 8: two reviewers, because one of them is blind

The loop that made parts one and two work was Claude grabbing a screenshot, looking at it and fixing what it saw.

That loop is great.

It is also how a moonwalk got shipped.

Vision models review a still picture.

Motion bugs live between the frames.

The 22 cm pop in the middle of a crossfade, the single frame where Miles hovered at a cut, the stance wider than the ledge he stood on, none of those survive a look at frame 1,201.

They all fail an assertion that measures where the feet are on every frame.

So every shot gets both reviewers:

Eyes: Claude looks at a contact sheet of draft frames and judges framing, lighting, mood and readability.
Asserts: scripts measure contact, bounds, continuity at cuts and missing textures, and fail loudly.

If the asserts fail, nobody wastes time looking.

The missing piece: a director skill

Everything above is a department.

What turns departments into a studio is one skill that takes my paragraph and writes the plan everyone else works from.

That plan should be data, not prose:

title: rooftop_siren
fps: 30
character: original_hero_v2       # not Miles, see above
shots:
  - id: SH010
    seconds: 4
    set: rooftop_night_rain
    camera: [establishing_wide, push_in]
    motion: { clip: "Standing Idle" }
    audio: { ambience: "rain on a rooftop, distant traffic, wind" }
  - id: SH020
    seconds: 2.5
    camera: [extreme_closeup]
    motion: { clip: "Look Around", turn: 30 }
    dialogue: { voice: hero, line: "You hear that?" }
    face: { lipsync: rhubarb, blinks: auto, look_at: siren }
  - id: SH030
    seconds: 3.2
    camera: [tracking, low_angle]
    motion: { clip: "Jump Down", speed: 1.2 }
    audio: { sfx: ["cloth whoosh", "web thwip", "heavy landing on concrete"] }

Look at what is in there.

Every value is a skill name, a library lookup or a prompt for an audio model.

Nothing asks the LLM to invent a joint angle, and nothing asks it to remember the whole movie at once.

The director writes this once.

Then each shot runs through the same loop, and a bad shot is re-run on its own instead of the whole film:

flowchart LR
    S[Shot from the YAML] --> A[Generate dialogue<br/>and SFX]
    A --> L[Rhubarb<br/>mouth cues]
    S --> M[Fetch motion<br/>and set]
    L --> B[Build shot<br/>bpy scripts]
    M --> B
    B --> D[360p draft]
    D --> T{Asserts pass?}
    T -->|no| B
    T -->|yes| V{Claude likes<br/>the contact sheet?}
    V -->|no, fix camera or light| B
    V -->|yes| F[Final frames]
    F --> X[FFmpeg mux]

    classDef src fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef order fill:#ff9a5c,stroke:#c4602a,color:#1a1a1a
    classDef fetch fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef build fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef out fill:#6ea8ff,stroke:#2f62c4,color:#1a1a1a

    class S src
    class A order
    class L,M fetch
    class B,D build
    class T,V decision
    class F,X out

It is CI for a movie.

Each shot is a job, the YAML is the config, the asserts are the tests, and Claude is the reviewer who also writes the fixes.

The order I am building it in

Not the order of the diagram, the order of the dependencies:

Bridge + skills. One MCP server, a camera and lighting skill library. Without reliable shots, nothing else matters.
Motion library. Mixamo and CMU behind one retargeter, Mixamo Rig for IK, Rokoko Vision for the clips nobody has.
Audio stems. Voice, SFX, music, all separate. This comes before the face, because the face reads the audio.
Face. Rhubarb glue, procedural blinks and eye targets, then close-ups stop being scary.
Asserts. Part two's checks, generalised to every shot.
Director skill. Only once every department works on its own. A director with no crew just writes very confident YAML.

And the question I will be honest about: will this make a Pixar short?

No.

What it can make is something I could not make at all a month ago: a 30 second scene with a character who moves like a person, talks with a mouth that matches the words, sounds like it happened somewhere, and was assembled by a machine while I wrote the next paragraph.

That is a wrap for part three.

If you have built any piece of this, especially the face, tell me what broke. I clearly have a lot of breaking ahead of me.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by tests…

View on GitHub

Click below to try LiveReview with your codebase:

Spider-Man Moonwalked Off a Billboard and Only My Test Suite Noticed

Athreya aka Maneshwar — Mon, 05 Oct 2026 20:26:54 +0000

Miles Morales moonwalked 6.5 metres past the edge of a billboard.

Not on the billboard.

Past it.

On nothing.

I had watched that shot more than once and did not notice.

The viewport did not notice either.

A 44 line Python script noticed, and it has been catching my mistakes ever since.

This is part two of my accidental Blender education.

Athreya aka Maneshwar

Oct 3

I Made Spider-Man Swing Without Animating a Single Frame

#blender #mcp #ai #beginners

13 min read

In part one I went from never having opened Blender to rigging a Spider-Man model, pulling animations from Mixamo and retargeting old Carnegie Mellon mocap onto him.

This one is about what happens next, when you try to turn that into an actual film and every mistake is suddenly a bug report.

The film, and why it is really a build system

The goal is a roughly 60 second explainer for LiveReview.

A masked Miles web-swings through a city at sunset, and every "slide" is part of the street.

A wall board, a rooftop billboard, an LED ticker, bus stop and subway lightboxes, a big street screen, and a runaway train for the finale.

Here is the twist that shapes everything below.

Nothing in the scene is hand edited.

The whole thing is generated headless by Python scripts against Blender 5.2, rendered with EEVEE on a 4 GB GTX 1650.

build/street.blend is not a file I open and tweak.

It is a build artifact, like a dist/ folder.

So every fix is a code change plus a rebuild, and the workflow ended up looking a lot more like CI than like art class:

flowchart LR
    S[Python scripts] -->|blender -b| B[street.blend]
    B --> C{support check clean?}
    C -->|no| S
    C -->|yes| D[--fast draft render]
    D --> T[review copy with timecode]
    T --> R{looks right?}
    R -->|no, f1201 is off| S
    R -->|yes| G[git commit]
    G --> F[full quality render]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef code fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef build fill:#6ea8ff,stroke:#2f62c4,color:#1a1a1a
    classDef render fill:#ff9a5c,stroke:#c4602a,color:#1a1a1a
    classDef done fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a

    class C,R decision
    class S,G code
    class B build
    class D,T render
    class F done

Almost every lesson in this post fell out of that loop.

"Can't you just use the GPU to render faster?"

This was my first question once renders started taking a while.

The answer was a little embarrassing.

It already was.

EEVEE is a GPU rasterizer, basically a game engine renderer.

While it rendered, the card sat at about 80% utilisation and 2.2 GB of VRAM.

There was no idle GPU waiting to be switched on.

And the other renderer, Cycles, is a path tracer, which on a GTX 1650 would have been slower, not faster.

So the real question was never "CPU or GPU".

It was "what is each frame paying for?"

Three settings turned out to dominate:

Setting	What it costs	Draft value
Screen-space ray tracing	reflections on the wet street, the single biggest per-frame cost	off
Motion blur	renders extra time steps for every frame	off
Samples	anti-aliasing and soft shadows, linear in render time	8 instead of 16 to 24

So the render script grew a --fast flag that flips exactly those three:

fast = "--fast" in args          # draft: no ray tracing, no motion blur, 8 samples
...
if fast:
    sc.eevee.taa_render_samples = samples or 8
    if hasattr(sc.eevee, "use_raytracing"):
        sc.eevee.use_raytracing = False
    sc.render.use_motion_blur = False

At 360p that took a frame from about 0.6 s to about 0.33 s.

Roughly 2x.

The full 58 second film is 1,760 frames, so a review render went from around 18 minutes to around 10.

What you give up is real but harmless.

The street looks flatter without the ray-traced reflections, and fast motion looks crisper without blur.

Nobody reviewing "does Miles land on the ledge" cares about either.

The finals still get everything.

A few other levers came along for the ride:

A 270p option (480x270) for the quickest "is the blocking right" passes.
Contact sheets. A strip of stills at 25% resolution and 8 samples, before committing to any video at all.
Partial re-renders. 05_render.py -- full 33 <from_frame> re-renders only from a given frame onward and reuses the earlier frames, for when only the ending changed.

The lesson is the same one you learn profiling a slow API.

Before you ask for a bigger machine, find out what the work actually is.

Most of the time you are paying for something nobody looks at.

"How do I go back? Is there a snapshot button?"

Blender has more undo than you would expect, in layers:

Ctrl+Z and Ctrl+Shift+Z for in-session undo and redo, plus Edit → Undo History to jump back several steps at once.
File → Revert to throw away unsaved changes and reload from disk.
.blend1 backups. Every save keeps the previous version next to the file, and Preferences → Save & Load → Save Versions sets how many (docs).
File → Recover → Auto Save for when Blender crashes, which it will.

All useful, and all beside the point here.

Because when the .blend is generated, it is not the source of truth.

The scripts are.

So the real snapshot button is git:

git log --oneline -- miles/scripts/04_street.py    # pick a commit
git stash                                          # park current work
git checkout <commit> -- miles/                    # that version's scripts and assets
blender -b build/character.blend --python scripts/04_street.py   # rebuild that scene
git checkout HEAD -- miles/ && git stash pop       # and come back

If the scene is generated from code, version the code, not the binary.

I commit after every review round now, so any version I ever showed someone can be rebuilt exactly.

The old rendered videos stick around too (street_v1, v4a and friends), because sometimes you just want to play two versions side by side without rebuilding anything.

A reader comment that found a real bug

Part one got a comment from @lewisywliu that was genuinely better than some docs I have read.

Two tips, paraphrased:

BVH files define their own euler rotation order per joint, and the importer does not always match it, so wrists and feet can end up subtly wrong even when the hips look fine. If a pose reads almost right but the hands are off, check euler order before touching offsets.

Consecutive clips with different hip heights, like land to idle or crouch to stand, need the vertical offset blended too, otherwise the character floats or sinks for a few frames while the horizontal path snaps correctly.

The first one turned out to be a near miss.

The second one was a bug I already had and had not seen.

Tip 1: euler order, or why the hands go wrong first

A BVH file, which is what the CMU mocap library ships as, stores each joint's rotation as three angles.

But it also stores the order those three rotations are applied in, and that order is per joint.

One joint can say Z then X then Y, the next can say Y then X then Z.

That matters because rotations do not commute.

Rotate 30 degrees around Z and then 45 around X, and you end up facing somewhere different from 45 around X and then 30 around Z.

Same numbers, different pose.

Now put that in a skeleton.

Every bone inherits its parent's rotation, so a small error at the shoulder moves the elbow, which moves the wrist, which moves the hand.

The error compounds down the chain.

The hips are at the top of the tree, so they look perfect.

The hands are at the bottom, so they are the first thing that looks "almost right but weird".

I checked the importer call, half expecting the worst.

It already passed rotate_mode="NATIVE" to bpy.ops.import_anim.bvh, which means "use each joint's own order from the file".

So we were safe.

But it is now the first thing on my list if a CMU clip ever shows odd wrists or ankles, before anyone reaches for an offset hack.

Tip 2: hip height across a blend, which bit me

Quick refresher on how the clips are chained.

Each clip is mostly "in place", so to make Miles actually travel, each clip's movement is added on top of wherever the last one ended.

A running sum.

That is root motion, the same idea game engines use.

Everyone remembers to blend the horizontal part of that path.

The second tip was that the vertical part needs blending too.

(Game engines usually call that Y. Blender is Z-up, because of course it is.)

On flat ground I never saw a problem.

On raised surfaces I had exactly this bug, and I only found it once I measured it.

The probe was simple: on every frame through each blend, take the gap between the lowest toe and the surface under him.

The target is about +5 cm, because the toe bone sits a little above the sole of the shoe.

Blend, on a raised surface	Before	After
Crouch → web shot, on a board's top ledge	sinks ~10 cm for 3 frames	+0.03 to +0.07 m
Crouch → moonwalk, on a billboard catwalk	pops up 22 cm, then settles	+0.02 to +0.10 m
Crouch → taunt, on a rooftop	drifts up 10 cm	+0.04 to +0.07 m

A 22 cm pop is a Spider-Man who briefly levitates for no reason.

Which, to be fair, is on brand, but not what I wanted.

The first fix did exactly what the tip says, and it did nothing.

I eased the root height smoothly across the blend window instead of letting it step.

The numbers barely moved.

The measurement showed why.

The hips were not the problem.

The crossfaded pose was.

Halfway between a crouch and a standing pose, the blended legs are half extended, so the feet rise on their own, no matter how gently the hips move.

A smooth curve for the hip height cannot follow what the legs are doing.

What worked was to stop predicting and start measuring.

For every frame in a raised shot, find the lowest toe, and move the root so that toe sits a fixed distance above the surface.

Then smooth it lightly so it does not jitter.

This is the real function, trimmed a little:

def contact(self, start, end, floor, toe=0.05, smooth=2, limit=0.35):
    """Keep the lowest toe `toe` above a surface at height `floor` for every frame."""
    frames = list(range(int(start), int(end) + 1))
    fc = anim.fcurve(self.root, "location", 2)
    base, low = [], []
    for f in frames:
        self.scene.frame_set(f)
        base.append(fc.evaluate(f) if fc else self.root.location.z)
        low.append(min((self.rig.matrix_world @ self.rig.pose.bones[b].head).z
                       for b in ("mixamorig:LeftToeBase", "mixamorig:RightToeBase")))
    corr = [max(-limit, min(limit, floor + toe - z)) for z in low]   # a hop in the clip stays a hop
    sm = [sum(corr[max(0, i - smooth):i + smooth + 1]) / len(corr[max(0, i - smooth):i + smooth + 1])
          for i in range(len(corr))]                                   # average over ±2 frames
    if fc:                                     # remove back to front: indices stay valid
        idx = [i for i, kp in enumerate(fc.keyframe_points) if start <= kp.co[0] <= end]
        for i in reversed(idx):
            fc.keyframe_points.remove(fc.keyframe_points[i])
    for f, b, c in zip(frames, base, sm):
        anim.key(self.root, "location", f, b + c, 2, "lin")          # a key on EVERY frame

That limit=0.35 is load bearing.

Without a cap, one clip that genuinely hops at the end got "corrected" by 8 metres.

The function saw feet in the air, decided the floor must be way off, and dragged the whole character to meet them.

The ±35 cm clamp means a real hop stays a hop.

And yes, that "remove back to front" comment is a scar.

Which brings us to the bugs.

Three bugs the measurement dug up

Once there was a probe printing a number per frame, the number started pointing at things I would never have seen in a viewport.

1. Miles, 18 metres up, for exactly one frame

An earlier floor-lock pass keyed the root height on every other frame, with linear interpolation in between.

That is a perfectly reasonable thing to do inside one shot.

It is a terrible thing to do across a hard cut.

At a cut from a rooftop to the street, the frame between two keys landed halfway between two shots.

Linear interpolation averaged "rooftop" and "street" and put Miles about 18 m up, in the sky above the street, for one frame.

At 30 fps that is a 33 millisecond glitch.

You would never catch it playing the video.

The probe flagged it instantly.

The fix: key every frame inside these ranges, so there is no in-between frame left to interpolate.

2. Hard cuts on fractional frames

Clips are scheduled in float frames, because blend lengths and speeds are computed.

So a "hard cut" could start at frame 1037.4.

Blender renders whole frames, which meant frame 1037 showed the old pose at the new position.

Half of one shot glued to half of another.

The fix: hard cuts snap to a whole frame with math.ceil, and a shot's last frame is ceil(end) - 1.

This is the animation version of an off-by-one, and it was hiding inside a float.

3. Deleting keyframes while iterating over them

This one is a classic, and I still walked straight into it.

The code collected references to the keyframes it wanted to replace, then looped over them calling keyframe_points.remove(kp, fast=True).

But removing a keyframe shifts every index after it.

So the next reference in the list now pointed at its neighbour, and the loop silently deleted the wrong key.

The first frame of two shots ended up with no height key at all.

It is the same bug as removing items from a Python list inside for x in items:, just wearing a 3D costume.

The fix is the one in the snippet above: collect indices, then remove from the back.

The whole lesson of this section is not about any one of those bugs.

It is that the first fix looked right in theory and did nothing in practice, and only a number per frame could tell me that.

Do not eyeball root motion. Measure it.

The probe became a test suite

Once the foot probe existed, it seemed silly to only run it when I suspected a problem.

So it became scripts/check_support.py, a permanent pre-render check.

It is about 40 lines and it answers one question, sampling every third frame of the film: is there something under his feet?

flowchart TD
    F[Next frame] --> A{airborne clip or upside down?}
    A -->|yes, swing, hang, flip| F
    A -->|no| T[Find the lowest toe]
    T --> R[Ray cast straight down]
    R --> M{hit Miles, a web, a spider or cash?}
    M -->|yes, step past it| R
    M -->|no| G{gap over 25 cm or sinking?}
    G -->|no| F
    G -->|yes| B[Flag the frame]
    B --> F

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef step fill:#6ea8ff,stroke:#2f62c4,color:#1a1a1a
    classDef bad fill:#ff9a5c,stroke:#c4602a,color:#1a1a1a
    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a

    class A,M,G decision
    class T,R step
    class B bad
    class F start

It uses Blender's scene.ray_cast, ignores anything that is part of the character or the props (a web is not a floor, sadly), skips the moments where he is supposed to be in the air, and groups flagged frames into runs so the output reads like a test report.

I run it before every render, the same way you would not deploy without running the tests.

And it caught things a visual check flat out missed.

Moonwalking on thin air.

The opening of this post.

Miles moonwalked 6.5 m past the edge of a billboard's frame, and, for bonus points, in the opposite direction to the customers walking off the screen behind him.

The fix had two parts.

A steel catwalk under the screen, because real billboards have one for maintenance anyway.

And the moonwalk now goes with the customers, which is a better joke.

A 40 cm stance on a 25 cm board.

He stood on top of a wall board whose top was 25 cm deep.

His stance was 40 cm.

One foot was hanging off the front, standing on air.

The fix was a 1.25 m service ledge on top of the board.

A turn that swung him 2 m off a ledge.

This was my favourite, because the cause was pure geometry.

In-place clips leave the rig's root pivot nowhere near the hips.

In this case it was about 4 m away.

So when a blend rotated the root to turn him 150 degrees, his whole body orbited a point 4 m away, like a door swinging on a hinge in the next room.

Mid-turn, his hips were 2 m off the ledge.

The fix was to key the root per frame during a turning blend, so the hips travel in a straight line while the body rotates around them.

A turn is a rotation about some point.

It is worth checking which one.

Small workflow wins

None of these deserve a section, but all of them saved real time.

Burned-in review copies.

Every render also writes a <name>_tc.mp4 with the Blender frame number and "time / total" stamped in the corner.

When someone says "f1201 looks off", that maps straight to a frame in the scene.

No more "uh, around the middle, after the bus thing".

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Risk-Aware AI Code Review for Business-Critical Systems

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

LiveReview is built for engineering teams that ship often, but also care about keeping production stable for customers. It approves some 40-50% of low-risk changes automatically, and it gives your team the tools to handle the rest:

Understand risky changes faster.

Fix problems in those changes, with human attention directed to the work that needs it.

Enforce your engineering standards.

Learn from production outcomes, such as incidents, rollbacks, and queue wait times.

Improve the review process over time and make it more efficient.

Connect to the major engineering tools that your team already uses.

Auto-Approve: Ship the Safe 50% Without a Human

Auto-Approve Upto 50% of Your Pull Requests, Safely. Risk scoring decides which MRs/PRs are safe enough to ship without a human, so your team only spends review time on risky and critical changes.
demo.mp4
LiveReview auto-approving a low-risk change while…

View on GitHub

Click below to try LiveReview with your codebase:

I Made Spider-Man Swing Without Animating a Single Frame

Athreya aka Maneshwar — Sat, 03 Oct 2026 18:05:25 +0000

Yes, that is Miles Morales.

He drops in on a web, lands a superhero crouch, fires a web off screen, and yanks a giant DEV board into frame.

I did not animate a single frame of that by hand.

I also had never opened Blender before this week.

This post is the story of how that happened, and the surprising amount of computer science hiding inside a "simple" 3D character.

Why Blender, of all things

I was trying to make a video.

Not a screen recording, an actual video with something moving in it.

My first stop was HyperFrames, which lets you write a video as HTML and render it out.

It is genuinely great at motion graphics.

Text that slides in, logos that pop, charts that draw themselves, all of that is lovely.

But the moment I wanted a character to do something, it hit a wall.

HTML can move rectangles around beautifully.

It cannot make a guy swing on a web.

So the next question was obvious, if a little scary: what do actual animators use?

The answer, for free and open source, is Blender.

And Blender has an MCP server, which meant I would not be learning it alone.

The MCP that drove the whole thing

The thing that made this possible for a total beginner is blender-mcp.

It connects Blender to an LLM client like Claude Code, so instead of hunting through 400 menus, you say "load this model and relink its textures" and it happens.

The architecture is worth understanding, because it is simpler than it sounds.

flowchart LR
    ME[Me typing a prompt] --> CC[Claude Code]
    CC -->|MCP over stdio| SRV[blender-mcp server]
    SRV -->|JSON over TCP socket| ADDON[Blender add-on]
    ADDON -->|runs Python| BPY[bpy API]
    BPY --> SCENE[Live Blender scene]
    SCENE -->|viewport screenshot| CC

    classDef human fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef agent fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef bridge fill:#6ea8ff,stroke:#2f62c4,color:#1a1a1a
    classDef blender fill:#ff9a5c,stroke:#c4602a,color:#1a1a1a

    class ME human
    class CC agent
    class SRV,ADDON bridge
    class BPY,SCENE blender

There are two halves.

An add-on inside Blender opens a small socket server and waits for commands.

An MCP server outside Blender speaks the Model Context Protocol to the LLM on one side, and forwards JSON commands down that socket on the other.

The most powerful tool it exposes is execute_blender_code, which runs arbitrary Python inside Blender through bpy, Blender's Python API.

Every button in Blender is just a thin skin over bpy.

So an agent that can write bpy can, in principle, press every button.

The loop that makes it work is the screenshot tool coming back the other way.

The agent changes something, grabs the viewport, looks at it, and decides what to fix next.

That is the same edit, run, inspect loop you use on code, just pointed at a 3D scene.

One honest warning: that tool is arbitrary code execution by design. Treat it like you would treat any local automation with a shell, and do not point it at a machine full of secrets.

Step one: get a character

I am not a 3D modeler, and I was not about to become one by Friday.

So I grabbed a ready made Miles Morales model, the Spider-Verse Spider-Man, from Sketchfab.

Even "just load it" had lessons in it.

The file came with four copies of Miles standing in a row: masked and unmasked, posed and T-posed.

The textures pointed at the original author's Windows folders, like C:\Users\someone\Desktop\..., so the agent had to relink them to the files in the zip.

And the zip was missing the normal and specular maps.

In Blender, a texture node pointing at a missing file renders hot pink.

It is the 3D equivalent of a 404, and it is very, very visible.

The fix was to disconnect those slots, at the cost of some surface detail on the suit.

Then I opened the unmasked Miles and his face looked like a horror movie, covered in white lines and dots.

Turns out the face was fine.

That model ships with a 143 bone face rig (eyelids, brows, lips, jaw, cheeks, even a tongue), and the armature was set to "draw in front", so every one of those bones was painted on top of his skin.

One checkbox later, he was handsome again.

The concept that unlocks everything: rigging

Here is where I learned the word that the rest of this post hangs on.

A 3D character is, at its core, a mesh: a big list of vertices (Miles has about 27k faces) glued into triangles.

A mesh is a statue.

You cannot make a statue wave.

To move it, you need three more things.

An armature, which is a skeleton.

It is a tree of bones, each one a parent of the next: hips to spine to chest to shoulder to upper arm to forearm to hand to fingers.

Rotate the shoulder and everything below it in the tree comes along for the ride, exactly like a transform hierarchy in any game engine or a nested <div> with a CSS transform.

Skin weights, which bind the mesh to the skeleton.

Every vertex stores a small table saying how much each bone influences it.

A vertex on the forearm might be 100% forearm.

A vertex right at the elbow might be 50% upper_arm and 50% forearm, which is what lets the elbow bend smoothly instead of folding like a paper straw.

The math is called linear blend skinning, and it is just a weighted average of matrices:

final_position(v) = Σ  weight(v, bone) × bone_matrix × rest_position(v)
                   bones

A rest pose, which is the reference everything is measured from.

That is why every character you download stands in a T-pose, arms straight out.

It is not a dance move.

It is the zero point, the git init of the body, and every animation is stored as rotations relative to it.

Put those together and you have a rig.

And then the naive plan fell apart, because the model's rig had bones but no controls.

Forward vs inverse kinematics

I had seen YouTube videos where an animator grabs a character's hips, drags them down, and the knees bend while the feet stay glued to the floor.

When I dragged Miles's hips, his whole body floated down through the floor like a ghost.

The difference is one of the nicest ideas in this whole field.

Forward kinematics (FK) is what a raw skeleton does.

You set the angle of every joint, and the hand ends up wherever the chain of rotations puts it.

Easy to compute, miserable to animate. Want the hand on the doorknob? Good luck guessing the shoulder, elbow and wrist angles that land it there.

Inverse kinematics (IK) flips the question.

You say where the hand (or foot) should be, and a solver works backwards to find joint angles that put it there.

It is literally an optimisation problem solved every frame, and Blender ships an IK constraint that does it for you.

Pin the feet with IK targets, drag the hips down, and the solver has to bend the knees because the feet are not allowed to move.

That is the magic in those YouTube videos.

It is not magic, it is a constraint solver.

Building a full IK control rig by hand is a real skill though.

And I wanted Miles doing backflips this week, not next quarter.

Mixamo: rigging as a service

Enter Mixamo, Adobe's free auto-rigger and animation library.

The flow is: export your character as an FBX in T-pose, upload it, and Mixamo builds a skeleton and skin weights for you.

The only thing it asks of you is to place a few markers.

Chin, wrists, elbows, knees, groin.

Five landmarks, with symmetry on, so really just three clicks per side.

From those it infers the whole skeleton, which is a nice reminder of how much structure a human body has. If you know where the joints are, the rest is mostly proportions.

Placement matters more than you would think.

My first pass had the wrists sitting on the forearms, which would have made his hands bend in the wrong place every time he shot a web.

I picked Standard Skeleton (65), which includes finger bones, because a two handed pull without fingers looks like a man wearing oven mitts.

And then it just... worked.

The rigged character goes straight into Mixamo's library of motion captured animations.

These are real performers in mocap suits, recorded and cleaned up, so they have the weight shifts and tiny wobbles that hand keyed animation struggles to fake.

Jumps, climbs, hangs, throws, dances, falls, the lot.

There was one trade off, and it hurt a little.

Mixamo builds its own body skeleton.

It does not know or care about the beautiful 143 bone face rig the model came with.

"With skin" and "without skin"

When you download an animation, Mixamo asks whether you want it with skin or without skin.

This is a good little lesson in separating data from presentation.

With skin gives you the mesh, the skeleton and the motion.

Without skin gives you only the skeleton and the motion, a few hundred kilobytes of bone rotations over time.

Because every clip targets the same 65 bone skeleton with the same bone names, you only need the skin once.

Everything else is just keyframe data keyed by bone name, and Blender happily applies it to any armature whose bones match.

So I downloaded the first clip with skin and the rest without.

And I downloaded a lot of them.

Seventy eight files.

Clicked one at a time, because Mixamo has no multi select.

Some of them even share a name with a different animation inside, because the site just appends (1) to whatever you downloaded last.

They are not the same picture.

The agent ended up scanning every FBX to see what was actually inside before trusting a single filename.

From a pile of clips to a performance

Here is the whole pipeline, end to end:

flowchart TD
    A[Sketchfab model] --> B[Clean up in Blender<br/>relink textures, drop pink slots]
    B --> C[Export T-pose FBX]
    C --> D[Mixamo auto-rig<br/>5 markers]
    D --> E[Download clips<br/>1 with skin, rest without]
    E --> F[Import as Blender actions]
    F --> G{Clip looks right<br/>from this angle?}
    G -->|no| E
    G -->|yes| H[Chain clips on NLA<br/>crossfades + root motion]
    H --> I[Webs, board, camera]
    I --> J[360p preview render]
    J --> K{Good enough?}
    K -->|no| H
    K -->|yes| L[Final render]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef source fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef rig fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef scene fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef out fill:#ff9a5c,stroke:#c4602a,color:#1a1a1a

    class G,K decision
    class A,B,C source
    class D,E,F rig
    class H,I scene
    class J,L out

Each Mixamo clip lands in Blender as an action, a named bundle of animation curves (an f-curve per bone per rotation channel).

On its own, an action is a single move.

A performance is several of them in a row, and that is what Blender's Nonlinear Animation editor is for.

The NLA is basically a video editor timeline for animation.

Each action becomes a strip, strips sit on tracks, and overlapping strips can blend into each other.

That blend is what stops a cut from "hanging" to "landing" from snapping like a broken GIF.

The agent wrapped all of this into a tiny sequencer, so the shot is described as a list of moves instead of a pile of frame numbers:

perf = Performer(rig)
perf.then("Hanging Idle", length=40)
perf.then("Hard Landing", to=50, speed=1.3, blend=6)
perf.then("Standing 1H Magic Attack 02", frm=16, to=40, speed=1.25, blend=6)
perf.then("Pull Heavy Object Stop", to=30, speed=1.1, blend=5, face=65)
perf.then("Victory Idle", blend=8, face=10, length=51)
perf.build()   # NLA strips, crossfades, stitched root motion

blend=6 is a six frame crossfade into the next clip.

face=65 turns him 65 degrees so the yank points at the board.

And the "web shot" is, I promise, a clip literally named Magic Attack. Mixamo does not have a "thwip".

The interesting problem hiding in build() is root motion.

Some clips move the character through space (a landing drops him, a pull leans him back), and each clip starts from its own origin.

Play two of them back to back naively and Miles teleports back to the start between moves.

So the sequencer reads where the hips ended in one clip and offsets the next one to begin there.

It is the same idea as stitching GPS tracks together: each segment is relative, so you keep a running sum.

Because all timing derives from that list, swapping one clip re-times the webs, the board and the camera automatically.

The web strands, the board and the camera are not Mixamo at all, they are plain Blender objects keyframed by script to fire on the frame where his hand reaches out.

Iterate at 360p, like a sane person

The first full render was 16 seconds at 720p and took about eight minutes on my GTX 1650.

That is fine once.

It is not fine when the feedback is "make it snappier", "make the board bigger" and "his suit is too dark" in the same breath.

So previews dropped to 360p, roughly 4x faster, and 720p became a finals only thing.

The same rule you know from code applies: make the feedback loop fast first, then make the output good.

Two more things that cost real time:

The suit rendered as a flat black blob.

The model used a toon shader that collapses lighting into a couple of hard bands, which looks great in a cartoon and terrible when you want to see muscles.

Swapping in physically based shading, plus a light just for him and red and blue rim lights, brought his shape back.

Light grey or dark studio?

A near black suit on a near black background is a silhouette of nothing.

Lighter backgrounds read better. Dark backgrounds look cooler. The GIF at the top is the dark one, because I have priorities.

Escaping the download button

After clicking "Download" 78 times, I asked the obvious engineer question: can I just do this locally?

Mixamo's rigger and library only live on Adobe's servers.

But motion capture data is not an Adobe invention.

The CMU Graphics Lab Motion Capture Database has about 2,500 free clips, and there is a BVH mirror on GitHub with a text index of what each one is.

Cartwheels, salsa, climbing, martial arts, "pulling a rope", all of it.

The catch is that CMU's skeleton is not Mixamo's skeleton.

Moving motion from one skeleton to another is called retargeting, and it is a lovely little linear algebra problem.

You cannot just copy rotations across, because the two skeletons have different rest poses. A 30 degree rotation from "arms slightly down" is not the same pose as 30 degrees from "arms straight out".

So the retargeter converts each bone's rotation out of the source rest pose, into a shared world frame, and back into the target's rest pose.

It also scales the hip travel by leg length, so a tall performer's stride does not make Miles moonwalk.

The first try rendered... nothing.

Miles was not in frame. He was not anywhere.

The BVH rest pose put the hips exactly at the origin, so "rest hip height" was zero, and the scale factor was a division by it.

Miles had been launched into orbit by a divide by zero.

Fixed by measuring the floor from the lowest foot in the clip instead, and suddenly he was walking, dancing and cartwheeling from a dataset recorded at Carnegie Mellon years ago.

What I actually learned

I set out to make a guy swing on a web.

I came back with a list of concepts I did not expect from an art tool:

A character is a mesh deformed by a tree of bones, via per vertex weights.
Animation is rotations relative to a rest pose, which is why everything T-poses.
IK is a constraint solver, and it is the difference between a puppet and a corpse.
Motion clips are data keyed by bone name, so one skin can wear a thousand animations.
NLA strips are a timeline of reusable actions, and chaining them means stitching root motion.
Retargeting is a change of basis, and it will divide by zero if you let it.
Blender is a Python program with a UI on top, which is why an MCP bridge can drive it at all.

There is a ton more in there for a CS person: shader node graphs are dataflow programs, Cycles is a path tracer, physics sims are numerical integrators, geometry nodes are basically a functional language for meshes.

But one step at a time.

Next on the list is the face rig I so cruelly abandoned in that pool, and maybe making Miles actually talk.

If you have played with Blender, rigging or the MCP, tell me what I should pick up next.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by tests…

View on GitHub

Click below to try LiveReview with your codebase:

What Happens When Two Databases Both Say Yes

Athreya aka Maneshwar — Thu, 01 Oct 2026 19:22:55 +0000

Your database has one primary and two replicas, and for about two years that is the best decision you ever made.

Reads scale out across the replicas. Writes all go to the one primary.

There is exactly one copy of the truth, and it lives on a machine you could point at.

Then sales closes a deal in Sydney.

Physics sends you an invoice

Your primary is in Virginia. Your new users are in Sydney.

Every write from Sydney now has to cross the Pacific, get acknowledged, and come back.

That is roughly a 200 millisecond round trip, and no amount of clever indexing touches it, because the delay is not your database being slow.

It is light being inconveniently finite.

Two hundred milliseconds does not sound fatal until you notice how many writes hide inside one user action.

A session token. An analytics event. A "last seen" bump. Suddenly a page that feels instant in Boston feels like dialup in Sydney.

And then there is the other problem, the one that shows up on a Saturday.

If us-east-1 has a bad day, nobody writes anything. Not Sydney, not Boston, not the replicas sitting right there with a full copy of the data.

They have the bytes. They are just not allowed to touch them.

One leader means one place to be slow and one place to fail.

So you give each region its own leader

This is multi-leader replication, and the idea is exactly as blunt as it sounds.

Put a leader in each region. Let each one accept writes from the users nearest to it. Then have the leaders ship their changes to each other in the background.

Sydney writes land in Sydney at 8 milliseconds. Virginia writes land in Virginia at 8 milliseconds.

A few seconds later, each change has made its way to the other side of the world and both leaders hold the same data.

You just bought two genuinely good things.

Write latency collapses for everyone who is not sitting next to your original region. Writes are now local.

Losing a region is survivable. Sydney goes dark, Virginia keeps taking writes, and when Sydney comes back it catches up from the log.

Your write availability is no longer the uptime of a single building.

This is the same shape as single-leader replication, just with the "single" removed. Which is a sentence that should make you slightly nervous, and it should.

Because the thing that made single-leader so pleasant to reason about was never the leader. It was the one.

The bill arrives as a row number

Alan is in Sydney. He opens row 1,000,000 and sets x = 7.

At more or less the same moment, John is in Virginia. He opens the same row and sets x = 9.

Sydney accepts Alan's write and returns a 200. Virginia accepts John's write and returns a 200.

Neither leader hesitated, because at the moment each of them decided, the other write did not exist yet as far as they could tell.

Now the replication streams cross in the middle of the ocean, and both leaders simultaneously learn that they were lied to.

So what is x?

There is no answer to that question hiding in the data. The database has already promised both values to two different humans.

Somebody is about to be wrong and nobody has told them yet.

This is the problem with multi-leader replication. Not a problem. The problem. Everything else in this post is a response to it.

And notice what single-leader gave you for free here.

With one primary, these two writes would have arrived at the same machine, one would have gone second, and it would have overwritten the first while knowing it was overwriting something. A SELECT in between would have seen a consistent story.

That is not a feature anybody shipped. It is a side effect of there being one queue.

"At the same time" is the wrong mental model

It is worth being precise here, because this is where people's intuition quietly breaks.

Alan and John did not have to write at the same instant for this to happen. They could have been forty seconds apart.

Two writes are concurrent if neither one knew about the other. That is the entire definition.

It has nothing to do with wall clock time and everything to do with what information had reached whom.

Leslie Lamport worked this out in 1978 in Time, Clocks, and the Ordering of Events in a Distributed System, and it is still the single most useful thing you can read on the subject.

If Alan's write had reached Virginia before John started typing, John's write would be a normal overwrite. Not a conflict, just an edit.

Replication lag is what turns an edit into a conflict.

Which means your conflict rate is not a property of your users. It is a property of how long your replication takes, multiplied by how much your users overlap.

The default answer is "last write wins", and it is worse than it sounds

Most systems that offer multi-leader replication ship with a default conflict resolution policy, and that default is almost always last write wins.

DynamoDB global tables do it. Cassandra does it. It is the policy you get when nobody chose a policy.

Each write carries a timestamp. When two versions of a row collide, the one with the bigger timestamp survives. The other one is discarded.

Not flagged. Not queued for review. Discarded.

Here is the part that gets people. LWW is a guaranteed data loss mechanism, and that is not a criticism, that is its job description.

If two writes conflict and you keep exactly one, you have deliberately thrown away a write that the database told a user it had accepted.

Sometimes that is completely fine. If the field is "last known GPS position", dropping the older one is correct and you should feel nothing.

If the field is "account balance", you have just invented a very exciting kind of bug.

And then there is the detail that turns a tolerable policy into a haunted one.

Whose clock?

LWW compares timestamps generated by two different machines, and those machines do not agree on what time it is.

NTP keeps them roughly in sync, where "roughly" means tens of milliseconds on a good day and considerably worse on a VM whose host is under load or whose clock just stepped.

So if Sydney's clock has drifted 120 milliseconds ahead, Alan's earlier write gets stamped with a later timestamp.

LWW dutifully keeps it and throws away John's write, which actually happened second.

The system did exactly what it was configured to do. It also reverted a value, told the user it had succeeded, and left no trace anywhere that it happened.

This is the reason Google built TrueTime for Spanner, with GPS receivers and atomic clocks in every datacenter, and then had Spanner deliberately wait out the clock uncertainty interval before committing.

They did not solve clock skew. They bounded it, paid for the bound in hardware, and then paid for it again in commit latency.

If you are doing LWW on timestamps from now() across regions, you are making the same correctness claim Spanner makes, with none of the equipment.

What you can do instead of picking a loser

The good news is that LWW is a default, not a law.

Once you accept that conflicts are going to happen, you get to choose what happens next, and there are better options than a coin flip weighted by NTP drift.

Keep both and let the application decide. CouchDB does this properly: a conflicting document keeps every version as a sibling, picks one deterministically so reads do not break, and leaves the rest sitting there for you to resolve.

Riak called these siblings too. It is the most honest option, and it is the one that forces you to actually write resolution logic, which is why people avoid it.

Write a merge function. For a lot of real data, "pick a winner" is just the wrong operation. Two users adding different items to a shared cart is a union, not a race.

Two users editing different fields of the same profile row is two independent edits that a column-level merge handles without anyone losing anything.

Change the data type so conflicts cannot exist. This is the good one.

Replicate the operation, not the value

Here is the move that makes a whole class of conflicts evaporate.

A conflict happens because two leaders made competing claims about what a value is. Sydney said the cart has 4 items.

Virginia said the cart has 4 items. They both read 3 and both added 1.

Last write wins picks one of those two 4s, and the answer is 4. The customer added two things and the cart grew by one.

Now watch what happens if the leaders stop shipping the result and start shipping the operation.

Sydney does not say "the cart is 4". It says "Sydney has contributed 1". Virginia says "Virginia has contributed 1".

The value of the cart is the sum of everyone's contributions, and both leaders compute 5 without ever disagreeing about anything.

Nothing had to win, because the two writes were never competing claims in the first place.

That is a CRDT, and the counter version of it is so simple you can build one in plain SQL with no special database at all:

-- one row per (cart, replica). each leader only ever touches its own row,
-- so two leaders can never write the same row and never conflict.
CREATE TABLE cart_counter (
  cart_id   bigint  NOT NULL,
  replica   text    NOT NULL,   -- 'syd' | 'iad' | 'fra'
  delta     bigint  NOT NULL DEFAULT 0,
  PRIMARY KEY (cart_id, replica)
);

-- Sydney's leader, adding an item. it writes replica = 'syd', always.
INSERT INTO cart_counter (cart_id, replica, delta)
VALUES (1841, 'syd', 1)
ON CONFLICT (cart_id, replica)
DO UPDATE SET delta = cart_counter.delta + 1;

-- the actual value, which every region computes identically
SELECT sum(delta) FROM cart_counter WHERE cart_id = 1841;

The write set of each leader is disjoint by construction. Replication can arrive in any order, duplicated, delayed by a week, and every region still converges on the same number.

This is the same principle behind Automerge and Yjs, which is how collaborative editors let twelve people type into one paragraph without a central referee.

And it is worth noticing that a Google Doc or a Figma file is a multi-leader system. Every laptop is a leader.

The offline mode you shipped last quarter made your mobile app one too.

The catch is that CRDTs are not free. They carry per-replica metadata, which grows, and they are only convenient for data types somebody has already worked out the merge semantics for: counters, sets, registers, sequences, maps.

The moment your business rule is "the total must never exceed the credit limit", you are back to needing coordination, because that is a global invariant and CRDTs specifically do not give you those.

The conflicts nobody warns you about

So far this has all been about one row, one column. The conflicts that actually ruin a migration weekend are the structural ones.

Auto-increment primary keys. Two leaders, both confidently handing out id 4817 to different rows. The fix is for each leader to generate ids from a disjoint space: offset sequences, UUIDs, or a Snowflake-style id with the region baked into the bits. If you are turning on a second leader and have not solved this, solve it first.

Unique constraints. Two users in two regions both register alan@example.com. Each leader checks its local table, sees nothing, and allows it.

The constraint held locally and is violated globally, which is a fun error to discover during replication rather than during signup. Uniqueness is a global invariant and asynchronous replication fundamentally cannot enforce one.

Foreign keys and deletes. Sydney deletes a parent row. Virginia inserts a child pointing at it. Both are valid locally.

Together they produce an orphan, and now your replication stream is stuck trying to apply something your schema refuses.

Triggers and defaults that call now(). Any logic that is not a pure function of the row will produce different results on different leaders. Replicate the computed value, not the computation.

Notice the pattern. Everything on that list is a constraint that spans rows your leaders do not coordinate on.

Single-leader hid all of them behind one serialisation point, for free, and you never knew they were being handled.

The strategy that actually ships

Here is what teams running this in production actually do, and it is less glamorous than any of the above.

They make the conflict impossible.

Give every record exactly one home leader that owns its writes. All writes for a given customer, tenant, or account are routed to that customer's home region, no matter where the request physically landed.

Every leader still replicates everything to everywhere, so reads stay local and failover stays possible.

Now the two leaders have disjoint write sets. Sydney writes rows Virginia never writes. There is no conflict to resolve because the two leaders are never making claims about the same data.

Conflict resolution logic you never run is the only kind that never has a bug.

This is why multi-leader works beautifully when your data has a natural ownership boundary. Per-customer SaaS. Per-tenant workspaces. Per-region marketplaces.

It works much less beautifully for a global leaderboard or a shared inventory count, where every write touches data everybody else touches too.

The routing itself is usually unexciting, which is the point:

def route_write(account_id, request_region):
    home = accounts.home_region(account_id)   # cached, changes rarely

    if home == request_region:
        return local_leader()                 # the fast path, ~8ms

    # cross-region write: slower, but still exactly one leader owns this row,
    # so it is an ordinary overwrite rather than a conflict waiting to converge
    return leader_in(home)

Note what that else branch is buying you. A European user acting on a US-homed account pays the 100 millisecond hop.

You have traded some users' write latency for never having to explain to your CFO why a balance went backwards.

That is usually the right trade, and it is a trade, not a free lunch.

flowchart TD
    A[Write arrives in Frankfurt] --> B{Which region<br/>owns this account?}
    B -->|Frankfurt| C[Local leader, 8 ms]
    B -->|Virginia| D[Forward to Virginia, 100 ms]
    C --> E[Commit, ack the user]
    D --> E
    E --> F[Async replication to every other leader]
    F --> G[All regions can serve reads<br/>and take over on failover]
    B -->|Home region is down| H{Can you wait<br/>for it to come back?}
    H -->|Yes| I[Reject the write, stay conflict-free]
    H -->|No| J[Reassign the home, accept<br/>that you may now conflict]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef good fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef warn fill:#ff9a5c,stroke:#c25f20,color:#1a1a1a

    class A start
    class B,H decision
    class C,D,E,F,G good
    class I,J warn

The honest caveat: homes move

Ownership routing is clean right up until the moment a region fails, which is the exact scenario you built multi-leader for in the first place.

Sydney goes down. Its accounts have no home.

You either reject their writes, which defeats the point, or you reassign their home to Virginia, which means accepting writes for rows Sydney might also have accepted writes for in the seconds before it died but not yet replicated.

That window is where your conflicts live. Not during normal operation, during failover.

So the real design is: disjoint write sets almost always, plus a conflict resolution policy you have genuinely thought about for the minutes per year when they are not disjoint.

Teams that skip the second half discover they needed it during an incident, which is the worst possible time to be reading about version vectors.

Two smaller things that bite, worth knowing before they bite you.

Reading your own writes gets harder. A user writes to their home leader in Virginia, then their next request hits a read replica in Frankfurt that is 400 milliseconds behind.

Their change is gone. You fix this by pinning a user's session to one region for a while after a write, or by having the client carry the version it last saw and the read wait for it.

Replication topology matters. All-to-all is the usual choice and is the most robust, but it lets messages overtake each other, so an update can arrive before the insert it depends on.

Circular and star topologies are cheaper on connections and much worse on failure, because one dead node breaks the chain. MySQL's circular replication is famous for exactly this.

So should you turn this on?

The uncomfortable truth is that most teams reaching for multi-leader replication do not need it.

If your problem is read latency, you want read replicas, and you already have them. Multi-leader only pays off when writes are the thing crossing an ocean.

If your problem is regional failover, a single leader with a well-rehearsed promotion runbook gets you most of the way there, with a fraction of the semantics to reason about.

flowchart TD
    A[Users in multiple regions] --> B{Are WRITES the<br/>slow part?}
    B -->|No, reads are| C[Read replicas per region<br/>Keep the single leader]
    B -->|Yes| D{Does the data split<br/>cleanly by owner?}
    D -->|Yes, per tenant or account| E[Multi-leader with home regions<br/>Disjoint write sets]
    D -->|No, everyone writes everything| F{Can you model it<br/>as a CRDT?}
    F -->|Yes, counters sets text| G[CRDTs, converge without picking]
    F -->|No, you need real invariants| H[Distributed SQL<br/>Spanner, CockroachDB, YugabyteDB]
    E --> I[Still need a policy for<br/>the failover window]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef good fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef alt fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef warn fill:#ff9a5c,stroke:#c25f20,color:#1a1a1a

    class A start
    class B,D,F decision
    class C,E good
    class G,H alt
    class I warn

And if you genuinely need strong consistency across regions, that is what CockroachDB and Spanner are for.

They run consensus under the hood and charge you for it in write latency, openly, instead of handing you an asynchronous system that looks consistent until it is not.

That is the actual choice. Not "multi-leader or single-leader", but where you would like to pay.

Single-leader charges distant users on every write. Distributed SQL charges everyone a smaller amount on every write.

Multi-leader charges nobody anything on the happy path and sends you one enormous invoice on the day two regions disagree about a row.

Pick the bill you would rather receive at 3am.

The one-line version

Multi-leader replication does not create conflicts.

It reveals that your data model always had them, and that a single primary was quietly doing your reconciliation for free.

The moment you add a second leader, that work moves to you.

You can do it with a timestamp comparison and hope, or with disjoint write sets and intent.

Only one of those lets you sleep.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by tests…

View on GitHub

Click below to try LiveReview with your codebase:

Building a Custom ERP Module on the MERN Stack: Data Modeling, Transactions and Audit Trails

Athreya aka Maneshwar — Tue, 29 Sep 2026 21:25:00 +0000

ERP software handles money, stock, and approvals, so small bugs get expensive fast. A stock count that drifts by a few units or a rounding error in an invoice can take days to trace.

Off-the-shelf ERPs cover the general case well, but many companies eventually need a custom module for a workflow the big platforms handle awkwardly.

The MERN stack works well for this. You get flexible schemas for evolving business rules, one language across the whole stack, and real multi-document transactions in MongoDB.

This guide walks through a small purchasing and inventory module and focuses on 3 things: data modeling, consistent writes, and audit trails.

The Example Module

The module has suppliers, products, purchase orders (POs) with line items, and stock levels per warehouse.

The critical operation is receiving goods: the system must update PO line quantities, change the PO status, and increase stock, all at once. If only half of that happens, the numbers stop matching reality.

Data Modeling: Embed What Belongs Together

A simple rule works well for ERP data. Embed things that live and die with the parent, like PO line items.

Reference things with their own lifecycle, like suppliers and products.

const { Schema } = require('mongoose');

const lineItemSchema = new Schema({
  product: { type: Schema.Types.ObjectId, ref: 'Product', required: true },
  sku: { type: String, required: true },            // snapshot at order time
  unitPriceCents: { type: Number, required: true, min: 0 },
  quantityOrdered: { type: Number, required: true, min: 1 },
  quantityReceived: { type: Number, default: 0, min: 0 },
});

const purchaseOrderSchema = new Schema({
  number: { type: String, required: true, unique: true },
  supplier: { type: Schema.Types.ObjectId, ref: 'Supplier', required: true },
  warehouse: { type: Schema.Types.ObjectId, ref: 'Warehouse', required: true },
  status: {
    type: String,
    enum: ['draft', 'approved', 'partially_received', 'received', 'cancelled'],
    default: 'draft',
  },
  lines: [lineItemSchema],
}, { timestamps: true, optimisticConcurrency: true });

purchaseOrderSchema.index({ supplier: 1, status: 1, createdAt: -1 });

const stockLevelSchema = new Schema({
  product: { type: Schema.Types.ObjectId, ref: 'Product', required: true },
  warehouse: { type: Schema.Types.ObjectId, ref: 'Warehouse', required: true },
  onHand: { type: Number, default: 0, min: 0 },
});
stockLevelSchema.index({ product: 1, warehouse: 1 }, { unique: true });

A few choices here are worth copying:

Snapshot fixed values. SKU and price are copied into each line, so old orders keep what was actually agreed even if the product changes later.
Store money as integers. JavaScript floats can't represent 0.1 exactly. Cents avoid a whole category of rounding bugs. If you need fractional cents, use Decimal128.
Turn on optimistic concurrency. If 2 people edit the same PO at once, the second save fails instead of silently overwriting the first.
Use a unique compound index on stock. One row per product per warehouse, guaranteed by the database.

It also pays to design indexes around the questions people will actually ask.

Procurement teams usually want "open orders from supplier X, newest first", and the compound index on supplier, status, and createdAt answers that without a collection scan.

Look at the reports and list screens in your mockups before writing schemas, and add indexes for them from day one.

Transactions: Keep Related Writes Consistent

Receiving goods touches 2 collections. Without a transaction, a crash between the 2 writes leaves you with phantom inventory.

MongoDB transactions need a replica set; locally, a single-node one (mongod --replSet rs0) is enough.

The easiest API is session.withTransaction(), which commits for you and retries on transient errors:

async function receiveGoods({ poId, receipts }) {
  const session = await mongoose.startSession();
  try {
    await session.withTransaction(async () => {
      const po = await PurchaseOrder.findById(poId).session(session);
      if (!po || !['approved', 'partially_received'].includes(po.status)) {
        throw new Error('Order cannot receive goods');
      }

      for (const { lineId, quantity } of receipts) {
        const line = po.lines.id(lineId);
        if (line.quantityReceived + quantity > line.quantityOrdered) {
          throw new Error(`Over-receipt on ${line.sku}`);
        }
        line.quantityReceived += quantity;

        await StockLevel.updateOne(
          { product: line.product, warehouse: po.warehouse },
          { $inc: { onHand: quantity } },
          { upsert: true, session }
        );
      }

      const done = po.lines.every(l => l.quantityReceived === l.quantityOrdered);
      po.status = done ? 'received' : 'partially_received';
      await po.save({ session });
    });
  } finally {
    await session.endSession();
  }
}

3 rules keep this safe.

1) Pass the session to every read and write.
2) Keep side effects like emails or payment calls out of the callback, since withTransaction may run it more than once.
3) Keep transactions short to avoid lock contention.

For side effects that must happen, the outbox pattern works nicely. Inside the transaction, insert a record like { type: 'po.received', poId } into an outbox collection.

A background worker picks these up after the commit and sends the email or notifies the accounting system.

If the transaction rolls back, the outbox record disappears too, so nothing gets sent for a change that never happened.

Retry semantics and write conflicts rarely show up in tutorials, which is why companies building finance-adjacent modules often look for MERN stack developers who have already debugged these issues under real load.

Warehouse networks are also flaky, so give each receipt an idempotency key, store it under a unique index inside the same transaction, and duplicate requests will fail harmlessly.

Audit Trails: Who Changed What, and When

Auditors will ask "who approved this order?" and "why did the price change?". A Mongoose plugin combined with AsyncLocalStorage answers these automatically:

const { AsyncLocalStorage } = require('node:async_hooks');
const get = require('lodash/get');
const auditContext = new AsyncLocalStorage();

// Express middleware, placed after authentication
function withAuditContext(req, res, next) {
  auditContext.run({ userId: req.user.id, requestId: req.id }, next);
}

function auditPlugin(schema, { entity }) {
  schema.post('init', function () {
    this.$locals.before = this.toObject({ depopulate: true });
  });

  schema.pre('save', async function () {
    const ctx = auditContext.getStore() || {};
    const changes = Object.fromEntries(this.directModifiedPaths().map(p => [
      p, { from: get(this.$locals.before, p), to: this.get(p) },
    ]));

    await AuditLog.create([{
      entity,
      entityId: this._id,
      action: this.isNew ? 'create' : 'update',
      changes,
      userId: ctx.userId,
      requestId: ctx.requestId,
    }], { session: this.$session() });
  });
}

purchaseOrderSchema.plugin(auditPlugin, { entity: 'PurchaseOrder' });

The key detail is this.$session(). The audit entry joins the same transaction as the change it describes, so a rollback removes both.

Small details like this are easy to miss, and a review from experienced MERN developers at this stage usually costs far less than cleaning up inconsistent audit data after a compliance check.

2 more things to handle. Query-level updates like the $inc on stock skip save hooks, so for inventory use a StockMovement ledger with one row per movement and treat onHand as a cached total.

The ledger also lets you rebuild stock for any date, which accountants love at month-end. And make the log append-only by giving the app's database user only insert and find rights on the audit collection.

On the React side, a simple timeline under each record showing who changed which field and when goes a long way.

Once users can see the history themselves, many "who changed this?" support tickets simply stop coming.

Testing the Hard Parts

mongodb-memory-server includes MongoMemoryReplSet, so transactions work in CI without a real cluster. Focus your integration tests on the failure paths:

2 concurrent receipts on the same PO, where one should fail on the version check.
A thrown error halfway through receiveGoods, after which stock, PO and audit rows should all be unchanged.
The same request sent twice with one idempotency key, which should change stock only once.

Wrapping Up

Embed data that belongs to its parent, store money as integers, and index for the reports people actually use.

Wrap related writes in withTransaction, move side effects to an outbox and write audit entries inside the same transaction.

Get these right, and the rest of the module becomes ordinary CRUD work, which is exactly how ERP code should feel.

Two-Phase Commit Is a Trap: How Real Systems Do Distributed Transactions

Athreya aka Maneshwar — Mon, 28 Sep 2026 21:55:38 +0000

When you are first building an application, a transaction is the cheapest guarantee you will ever get.

A customer places an order. You charge their card, reserve the inventory, write a ledger entry for accounting.

You wrap the whole thing in BEGIN and COMMIT, and if any one of those writes falls over, the database quietly rewinds the other two.

It is as if the order never happened. You probably wrote that code in four minutes and never thought about it again.

What you were leaning on there are two of the ACID guarantees, doing unpaid labour on your behalf.

Atomicity. All three writes land, or none of them do. There is no universe where the card gets charged but the stock never gets reserved.

Isolation. While the transaction is in flight, nobody else can see the half-finished middle. Another request checking the customer's balance will not catch a charge for an order that has not finished processing.

You wrote the SQL. The database did the genuinely hard part, for free, and never asked for credit.

Then you split the database, and the guarantee quietly leaves

Traffic grows. Writes grow. That one box starts breathing heavily.

So you do what everyone does. You shard, or you break the monolith into services and give each one its own database.

The details vary. The outcome is identical: your data now lives on several independent machines that have never been introduced to each other.

That one payment flow is now three separate operations against three separate databases.

Card charged in the payments database. Stock reserved in the inventory database. Ledger entry recorded in accounting.

There is no BEGIN big enough to cover all three, because none of them knows the others exist.

So when the card charge commits and the inventory reservation then fails because the last unit just sold, nothing rolls back.

That charge is already durable, on a different machine, in a database that has no idea an order died somewhere else.

At a few hundred orders a day, you will see this maybe once and fix it by hand. At a few thousand transactions a second across distributed infrastructure, partial failure stops being an incident and starts being a Tuesday.

This whole shape of problem has a name: the distributed transaction. One logical operation spanning multiple independent databases or services, where every step needs to either succeed together or get cleaned up when something goes sideways.

The literature offers you two answers. The industry has very firmly picked one of them, and knowing why will save you a genuinely painful quarter.

The textbook answer: two-phase commit

Two-phase commit is the classic academic solution, formalised decades ago in the X/Open XA specification.

You introduce a new component, a coordinator, whose entire job in life is to make every participant agree on the outcome before anybody makes anything permanent.

It runs in two phases, which is where the name comes from.

Phase one, prepare. The coordinator asks every participant: can you commit this? Each database does the real work, durably records the change so a crash cannot lose it, locks the affected rows so nothing else can touch them, and then answers yes or no.

Phase two, commit. If anyone voted no, the coordinator tells everyone to abort and release their locks. If everyone voted yes, it tells everyone to commit, the locks come off, and you are done.

What you get out of this is the same strong consistency you had with one database. Every participant agrees before anything is finalised, so there is no visible window where the system is half-done.

On a whiteboard, this is exactly what you want. It is beautiful. It is also why almost nobody runs it across services.

Blocking is the whole problem

2PC is a blocking protocol, and blocking in a distributed system means you have made your availability the product of everybody else's uptime.

Picture the coordinator collecting all three yes votes. Then, in the gap between collecting them and broadcasting the decision, it dies.

Now look at the participants.

They cannot commit on their own, because the coordinator might have been about to say abort.

They cannot abort on their own either, because the coordinator might have been about to say commit, and one of their peers might have already gone through with it.

So they wait. Holding locks. Indefinitely.

And every other transaction in your system that wants to touch any of those rows is now queued behind a decision that nobody alive can make.

Crashes are not even the interesting failure. The boring one is worse.

A single slow participant stalls the entire transaction. If the ledger service takes ten seconds to answer the prepare message, the payment service and the inventory service both sit there holding locks for ten seconds, doing nothing productive whatsoever.

Your whole system now moves at the speed of its slowest participant, which in a microservice estate is whichever team deployed most recently.

Add a network partition and it gets worse still, because the coordinator cannot distinguish "the message did not arrive" from "the message arrived and the reply got lost". There is no safe default to pick.

Pat Helland's Life beyond Distributed Transactions made this argument in 2007 and it has aged extremely well. Distributed transactions across autonomous services do not work at internet scale, so build systems that do not need them.

Where 2PC actually does live

To be fair to the protocol, 2PC is alive and shipping. It is just not shipping where you would write it yourself.

It lives inside distributed databases. Google Spanner and YugabyteDB both run two-phase commit under the hood to coordinate writes across shards.

The difference is coupling. In there, the coordinator and the participants are components of one system, with one deployment, one failure model, one team, and a consensus protocol like Paxos or Raft underneath to make the coordinator itself fault-tolerant.

Crucially, the database is absorbing that complexity so that you, the caller, never see it.

The moment you try to rebuild that arrangement across independent services with different release trains, different on-call rotations and different opinions about timeouts, every assumption that makes it work in Spanner evaporates.

Sagas: give up atomicity, keep your weekends

When companies genuinely need to coordinate work across services, the pattern they reach for is the saga. Uber, Netflix, Amazon and DoorDash all run this in production.

Sagas start from a deliberately humbler assumption.

You do not actually need all-or-nothing atomicity spanning five services. You need a way to reliably end up in a consistent state, even when a step blows up along the way.

So instead of one big locked distributed transaction, you break the work into a chain of independent local transactions. Each service does its piece and commits to its own database, on its own terms, immediately.

When something fails halfway down the chain, there is no rollback available, because the earlier steps are already committed somewhere else. Instead you run a compensating action.

That is a business-level undo. A refund instead of a rollback. A cancellation instead of an abort.

What you trade away is strong consistency. What you get is eventual consistency.

The system can genuinely be inconsistent for a few seconds. The customer might see a charge land before they see the refund arrive.

But it always converges, and nothing is blocked while it converges. Every other transaction in the system keeps flowing the entire time, because no saga step ever holds a lock that a stranger on another machine is responsible for releasing.

That is the actual trade. Not "correct versus incorrect", but "briefly visible mess" versus "system-wide stall".

Choreography or orchestration

Something has to notice the failure and kick off the compensations. Who that something is turns out to be the design decision that matters most.

Choreography is the decentralised option. It is publish and subscribe: each service broadcasts an event when it finishes, and whoever cares picks it up.

Payments charges the card and publishes CardCharged. Inventory is listening, reserves the stock, publishes StockReserved. Ledger picks that up and records the entry. On failure, a service publishes a failure event and upstream services run their own compensations.

Orchestration is the centralised option. A dedicated orchestrator drives the flow, calling one service at a time and waiting for confirmation before moving on.

Choreography is lovely at three steps. It is miserable at eight.

Once a dozen services are publishing and reacting to each other, "where is order 1841 right now?" becomes an archaeology project. Which step failed? Which compensations already ran? Did the refund actually go through, or is it still retrying?

Nobody owns the answer, so you go and read twelve services' logs to reconstruct it.

Orchestration costs you a central component, and buys you the one thing choreography cannot give you: a single place that knows the state of every in-flight transaction.

Temporal, built by the people behind Uber's Cadence workflow engine, and AWS Step Functions both exist specifically to be that component.

And here is the part that matters most, the reason the orchestrator is not just 2PC wearing a new hat.

When a saga orchestrator crashes, nothing is holding locks.

It is durable. It restarts, reads its own state back from its own database, and resumes from exactly where it stopped. Meanwhile the rest of your system never noticed, because no rows were ever frozen waiting on it.

A dead 2PC coordinator freezes your data. A dead orchestrator just delays one workflow.

Compensating actions are where the honesty comes in

"Just undo the previous step" sounds tidy. In practice, it is where sagas get their reputation.

The card charge committed, the inventory reservation failed, so you refund. Fine. Except that refund is not the invisible cleanup a database rollback gives you.

The customer sees a charge appear on their card. Then, seconds later, a refund. Their bank probably sent them a push notification for each one. It is correct, and it still generates a support ticket.

Some actions are worse than visible. Some are simply not undoable.

You cannot unsend an email. You can send a follow-up saying "ignore that last one", which is a different thing pretending to be the same thing.

If you fired a webhook at a third party, you can send a cancellation, but you cannot make them process it in time, or at all.

Every step in your saga needs a compensating action defined up front, and you should be clear-eyed that some of them are inherently imperfect.

Then there is the recursive problem: compensations fail too.

What happens when the refund API is down at exactly the moment you need to refund? Now you need retries on your failure handling. And the instant you have retries, you need idempotency, otherwise a flaky network refunds the customer twice and you have invented a new bug that is much harder to explain in a postmortem.

The trick is to derive the idempotency key from the thing you are undoing, never from the attempt:

def compensate_charge(order_id, amount_cents):
    # keyed on the order, not on this attempt, so ten retries of this
    # compensation collapse into exactly one refund on the customer's card
    key = f"refund:order:{order_id}"

    return stripe.Refund.create(
        payment_intent=lookup_payment_intent(order_id),
        amount=amount_cents,
        idempotency_key=key,
    )

Generate a fresh UUID per attempt and you have an idempotency key that guarantees nothing.

The uncomfortable summary is that your unhappy path needs the same reliability engineering as your happy path. Most teams budget for one of those.

The dual write problem, quietly waiting behind all of this

Even with perfect compensation logic, there is one more failure mode that catches teams off guard, and it is the one I would bet money is already lurking in your codebase.

When the payment service finishes charging the card, it has to do two things. Save the result to its own database, and publish an event so the next step knows to run.

Those are two writes, to two different systems, with no shared transaction between them.

If the database write succeeds and the event publish fails, the next step never fires and the saga stalls silently. No error, no compensation, just an order sitting in charged forever until a customer emails you.

If the event publishes and the database write fails, downstream services are now reacting enthusiastically to something that never happened.

The fix is the transactional outbox, and it is one of those patterns that feels like cheating the first time you see it.

Stop treating the event as a separate write. Put it in the same database, in the same transaction, as the data.

BEGIN;

UPDATE payments
   SET status = 'charged'
 WHERE order_id = 1841;

INSERT INTO outbox (aggregate_id, event_type, payload)
VALUES (1841, 'CardCharged', '{"order_id":1841,"amount_cents":4999}');

COMMIT;

One local transaction. Both rows commit, or neither does. The dual write is gone, because there is no longer a second write to fail.

A separate background relay then reads that outbox table and publishes to the broker. It can tail the database's own transaction log via change data capture, which is what Debezium does, or it can just poll the table on an interval, which is unglamorous and works fine at most scales.

flowchart TD
    A[Card charged successfully] --> B[BEGIN local transaction]
    B --> C[Write the payment row]
    C --> D[Write the event into the outbox table]
    D --> E[COMMIT: both rows, or neither]
    E --> F{How does the relay<br/>pick it up?}
    F -->|CDC, tail the WAL| G[Publish to the message broker]
    F -->|or just poll the table| G
    G --> H[Next saga step runs]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef db fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef out fill:#6ea8ff,stroke:#2f5fbf,color:#1a1a1a

    class A start
    class F decision
    class B,C,D,E db
    class G,H out

Note that this gives you at-least-once delivery, not exactly-once. The relay can crash after publishing and before marking the row as sent, so the event goes out twice.

Which brings us back to idempotency. Every consumer in your saga needs to handle seeing the same event twice, because it will.

So what should you actually build?

Before either of these patterns, ask the question that people skip because it feels like a cop-out.

Do you need a distributed transaction at all?

If you can draw your service boundaries so that data which changes together lives in the same database, do that. Move the inventory and ledger tables next to the payments table if they are always updated together.

A local transaction is simpler, faster and more reliable than any distributed thing you can build on top of it. This is always the best answer when you can get away with it, and it is enormously easier to get right up front than to retrofit two years and forty services later.

flowchart TD
    A[One operation spans several services] --> B{Can that data live<br/>in a single database?}
    B -->|Yes| C[Just use a local transaction]
    B -->|No| D{Is eventual consistency<br/>acceptable here?}
    D -->|No| E[Put it in a distributed SQL database<br/>Spanner or YugabyteDB]
    D -->|Yes| F{Many steps, branching,<br/>or tricky compensations?}
    F -->|No| G[Saga with choreography]
    F -->|Yes| H[Saga with orchestration]
    H --> I[Temporal or AWS Step Functions]
    G --> J[Transactional outbox<br/>+ idempotent consumers]
    H --> J

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef good fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef alt fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef tool fill:#6ea8ff,stroke:#2f5fbf,color:#1a1a1a

    class A start
    class B,D,F decision
    class C,H good
    class G,E alt
    class I,J tool

If you truly cannot keep it in one database, you are writing a saga. That part is not really up for debate anymore.

The remaining question is which flavour.

Choreography if the flow is short, the services are genuinely independent, and nobody needs a central view of where a transaction stands. An order placed event triggering an email and a push notification is a perfect fit, and wiring up an orchestrator for it would be theatre.

Orchestration for anything with branching logic, more than a handful of steps, or compensation logic you would rather define in one file than scatter across six repos. Most teams end up here eventually, and the tooling has got good enough that starting here is no longer expensive.

And if eventual consistency is genuinely unacceptable for some specific slice of your data, put that slice in a distributed database that handles strong consistency internally. That is a completely different proposition from hand-rolling 2PC across services you do not control.

The shape that keeps winning

Strip it all back and the production answer at basically every company operating at scale looks the same.

A saga, orchestrated by something durable. Idempotent operations at every step, so retries are always safe. A transactional outbox, so events are as reliable as the database writes they describe.

It means accepting that your system is briefly, visibly inconsistent sometimes.

That is not a compromise the industry stumbled into. It is one it chose, on purpose, after watching two-phase commit deadlock enough production databases to make the decision for everyone.

The database gave you atomicity for free. The moment you split it, the bill came due, and sagas are simply the cheapest way anyone has found to pay it.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by tests…

View on GitHub

Click below to try LiveReview with your codebase:

Do You Really Need to Be in the Top 0.1% of Your Field?

Athreya aka Maneshwar — Sun, 27 Sep 2026 17:52:14 +0000

Today I found a post on LinkedIn that hit a nerve.

If you're not best at what you do, nothing matters bhai !

I literally mean it, coz I felt it myself.

This is not some post which I am posting coz I am happy, sad or any XYZ emotion.

If you're not on top, you're just an average guy, and trust me no one respects an average guy.

If you're average, just keep your head down, and keep working until you can say,
"Yes, I belong to 0.1% of my field".

Not to motivate, or demotivate. It's just what I felt.

Keep GRINDING bhai!!

I read it twice.

The first read got me fired up, which is exactly what a post like that is built to do.

The second read left me with a question I could not shake: is this actually true? Do I need to be in the top 0.1% of my field for any of my work to matter?

So I went looking. Not for motivational quotes, but for what people who genuinely got to the top of their industries say about how they kept score along the way.

What I found changed how I read that post entirely.

The problem with "top 0.1%" is not the ambition

Let me be clear about what I am not saying.

I am not saying aim lower. The intensity in that post is real and I respect it.

The problem is not the height of the bar. The problem is that "top 0.1% of my field" is a terrible thing to point your daily effort at, because you cannot actually do it.

Try it right now. Open your calendar for tomorrow and schedule "be in the top 0.1%".

You can't. There is no task there. There is no first step. There is nothing you can start at 9am and finish by lunch.

You can schedule "read the Raft paper", or "rewrite that gnarly service with tests this time", or "review three PRs properly instead of rubber stamping them". Those are things.

"Top 0.1%" is not a thing you do. It is a thing that gets said about you, later, by other people, after a lot of stuff outside your control goes your way.

Buffett's answer: which scorecard are you using?

The first thing I ran into was Warren Buffett's idea of the Inner Scorecard, from Alice Schroeder's biography The Snowball.

Buffett frames it as a single question:

The big question about how people behave is whether they've got an Inner Scorecard or an Outer Scorecard. It helps if you can be satisfied with an Inner Scorecard.

Then he makes it concrete with a test that is hard to wriggle out of:

Lookit. Would you rather be the world's greatest lover, but have everyone think you're the world's worst lover? Or would you rather be the world's worst lover but have everyone think you're the world's greatest lover?

Swap in your own field and answer honestly.

Would you rather be a genuinely excellent engineer that nobody has heard of, or a mediocre one that everybody thinks is excellent?

Most of us want to say the first one. Very few of us actually behave like it.

The Outer Scorecard is your rank, your title, your follower count, your comp band, what people say about you when you leave the room.

The Inner Scorecard is your effort, your intensity, your training, your consistency, and whether you were honest with yourself about all four.

Here is what makes the difference practical rather than philosophical.

The Outer Scorecard moves even when you do nothing.

Somebody more talented joins your field and you drop a slot. A hiring freeze lifts somewhere else and you drop two more. A guy who is worse than you posts better and passes you anyway.

The Inner Scorecard only moves when you move it.

Nobody can take a day of real work away from you. Nobody can devalue the hour you spent actually understanding something instead of pretending to.

Buffett did not pick this up from a book, by the way. He got it from his father Howard, who he described as "a hundred percent Inner Scorecard guy", a congressman who kept voting his conscience into losing causes and never seemed bothered that it cost him.

Bezos says the same thing in engineer language

Buffett's version is about character. If that feels a bit soft, the same idea shows up in a much more mechanical form.

Jeff Bezos, in the 2009 Amazon shareholder letter:

We take these financial outputs seriously, but we believe that focusing our energy on the controllable inputs to our business is the most effective way to maximize financial outputs over time.

And he is blunt about why:

I have no idea how to drive up the stock price. I can't manage that directly. It's not a controllable input. But I can make your picking algorithms more efficient, and that will reduce cost structure.

That last sentence is the whole thing.

The stock price is an output. It is the score other people give you. You cannot touch it.

Picking efficiency in a warehouse is an input. You can walk over there tomorrow and improve it.

The method Bezos describes is just working backwards until you hit something you can actually put your hands on:

You keep working backwards until you get to something that's controllable.

So let's run that on "top 0.1% of my field".

Top 0.1% is the output. What feeds it? Reputation, who happened to give you a shot, and the timing of the market you walked into. Those are half yours at best.

Keep going backwards. What feeds those? Hours of deep work. Hard problems you actually shipped instead of talking about. Feedback you asked for and then used. Showing up again the next day when it stopped being fun.

That bottom row is where you live. You can do every single one of those tomorrow, without anyone's permission, and nothing in the world can stop you.

Once you write it out like that, the top row stops looking like a goal and starts looking like a weather report.

Both men are describing the same trap

Read them together and it is one idea wearing two outfits.

Buffett: do not let other people hold your scorecard.

Bezos: do not aim at a number you cannot move.

The failure mode is identical either way, and I have watched it happen to good engineers.

You pick an outer metric. Staff title, follower count, top 0.1%, whatever it is.

You work hard. The metric does not move, because it depends on six things that have nothing to do with you.

Now you have a problem. The honest read is "the inputs were fine, the output lagged". But the outer scorecard does not offer that read. It just shows you a number that did not go up.

So you conclude the work was worthless. You either burn out or you start optimising for the thing that actually moves the number, which is almost never the work itself.

That is how you end up with people who post about engineering more than they engineer.

What I actually do with this

I am not going to pretend I have solved this. I check my own outer scorecard more than I would like to admit.

But one small thing has helped, and it is basically Bezos with the corporate serial numbers filed off.

At the end of the day, I do not ask "am I getting closer to the top 0.1%".

I ask: did today's inputs deserve a good output?

Did I do real work, or did I do busy work that looked like real work?
Did I learn something I did not know this morning?
Was I honest about the thing I got stuck on, or did I quietly route around it?

Some days the answer is no. That is fine, it is information, and it is information I can act on tomorrow.

The important part is that this question has an answer I own. "Am I top 0.1% yet" does not.

Keep the grind, move the scoreboard

None of this is an argument for wanting less.

The grind part is real, and if you have the intensity for it, that is a genuine advantage over most people. Do not talk yourself out of it.

Aiming for the top 0.1% is a fine direction to have. It is just a bad thing to measure yourself by.

Being in the top 0.1% is an output. It depends on a lot of things we don't control, like timing, competition, and luck.

The inputs are what we do control: our effort, our training, our intensity and consistency.

If we get those right, over time the output takes care of itself.

So keep the intensity. All of it.

Just move it off the scoreboard and onto the thing that actually feeds the scoreboard.

Now I want to hear from you, because I know this community is full of people quietly keeping score on themselves.

Which scorecard are you running on right now?

And if you switched from outer to inner at some point, what was the thing that made you switch?

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by tests…

View on GitHub

Click below to try LiveReview with your codebase:

GFS Cost More Than 50 Cents a Run. Now Each Run Costs 4 Cents

Athreya aka Maneshwar — Sat, 26 Sep 2026 18:02:42 +0000

The bill was 50 cents a run.

Not a month. Not a thousand runs. One run.

Paste a design proposal in, get back a grounded review with precedent from our shelf of books and internal postmortems, and watch half a dollar evaporate.

Comments warn about API key store isolation quirks

Athreya aka Maneshwar

Sep 22

Cheap RAG in Go with Gemini File Search: no vector DB, two calls, one hosted store

#go #ai #rag #gemini

14 min read

Part 1 was the happy version of this story: a Go binary, Gemini's hosted File Search, no vector database, and a free tier that made the whole thing look like a free lunch.

Then the free tier ran out and the paid rates arrived, and the free lunch turned out to be a tasting menu.

Five people on the team, several runs a day each, and a number that scaled with how hard the tool was thinking.

This post is what we replaced it with, what we measured, and the one component we built, benchmarked, and then deleted.

The word "free" was doing a lot of work

Here is the thing the pricing page tells you about File Search, and it is all true.

Storage is free. Query-time embeddings are free. You pay once at indexing time.

Here is the thing it does not put in bold.

The search is performed by a model. It is a tool the model calls mid-answer. Which means the retrieved chunks land in that model's context as ordinary input tokens, and the model reasons its way through them before replying.

And on Gemini 3.5 Flash, reasoning costs $9.00 per million output tokens.

Look at that output row.

9,529 tokens out, of which 8,493 were the model thinking. For a call whose entire job is "find the relevant passages and hand them over."

We paid a reasoning model to reason about which paragraphs to copy.

The first fix was structural and it helped a lot. The draft step used to do its own search, which meant three drafts meant three searches. Splitting search out so one search feeds every draft took a run from about $0.70 to about $0.12.

Six times cheaper, and still the wrong shape.

Because the expensive part was never the searching. It was the thinking attached to it.

Retrieval does not need a brain.

It needs an index, a similarity function, and a tiebreaker. Every one of those is a thing you can run on a laptop while it charges.

What we swapped in

A local Chroma store, embedded with Qwen3-Embedding-0.6B, searched with dense vectors plus BM25, fused with Reciprocal Rank Fusion.

Every model call moved to DeepSeek V4 Flash on Atlas Cloud at $0.14 in and $0.28 out per million.

Roughly ten times cheaper per token than what we were on, with a 1M context window and JSON mode.

The retrieval service is Python, running as a child process of the Go binary, on a port nobody else talks to. It starts with make run and dies with it.

I did try to talk myself out of that boundary.

Go has no mature CUDA story. The honest route is exporting the model to ONNX and linking a Go runtime through cgo, matching driver and CUDA and cuDNN versions exactly. PyTorch is the path a million people have already debugged, including on WSL2, where CUDA passthrough breaks in creative ways.

Adding a process boundary was cheaper than adding a whole new class of "works on my machine."

Chunking is where the accuracy actually lives

Everyone talks about which embedding model to pick. Almost nobody talks about what you feed it, and that is where the wins were.

Our corpus is 71 markdown files, and most of them are books that used to be PDFs. PDFs converted to markdown are full of things that are not text.

The cleaner strips the converter's own footer, lines that are only a page number, picture-text blocks, and hyphenated line breaks that split a word across two lines.

The one that surprised me is running headers.

A converted book repeats the chapter title at the top of every single page, usually promoted to a markdown heading.

Left in, it shreds the text into confetti, and a passage about paper reactors comes back with "Ship Project and Civilian Power" wedged into the middle of a sentence.

The rule that fixed it is embarrassingly simple: a short line that appears five or more times in one file is page furniture, not prose.

Then the chunking. About 300 words, hard cap 450, split on sentence boundaries, with two sentences carried into the next chunk so a quote that straddles a boundary survives in at least one piece. A real section heading starts a new chunk, once the current one has enough in it to stand alone.

And then the part I would steal even if you take nothing else from this post.

Embed more than you store.

The chunk you store is the text the draft is allowed to quote. The string you embed has a header glued on top of it:

EMBED_MODEL = "Qwen/Qwen3-Embedding-0.6B"

# what goes into the embedding, per chunk
f"{title} ({kind}) › {heading_path}\n\n{text}"

# and the query side gets an instruction, because Qwen3-Embedding is
# instruction-tuned on queries only. documents are embedded as they are.
QUERY_PROMPT = (
    "Instruct: Given an abstract pattern or principle, retrieve historical cases, "
    "documented examples, and named principles from books and essays that show "
    "the same pattern\nQuery: "
)

A chunk in the middle of chapter nine might only say "he decided otherwise."

With the header, that vector still knows it is Rickover, in a book about Rickover, in a section about paper reactors. Without it, it is a pronoun floating in space.

The query instruction is the Part 1 idea, "search with the pattern, not the post," pushed one layer down into the embedding itself. The corpus is cases. The queries are patterns. Saying so out loud to a model that was trained to listen costs nothing.

Dense finds the paraphrase, BM25 finds the name

Vector search is great at "these two paragraphs mean the same thing" and oddly bad at "this paragraph contains the word Rickover."

Keyword search is the reverse.

So we run both, take 40 candidates each, and fuse them.

flowchart TD
    Q[one query from call 1] --> DN[dense top 40, Qwen3-Embedding]
    Q --> BM[BM25 top 40, title + text]
    DN --> RRF[Reciprocal Rank Fusion, k=60]
    BM --> RRF
    RRF --> SEL[keep the best 4 per query]
    SEL --> CAP{2 chunks from this file already?}
    CAP -- yes --> SKIP[skip, so no book dominates]
    CAP -- no --> KEEP[keep it]
    KEEP --> RR[round-robin merge, 3 queries]
    SKIP --> RR
    RR --> DUP{near-duplicate of one picked?}
    DUP -- yes --> DROP[drop: a post synced twice]
    DUP -- no --> OUT[12 passages go to the draft]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start    fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef dense    fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef lex      fill:#6ea8ff,stroke:#2f5fc4,color:#1a1a1a
    classDef good     fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef bad      fill:#ff9a5c,stroke:#c4602a,color:#1a1a1a

    class CAP,DUP decision
    class Q start
    class DN,RRF dense
    class BM,SEL,RR lex
    class KEEP,OUT good
    class SKIP,DROP bad

def hybrid(self, query: str, k: int = CANDIDATES) -> list[tuple[int, float]]:
    score: dict[int, float] = {}
    for ranked in (self.dense(query, k), self.lexical(query, k)):
        for r, i in enumerate(ranked):
            score[i] = score.get(i, 0.0) + 1.0 / (RRF_K + r + 1)
    return sorted(score.items(), key=lambda x: -x[1])[:k]

That is the whole fusion. Six lines.

The reason it works is that it never compares the two scores. A cosine similarity of 0.82 and a BM25 score of 14.3 have nothing to say to each other. Ranks do.

A document at position 3 in both lists beats one that is first in a single list, and the constant k (60 is the number the literature settled on) keeps the top of each list from steamrolling everything else.

BM25 indexes the title and heading alongside the text, same as the embedding header does, so naming a book in your query actually finds pages from that book.

A few rules keep the final twelve honest. At most two chunks per file per query, so one 400-page book cannot fill every slot. Queries merge round robin, so each of the three contributes its best passage before any of them gets its fourth.

And anything with a word-overlap above 0.6 against something already picked gets dropped, because our blog corpus has a couple of posts that got synced twice and they were politely returning themselves as two independent sources.

The reranker that got fired

Standard advice says: retrieve broadly, then rerank with a cross-encoder. So we did that, with bge-reranker-v2-m3.

Then a run took six and a half minutes and I went looking.

The retrieval service's own log had it in one line: search: 3 queries -> 8 chunks in 117.5s.

Three queries times 40 candidates is 120 cross-encoder forward passes on a 4 GB GTX 1650 that is also drawing the desktop. It was not thrashing. It was just honest work on unfit hardware.

Before ripping it out, we measured. The golden set builds itself out of finished sessions: take Call 1's retrieval queries, pair them with the files the accepted draft actually cited, and you have a retrieval test built from real usage rather than from vibes.

The reranker won on MRR and hit@4. It genuinely put better passages nearer the top.

It also did not change recall@12 at all, and recall@12 is the only number with a consumer.

The draft call gets all twelve passages in its prompt. It reads all twelve. There is no top-4 cutoff downstream, no truncation, nothing that treats passage 1 differently from passage 9.

So the reranker was spending 108 seconds improving an ordering that nothing downstream reads. It was optimising a metric we had accidentally chosen because it appears in every retrieval paper, not because our pipeline consumed it.

Out it went, with the reasoning written into the top of retriever.py so the next person does not "fix" its absence.

And the honest caveat, which lives there too: this is one golden session. A harder query might genuinely need reranking to pull the right passage into the top twelve. The code is in git history, the eval is a make target, and when the golden set is fat enough to mean something we will run it again.

Measure before you delete. Also measure before you keep.

Two hours of embedding, four laptops

3,639 chunks at roughly 1.8 seconds each on a shared 4 GB GPU is about two hours, which is about one hour and fifty minutes more than anyone wants to wait.

But embedding is deterministic and the corpus splits cleanly by file. So it parallelises across people, not just across cores.

flowchart TD
    C[71 files, 3,639 chunks] --> S[shard_files: disjoint quarters]
    S --> M[four laptops, --shard i/4]
    M --> R{sha256 AND chunk count match?}
    R -- yes --> SK[already embedded, skip]
    R -- no --> EM[embed, halve the batch on OOM]
    EM --> G[commit the shard, hand it back]
    SK --> G
    G --> I[rag/integrate.py]
    I --> V{same embedding model everywhere?}
    V -- no --> X[refuse: mixed vectors lie quietly]
    V -- yes --> O{every file in exactly one shard?}
    O -- in two --> X
    O -- in none --> W[warn, merge what arrived]
    O -- yes --> F[copy vectors into db/chroma]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start    fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef work     fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef box      fill:#6ea8ff,stroke:#2f5fc4,color:#1a1a1a
    classDef good     fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef bad      fill:#ff9a5c,stroke:#c4602a,color:#1a1a1a

    class R,V,O decision
    class C start
    class M,EM work
    class S,G,I,SK box
    class F good
    class X,W bad

# on four different machines, one quarter each
uv run rag/ingest.py --shard 1/4 --out db/chroma-shards/1
uv run rag/ingest.py --shard 2/4 --out db/chroma-shards/2
# ... then, once everyone commits their shard back
uv run rag/integrate.py db/chroma-shards/*

The merge does no embedding at all. It checks that every shard used the same embedding model, that no file landed in two shards, and that no file landed in none, then copies the vectors into one store.

Those checks are not paranoia. Mixed embedding models do not crash. They return confidently wrong neighbours forever, which is a far worse failure than a stack trace.

Building this also shook out a real bug in the incremental logic. Resume was deciding "already done" by comparing the file's sha256 against what was in the store.

A run killed halfway through a file leaves that file's sha perfectly correct and its chunk count short, so it would have been marked done forever, silently missing half a book.

The fix is one AND: sha256 and chunk count both have to match.

Then we committed the finished store. db/chroma is 69 MB in git, which is nothing, and it means nobody else on the team ever embeds anything. Clone, run, search.

Now that retrieval is free, spend it on drafts

The nice thing about killing your most expensive call is that the cheap calls get interesting.

A draft is now a few tenths of a cent. So instead of drafting once and retrying on failure, the pipeline fires three or four drafts at once at temperature 0.7 and lets them race.

Each reply gets resolved and checked in Go as it lands. Drafts that fail the checks are rejected on the spot. The first one that passes goes on to the review call. Only if all of them fail does the batch retry, with the closest draft's failures appended to the prompt.

Every draft is kept and shown, including the rejected ones with the checks they failed, because "here are four attempts and why three of them were bad" is more useful to the person reading than one draft and a shrug.

This did produce one genuinely dumb bug, which parallelism made much more likely.

In one run the first batch produced a draft that passed every check. The reviewer then asked for a revision. Eight redrafts later, none of them passing, the pipeline shipped the closest failing redraft.

It had a passing draft in hand and threw it away for a worse one. The fix is the obvious fallback: if no redraft passes, ship the draft that already did, with the reviewer's notes attached.

Retries are cheap. Losing work you already paid for is not.

The check that replaced trusting the grounding metadata

Handing the passages in ourselves unlocked the thing I actually care about.

When a hosted search tool returns grounding metadata, you know which chunks the model looked at. You do not know that the words it put in quotation marks are in any of them.

Now the pipeline can prove it.

Every history.sources[].passage in the output has to appear word for word in a chunk from the file it names. Same for an authority's exact_words when it cites one of the given files. Markdown, line wrapping and quote style are normalised away first, and a ... marks an omission, with the remaining pieces required to appear in order.

A failure is not a warning. It is a check failure, exactly like a length violation or an unresolved law citation, and it feeds the same retry loop that everything else does.

This is the difference between "the model had access to the right book" and "the model quoted the right book correctly," and only one of those is worth showing a reviewer.

What the bill looks like now

One real run, end to end, with 15 model calls including 12 drafts across three batches:

210,976 input tokens, 55,285 output tokens
$0.045, about ₹4.3
retrieval: 0.5 seconds of it, and none of the money

Against $0.70 for the same pipeline shape on hosted search. Fifteen times cheaper, and the expensive part is now the part that does the actual writing, which is how it should be.

Two honest footnotes on that number.

Atlas reports most of each draft's input as cached, and we charge every input token at the full rate, so $0.045 is a ceiling, not an estimate.

And latency went up, not down. Retrieval dropped from 117 seconds to half a second, but DeepSeek thinks hard before each draft, so a full run is still two to four minutes.

We bought cost, not speed. For "paste a design doc, come back with a coffee," that is the right trade. For a chat box it would be the wrong one.

What I would steal

If your retrieval is a model call, you are paying reasoning prices for a lookup.

Pull it onto your own machine, spend the effort on cleaning and chunking rather than on model selection, embed a contextual header you never show anyone, fuse dense and lexical on ranks instead of scores, and check the quotes rather than trusting them.

Then build the eval before you build the clever part. Ours told us to throw the clever part away, which saved 108 seconds a search and a permanent dependency we did not need.

The best component in this system is the one that is not in it.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by tests…

View on GitHub

Click below to try LiveReview with your codebase:

Cheap RAG in Go with Gemini File Search: no vector DB, two calls, one hosted store

Athreya aka Maneshwar — Tue, 22 Sep 2026 07:12:46 +0000

Every team I have been on has the same shelf.

Nine engineering books somebody swears by, a couple of long essays that get linked in every third design review, and sixty-odd internal blog posts and postmortems that nobody rereads.

About 1.4 million tokens of "we already learned this once."

I wanted a tool where you paste a design proposal, an ADR, a postmortem draft, and it comes back with: here is what is actually being proposed, here is the pattern underneath it, and here are the three places on the shelf where we, or someone smarter, already ran into this exact shape.

Grounded. With citations. Not "an LLM read your doc and had feelings about it."

The obvious build is a vector database, an embedding pipeline, a chunker, a reranker, and a weekend.

I built it with none of those, in Go, on Gemini's free tier, and the retrieval side of it costs nothing to run.

This post is about how, and about the four things that bit me on the way.

RAG without owning the R

Gemini has a thing called File Search.

You create a "store", upload documents into it, and Google chunks them, embeds them, and indexes them.

Then you attach that store to a normal generateContent call as a tool, and the model searches it by itself, mid-answer, and hands you back the chunks it used as groundingMetadata.

No pgvector. No Pinecone bill. No embedding model to pick and then regret.

The pricing is the part that made me sit up.

Storage is free. Query-time embeddings are free. You pay once at indexing time, at embedding prices, and then the chunks the model pulls in are billed as ordinary context tokens on the call you were already making.

On the free tier the store caps at 1 GB. My entire shelf, every book and every post converted to markdown, is 5.3 MB.

So the architecture is embarrassingly short.

One Go binary. SQLite via modernc.org/sqlite, so no cgo. A REST client for Gemini with no SDK, because I wanted to log the exact request and response bodies verbatim, and SDKs love to hide those.

Two model calls per question. One hosted store. That is the whole thing.

Step zero: getting books into markdown without losing the spaces

Before any of the clever parts, the corpus has to exist as text, and this cost me an evening.

The first tool I reached for was markitdown. It gave me headings, sort of, and it also gave me this:

Theubiquityoffrustrating,unhelpfulsoftwareinterfaceshasmotivateddecadesofresearch

Every word in the PDF glued to its neighbour. The embedding model does not know what "Theubiquityoffrustrating" is, and neither does the retrieval.

pdftotext fixed the spacing and threw away every heading, so a 300-page book became one undifferentiated scroll.

The one that won was pymupdf4llm. It looks at font sizes to decide what is a heading, keeps bold and italics, and its text extraction handled the spacing correctly on the same PDF.

uv run --with pymupdf4llm python -c "
import pymupdf4llm, pathlib
pathlib.Path('out.md').write_text(pymupdf4llm.to_markdown('in.pdf'))
"

Forty-four real headings out of one essay, versus zero.

Headings matter more than they look like they should here, because File Search chunks on whitespace with a token budget. A chunk that starts at a heading is a chunk that means something on its own. A chunk that starts mid-sentence in a wall of text is noise with an embedding attached.

I put the conversion behind make process-data so nobody on the team has to rediscover this. EPUBs, PDFs, and a sync of our blog repos, all into one post_processed_data/ tree of markdown.

The store belongs to the key, not to you

This is the first thing that bit me, and it is not in the big print.

A File Search store lives inside the Google Cloud project behind the API key that created it.

Create a store with key A, try to search it with key B from a different project, and it does not error in a helpful way. It just is not there.

That matters the moment you have more than one key, which on the free tier you will, because the per-minute caps are real and the fix everyone reaches for is "rotate across a few keys."

You cannot rotate a tool-attached call across keys. The store pins you.

So keys got roles.

keys/store.txt is the short list. Each key in it owns a complete copy of the corpus in its own store. Ingest uploads to every one of them. The search call tries store 0 with key 0, and only on a quota or auth failure falls through to store 1 with key 1.

keys/rotate.txt is the long list, for any call with no tool attached: the analysis call, the corpus-index summaries. Those genuinely do not care which key answers, so they rotate on 429, 401, 403, and anything 5xx.

Two things I got wrong before I got them right:

503 is not a key problem. Gemini returns 503 "high demand" when Google is busy, and switching keys does nothing except burn another key's quota. So the client waits and retries the same key a couple of times before falling over.

The key goes in the x-goog-api-key header, never the URL. Put it in the query string and the first transport error prints your key into your own logs. Ask me how I know.

Yes, two stores means the one-time indexing cost happens twice. It is the price of the search call never dying on a single quota, and at embedding prices for 5 MB, it is coffee money.

Ingest is a diff, not an upload

The ingest step is where "just upload the folder" turns into an actual program.

Every file gets a sha256. A manifest in SQLite records path -> sha -> document id for each store. Then:

flowchart TD
    A[walk data/**/*.md, sha256 each] --> B{path in manifest?}
    B -- no --> U[upload to store]
    B -- yes --> C{sha changed?}
    C -- no --> S[skip, unchanged]
    C -- yes --> D[delete old doc id] --> U
    U --> P[poll the indexing operation until done]
    P --> M[upsert manifest: path, sha, doc id]
    A --> R{manifest path missing on disk?}
    R -- yes --> X[delete doc from store, drop manifest row]
    R -- no --> S
    M --> I{set of path+sha changed?}
    S --> I
    X --> I
    I -- no --> K[corpus index unchanged]
    I -- yes --> G[summarize new files with flash-lite, one line each] --> J[store corpus_index]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start    fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef net      fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef local    fill:#6ea8ff,stroke:#2f5fc4,color:#1a1a1a
    classDef idx      fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a

    class B,C,R,I decision
    class A start
    class U,D,P,X net
    class S,M,K local
    class G,J idx

Unchanged files are skipped. Changed files have their old document deleted first, then get re-uploaded. Files that vanished from disk get deleted from the store.

Each upload is a multipart POST that returns a long-running operation, and you poll it until done before you trust the document id. Four uploads run in parallel. Serial, a 71-file corpus takes about ten minutes. Parallel, a few.

The chunking is set per upload, and I landed on 400 tokens with 60 of overlap. The docs' example is 200 and 20, which for a book felt like reading through a letterbox.

meta := map[string]any{
    "displayName": displayName,
    "chunkingConfig": map[string]any{
        "whiteSpaceConfig": map[string]int{
            "maxTokensPerChunk": 400,
            "maxOverlapTokens":  60,
        },
    },
}
// multipart/related: part 1 is this JSON, part 2 is the markdown bytes
// POST /upload/v1beta/{store}:uploadToFileSearchStore
// with X-Goog-Upload-Protocol: multipart

The last box in that flowchart is the corpus index, and it is small but it is the thing that makes the next section work.

For every file, a one-sentence summary from the cheapest model available (gemini-3.5-flash-lite), keyed by the file's sha so it is only ever generated once. All the lines get joined into one block, "path, summary", that gets pasted into the analysis prompt.

That way the model deciding what to search for knows what is actually on the shelf. It aims at sources that exist instead of guessing.

Search with the pattern, not the post

This is the idea in the post that I would keep if I had to throw everything else away.

Embedding search finds things that sound alike.

Paste a proposal that says "we should standardize on one MCP transport across all our agents" straight into retrieval, and you get back every chunk that contains the words "MCP", "agents", and "standardize". Which is your own recent blog posts about MCP and agents.

That is not precedent. That is a mirror.

What you actually want is the shape of the situation with the nouns removed.

"Fragmented, incompatible implementations, then one open standard, then mass adoption."

Search the shelf with that, and the browser wars come back. Rickover on standardizing the nuclear navy comes back. A 2,300-year-old Legalist essay on uniform law comes back.

Same pattern, different clothes. That is what a reviewer with thirty years of reading brings, and it is what the model cannot do if you let it search with the raw text.

So the pipeline never lets it.

Call 1 gets the lawbook, the corpus index, and the proposal, with no tools attached. It returns JSON: the motive, what is actually happening, who the actors are, the generalization, and two or three retrieval queries derived from that generalization.

Call 2 gets the analysis from call 1, the File Search tool pointed at the store, and the instruction to search with those queries and then draft.

The raw document is in call 2's context, so the draft can quote it. But the search terms came from call 1, and they are about the pattern, not the subject.

{
  "systemInstruction": { "parts": [{ "text": "...the lawbook..." }] },
  "contents": [{ "role": "user", "parts": [{ "text": "PROPOSAL:\n...\n\nANALYSIS FROM CALL 1:\n{...}\n\nSearch the corpus with the retrieval queries above, then reply with the JSON contract." }] }],
  "tools": [{ "fileSearch": { "fileSearchStoreNames": ["fileSearchStores/abc123"] } }],
  "generationConfig": { "responseMimeType": "application/json", "temperature": 0.7 }
}

The response carries candidates[0].groundingMetadata.groundingChunks, each with the file's display name and the retrieved text. Those get shown to the user, in full, next to the draft. If the "authority" the review leans on is not in that list, it did not come from the shelf, and the checks catch that.

Every instruction is a law, every reply cites its laws

There is no free-text system prompt anywhere in this thing.

The prompts are an AgentLaws lawbook: a folder of markdown where every instruction the model sees is a numbered law, grouped into chapters, versioned in git, and compiled once at startup.

The model is required to return applied_laws, the numbers it relied on, and the pipeline resolves each one back to file:line in the lawbook.

A number that does not resolve fails the run. Which sounds pedantic until the first time the model confidently cites law 7.2.3 in a lawbook with six chapters.

The practical win is that "why did it do that" has an answer that is a file and a line number, and "make it stop doing that" is a pull request against a markdown file, not archaeology in a Go string.

The output contract is the last thing in the prompt, because that is where the model weights it most. Laws about tone and structure go first, the JSON schema goes last.

Trust, but run the checks

The model also fills a checks block in its own reply, a self-audit. I do not trust it, but it is useful as a second signal.

Before that gets read, a small pile of deterministic checks in plain Go runs on every draft: length bounds, no markdown, at most one question, no "see point N" references, every cited law resolved.

flowchart TD
    A[call 2: search + draft, JSON mode] --> B{reply parses as JSON?}
    B -- no --> F[resend once without JSON mode, extract first object] --> C
    B -- yes --> C[resolve every cited law number to file:line]
    C --> D{all citations resolve?}
    D -- no --> R
    D -- yes --> E[deterministic checks: length, no markdown, one question max, distinct openers]
    E --> G{checks pass?}
    G -- no --> R{attempts left?}
    R -- yes --> H[append the failures to the user message] --> A
    R -- no --> S[ship the draft with failures listed]
    G -- yes --> V[call 3: reviewer pass, verdict + objections]
    V --> W{verdict?}
    W -- ship --> O[return JSON with sources and applied laws]
    W -- revise, rounds left --> H
    W -- revise, no rounds --> S

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef llm      fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef local    fill:#6ea8ff,stroke:#2f5fc4,color:#1a1a1a
    classDef bad      fill:#ff9a5c,stroke:#c4602a,color:#1a1a1a
    classDef good     fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a

    class B,D,G,R,W decision
    class A,F,V llm
    class C,E,H local
    class S bad
    class O good

A failure gets appended to the user message as a plain list, "the previous attempt failed these checks, fix them", and call 2 runs again. Two retries, then the draft ships anyway with the failures listed, because a draft with a warning is more useful than a spinner that gave up.

One box in there deserves its own sentence: JSON mode plus a tool is not guaranteed. responseMimeType: application/json together with File Search worked most of the time and then, occasionally, returned prose with a JSON object somewhere in it. The fix is boring: if the reply does not parse, resend once without JSON mode and pull the first {...} out of the text. Both attempts get logged.

The bug that made me split the database

The tool is used by five people, and each of them runs it against their own SQLite file, so history is per person.

The first version put everything in that one file: sessions, steps, and the ingest manifest.

Spot it?

A new teammate makes a new database. New database, empty manifest. Empty manifest means every file on disk is "new", so all 71 get uploaded again. And because the manifest is empty, ingest does not know the old document ids, so nothing gets deleted.

The store now has two of everything. Do it five times and search quality quietly rots, because every query returns the same chunk five times and crowds out the second-best hit.

The fix is one sentence: the manifest is a property of the store, not of whoever is asking questions today.

So there are two databases now. corpus.db holds the store names, the manifest, and the corpus index, and every session database reads from it. alice.db holds Alice's sessions and nothing else.

Delete corpus.db and you get a full re-ingest. Delete alice.db and nothing about the corpus changes.

What the free tier actually gives you

Since "cheap" is in the title, the honest numbers, as of when I built this.

Models. Flash only. gemini-3.5-flash for the real calls, gemini-3.5-flash-lite for the throwaway summaries. Pro returns a 429 with limit: 0 on free keys. A paid key unlocks it with one env var, and I have not needed to.

Rate limits. Per-minute caps are the thing you hit, not per-day. Ingest backs off with a growing wait when every key is rate limited, and the UI shows each key fallover as it happens instead of hiding it behind a spinner.

Storage. 1 GB per store. My 5.3 MB corpus does not register.

Cost per question. Two Flash calls, sometimes three with the review pass, around 10k tokens in and 2k out. The retrieved chunks are already counted in that. On the free tier, zero. On a paid key, well under a cent.

Latency. 30 to 90 seconds end to end, most of it call 2 doing the search and the draft. That is fine for "paste a design doc, get a review" and would be wrong for a chat box.

The catch. On the free tier, Google may train on what you upload. My shelf is published books and public blog posts, so I do not care. If your corpus is your customer contracts, read the terms before you make ingest.

What I would tell you to steal

If you have a shelf, and a question shaped like "have we seen this before", you do not need a vector database to answer it.

You need markdown with real headings, a hosted store you diff against instead of re-uploading, keys with roles because the store pins you to one, and a first model call whose only job is to turn the question into the pattern underneath it.

The rest is checks and logging, and the checks are the part you will thank yourself for.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by tests…

View on GitHub

Click below to try LiveReview with your codebase: