DEV Community: Daniel Romitelli

I Had to Build What the Bumper Could See

Daniel Romitelli — Wed, 30 Sep 2026 07:12:07 +0000

The carrier’s chrome reflected my light card as a white rectangle. I moved the light to the wet road and rebuilt the shot around what the bumper saw.

1. The rectangle in the bumper

With the card gone, I traced the bumper’s reflection back to the apron.

A polished surface reflects along a direction set by its angle and the camera’s position. Roughness spreads that reflection. It cannot bring a lit patch of road into metal that faces somewhere else. I checked the reflected direction before touching the shader again.

The teaching plate is flat. The bumper is not. Each patch of chrome looks at a different piece of apron, so one lit mark on the road cannot serve the whole bar. I had to use the shot camera and light the ground that bar was actually returning.

The scripts below use generic geometry and no production assets. They run with Python 3.10 or newer and need no extra packages. In mirror_ray.py, a vertical plate reflects a flat floor at teaching coordinates:

"""Adapted educational example; no film assets, scene, or production settings."""
from math import sqrt


def unit(vector):
    length = sqrt(sum(v * v for v in vector))
    if length == 0:
        raise ValueError("A direction must have nonzero length")
    return tuple(v / length for v in vector)


def ground_hit(point, normal, camera):
    normal = unit(normal)
    view = unit(tuple(c - p for c, p in zip(camera, point)))
    dot = sum(n * v for n, v in zip(normal, view))
    reflected = tuple(2 * dot * n - v for n, v in zip(normal, view))
    if reflected[2] >= 0:
        return None
    distance = -point[2] / reflected[2]
    return tuple(p + distance * r for p, r in zip(point, reflected))


if __name__ == "__main__":
    point, normal = (0, 0, 0.6), (0, -1, 0)
    for height in (1.4, 2.2):
        hit = ground_hit(point, normal, (0, -4, height))
        assert hit is not None and abs(hit[2]) < 1e-9
        print(f"camera_z={height:.1f} -> ground_hit={tuple(round(v, 3) for v in hit)}")
    assert ground_hit(point, normal, (0, -4, 0.6)) is None

Run python3 mirror_ray.py. The tested output is:

camera_z=1.4 -> ground_hit=(0.0, -3.0, 0.0)
camera_z=2.2 -> ground_hit=(0.0, -1.5, 0.0)

The plate stayed still. Raising the camera by 0.8 meters moved the reflected floor point 1.5 meters, from three meters ahead of the plate to 1.5 meters ahead. In the carrier scene, I traced the bumper’s reflected directions to the actual ground and put fill there. In the later frame, the chrome held dark ground structure and the card was gone.

2. Put the road where the reflection lands

Two wet paths lead toward the service bay. Snow is visible near the camera and around the door.

I built the winter lot around the service building. Its lit door gave the camera a destination; parked rows kept the route narrow. The road starts broad in the foreground and runs straight to that door. Near the lens, snow crosses in long streaks. At the far wall it becomes flecks against the light. I judged the snowfall by where I could see it.

A flat dark stripe gave the lamp little to describe. I cut shallow depressions into the terrain and shaped uneven slush at their edges. Their shoulders caught light. The wet middle sent a different reflection toward the camera. A worn path has a cross-section. Snow can remain beside it.

Cover: snow on the upper-deck vehicles, amber markers along the carrier, dark ground in the chrome.

I bought a hauler model, then rebuilt it into Online Auto Connection truck #171 with permission from the owner, Mark Subjeck. The other vehicles also began as licensed models that I modified for the film.

The loaded carrier beside the parked rows.

I placed the overhead camera to check where the deck sat among those rows. At road level, the rows compress toward the service building. I returned to the low camera to judge the paths and the snow.

3. Build a smaller scene to test the reflection

Three renders, same plate: road light off, road light on, then a second camera with the light left alone. Look at the floor first. If the bright patch is not the point the plate is reflecting, more power only lights the wrong ground. Then look at the metal.

A short road, a box on wheels, and a polished front plate are enough. The adapted rut_strip.py writes a 10 m road strip with two shallow troughs. Run python3 rut_strip.py demo-ruts.obj with a new output filename. It reports 3,321 vertices and a lowest height of 0.007 m. The full generator is under Scripts.

Import the OBJ into Blender at one unit per meter and check that its upper face normals point up. The surface starts at 0.025 meters. Each track center drops by up to 0.018 meters. Change the track width, then inspect the shoulder from the camera.

Use a beveled cube for the body and four cylinders for wheels. Check their contact with the road. Put a thin plate at the front, facing back toward the first camera.

For the plate’s Principled shader, start with metallic 1 and roughness around 0.06. Start the exposed track around 0.18 roughness and the pale snow around 0.65, with fine Noise Texture feeding a small Bump node. These are starting values for this test.

To keep snow beside the tracks, take the X coordinate from Geometry Position through Separate XYZ. Measure the distance to track centers at -0.7 and 0.7 meters and keep the smaller distance. That drives a Mix between wet material and snow, with a starting blend from 0.14 to 0.33 meters from either center. If the tracks read as ink marks, inspect the height profile, transition width, and light direction one at a time.

Put the first camera low and aim it at the plate. Use a broad area light to illuminate the road ahead of the box. Place the second camera about 0.8 meters higher, still aimed at the plate. Compare the third render with the second before changing roughness.

For a moving test, key a straight camera push and inspect its first, middle, and last frames. Check wheel contact and whether flakes remain visible near the lens and farther into the scene. Place a recorded mechanical contact at a visible action, then listen on speakers and headphones.

4. Know which frame you judged

At thumbnail size the truck reads and the chrome does not. At 1280 × 720 I could see dark ground in the bumper, snow on the upper deck, and whether a wheel sat on the lot. That is the file I kept a receipt for.

Run python3 frame_receipt.py your-render.png to print image dimensions, byte count, and SHA-256. For the cover it returned 3,219,108 bytes and a hash matching the handoff receipt. The full script is under Scripts. Its pixel_review field reads NOT_ESTABLISHED_BY_THIS_SCRIPT. The hash is not the review.

5. Four questions from the directors

Adam Greenberg described wetting the streets for Terminator 2 to darken gray pavement, and separately described moving reflections in car windows and hoods. Where did my wet road enter the bumper’s reflection?

Which visible fixture revealed the snow? Dan Laustsen’s backlit rain in John Wick: Chapter 3 sent me to the foreground, then the service door. The overhead posed a different question after Chad Stahelski’s discussion of readable wide action: could I place the carrier among the parked rows before cutting closer?

Christopher Nolan’s account of building sound effects and music together on Dunkirk left one question: which mechanical action deserved the sound cut?

6. The cold open

The 12-second cold open below is a 390 × 220 phone-size preview with stereo sound. It starts in darkness, then reveals exhaust, falling snow, a headlamp, and red bodywork. Play it with sound on.

Your browser does not support embedded video.

Open the 12-second preview with sound.

The clip is from One Lap, my commercial for AutoLensAI.

Glass, wet asphalt, and chrome show pieces of a set that sit outside the frame.

Scripts

rut_strip.py — road mesh generator

"""Adapted educational OBJ generator; a new toy surface, not film geometry."""
import argparse
import json
from math import exp
from pathlib import Path


def surface_height(x):
    distance = min(abs(x - 0.7), abs(x + 0.7))
    return 0.025 - 0.018 * exp(-((distance / 0.19) ** 2))


def write_obj(path):
    nx, ny = 80, 40
    vertices = [(-2 + 4 * i / nx, -1 + 10 * j / ny,
                 surface_height(-2 + 4 * i / nx))
                for j in range(ny + 1) for i in range(nx + 1)]
    with path.open("x", encoding="utf-8") as stream:
        stream.write("# Adapted educational rut strip, units are meters\n")
        for x, y, z in vertices:
            stream.write(f"v {x:.6f} {y:.6f} {z:.6f}\n")
        for j in range(ny):
            for i in range(nx):
                a = j * (nx + 1) + i + 1
                stream.write(f"f {a} {a+1} {a+nx+2} {a+nx+1}\n")
    assert len(vertices) == 3321
    assert abs(min(v[2] for v in vertices) - 0.007) &lt; 1e-9
    return {"vertices": len(vertices), "quads": nx * ny,
            "minimum_z_m": round(min(v[2] for v in vertices), 6),
            "geometry_checked": True, "rendered": False}


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("output", type=Path)
    args = parser.parse_args()
    print(json.dumps(write_obj(args.output), sort_keys=True))

frame_receipt.py — PNG file receipt

"""Adapted read-only PNG receipt; integrity is not visual approval."""
import argparse
import hashlib
import json
import struct
from pathlib import Path


def inspect_png(path):
    digest = hashlib.sha256()
    with path.open("rb") as stream:
        header = stream.read(24)
        if header[:8] != b"\x89PNG\r\n\x1a\n" or header[12:16] != b"IHDR":
            raise ValueError("Expected a PNG with an IHDR header")
        digest.update(header)
        for block in iter(lambda: stream.read(1024 * 1024), b""):
            digest.update(block)
    width, height = struct.unpack("&gt;II", header[16:24])
    return {"file": path.name, "width": width, "height": height,
            "bytes": path.stat().st_size, "sha256": digest.hexdigest(),
            "pixel_review": "NOT_ESTABLISHED_BY_THIS_SCRIPT"}


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("image", type=Path)
    args = parser.parse_args()
    print(json.dumps(inspect_png(args.image), indent=2))

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

I Had to Build What the Bumper Could See

Daniel Romitelli — Wed, 30 Sep 2026 05:39:50 +0000

After I removed the light card, I checked the bumper from the camera view and worked backward to the part of the apron it reflected. The road had to carry the light now. Its surface and the camera angle had to agree.

That changed how I worked on the winter lot in One Lap. The bumper needed ground to reflect. The road needed worn paths with actual shape. Snow had to remain visible between the camera, the parked cars, and the lit service building. Each part affected what the next camera view could show.

1. The rectangle in the bumper

A polished surface responds to the direction it faces and the position of the camera. Roughness spreads a reflection; it cannot bring an unlit patch of ground into a bumper that is looking somewhere else. I had to inspect the truck from the camera position, then work on the part of the lot that appeared in its metal.

A small numerical exercise makes the camera change plain. It uses a vertical reflective plate and a flat floor. Raise its camera by 0.8 meters and the floor point appearing in the plate moves 1.5 meters. These are adapted teaching coordinates, separate from my film scene. The scripts and Blender exercise below use generic geometry and no production assets or settings.

Save this as mirror_ray.py. It runs with Python 3.10 or newer and needs no packages:

"""Adapted educational example; no film assets, scene, or production settings."""
from math import sqrt


def unit(vector):
    length = sqrt(sum(v * v for v in vector))
    if length == 0:
        raise ValueError("A direction must have nonzero length")
    return tuple(v / length for v in vector)


def ground_hit(point, normal, camera):
    normal = unit(normal)
    view = unit(tuple(c - p for c, p in zip(camera, point)))
    dot = sum(n * v for n, v in zip(normal, view))
    reflected = tuple(2 * dot * n - v for n, v in zip(normal, view))
    if reflected[2] >= 0:
        return None
    distance = -point[2] / reflected[2]
    return tuple(p + distance * r for p, r in zip(point, reflected))


if __name__ == "__main__":
    point, normal = (0, 0, 0.6), (0, -1, 0)
    for height in (1.4, 2.2):
        hit = ground_hit(point, normal, (0, -4, height))
        assert hit is not None and abs(hit[2]) < 1e-9
        print(f"camera_z={height:.1f} -> ground_hit={tuple(round(v, 3) for v in hit)}")
    assert ground_hit(point, normal, (0, -4, 0.6)) is None

Run python3 mirror_ray.py. The tested output is:

camera_z=1.4 -> ground_hit=(0.0, -3.0, 0.0)
camera_z=2.2 -> ground_hit=(0.0, -1.5, 0.0)

The plate did not move. At the lower camera height, it showed ground three meters from the plate. At the higher position, it showed ground only 1.5 meters away. If I light the first patch and then raise the camera, the highlight can vanish from the metal. The script computes an intersection; it does not render or judge an image. I still inspect the actual frame at native size.

For the carrier section, I checked reflected directions against the actual ground surfaces in the scene and put fill on the ground the camera could see through the bumper. In the later carrier view, the chrome carries a broad silver response with dark ground structure through it. I can no longer pick out the card as a rectangle.

2. Put the road where the reflection lands

The road-level approach. Two wet paths lead toward the service bay; its lamps catch snow at several distances.

The building gives this frame a destination. Parked vehicles close in from both sides. Snow near the camera crosses in long streaks; smaller flecks remain around the lit door. The exposed road pulls the eye through the middle.

I judged the snowfall by where it could be seen. The nearest flakes cut across the foreground. Around the building they become small marks against the door and work lights. That distance change matters when the storm and the road have to occupy the same space; an even screen of white would cover the scene without telling me how deep the lot is.

A dark stripe could have occupied those pixels, but a flat stripe gave the lamp little to describe. I shaped the exposed paths as shallow depressions with uneven slush at their edges. Their shoulders catch light. The wet middle sends a different reflection toward the camera. A worn path has a cross-section. Snow can remain beside it.

The carrier needed the same weather as the lot around it. The overhead view helped me check the arrangement before returning to the low camera:

Snow stays on the upper-deck roofs while amber markers trace the carrier’s length.

From above, I could see where the loaded deck sat among the parked rows. At road level, those same rows compress toward the service building. I used the two views for different checks: one for the lot’s geography, one for the path and weather that a viewer would feel while approaching it.

The selected cover let me inspect a third relationship at native size. Snow covered the hoods, roofs, and windshields of vehicles on the upper deck, while the carrier’s small markers still described its length. The weather was present on the payload as well as in the air and on the road. That gave the truck a place in the winter lot instead of treating it as a clean object dropped in front of one.

3. Build a smaller scene to test the reflection

Make three renders: the first camera with the road light off, the same camera with it on, then a second camera with the light unchanged. Keep the polished front plate material fixed. That sequence lets you see whether the road enters the reflection and what the camera changes.

The film scene used licensed vehicle models. For this exercise, make a short road and an unbranded vehicle from simple shapes. The plate at its front is the surface to watch.

Shape the road

Save this second adapted script as rut_strip.py. It writes a four-meter-wide, ten-meter-long Wavefront Object (OBJ) mesh with two shallow troughs:

"""Adapted educational OBJ generator; a new toy surface, not film geometry."""
import argparse
import json
from math import exp
from pathlib import Path


def surface_height(x):
    distance = min(abs(x - 0.7), abs(x + 0.7))
    return 0.025 - 0.018 * exp(-((distance / 0.19) ** 2))


def write_obj(path):
    nx, ny = 80, 40
    vertices = [(-2 + 4 * i / nx, -1 + 10 * j / ny,
                 surface_height(-2 + 4 * i / nx))
                for j in range(ny + 1) for i in range(nx + 1)]
    with path.open("x", encoding="utf-8") as stream:
        stream.write("# Adapted educational rut strip, units are meters\n")
        for x, y, z in vertices:
            stream.write(f"v {x:.6f} {y:.6f} {z:.6f}\n")
        for j in range(ny):
            for i in range(nx):
                a = j * (nx + 1) + i + 1
                stream.write(f"f {a} {a+1} {a+nx+2} {a+nx+1}\n")
    assert len(vertices) == 3321
    assert abs(min(v[2] for v in vertices) - 0.007) < 1e-9
    return {"vertices": len(vertices), "quads": nx * ny,
            "minimum_z_m": round(min(v[2] for v in vertices), 6),
            "geometry_checked": True, "rendered": False}


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("output", type=Path)
    args = parser.parse_args()
    print(json.dumps(write_obj(args.output), sort_keys=True))

Run it with a new filename:

python3 rut_strip.py demo-ruts.obj

The tested run produced 3,321 vertices and 3,200 faces, with a lowest height of 0.007 meters. The generator refuses to overwrite an existing file. I checked its geometry and counts.

Import the OBJ into Blender with one unit per meter. Check that the upper face normals point up. The height function starts at 0.025 meters and lowers each track center by up to 0.018 meters. Change the width value, then look from the camera again. You are changing the shoulder that catches light, not merely painting a darker line.

Give the metal a place to look

A beveled cube can stand in for a vehicle body. Put four cylinder wheels under it and check that they meet the road. Add a thin plate at the front, facing back toward the first camera. No badge, grille, or purchased asset is needed.

For the material test, use a Principled shader on each surface:

Start the front plate at metallic 1 and roughness around 0.06. Its reflection should reveal the set around it.
Make the exposed track darker and smoother, with roughness around 0.18. Give the snow a pale material at roughly 0.65 roughness, with fine Noise Texture feeding a small Bump node. Keep the wet road smoother than the snow.
To place snow beside the tracks, take the X coordinate from Geometry Position and Separate XYZ. Measure its distance to track centers at -0.7 and 0.7 meters, keep the smaller distance, and map that value into a Mix between wet material and snow. Start the blend around 0.14 to 0.33 meters from a center. The mesh supplies the dip; the shader supplies the surface response.

Those roughness and blend values are starting controls, not measured settings from One Lap. If the tracks read as two ink marks, inspect their height profile, transition width, and light direction one at a time.

Move the camera, keep the light

Put one camera low and aimed at the front plate. Place a broad area light so it illuminates the road ahead of the vehicle, rather than presenting a bright card directly to the plate. Make a second camera about 0.8 meters higher or to the side. Keep both cameras aimed at the plate.

Compare the third render with the second. Its reflection may change even though the light stayed put. Return to the ray script before adjusting roughness.

For a short moving study, key a straight camera push that clears the wheels and ground. Inspect its first, middle, and last frames before rendering the full range. A sparse Geometry Nodes field of instanced flakes can test whether snow remains legible near the lens and at the building. Record your own wind and one mechanical contact for sound. Place that contact at a visible action, then listen once on speakers and once on headphones.

4. Know which frame you judged

I inspected the selected carrier render at its native 1280 × 720 size. That let me see the upper-deck snow, marker lamps, bumper response, and dark parts of the lot. I also needed to know that I was opening the selected file rather than an earlier lighting pass or a smaller preview.

This third adapted script, frame_receipt.py, records a Portable Network Graphics (PNG) file’s dimensions, byte count, and SHA-256 digest. It reads the file without changing it:

"""Adapted read-only PNG receipt; integrity is not visual approval."""
import argparse
import hashlib
import json
import struct
from pathlib import Path


def inspect_png(path):
    digest = hashlib.sha256()
    with path.open("rb") as stream:
        header = stream.read(24)
        if header[:8] != b"\x89PNG\r\n\x1a\n" or header[12:16] != b"IHDR":
            raise ValueError("Expected a PNG with an IHDR header")
        digest.update(header)
        for block in iter(lambda: stream.read(1024 * 1024), b""):
            digest.update(block)
    width, height = struct.unpack(">II", header[16:24])
    return {"file": path.name, "width": width, "height": height,
            "bytes": path.stat().st_size, "sha256": digest.hexdigest(),
            "pixel_review": "NOT_ESTABLISHED_BY_THIS_SCRIPT"}


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("image", type=Path)
    args = parser.parse_args()
    print(json.dumps(inspect_png(args.image), indent=2))

Run python3 frame_receipt.py your-render.png. Under Python 3.10.12, the script was run against the selected cover and another reviewed image. The cover’s dimensions, byte count, and digest matched its handoff record.

The last field has a narrow meaning. Matching bytes establish which file I inspected. They say nothing about whether the snow has depth or a wheel touches the ground. Open the frame at its actual size for those decisions. For the exercise, save the three comparison renders with their camera and light choices, then make a receipt for each.

5. Four questions I brought back from the directors

I used interviews as checks on my own shots. Each source gave me a question I could ask of the lot.

Adam Greenberg: When he described wetting streets for Terminator 2, he also discussed moving lights appearing in a car’s windows and hood. Where should the road carry light, and what does the carrier reflect? Greenberg in American Cinematographer.
Dan Laustsen: His account of backlit rain in John Wick: Chapter 3 pushed me to check snow at more than one distance. Which visible fixture reveals it? Laustsen in American Cinematographer.
Chad Stahelski: His discussion of readable wide action sent me back to the overhead image. Can a viewer place the carrier among the parked rows before the cut moves closer? This is how I used the principle, not a copied John Wick setup. Stahelski in Filmmaker.
Christopher Nolan: His account of building sound effects and music together during Dunkirk raised an editing question. Which mechanical action deserves the sound cut? Nolan at Cannes.

6. Coming soon: the AutoLensAI commercial

The 12-second clip below is a phone-size preview from One Lap, the upcoming commercial for my AutoLensAI application. It has a stereo audio track. The picture starts in darkness, then reveals exhaust, falling snow, a headlamp, and red bodywork. Play it with sound on.

Your browser does not support embedded video.

Watch the 12-second preview of the One Lap commercial with sound.

The AutoLensAI commercial is coming soon. Glass, wet asphalt, and chrome can all show pieces of a set that sit outside the frame.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

You Want Engineers Using AI on the Job. Why Turn It Off in the Interview?

Daniel Romitelli — Sun, 20 Sep 2026 08:58:52 +0000

Forty-five minutes in, I'd walked through systems I built. The architecture, the integrations, the auth decisions, what broke in production and what I did about it. There are working products online with demos anyone can open. Nobody opened them.

Then came the coding exercise. Share your screen. Open the editor. Turn off AI, disable autocomplete. Write a program that calls an API and returns a picture of a bird and a fact about it.

The endpoint was supplied. So were the library, the response shape, and the expected output. The product and architecture decisions were already out of the picture. What remained was implementing a tiny, prescribed task without the tools I'd normally use.

I've shipped a lot of things. I have never once shipped a bird.

When I talk to someone who knows a field I know, I can usually hear it inside five minutes. They ask about a condition I haven't mentioned. They tell me why an approach works here and turns into a problem two systems over. Change an assumption on them and they follow it downstream without stopping to reload.

I've found that in software and I found it in a woodworking shop. People who have actually done the work hand you details you can push on. The part that took three times longer than the estimate. The shortcut they took once and will never take again. You always have somewhere left to go.

A first impression still needs checking. Keep going with the follow-ups, because somebody can stumble over a sentence and still understand the problem cold, and somebody else can be very smooth and very empty. But by the end of a real technical conversation, an interviewer should be holding evidence.

So why spend forty-five minutes asking about my experience, decline to look at the results, and then let a small unaided API call cast the deciding vote?

1. The woodworking shop

I owned a woodworking shop with expensive equipment in it. A jointer, a planer, a table saw with a fence I could trust. That equipment took rough lumber and turned it into square, four-sided, dimensioned stock faster than I could ever do it by hand. That was the entire point of buying it.

A stack of perfectly milled boards still wasn't furniture. I had to know what I was building, how the joints would carry load, where the material would move across a winter, and whether the finished piece was any good. If I couldn't put it together, the machinery didn't excuse that. The gap was mine.

Some excellent woodworkers can walk onto a site with basic tools and build something beautiful. I respect that ability. It doesn't follow that using a planer proves somebody lacks it, and making a person repeat every prep operation by hand will not tell you whether they can deliver the finished piece. It'll tell you they can push a hand plane.

I bought equipment to take back time that didn't need to be spent, and to get consistency I couldn't hold by hand at volume. What I got in exchange was attention I could spend on the design and on the parts of the job the machines hadn't solved.

AI isn't a planer. It can hand you something confidently wrong, which makes verification part of the engineering problem instead of a formality. But verification is a thing you can design, automate, and hold to a standard.

The objective is a well-built result. I don't hand out extra credit for preserving unnecessary labor on the way there.

2. Nobody panics about the teleprompter

Here's a tool that's been around since the 1950s.

An executive can deliver a prepared speech from a piece of glass. A news anchor can read a newsroom's script. We understand that the person on camera may be working from words prepared with other people. Imagine interrupting a CEO halfway through a town hall: hang on, kill the prompter, let's find out if he actually believes any of this.

I'd have questions about what he said. Whether he'd memorized it wouldn't settle them.

Apply the same test to a quantitative analyst: take away the computer and ask for long division on a legal pad. You could learn something about their arithmetic. You'd still have questions about whether they understand the risk they're being asked to model.

Spellcheck. Route planning. CAD. Compilers, for that matter. We could spend the afternoon removing assistance and congratulate ourselves on how much harder we'd made the job.

The honest objection is that these tools aren't equivalent. A teleprompter displays a prepared script; a generative model can supply reasoning the person never had. Fine. That difference is worth examining. Tell me which ability this role requires unaided, why this exercise measures it, and I'll take that seriously.

What I won't accept is the reflex. Finding a tool in someone's hand is the beginning of a question, not the answer to it.

3. Walowitz is mine

I've spent the better part of a decade building my own assistant. Walowitz is part of it, in my editor and across the other places I use it. I haven't released it publicly. I built it for myself.

Cognitive relay is one reason I built it. I can read a raw log, see the relationship in it fast, and still lose the explanation somewhere between knowing the answer and getting it out of my mouth on demand. Anxiety makes that worse. I have understood a problem and blown the explanation in front of people. It doesn't feel like a communication problem. It feels like being found out.

For recall, the material I want back is my own writing: notes, comments, ledgers, and decisions I recorded while doing the work. Retrieved passages carry references back to their source. That lets me return to the record rather than pretend a new model-generated explanation is something I wrote years ago.

Exact identifiers and semantic search do different jobs, so I run both and put exact matches first when they're available. Asking for the function where I handled a specific case isn't the same request as asking for everything related to an idea. The combined results are deduplicated before entering the context.

Walowitz cancels retrieval when a newer request overtakes it and expires temporary context on purpose. Material from the previous subject can be completely accurate and completely wrong for the conversation happening now. Skip that and you've built a very sophisticated way to remember the wrong thing at exactly the right moment.

Recall isn't permission to act. As I extend the assistant into screens for my kids, a request to trace the letter B shouldn't inherit authority to touch the router. If I ever confuse those scopes, I have considerably larger problems than a coding interview.

The practice ledger has separate states for aided, unaided, and unrecorded turns. Zero means nobody measured this. It doesn't mean unaided. Within a tracked turn, an aided label doesn't revert merely because the display later clears. The ledger still has paths that don't record aid. I won't pretend it is finished.

I chose those states because a ledger based only on the session's opening mode would record a preference and call it behavior. A flattering default becomes indistinguishable from a real measurement once somebody writes it down.

Pick one of those decisions. Keep asking. Change a condition and see whether I can follow the consequences. We can discuss those choices without distributing private source or a former employer's work.

But first, apparently, we need to establish whether I can fetch the bird unassisted.

4. Knowing what not to build

The first decision on anything is whether it should exist. Who needs it, what are they trying to do, what would make it worth using.

While one team builds a capability from scratch, somebody else may find a maintained service or an open model that already does it and spend those weeks shipping the part nobody has solved. Checking that is part of the job. License, operating cost, what you'd depend on, what happens when it changes under you.

There are real reasons to build your own. The available option might require sending customer records somewhere they're not allowed to go, or become unaffordable at the volume you're planning for. I want those limits investigated before anybody commits the team to months of work, not discovered afterward.

That judgment is invisible in a test where the build-versus-buy decision was made for me before I opened the editor.

5. Companies already changed the test

Karat surveyed 400 engineering leaders across the United States, India and China. Seventy-one percent said AI was making technical skills harder to assess, while 62% of organizations still prohibited it in technical interviews.

Karat co-founder Mo Bhende put it to IEEE-USA plainly: "the fundamental job has changed, but technical interviews, as we know them, have not."

Canva announced in June 2025 that backend, frontend and machine-learning candidates were expected to use AI tools. Its pilot found that some candidates who could code struggled anyway, because they couldn't guide the tools or recognize a bad suggestion. Canva kept code fluency and technical depth as requirements. Allowing assistance exposed a weakness the old exercise hadn't been designed to assess.

Meta's July 2025 internal memo gave two reasons for developing AI-enabled interviews: they better represent the working environment, and they make AI-based cheating less effective. 404 Media reported the memo with WIRED. Read that second one twice. Allowing visible assistance was part of the stated rationale for protecting the assessment.

CoderPad's 2026 survey found its hiring respondents split 34% prohibiting, 46% allowing broadly or with constraints, and 20% deciding case by case. Asked what shows skill when AI is allowed, 66% picked catching and fixing its mistakes and 56% picked explaining trade-offs and correctness.

Excluding AI is a choice. It isn't a settled professional requirement, and it shouldn't be presented to a candidate as one.

Anthropic's candidate guidance, updated July 2025, prohibits AI in live interviews and take-homes unless permission is given. The same page describes using Claude to develop questions, draft communications, analyze hiring metrics and source candidates. Claude doesn't make the hiring decisions.

Preparing a question and answering one are different activities, so different rules on the two sides aren't automatically hypocrisy. But the restriction still owes the applicant an explanation. Which ability does this job require unaided, and why does this exercise deserve the weight you're giving it?

6. What about cheating

The concern is legitimate and I'm not going to wave at it.

Fabric's January 2026 report analyzed 19,368 interviews on its own AI interview platform. It flagged 38.5% of candidates for cheating behavior, and roughly 61% of those flagged scored above its passing threshold. That's one vendor's detection and scoring, not an independently established industry rate, and it doesn't establish that every flagged person cheated.

If the people you believe haven't demonstrated competence are clearing your threshold at that rate, the scoring deserves investigation alongside their behavior. Adding a ban doesn't establish that the score measures the ability you're hiring for.

There's risk pointed the other way too. A 2020 NC State and Microsoft study put 48 computer science students through a whiteboard problem. The ones solving it while watched and narrating performed about half as well as the ones working privately. Observation and narration both changed at once, and they were students, not principal engineers. The size of that gap is still worth noticing before anyone insists the stressful version is the honest one.

AI can also undermine learning, and I'm not leaving that out because it complicates my case. In Anthropic's January 2026 randomized study, 52 mostly junior engineers learned an unfamiliar Python library. The AI group averaged 50% on a quiz covering concepts they'd used minutes earlier, against 67% unaided. The speed gain wasn't statistically significant.

Inside the AI group, people who asked conceptual questions or followed generated code with questions to understand it scored better than people who mostly delegated. That comparison was exploratory and can't establish cause. It still gives an interviewer a behavior worth investigating rather than treating every use of assistance as equivalent.

I'm not claiming AI always makes experienced engineers faster, either. METR's February 2026 update revisited its earlier 19% slowdown finding and reported that the follow-up couldn't reliably estimate the current effect, largely because developers declined to participate or withheld tasks they didn't want to attempt without AI. The experiment was missing people and work that could have changed the answer.

A fast, plausible mistake is still a mistake.

7. Keep the birds, then change the response

I'm not asking anybody to stop testing me. I spent part of my twenties in military aviation, where competence was checked by people qualified to check it. I have no objection to proving I can do the work.

A failure scenario can justify taking a capability away. Then the question is how the person handles that failure in the job they're trained to do. That's an explanation I can understand. "We always turn the tools off" isn't one.

So keep the bird exercise. Then make it worth the hour.

Change the response halfway through. Introduce a timeout. Hand me an implementation that looks reasonable and has a defect sitting in it, and ask what needs testing before anyone should trust it. Let me use my tools, including automated review, and watch whether my checks find the problem and what I do once they have. Make me explain why the evidence I produced addresses the changed requirement.

Then take a problem with real decisions in it. Talk about what the user needs and let me investigate the options. Ask why I'd use an existing service instead of building one. Change a constraint and follow it with me. I can use research or a model to explore it, and you can ask what made me accept one approach and abandon another.

Work I can legitimately share gives you another place to start. A demo leads into the decisions behind it without exposing a former employer's code.

Nobody needs to complete an unpaid production project for this. A bounded exercise plus a serious technical conversation gives you several kinds of evidence instead of one convenient proxy. Give candidates comparable access and consistent criteria. That addresses an important part of fairness without pretending the working environment doesn't exist.

If your process can make time to watch my screen share but not to open what I built, the priorities are backward.

And if you're using a tool to assess me, own how you interpret its output. That standard runs both directions or it isn't a standard.

8. What I'd want to know before I said yes

I wouldn't want to join a company that expects end-to-end AI engineering and treats an unaided API call as the deciding test, particularly if the people running it can't connect it to the role. I'd question whether their understanding of the job had caught up to what they're asking for. I don't need to prove their whole stack is obsolete before deciding to pass.

There's a question worth asking out loud in either direction. Am I being hired to work inside an established engineering practice, or to modernize the software and change how it gets built? The second one comes with budget decisions, approval processes, and people outside the development team who have to agree. That's a different job, and whoever takes it needs the authority and the resources to do it.

A paycheck doesn't resolve a mandate nobody defined. I'm not interested in spending my first six months arguing for the methods I was supposedly hired to bring.

I'm asking you to put the bar where the work is.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

CommandCanvas: Building Around Shared Objects

Daniel Romitelli — Sat, 05 Sep 2026 05:27:41 +0000

In CommandCanvas, a sketch can become a structured diagram beside its source. A person can rearrange the result, a collaborator can edit it, and a supported agent host can address it through WebMCP. Each operation refers to objects in the same room, with identities and versions that survive changes in how people interact with them.

I built the shared canvas mutations around one commit path. Pointer input, voice, and tools can propose changes, but the server must establish which member authorized a change and whether it still applies to the current room state. Adding another input should not introduce another definition of what counts as saved.

Objects that people and tools can both address

A note in CommandCanvas has an identity, a title, a position, dimensions, and a version. Its content follows a note schema. A task board has columns and tasks. A sketch has strokes and points. A structured diagram has its own payload and can retain a reference to the sketch it came from.

These objects share spatial behavior while keeping their different kinds of content. Moving a board should preserve its tasks. Resizing a sketch should preserve the drawing. An operation can address a particular object and check its version before changing it. The object model makes those distinctions explicit.

That gives a tool request something concrete to address: the selected sketch, a populated board, or an existing object to transform. It also constrains the result. A model response has to fit a supported object schema before the application can submit it as a canvas change.

The distinction is useful when people and an agent take turns. They can refer to the same object without reconstructing its content from a screenshot or carrying a separate copy into another conversation.

Where a canvas change commits

The browser holds the visible objects, current selection, and room state. Direct input produces canvas commands. WebMCP and optional embedded voice pass through a capability runtime that validates their arguments and current context before invoking the command adapters.

WebMCP exposes capabilities to a supported agent host. Embedded voice uses a separate OpenAI Realtime connection and a narrower set of capabilities. Both reach the application's command adapters; neither gets its own route around room membership or revision checks.

The durable room path looks like this:

flowchart TD
  subgraph browser[Browser workspace]
    direct[Pointer, touch and hand intents] --&gt; command[Canonical canvas command]
    tools[WebMCP and optional voice] --&gt; capability[Capability validation]
    capability --&gt; command
  end
  subgraph server[Server and database]
    api[Authenticated room API] --&gt; guards[Membership and revision checks]
    guards --&gt; commit[Postgres mutation transaction]
    commit --&gt; records[Objects, room revision and receipt]
  end
  command --&gt;|Command, source and base revision| api
  records --&gt;|Authoritative readback| origin[Originating canvas]
  records --&gt;|Revision notification| reload[Other clients reload room state]
  reload --&gt; peers[Collaborator canvases]
  presence[Presence and cursor Broadcast] -.-&gt; origin
  presence -.-&gt; peers

The browser proposes a command; the server establishes who may commit it. Solid arrows trace the durable command and synchronization path. Dotted arrows show participant presence and cursor traffic, which do not create mutation receipts. On narrow screens, scroll the diagram horizontally.

The request carries a command ID, its reported input source, and the room revision the client worked against. The HTTP route verifies the bearer token. The room service loads membership, derives the actor, reads the current canvas, and builds a mutation plan. The database operation checks the expected revision again when committing the change.

That second check matters because the room can change between the server reading it and attempting the write. A command prepared against revision 12 cannot commit against revision 13 by assuming its original view still applies. The stale request is rejected.

The shared revision makes the order of committed changes explicit. It also makes conflicts coarser: two people working on different objects can still contend for the same room revision. The application does not automatically merge those edits. That is the cost of using a single room revision as a commit precondition.

The transaction persists the object changes, advances the room revision, and writes the receipt. The server reloads the authoritative state and checks for the expected receipt before returning success. Other clients receive a compact revision notification and reload the room. The room service contains that commit and readback path.

Why drag previews stay out of history

Applying that commit path to every cursor sample would make ordinary movement compete with object edits for room revisions. It would also fill history with intermediate positions that are of little use when someone wants to undo a completed action.

Cursors and movement previews therefore remain ephemeral. Supabase Presence describes connected participants, and Broadcast carries the frequent updates. Stable object changes go through the mutation path and produce the receipt that other clients can verify.

The room can show motion before it has a new committed state. That distinction is necessary for responsive interaction, and it has to survive into the UI: seeing another person's cursor move is not evidence that their edit was saved.

Turning a sketch into another object

Sketch interpretation introduces a longer gap between preparing a change and trying to commit it. A provider can finish its work after the drawing it received has changed.

The transformation captures the selected sketch's ID and version, then rasterizes it to a PNG in the browser. A provider interprets that image and returns structured output. The application validates the payload, checks the source reference, and checks that the requested output kind agrees with the result.

The person can keep drawing while the provider works. Before submitting the new object, the transformation checks the source version again. If the sketch changed, interpretation may have succeeded, but the application refuses to create the diagram from that result. The provider time has already been spent; accepting its answer anyway would attach an outdated interpretation to the current work.

sequenceDiagram
  participant C as Canvas
  participant P as Interpretation service
  participant R as Room API
  C-&gt;&gt;C: Capture sketch ID and version
  C-&gt;&gt;P: Rasterized sketch and instructions
  P--&gt;&gt;C: Structured result
  C-&gt;&gt;C: Validate payload and recheck source
  alt Source unchanged
    C-&gt;&gt;R: Create linked diagram at current room revision
    R--&gt;&gt;C: Committed object and receipt
  else Source changed
    C-&gt;&gt;C: Refuse the stale transformation
  end

The generated diagram becomes a separate canvas object. The original sketch remains available, including when interpretation fails or its source has changed. On narrow screens, scroll the diagram horizontally.

Creating the result still uses the ordinary command path. After the save, the transformation looks for the new diagram and its receipt in the authoritative returned state. A provider response alone is not enough to report that the room contains the diagram.

The source-version check and the room-revision check address different changes. One checks the drawing used for interpretation; the other protects the shared commit. Neither establishes that the model understood the drawing correctly.

That is why the result appears beside the preserved source. Someone can compare them, point out a missing relationship, or try another interpretation without recovering an overwritten sketch. The output remains a structured object that later commands can address. These checks and the separate-object creation are implemented in the sketch transformation orchestrator.

Deciding when a hand is drawing

The commit checks only help after an input has produced the right command. Hand tracking has an earlier decision to make: whether a moving fingertip should create ink at all. A valid room membership and a current revision cannot answer that.

On the default local path, an opt-in camera feeds MediaPipe Hand Landmarker in a browser worker. The interaction layer receives landmarks and decides what action they may produce. The index fingertip supplies the pen position. A separate thumb-to-middle-finger clutch supplies pen-down intent.

That separation lets someone move their hand to the start of another stroke without drawing a line across the intervening space. Closing the clutch engages the pen; opening it lifts the pen. A later engagement begins another stroke with its own identity.

The clutch uses different engagement and release thresholds, along with temporal confirmation. This hysteresis prevents a small amount of motion near one threshold from repeatedly switching the pen on and off. The drawing policy also distinguishes provisional thresholds from calibrated ones.

Those rules are testable, but a state-machine test cannot tell me whether the interaction feels comfortable after ten minutes, or whether it holds up under poor lighting and partial occlusion. Physical-hand usability remains experimental. Pointer, touch, and typed controls keep the workspace usable while that work continues.

The default hand path processes camera frames locally. Asking for sketch interpretation is a separate operation: it sends an image of the selected drawing for interpretation. The distinction matters when explaining what leaves the device.

The agent is not the actor

Once several inputs could change the same room, the receipt needed to describe both the member responsible for a committed change and the path that requested it.

The earlier schema already had a separate source column. Its rule was too restrictive: a WebMCP source could only accompany an agent actor. The receipt still retained the authorizing user's ID and required room membership, but its actor classification and display name presented the action as belonging to a generic agent.

The correction binds durable WebMCP canvas mutations to the authenticated member. A host is classified as human; another room member is classified as participant. The source remains webmcp. The SQL wrapper derives the classification from stored membership:

v_effective_actor_type := case v_member_role
  when 'host' then 'human'
  when 'participant' then 'participant'
  else null
end;

This aligns the receipt's visible attribution with the identity that authorized the request. It also allows the reader to distinguish a tool-originated change from direct interaction.

The compatibility work was less visible. The wrapper accepts request shapes from both releases and canonicalizes them before persistence. The table constraint continues to permit the historical agent-plus-WebMCP shape so older rows remain valid; a stricter validator in the write path enforces the new behavior. The migration adds the broader constraint without scanning historical rows immediately, and a separate migration performs validation. The attribution migration shows both responsibilities.

There are limits to what this receipt says. The source field records an application input path; it does not authenticate a particular external agent host. The current service also normalizes most participant inputs to collaborator, preserving webmcp and system separately. It therefore cannot answer every question about whether a remote participant used touch, voice, or a pointer.

The shared write path gives those receipts a consistent meaning across supported inputs. Each refers to committed work, identifies the affected objects, and can support guarded undo.

Reviewing the work before it leaves the room

The optional participant filmstrip keeps a conversation beside the canvas. Its small peer-to-peer WebRTC connection uses Supabase signaling and remains separate from canvas persistence and embedded voice.

Sending a meeting packet introduces another boundary. A saved canvas object does not authorize an email. The packet has its own workflow: preparation creates a snapshot, and approval binds the content and recipients being reviewed. A tool can stage a send request, but the host must perform the final Send action before the configured server transport may submit it.

That requires an explicit review step after the room has produced useful work. The host approves the packet that will leave the room, with its intended recipients. Actual email delivery remains a separate result to verify.

What the recorded checks establish

I built CommandCanvas intending to submit it to the WebMCP Challenge, but didn't submit it.

The verification ledger records native Chrome WebMCP execution, a real voice-provider call that created a board, a real sketch-interpretation request that created structured output beside its source, and two browser clients converging on a durable mutation and receipt. Those are individual recorded checks, rather than proof that the whole intended experience has passed on physical devices.

The documented gaps include a ChatGPT built-in-browser Site Tools invocation, final physical-hand and microphone ergonomics, real email delivery, and cross-network TURN traversal. WebMCP's shared-page model is described in the official Site Tools documentation; host support and an observed invocation still have to be established separately for CommandCanvas.

The next useful rehearsal is a complete session with two people: draw something, explain it, create structured work from it, let the other person change it, and review the outcome together. I want to see whether revision conflicts interrupt that sequence, where selection becomes unclear, and whether the hand controls earn their place beside the pointer. Those are questions about working in the room that individual commit checks cannot settle.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

The Approval Machine That Refuses Before It Drafts

Daniel Romitelli — Wed, 26 Aug 2026 12:10:28 +0000

A message can be wrong before anyone writes it.

That sounds strange until the queue asks for a reply the system has no business drafting. If the member is under a policy flag that disables that intent, the safe answer is not a cautious paragraph. The safe answer is no draft. No retry buffer. No trace payload with forbidden text inside it. Nothing to accidentally send later.

I built an AI message review console around that shape. A card enters the day’s queue and leaves through exactly one exit: blocked before generation, held by a hard rule, or eligible for score-based release. The large language model can draft, classify, and explain. It does not get the final word.

1. The spine is the product

The whole system is readable from pipeline/run.py. That file calls five stages in order: policy, generation, hard rules, scoring, then decision. The ordering matters more than the individual model prompts.

The first gate runs before any model call. The third gate runs after generation and before scoring. The scoring stage is late on purpose, because a score is a judgement about quality. It is not permission to ignore an eligibility flag, a contraindication, missing equipment, scope drift, a stale thread state, or tone that misses a recent life event.

flowchart TD
  queue[Draft request enters queue] --> policy[Pre generation policy]
  policy -->|intent disabled| blocked[Blocked before generation]
  policy -->|allowed| generation[Generate grounded draft]
  generation --> hardRules[Deterministic hard rules]
  hardRules -->|rule fails| held[Held by hard rule]
  hardRules -->|rules pass| scoring[Probabilistic scoring]
  scoring --> decision[Eligible for score based release]

That diagram is the contract I wanted the code to enforce. The interesting part is the missing arrow: there is no route from scoring back into policy or hard rules. A high score cannot wash out a refusal, which is the same fail-closed shape Abhijat Chaturvedi argues for in Fail Closed, Not Open: Designing an AI Gateway for Regulated Enterprises.

2. Pre-generation refusal is a privacy boundary

The naive version of this system is easy to build. Generate the message, run checks, discard the message if the checks fail. It feels safe because the user never sees the rejected text.

It is not, because discarding is not the same as never creating. Once generated, text exists in process memory, logs, traces, retries, recordings, and whatever future evaluation set someone builds from today’s artifacts.

pipeline/stage1_policy.py makes that ordering explicit. The file’s own comment is blunt: if a member has an active policy flag that disables the intent the day’s queue asked for, the stage returns blocked and nothing is generated. The table has two rows, and only one suppresses. A flag can raise scrutiny without disabling generation, which keeps “this topic needs checking” separate from “this topic is not ours.”

From core/models.py, the policy row is plain data:

class PolicyRule(Strict):
    """One row of the pre-generation policy table.

    disables_intents is the whole mechanism. A flag that disables nothing is
    still recorded in the trace, because the difference between a flag that
    suppresses and one that merely raises scrutiny is exactly what stage 1 is
    demonstrating.
    """

    flag: PolicyFlag
    disables_intents: list[DraftIntent]
    route_to: list[CareRole]
    coach_explanation: str

What I like about this shape is that the refusal is auditable without being clever. disables_intents is the mechanism. A route and explanation travel with it, so the operator sees why the machine refused instead of getting a silent missing card.

The cost is that policy has to be modeled before generation. You cannot hide vague eligibility rules in a prompt and hope the model declines. If the rule is meant to prevent text from existing, it belongs before the generator. Ahmed P makes the case in An LLM is not a security boundary: he builds a moderation layer, then walks through it, because a model reads instructions and data from the same token stream.

3. Generation is allowed only after policy clears it

Stage 2 is the point where the model can do useful work. In this console, generation runs a real tool loop. The generation mode is recorded in the trace and response, because showing generated text while implying it was queued text would overstate what happened.

That detail matters because the console is reviewing AI-drafted messages, not pretending every draft has the same origin. The downstream gates do not change based on the mode. Policy has already cleared the request. Hard rules still run. Scoring still waits.

pipeline/run.py records the generation stage with the provider, model, mode, turn count, and tool call count. If the generator proposes different wording while seeded mode is in use, the trace says the queued text was kept. The review layer should not blur authorship just because both paths pass through the same adapter boundary.

The tradeoff is ceremony. A simple demo could have skipped modes, traces, and tool counts. I kept them because the system is making approval decisions, and approval decisions need receipts.

4. Hard rules are post-generation, deterministic, and score-proof

Some checks need the text. Equipment is a good example: the system cannot know whether a message asks for something impossible until there is a message to inspect. Thread state and tone also depend on the relationship between the draft and recent messages.

So hard rules live after generation.

pipeline/stage3_hard_rules.py says the rule set ignores every score and fails closed. Five rules run there. Three are pure code. Two use a model to classify something, then keep the decision in Python by combining strict enum values with an and.

The thread-state rule in pipeline/rules/thread_state.py is deterministic and carries its limit in the file comment. It reads who spoke last, not whether the last reply and the draft are about the same thing. The window is explicit:

ANSWERING_INTENTS = {DraftIntent.reschedule, DraftIntent.check_in}
WINDOW = timedelta(days=7)

I prefer that kind of visible weakness to a hidden prompt instruction. At seventeen cases the distinction between “coach spoke last” and “coach answered this exact topic” does not bite. At scale, the fix would be topic matching, not stretching the window and pretending the predicate became smarter.

The hard-rule result model also keeps the decision state concrete. From core/models.py:

class Severity(str, Enum):
    """What a failed rule costs.

    block: the message must not be sent as written.
    hold: a coach has to look at it. Cheaper to be wrong about.
    """

    hold = "hold"
    block = "block"

class Highlight(Strict):
    target: Literal["draft", "thread"]
    start: int  # located by exact substring match; never fabricated
    end: int
    label: str
    kind: Literal["movement", "claim", "tone", "message"] = "movement"

class HardRuleResult(Strict):
    rule: HardRuleName
    passed: bool
    severity: Severity | None = None  # set only when passed is False
    reason: str
    triggering_message_id: str | None = None
    highlights: list[Highlight] = Field(default_factory=list)
    detail: str = ""  # engineering detail; stays out of the coach copy

That comment on start is a rule about evidence. Where a rule has a span to point at, the offsets are found by exact substring match rather than generated, so the console cannot underline text it invented after the fact.

The thread-state rule is the exception. Its finding is about who spoke last, not about any phrase, so it has no span:

highlights=[Highlight(target="thread", start=0, end=0, label="already answered", kind="message")],

kind="message" is the field that matters, and the reference is triggering_message_id, set beside it to the id of the message that triggered the hold. The zeros mean no span. This rule points at a whole message, which is the right granularity for it and the wrong thing to call an exact offset.

The cost is that hard rules need measurements. pipeline/stage3_hard_rules.py does that measuring: it scans the draft for movements and makes the classifier calls. The rule modules receive measurements and return blockers. That shape is why the rules can be tested without a network.

5. A model can classify without becoming the judge

Two hard rules use a model because pattern matching is the wrong tool for the job. The scope rule in pipeline/rules/scope.py is about whether a draft interprets a clinical value instead of pointing at a clinician. The failure is semantic: two messages can share most words while one crosses scope and the other routes correctly.

The tone rule has the same shape. A high-energy push can be fine on most days and wrong when a recent thread contains a hardship. The problem may not be inside the draft alone. It can be in the relationship between the draft and a message from a few days earlier.

The model returns strict enums. Python decides.

That choice costs some flexibility. If the enum set is too small, the classifier has to force an edge case into a label that does not fit. But I would rather expand a typed vocabulary than let a free-form model answer become an approval predicate.

6. Typed failures are decisions, not missing values

The approval machine also has to handle broken model calls. A timeout, malformed structured output, and an unusable rubric are different failures. Treating all of them as zero would make the system look more numeric while hiding the reason it refused.

core/errors.py names the failures below the adapter boundary:

class ProviderError(RuntimeError):
    """Anything that went wrong below the adapter boundary."""

    def __init__(self, message: str, *, provider: str, model: str) -> None:
        super().__init__(message)
        self.provider = provider
        self.model = model

class ProviderUnavailable(ProviderError):
    """Timeout, connection failure, 5xx, or rate limit. Triggers failover."""

class StructuredOutputError(ProviderError):
    """The call returned, but not something that validates.

    raw_excerpt is the first 400 characters of what actually came back. It goes
    in the trace so a reader can see the malformation rather than trust the
    label on it.
    """

    def __init__(
        self, message: str, *, provider: str, model: str, raw_excerpt: str | None = None
    ) -> None:
        super().__init__(message, provider=provider, model=model)
        self.raw_excerpt = (raw_excerpt or "")[:400] or None

class TruncatedJson(StructuredOutputError):
    """Output stopped mid-token. Unparseable."""

class IncompleteObject(StructuredOutputError):
    """Parsed as JSON, but a required field is missing."""

class NullResponse(StructuredOutputError):
    """The provider returned no content at all, or an explicit null."""

class OutOfRange(StructuredOutputError):
    """A value parsed and is the right type but violates its bound."""

A provider timeout, output that stopped mid-token, an object missing a required field, and a score of 1.4 on a zero-to-one dimension are four different failures, and each gets a name instead of a fallback value. OutOfRange is not clamped: rounding that 1.4 down to 1.0 would turn a broken judge into a passing grade.

core/models.py carries the same philosophy. The file comment says absence refuses, it never defaults. A missing measurement is a blocker, not a zero. That is why rubric scores are optional on the gate decision and why judge failure has its own type instead of a fallback value.

This is less convenient than filling blank fields with zeros. It also stops a dashboard from mixing “bad answer” with “no answer.” Those are different operational problems.

7. Scoring is late because it is allowed to be uncertain

Only after policy and hard rules pass does the probabilistic part get a vote. The threshold stage clears a card only if the samples agree that it clears, using the minimum of three dimensions rather than an average.

The judge is not asked once. core/settings.py sets ensemble_judge: bool = True and judge_samples: int = 5. The stage header gives the reason: on this fixture set, five identical calls move the weakest score by up to 0.28, wider than the 0.70-to-0.80 range the threshold slider was built around.

So the decision comes from how often the samples agree, not from where one of them landed. Three or four usable samples out of five still support an agreement rate, and the failures are recorded. Below three, the stage blocks.

The numbers below come from one run. make eval sweeps the threshold across the seventeen-case fixture set in fixtures/cases.yaml and writes eval/results.json. It uses recorded provider responses rather than live calls, which is what provider: "replay" in that file means and what makes the sweep reproducible offline. Everything here is that run at its default threshold of 0.70.

Of seventeen cards, three clear automatically, nine go to the coach, and five are blocked. Nothing labelled as needing a human was auto-cleared.

Four of the fourteen that did not clear are over-holds: cards labelled as not needing a human that the gate declined to release. One card is undecidable, where the judge could not resolve it and a coach looks at it instead. eval/metrics.py excludes those from the over-hold count, because a judge that could not answer and a gate that held a card it should have released are different failures. Mean spread across the samples is 0.148.

I kept the over-hold cost in the system view and the decision notes instead of tuning it away. A fail-closed system can still be lazy if it hides that cost, because holding too much burns human attention. The point is not to make refusal free. The point is to make the refusal cost visible.

8. What the design costs

The cost lands in maintenance, not at runtime.

Every refusal here is something a person has to keep current: a policy row, an enum member, a threshold, a fixture case proving the rule still fires. That is more surface than a prompt asking a model to be careful. It buys artifacts that can be diffed, tested, and argued with in a pull request. A paragraph of instructions cannot be tested the same way.

The number I would watch in production is the over-hold count, not the block count. Four out of seventeen is survivable on a fixture day and would be a staffing problem at a thousand cards a morning.

Fail-closed systems rarely fail by letting something through. They fail by becoming more expensive without anyone writing down why, until somebody raises the threshold to make the queue manageable. That is a decision to make against a measured over-hold rate, which is why the sweep exists and why the over-held column stays on the screen.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

Confidence Procedures Are Production Code

Daniel Romitelli — Wed, 26 Aug 2026 11:48:12 +0000

A clean sample can sound stronger than it is. Someone checks a handful of items, finds no exceptions, and the sentence wants to become broader than the evidence: the process worked, the control operated, the population was clean.

That sentence is where I stopped trusting the product unless the math had the same status as the API. If a compliance audit sampling tool emits a confidence claim, the confidence procedure is production code. It needs adversarial tests. A formula in a notebook is too far away from the thing users read.

This post is about the bound in core/bounds.py, the inverse sample-size question in the same module, and the evaluation gate in eval/coverage.py and eval/run.py. The tool reports a one-sided upper confidence bound on the number of failures in a finite population. Then it tries to prove that bound covers at the confidence it claims.

1. The number is a frontier, not a fact

The bound answers a narrow question: after drawing n items from a population of N and observing k exceptions, what is the largest true failure count D still compatible with the sample at the requested confidence?

That wording matters. The tool is not estimating the true number of failures. It is stating what the sample rules out. A clean sample does not mean the population is clean. It means failure counts above the frontier are no longer supported by the observed draw at the selected confidence.

The implementation uses the hypergeometric distribution because audit sampling draws without replacement. The same item cannot be selected twice. That is a small detail in code and a large detail in interpretation when the sample is a meaningful fraction of the population, which is exactly the situation where finite-population correction starts to matter (MeasuringU).

DECISIONS.md has the blunt version of the tradeoff. At N=1,000 with n=25, the exact finite-population bound is 11.1 percent; the binomial comparison is 11.3 percent. Close enough that the distinction feels academic. At N=64, the gap opens as the sampling fraction rises: at n=5 the binomial is 7 percent high, at n=25 it is 45 percent high, at n=48 it is 94 percent high, and at a full census of 64 it reports 4.6 percent where the true answer is zero.

That last case decided the design. If I examine every item in a finite population, the upper bound on unobserved failures cannot stay above zero after a clean census. The binomial approximation has forgotten that the population can be exhausted.

def exact_upper_bound(N: int, n: int, k: int, confidence: float = 0.95) -> Bound:
    """What a sample of n from N, showing k exceptions, actually supports."""
    _validate(N, n, k, confidence)
    D = _max_failures(N, n, k, confidence)
    return Bound(
        N=N,
        n=n,
        k=k,
        confidence=confidence,
        max_failures=D,
        max_failure_rate=D / N,
        rule_of_three=(3.0 / n) if (k == 0 and n > 0) else None,
        binomial_max_failure_rate=_clopper_pearson_upper(n, k, confidence),
    )

I kept the binomial value in the returned model as a comparison, never as the answer. That made the approximation visible without letting it drive the verdict.

The other choice in this function is less obvious: the result carries N, n, k, and confidence alongside the bound. core/models.py says the rule directly: no bare float crosses a signature. A number without its inputs is exactly the failure mode this tool is trying to avoid.

2. The hypergeometric inversion is the product behavior

The core search lives in _max_failures. It inverts the probability statement instead of sampling possible worlds. For each possible true failure count D, the observed exception count K has a hypergeometric distribution. The reported bound is the largest D for which observing at most k exceptions is still probable enough.

The implementation depends on monotonicity: adding more failures to the population can only make small observed counts less likely. That turns the search into an exact binary search over integer failure counts, rather than a heuristic over percentages.

if n == 0:
    # Zero draws exclude nothing. Every item could be a failure.
    return N
if k == n:
    # Every draw was an exception. P(X <= n) == 1 for every D.
    return N

alpha = 1.0 - confidence
lo, hi = k, N  # D = k always satisfies: P(X <= k | D = k) == 1
while lo < hi:
    mid = (lo + hi + 1) // 2
    if _cdf_at_most_k(k, N, mid, n) >= alpha - _ALPHA_TOL:
        lo = mid
    else:
        hi = mid - 1
return lo

The edge cases are part of the contract. With zero draws, the sample excludes nothing. If every draw is an exception, every possible failure count remains compatible with observing at most n exceptions, so the upper bound is the whole population.

The middle is the frontier. For fixed N and n, every observed k cuts the lattice of possible true failure counts into allowed and ruled-out values. The product prints the frontier, not the hidden truth.

flowchart TD
  population[Finite Population N] --> draw[Draw n Without Replacement]
  draw[Draw n Without Replacement] --> observed[Observed Exceptions k]
  observed[Observed Exceptions k] --> inversion[Hypergeometric Inversion]
  inversion[Hypergeometric Inversion] --> frontier[Reported Upper Bound D]
  frontier[Reported Upper Bound D] --> claim[Allowed Claim]
  population[Finite Population N] --> lattice[All True Failure Counts D]
  lattice[All True Failure Counts D] -.-> inversion[Hypergeometric Inversion]
  binomial[Binomial Approximation] -.-> wrongFrontier[Forgets Finite Population]
  alwaysN[Always Return N] -.-> vacuous[Always Covers But Says Nothing]
  inversion[Hypergeometric Inversion] ==> exactPath[Exact Finite Population Bound]

I folded the bad alternatives into the same diagram because they fail for different reasons. The binomial approximation can be too loose when the sampling fraction rises. Always returning N would pass a naive coverage check, because it covers everything. The exact finite-population inversion has to satisfy both constraints: cover at the stated confidence and still say something.

3. The inverse question is where product pressure shows up

Once the tool can say what a sample supports, the next question is predictable: what sample size would support a stronger sentence?

That inverse question also belongs in core/bounds.py. It uses the same finite-population bound, only flipped around. Instead of asking, “given N, n, and k, what is the maximum supported failure rate,” it asks what draw size would be needed for a target cap under an assumed exception rate.

I am careful with that framing because this is where tools start to lie politely. A target cap is not a measurement. An assumed exception rate is an input. If those are treated as facts, the sample-size panel becomes a confidence costume for planning assumptions.

The model boundary helps here. In core/models.py, Bound validates internal consistency: k cannot exceed n, n cannot exceed N, the bound cannot fall below the observed exceptions, and the maximum failures cannot exceed the population. That is plain defensive programming, but in this domain those checks are also argument hygiene.

A product that emits statistical language has two outputs. There is the value on the screen, and there is the sentence a human will say after reading it. The inverse sample-size path exists because the second output matters. If the current sample cannot support the stronger sentence, the tool should state the cost of that sentence rather than rounding the current evidence upward.

4. Coverage is a gate, not a chart

The evaluation runner treats coverage as the first measurement. eval/run.py describes four measurements, but coverage runs first and stops the run if it fails. The target is never adjusted to match the result.

That is the right shape for a statistical product. Detection rates, draws to first exception, and prior sensitivity are useful after the confidence procedure is valid. Before that, they are decoration.

eval/coverage.py checks the confidence claim two ways. The exact path enumerates outcomes analytically. The Monte Carlo path replays the same quantity through the random draw mechanism. They are not redundant. Exact enumeration tests the math. Simulation tests that the implementation path agrees with the math.

def exact_coverage(N: int, n: int, D: int, confidence: float = NOMINAL) -> float:
    """P(true D falls inside the reported bound), summed over all outcomes."""
    ks = np.arange(0, n + 1)
    pmf = hypergeom.pmf(ks, N, D, n)
    covered = np.array(
        [exact_upper_bound(N, n, int(k), confidence).max_failures >= D for k in ks]
    )
    return float(pmf[covered].sum())

This is the cleanest test in the project. For a fixed N, n, and true failure count D, every possible observed k is finite. So the test sums the probability mass for the outcomes where the reported bound contains the true D.

There is no sampling error in that number. If it falls below the nominal confidence, the procedure does not cover. The product is wrong.

The Monte Carlo version uses NumPy’s hypergeometric draw and then runs the same bound calculation on the observed exception counts. The default is 10,000 runs with seed 0.

def mc_coverage(
    N: int,
    n: int,
    D: int,
    confidence: float = NOMINAL,
    runs: int = MC_RUNS,
    seed: int = 0,
) -> float:
    """The same quantity by simulation, through the real draw path."""
    rng = np.random.default_rng(seed)
    ks = rng.hypergeometric(ngood=D, nbad=N - D, nsample=n, size=runs)
    bounds = {
        int(k): exact_upper_bound(N, n, int(k), confidence).max_failures
        for k in np.unique(ks)
    }

The dictionary cache is a small detail, but I like it. A simulation can draw the same k many times, and the bound for that k does not change. Cache the unique observed counts, then compare repeated outcomes to the same computed frontier.

The readme states the result before anything else: verdict passed. Minimum coverage observed was 0.9500 against nominal 0.9500, across 5,943 exhaustively enumerated configurations. The worst configuration was N=64, n=25, D=16, at 0.9505. The largest disagreement between exact and simulated coverage was 0.0039.

Those numbers are not marketing. They are the release condition for every sentence that depends on the bound.

5. A passing coverage test can still be useless

A malicious implementation can pass a lower-bound coverage check by returning N for every input. It will always contain the true failure count, because the true count cannot exceed the population.

That is the trap in testing confidence procedures. “Covers at least 95 percent” is necessary. Alone, it rewards vacuity.

The suite adds an anti-vacuity ceiling. The readme states it as a test: minimum coverage must not rise above 0.96. A bound that always returned N would cover at 100 percent and say nothing, so that implementation should fail.

This is one of those tests that looks strange until you need it. Most test suites only reject undercoverage. Here, overcoverage can also be a bug because the product is supposed to say the strongest true sentence supported by the evidence. Too safe becomes uninformative.

There is a second guard: shrink the bound by one and confirm coverage breaks. That catches the opposite fake. Without it, the coverage test could be passing while failing to detect that the frontier is load-bearing. If reducing every reported upper bound by one still passed, the original test would not be sharp enough to prove the implemented boundary.

The two guards create pressure from both sides. The bound cannot be loose enough to say nothing, and it cannot be tightened without losing the claimed confidence.

6. Ground truth stays outside the estimator

The simulator needs hidden failure rates. The estimator must never see them.

That boundary is explicit. simulation/draw.py is the only module permitted to read Stratum.true_failure_rate. Its docstring says why: everything under core/ estimates, while the simulation knows the answer. If an estimator could read ground truth, the evaluation would be theatre.

The same file says the boundary is enforced three ways: parse every file under core/ and api/ for the identifier, assert that no estimator module imports the simulation package, and perturb every true rate in the fixtures while checking that estimator output does not move by a single digit.

I care about this boundary more than I expected to when I started. Statistical code has an especially ugly failure mode: it can look more correct after it cheats. If ground truth leaks into the estimator, the bounds become beautifully calibrated to information the auditor never had.

The HTTP surface follows the same rule. server/main.py says every response is a public projection. Stratum carries simulator ground truth, so nothing of that type is serialized. The browser receives projections, never the domain object that knows the answer.

That separation costs some convenience. You write more models. You pass more values around. You make tests about identifiers and imports, which feels crude until the first time it prevents a silent category error.

7. The formula was the easy part

The finite-population bound is compact. The production problem is larger: keep the approximation visible but out of the verdict, carry the inputs with every output, invert the distribution over integer counts, prove coverage by enumeration, replay it through simulation, reject vacuity, and make ground truth unreachable from estimators.

I do not think of that as “extra validation” around the math. It is the math becoming a product surface.

A confidence number is a promise about repeated behavior. Once that promise appears in software, the test suite has to attack the promise itself, because the user will build a sentence from the number whether the code earned it or not.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

A Rehearsal Is Only Cheap In Distribution

Daniel Romitelli — Mon, 24 Aug 2026 03:19:27 +0000

TL;DR In generative video pipelines, running cheap low-step sketches to pick parameters sounds like free optimization. But when prompts go out-of-distribution, surrogate scorers return noise, turning a \$0.002 check into a bad decision that triggers a \$15 compounding failure. Here's why skipping the cheap step is sometimes the cheapest option.

Three numbers run Scenematic's generation loop. A think-frame costs \$0.002. A full render costs \$0.50. A bad scene that slips through and gets built on costs about \$15.50, because the scene chain compounds it before anyone looks. The constant in lib/generation-loop.ts carries the arithmetic in a comment: 15.502, // CALIBRATION_TARGET: 0.002 + 0.50 + 15.00.

Most of the pipeline exists to keep spend at the cheap end of that ladder. One module decides when the cheap step should be skipped entirely. A hundred-contract baseline then put numbers on how often that decision was wrong.

1. The rehearsal

lib/think-frames.ts generates quick, low-inference-step sketches before committing to a full-quality keyframe. The file header credits DeepGen's think tokens as the inspiration. Each sketch tries a different preservation focus, character, environment, mood, composition, or atmosphere, with its own image-to-image strength and seed. The reward mixer scores the batch and the winner's parameters go to the full render.

The economics only work if those scores mean something. That assumption fails quietly, and it fails hardest on the prompts where a rehearsal looks most useful.

2. Where the scores stop meaning anything

Scoring a sketch of A detective leans forward across a metal table, interrogating a nervous suspect under fluorescent lights works fine. The scorer has seen a thousand shots like it. Scoring A sentient equation writes itself across a blackboard that extends infinitely in all dimensions does not fail loudly. It returns a number, and the number is noise. Both prompts are verbatim from the baseline harness.

lib/ood-detector.ts measures the distance instead of hoping. It keeps a reference corpus of 25 in-distribution prompts, roughly 8 dialogue, 9 cinematic, 8 multi-asset, and embeds them with all-MiniLM-L6-v2 through @xenova/transformers. Local model, no API call.

The embeddings mean-pool into a unit centroid, computed once per process and cached. An incoming prompt gets embedded the same way. Epistemic uncertainty is one minus the cosine similarity to that centroid, and at or above the threshold the detector sets bypass_surrogate: true.

Bypass means skipping the cheap step, so the prompt the system understands least is the one that goes straight to the expensive render. Backwards from most cost optimizations, and deliberate. Out past the corpus a rehearsal is a \$0.002 lie that steers a \$0.50 decision toward a \$15 mistake, and skipping it buys the removal of a bad witness at the price of one render. The gate recuses an unqualified judge rather than filtering bad prompts.

Every evaluation writes a row to an ood_events table: the uncertainty, the threshold that was applied, the bypass flag, and cost_incurred at either \$0.50 or \$0.002. That table is where everything else in this post comes from.

flowchart TD
  P["Compiled prompt"] --> U["Uncertainty = 1 - cosine vs corpus centroid"]
  U --> G{"At or above the category threshold?"}
  G -->|"BYPASS"| F["Straight to full render, $0.50"]
  G -->|"SURROG"| T["Think-frame rehearsal, $0.002"]
  T --> W["Full render using the winning sketch's parameters"]

3. One hundred contracts

scripts/run-phase1-baseline.ts runs 100 steering contracts through the gate. A steering contract is the typed record the compiler emits for one scene: the prompt, the chosen model and seed, the quality targets, and the audit trail of what happened to it. The batch splits into 80 in-distribution prompts across dialogue, cinematic, and multi-asset, plus 20 out-of-distribution (OOD) prompts, abstract and high-complexity. Seeds are fixed at 42 plus the contract index. The renders are real, on RunPod pods running ltx2, wan22, and hunyuan behind per-model semaphores of 3, 2, and 2. Each console line prints SURROG or BYPASS next to the measured uncertainty.

A caveat before the numbers. The reward heads, the four per-dimension scores that judge a finished render (R_smooth, R_motion, R_semantic, R_narrative), are simulated in this harness: in-distribution signals drawn around 0.65 with jitter, OOD around 0.35, because real per-head scoring of rendered video was not wired up yet. The embeddings, the routing, the renders, and the telemetry are real. The simulation still surfaced two real bugs, covered below.

The cost waterfall, straight out of fig4_cost_waterfall.csv:

Stage	Count	Cost
Contracts submitted	100
OOD bypassed	31	\$0.00 extra
GPU rendered	69	\$15.64
Passed quality floor	43
HITL flagged	26
False negatives	53	\$821.60 compound

Twenty-two of the 26 human-in-the-loop (HITL) flags blamed the narrative head. The Phase 2 commit message gives the aggregate: a 25.3% false negative rate, with R_narrative dominating 72.4% of HITL flags. Fifty-three contracts finished under the quality floor, and at \$15.50 of modeled compound damage each the ledger prices those misses at \$821.60, against \$15.64 for every render in the run combined.

4. Thresholds from the misses

Phase 2 swept the threshold from 0.30 to 0.95 in steps of 0.01 and computed a full confusion matrix at each stop. The F1-optimal global threshold came out at 0.31, which flags 97 of 100 contracts and puts the cost column at \$48.51 per hundred. Flagging nearly everything does maximize F1. It also defeats the point of having a surrogate at all.

So the shipped config ignored F1. The calibration_metadata block in quality-thresholds-v2.json records the target it optimized instead: Catch ALL known FNs per category with margin. Take the known misses, find the minimum uncertainty among them per category, set the threshold just under it. DIALOGUE's one known false negative sat at 0.631, so DIALOGUE got 0.62. SCENIC's four bottomed out at 0.506, so 0.50. ACTION's five at 0.495, so 0.49. The global fell from 0.72 to 0.55 and now serves as the fallback for categories with no data.

Per category matters because the sweep exposed an inversion a single global threshold cannot encode. The config file says it in one line: SCENIC and ACTION had LOWER uncertainty but HIGHER FN rates — category thresholds fix this.

A scenic prompt reads as familiar to the embedding space. Sweeping drone shot, golden light, the corpus is full of that texture. The renders still miss. Uncertainty and failure rate are correlated across the whole population and inverted inside two of its slices, and one knob cannot express that.

ABSTRACT kept the old 0.72, since genuinely weird prompts carry high uncertainty on their own. NARRATIVE got 0.65 with an annotation calling it a conservative estimate with no false negative data behind it. That one is a guess with a label on it, and it stays a guess until a NARRATIVE prompt fails in a logged run.

The detector loads this file at runtime and falls back to the legacy global 0.72 if it is missing. The sweep range and step size are recorded next to the thresholds they produced, along with the per-category evidence.

5. Two gates that could not fire

The baseline caught two bugs I would not have found by reading the code, because the code looked fine.

The HITL gate required the composite score to be under the floor AND at least one reward head to breach a z-score of -2.0. With simulated signals at 0.65 ± 0.1, the commit message for 20aa5ec describes the result: the z-score condition was mathematically impossible to satisfy with simulated signal distribution of 0.65 ± 0.1. Two conditions joined by AND, one of them unsatisfiable. The gate sat silent while contracts failed under it, and the dashboards looked calm the whole time.

The fix inverted the roles. The reward floor of 0.65 is now the primary trigger, and per-head z-scores only attribute which head gets blamed.

The second bug was in the false negative counter itself. It only counted SURROG contracts, on the theory that a false negative means the surrogate path trusted a prompt it should not have. That definition missed 8 contracts the gate bypassed which still rendered below the floor. A bypassed prompt that fails is still the pipeline failing, whatever path it took, so the counter now takes any contract under the floor.

6. The head that ran cooler

R_narrative took 72.4% of the HITL blame in the phase 2 baseline. The narrative scoring was doing its job. Its head just runs cooler than the others: population mean 0.55, against 0.60 for R_motion and 0.70 for R_smooth and R_semantic. Four heads held to the same absolute bar, and the coolest one took nearly all the blame.

The fix in lib/reward-mixer.ts is z-score normalization per head. normalizeHeadScore computes (raw − mean) / std against per-head population stats stored in the same versioned config, so a head only triggers when it is unusual for itself. computeSubReason then cross-references the other heads whenever R_narrative does trigger: motion also low means pacing, semantic also low means fidelity, narrative alone means coherence.

7. Infra noise is not model failure

Twenty hunyuan events in the baseline were graphics processing unit (GPU) failures, out-of-memory and handler crashes, sitting in ood_events and dragging the quality numbers around. So the table grew gpu_error and superseded columns. evaluateOOD returns the row id of the event it just logged, and the runner calls markOODEventGpuError on a GPU error so the dashboard can segment it. Re-running contracts with --rerun-contracts="81,82,100" marks the old contaminated rows superseded instead of deleting them. queryDashboards takes a view argument, clean or all, and clean excludes both flags.

Even total infrastructure collapse gets a row: ALL_MODELS_DOWN logs an OOD event with gpu_error=true. Without the segmentation, a crashed pod reads as a quality regression, and one afternoon of flaky hardware quietly recalibrates your thresholds for you.

8. What is still provisional

The detector landed at 234 lines and sits just over 300 after the Phase 2 changes. The scoring inside it is a cached centroid and a cosine. Nothing is learned and none of it touches a GPU. The cost constants carry a CALIBRATION_TARGET marker with a comment saying to recalibrate after 100 real runs or a provider switch, so the provisional numbers announce themselves and are greppable.

Plenty is still provisional. The reference corpus is 25 prompts, and every threshold in the v2 config is calibrated against the centroid those 25 produce. Adding corpus prompts moves the centroid, which shifts every uncertainty measurement, which invalidates the per-category thresholds. Corpus and thresholds have to version together or the calibration quietly stops describing anything. The reward heads are still the simulated ones, so the head_stats block needs re-deriving from real scores before the z-score triggers mean what they claim. And NARRATIVE still has no false negative data.

The part I trust is the ledger. Both dead gates were invisible in code review and obvious in the event counts, and the event counts only exist because every decision writes a row, including the decision to spend more.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

Calibration Is Bet Sizing

Daniel Romitelli — Sun, 23 Aug 2026 12:19:16 +0000

The last post was about making a number trustworthy. Leakage geometry, purge widths, de-overlap, a baseline that could not cheat. It ended with a minute-scale ceiling that held at 52% across seven configurations and a model family swap.

This one is about what happens after you trust the number. Because a probability you are going to bet on is a different object from a probability you are going to report.

The probabilities are not decorative

The path-passage classifier is a three-class LightGBM. It returns p_up, p_down, p_none. Those go straight into the expected-value score that decides whether to take a trade and how big:

long_score  = p_up * (B - C) + p_down * (-B - C) + p_none * (-C)
short_score = p_up * (-B - C) + p_down * (B - C) + p_none * (-C)

B is the barrier, C the cost. Read the arithmetic. Every term is linear in a probability. Scale p_up by 1.2 and you scale the long score by very nearly 1.2.

So miscalibration does not stay in the model. It becomes a bet-sizing error, in proportion, in the bins where the gate actually fires. A classifier that is right 70% of the time while claiming 90% is not 20 points wrong. It is sizing every position in that bin as though the edge were far larger than it is.

Boosted trees are known for uncalibrated softmax output. I had been consuming it as if it were a probability.

The audit

Seven live assets. For each one, fit an Inductive Venn-Abers wrapper on the time-ordered older 80% of that model's training data, 6,988 rows, and evaluate against a 500-row uniform-random sample of the newer 20%, seed 42. The LightGBM models are reloaded from disk and left alone. Only the wrapper is fit.

Measure Expected Calibration Error and log-loss, before and after.

Asset	ECE before → after	ECE Δ	Log-loss Δ
BTC	0.1272 → 0.0621	-51.2%	-5.5%
ETH	0.1795 → 0.0298	-83.4%	-11.5%
SOL	0.1680 → 0.0386	-77.0%	-10.6%
XRP	0.2219 → 0.0645	-70.9%	-17.7%
ADA	0.1419 → 0.0369	-74.0%	-8.0%
LINK	0.1260 → 0.0737	-41.5%	-1.2%
LTC	0.1508 → 0.0603	-60.0%	-14.1%

Every asset has a real gap. That settles the first question, which was whether this was one bad model or a property of the setup. It is systematic.

The second question is the interesting one.

The direction is asset-specific, and that rules out the easy fix

ETH fails the way boosted trees are supposed to fail. Its worst reliability bin is [0.90, 1.00]. Eleven samples. Mean stated confidence 93.8%. Empirical accuracy 45.5%.

Most certain, least reliable. That is the bin where the EV gate fires hardest and the position sizes are largest.

The other six fail in the opposite direction.

XRP, in the [0.60, 0.70] bin: 93.1% accuracy at 64.9% stated confidence, n=116. A 28-point understatement.

LTC, same bin: 90.4% accuracy at 64.7% confidence, n=115.

That is a suppressed-signal failure. The gate does not fire often enough, because the stated confidence lags what the model actually delivers. It costs money quietly, by declining trades that were good.

One asset over-confident. Six under-confident.

Which kills the convenient answer. Platt scaling and temperature scaling apply one monotone correction. They cannot pull ETH's tail down and push XRP's middle up at the same time, because those are corrections in opposite directions. A single global calibrator fits the average of two failure modes and helps neither.

Per-asset Venn-Abers works here because it fits each asset's own reliability curve and does not assume a shape.

Where the split came from

This is the part I did not expect, and it is the reason I keep the training params in the same document as the audit.

The hardened LightGBM settings were min_data_in_leaf=400, num_leaves=8, max_depth=3. I introduced those specifically to stop the ETH-style saturation, where terminal leaves go to 1.0 and the model claims certainty it has not earned.

They worked. They also worked too well.

Constraining the leaves prevented the over-confidence failure and produced a structural under-confidence pattern across the rest of the universe. The fix for one failure mode manufactured the opposite failure mode in six assets.

ETH is the lone holdout that still saturates, because in the rare cases where it is genuinely certain its leaves still reach the top of the range. Eleven samples in that bin tells the story: sparse and extreme.

That is a straight tradeoff I made without knowing I was making it. Reliability diagrams are what showed it. An accuracy score would have shown a modest improvement and nothing else, because averaging is exactly the operation that hides a bin.

The bar was set before the numbers came back

ECE has to improve by at least 50%, and log-loss by at least 5%. Both, not either.

Six assets clear it. LINK-USD misses both, at -41.5% and -1.2%.

LINK is not a data problem. It has the same 8,736 training rows as everything else, so this is model quality rather than availability. The LightGBM may simply be better calibrated for LINK already, in which case there is less for the wrapper to do and the small lift is honest. Or the calibrator needs different hyperparameters. Both are worth a follow-up and neither is resolved today.

So the rollout is selective. Six of seven, and LINK stays on raw softmax until somebody investigates it.

The mechanism is deliberately boring. The loader falls back to raw softmax for any asset with no calibrator on disk, so excluding LINK means either leaving its pickle off the deployment image or adding an allow-list:

allow = set(os.environ.get(
    "VENN_ABERS_ASSETS",
    "BTC-USD,ETH-USD,SOL-USD,XRP-USD,ADA-USD,LTC-USD",
).split(","))

One environment variable and three lines in the loader. A rollout that cannot express "these six and not that one" ends up shipping the failure with the successes.

What this audit does not prove

The status line at the top of the integration guide says prototype, not integrated into the live inference path. That is still true, and the caveats are worth having in the open.

The holdout is a 500-row uniform sample, seed 42. The 80/20 split is on the model's own training data rather than on live signals.

There has been no live-data refit, because signals_history does not yet carry realized 24-hour outcomes for the p_up/p_down/p_none rows. Migration 028 landed on 2026-05-21 to start collecting them.

BTC's barrier is 150 bps and the rest are 200, so the ECE deltas compare cleanly but the absolute ECE values across assets carry a class-balance shift. I treat those as informal.

LINK's failure in particular should be re-run once live outcomes exist. A training-data fit can hide a different live picture, and that is exactly the asset where I would expect it.

Why this is the second half of the last post

The geometry work answered whether the measurement could be trusted. Purge, embargo, de-overlap, a split that cannot leak. All of that gets you a number you can believe.

Believing the number is not the same as being able to bet on it. The ceiling post established that the minute-scale direction signal is capped near 52% and that the cap is real rather than an artifact. This audit establishes something narrower and more immediate: on the horizon where there is signal, the probability the model hands the sizer is not the probability it should act on, and the correction is different for every asset.

Accuracy averages. A reliability diagram does not. The bin where the model was most confident and least correct is worth more attention than any figure computed across the whole set, because it is the bin where the money goes.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

Experimentation Is the Missing Evaluation Layer for Agent Memory

Daniel Romitelli — Sun, 23 Aug 2026 10:43:40 +0000

An agent can fail because it missed the right material. It can also fail because I gave it the wrong material with confidence. The worse case is quieter: the wrong material sits beside a successful task, then looks useful afterward.

A skilled user may ask for architecture notes because they know where to look. Those notes can travel with a good result even when they did not cause it. If the system learns from that trace alone, the next session may get extra text that feels justified and still wastes the agent's attention.

That failure has a name and a direction. The same expertise that makes someone request the right document also makes them likelier to finish the task without it. Skill causes both the request and the outcome, so skill is a common cause sitting upstream of the thing I am trying to measure. A system that reads the trace and credits the document has attributed to the context what belonged to the person. This is not noise that averages out as traffic grows. More sessions from confident users make the estimate tighter and no less wrong, which is the property that makes it dangerous: the system becomes more certain of a relationship it never established.

There is no way to subtract that bias afterward from the trace alone, because the trace does not record why the material was requested. The only cheap instrument that removes it is deciding who gets the material by coin flip instead of by request. Randomization makes assignment independent of skill, so whatever difference survives between the two branches is attributable to the material rather than to the person holding it.

This is the problem I built the active experimentation layer for in Zero Context Loss (ZCL). ZCL is the context learning platform I use for my AI agents. This post is about one part of it: the evaluation layer that tests changes to context provisioning while the agent is doing real work. It is also, by the end, an honest account of how much that layer can currently prove, which is less than its own vocabulary suggests.

1. A context change becomes a hypothesis

The active learning code lives in zcl_core/learning/active_learning.py. The central object is Hypothesis. I made the candidate change carry its own treatment, control, task type, metric, and minimum sample threshold.

That shape is deliberate. Reordering two documents has a different target from including a document whose value is uncertain. The first can be judged by time. The second can be judged by success. If those choices stay implicit, the system can promote a vague improvement without saying what improved.

Declaring the metric before the data arrives matters more than it looks. A hypothesis that names its outcome in advance can only be judged on that outcome. A hypothesis that stays vague can be judged on whichever of success, duration or token count happens to have moved, and something almost always has. Writing the metric into the object is the cheapest available guard against choosing the comparison after seeing the result.

The hypothesis object uses a UUID, a type string, a description, the task kind, treatment and control payloads, a metric name, and min_samples set to 20. The default threshold sits on the object because I wanted the brake to travel with the proposed change. Section 6 returns to whether that brake is connected to anything.

The generator in ActiveLearning stays close to context the system can actually apply. It looks at documents already used for a task type, then builds order hypotheses from the top five documents against the next slice. It also asks for uncertain documents and creates inclusion tests for up to three of them. Each hypothesis takes its id from uuid4() — called, not referenced.

A context system with limited traffic can burn every session it has on combinations that will never gather enough evidence to matter, so the generator stops early and some pairings never become live candidates at all, and I gave that coverage up on purpose. The generator is not there to enumerate everything that could be tested. It is there to keep the number of live tests small enough that each one can actually finish.

2. Assignment happens while context is being built

The integration point is zcl_core/provision/context_provider.py. The context provider creates the session id with uuid4(), decides whether the request should explore, asks value attribution for learned document values, handles cold start, then filters the selected document ids through causal evidence before the final bundle goes back to the agent.

Assignment has to happen before the agent consumes the material, because the branch has to be decided by the coin rather than by the request, and that ordering is doing more work here than anything else in the file. A report run later can describe what happened, but by then the material has already been chosen by whatever mixture of habit, skill and retrieval score produced it, and the confound from the opening is already baked into the data. Deciding during provisioning is what converts an observation into an experiment.

The returned ContextBundle carries is_experimental, experiment_id, and experiment_group. Those fields are plain bookkeeping, and they are the difference between analysis and guesswork. If a session was in treatment, the bundle says so. If it was in control, the evaluator does not have to reconstruct that from log order.

flowchart TD
 candidate[Candidate context strategy] --> assignment[Randomized assignment]
 assignment --> treatment[Treatment context]
 assignment --> control[Control context]
 treatment --> outcome[Task outcome]
 control --> outcome
 outcome --> evaluation[Statistical evaluation]
 evaluation --> promoted[Promoted strategy]
 evaluation --> retired[Retired strategy]```



The diagram lies in one place, and it is the place most likely to matter. Randomizing sessions treats sessions as independent units. They are not. The same person returns, so their skill appears in both arms across the week rather than being held constant within one. Learned document values also update between sessions, which means the control arm is not fixed while the experiment runs; the baseline drifts under the test. Randomizing the person rather than the session would remove the first problem, and freezing learned values for the duration of an experiment would remove the second. The current design does neither, and the effect is that the estimate is noisier than a clean trial of the same size.

## 3. How much of the traffic gets spent on trials

`ActiveLearning` starts with `base_exploration_rate = 0.2`, uses `uncertainty_multiplier = 0.5`, and caps the final probability at 0.4. These are configuration values in the implementation. They are not presented as observed production rates, throughput measurements, or hardware-dependent results.

The decision is small: compute task uncertainty for the organization, add the uncertainty bonus to the base rate, cap it, then compare that probability with `random.random()`.

The uncertainty calculation is scoped by organization. That stops one organization's history from making another organization's context appear more certain than it is. In the helper, fewer than 10 sessions returns maximum uncertainty. After that, the code reads outcomes for the task, computes success-rate variance as a Bernoulli variance, and reduces the result as sample count grows using the log of the count. With no outcomes, it uses 0.25 as maximum variance.

It is a cheap signal for where to spend trials, and it makes no claim at all about why a document helped.

There are two traditions tangled together in that decision, and they do not want the same thing. Spending more trials where uncertainty is high is bandit reasoning, and a bandit's goal is to minimise regret: give as many sessions as possible the best-known context while still learning. A controlled experiment has a different goal, which is an unbiased estimate of an effect, and it is happiest with a fixed allocation decided in advance. The two are not the same discipline. Adaptive allocation is known to bias the naive difference in means, because the amount of data each arm receives depends on how the arm has been performing.

Here the two are only loosely coupled: the uncertainty rate governs whether a session explores at all, while assignment within a live experiment is a fair split. That keeps the bias small. But the honest description of this layer is that it uses a bandit to decide when to run trials and a fixed randomization to run them, and if the exploration rate ever starts responding to the results of a specific live experiment, the estimator stops being trustworthy.

Capping it at 0.4 means the areas with the least evidence still spend most of their sessions on the path already known, which slows learning down badly. I kept the cap anyway. Without it a sparse task type turns into churn, every request treated as a trial, and nothing ever settles into a default.

## 4. The schema keeps the branches separate

The experiment tables are defined in `migrations/004_experiments.sql`. The migration separates the proposed change from the per-session assignment. One table stores the hypothesis text, task type, treatment, control, status, results, and timestamps. The other stores the session, experiment id, group name, and assignment time. Both take their ids and timestamps from `uuid_generate_v4()` and `NOW()`, called rather than named.



```sql
CREATE TABLE zcl_experiments (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    hypothesis TEXT NOT NULL,
    task_type VARCHAR(100) NOT NULL,
    treatment JSONB NOT NULL,
    control JSONB NOT NULL,
    status VARCHAR(20) NOT NULL DEFAULT 'active', -- active, completed, cancelled
    results JSONB,
    started_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    completed_at TIMESTAMPTZ,

    CONSTRAINT valid_status CHECK (status IN ('active', 'completed', 'cancelled'))
);

CREATE TABLE zcl_experiment_assignments (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    session_id UUID NOT NULL REFERENCES zcl_sessions(id) ON DELETE CASCADE,
    experiment_id UUID NOT NULL REFERENCES zcl_experiments(id) ON DELETE CASCADE,
    group_name VARCHAR(20) NOT NULL, -- treatment, control
    assigned_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),

    UNIQUE(session_id, experiment_id),
    CONSTRAINT valid_group CHECK (group_name IN ('treatment', 'control'))
);

JSONB is PostgreSQL's binary JSON type. I used it here because treatment and control payloads vary by hypothesis type. Ordering two documents is shaped differently from excluding one document, and forcing those into a rigid column set would add ceremony without improving the evaluator.

The two constraints are doing real work, and they are doing the kind of work that is easy to skip and expensive to skip. UNIQUE(session_id, experiment_id) means a session is assigned once to a given experiment. Without it, a retry or a duplicated provisioning call would let one session contribute two rows to the same arm, which inflates the sample count with correlated data and makes the test more confident than the evidence warrants. The group check keeps the result set to treatment or control, so a typo cannot open a third bucket that then silently reduces the size of both real arms.

The migration also adds indexes for task type, status, active experiments, assignment lookup, session lookup, and group lookup. The write path needs to find active tests during provisioning. The read path needs to count assignments when judging results. Those are different access patterns, so the schema names both.

5. Outcomes are measured after the task

Retrieval can produce tidy scores before the agent acts. Rank, similarity, token fit, and document presence all arrive early. None of them says whether the work succeeded.

This is the gap the title is about, and it is worth stating plainly rather than implying it. The standard instruments for retrieval quality — recall at k, mean reciprocal rank, normalised discounted cumulative gain — all score a ranking against a set of documents somebody labelled relevant in advance. They answer whether retrieval found what a human said was relevant. Agent memory has to answer something else: whether the material changed what the agent did. Those questions come apart precisely where the interesting failures live. A document can be topically relevant, rank first, be judged relevant by any labeller, and still cost the agent attention it needed elsewhere. Recall at k cannot see that, because the harm is not in the ranking. It is in what happened next.

Asking a model to grade the retrieval has the same shape. It scores the plausibility of the material against the request, which is a judgement made before the work and without knowing how the work went. Both instruments measure the retrieval, and both stop exactly where the question starts.

The migration defines get_experiment_outcomes. It joins experiment assignments to recorded outcomes, grouped by branch. The function returns group name, total sessions, successful sessions, success rate, and average time in minutes.

CREATE OR REPLACE FUNCTION get_experiment_outcomes(p_experiment_id UUID)
RETURNS TABLE (
    group_name VARCHAR,
    total_sessions BIGINT,
    successful_sessions BIGINT,
    success_rate FLOAT,
    avg_time_minutes FLOAT
) AS $$
BEGIN
    RETURN QUERY
    SELECT
        ea.group_name::VARCHAR,
        COUNT(*)::BIGINT as total_sessions,
        SUM(CASE WHEN o.success THEN 1 ELSE 0 END)::BIGINT as successful_sessions,
        AVG(CASE WHEN o.success THEN 1.0 ELSE 0.0 END)::FLOAT as success_rate,
        AVG(o.time_minutes)::FLOAT as avg_time_minutes
    FROM zcl_experiment_assignments ea
    JOIN zcl_outcomes o ON ea.session_id = o.session_id
    WHERE ea.experiment_id = p_experiment_id
    GROUP BY ea.group_name;
END;
$$ LANGUAGE plpgsql;

This is where the layer stops being retrieval self-scoring. An inclusion hypothesis can be judged on whether the session succeeded; an order hypothesis is better judged on how long it took. Either way the metric was declared up front, and the outcome table supplies whatever was actually measured.

There is a floor under all of this that randomization cannot reach. zcl_outcomes.success is a bare BOOLEAN NOT NULL, and nothing in the schema says who decides it or on what evidence. Randomizing the assignment removes the confound between the user and the branch they got. It does nothing whatsoever about a mismeasured outcome. If success is set by a heuristic, or reported by the same person whose skill I was trying to control for in the first place, then the bias I built this entire layer to remove walks back in through the dependent variable — and this time the system reports it with a p-value attached, which makes it harder to argue with rather than easier.

If I want to judge some other behavior later, the outcome schema has to carry it first, or the candidate cannot be promoted inside this loop at all. It is a narrow door to walk through, and I would still rather have it than a soft win assembled from whatever trace happened to be lying nearby.

6. Minimum samples slow the system on purpose

ActiveLearning sets alpha = 0.05, with the code comment p < 0.05 for significance. The result object carries effect_size, p_value, significant, and recommendation. Evaluation compares the two arms' success counts with a chi-square test on a two-by-two contingency table, and the migration includes experiment_ready_for_analysis, which checks whether an experiment has enough samples before evaluation.

This is the guardrail I wanted most. A document can land in treatment, ride along with one successful session, and look useful if the system is hungry for reinforcement, so the threshold is supposed to make it wait.

At least that is what I had been telling myself. Writing this section I went to check the number, and the number is not wired to anything.

Hypothesis.min_samples is set to 20. Nothing reads it. It sits on the dataclass, rides into the row, and not one code path consults it before a result gets judged. The gate that actually runs is in analyze_experiment, which bails out only when an arm holds fewer than five outcomes, and its SQL counterpart experiment_ready_for_analysis takes p_min_per_group INTEGER DEFAULT 5. So the brake is five, not twenty. The twenty is decoration, and it is the kind that looks clean and fails quietly.

Five per arm cannot carry the conclusion the code is willing to draw from it. Work the table: five sessions in each branch, chi-square at alpha 0.05, and exactly two of the thirty-six possible outcomes clear significance. Five successes against zero. Zero against five. Both land at p equal to 0.011, and every other cell in that table is a null before the experiment starts. At this size the test is not measuring an effect. It is asking whether the two arms disagreed about every single session, which is not the question I meant to ask.

Chi-square is the wrong instrument here anyway. It wants expected cell counts of about five or more, and a two-by-two table built from five observations per arm cannot give it that. Fisher's exact test handles tables this small and the swap is one line. Raising the gate to the twenty already written on the object does not rescue it either — at twenty per arm the smallest difference the test can see is still around thirty percentage points.

Here is the scale I should have worked out before writing any of it. A ten point improvement in success rate, 0.50 to 0.60, at eighty percent power and the same alpha, needs roughly 194 sessions per arm. Five points needs about 782. Those are the numbers at which the words sitting in recommendation — ADOPT, REJECT — mean what they claim. Ten points is a substantial win for a context change. This layer would need forty times the evidence it currently demands to notice one.

There is a related exposure in how many tests run at once. The generator can open five order hypotheses and three inclusion hypotheses for a single task type. Eight independent tests at alpha 0.05 carry roughly a 34 percent chance that at least one clears significance by luck alone, and the one that clears is the one that gets promoted into every future session's context. A Bonferroni correction, or simply refusing to run more than one live experiment per task type at a time, is the cheap fix.

One thing the design gets right by accident of structure is worth crediting, because it is the error most A/B systems make. analyze_experiment completes the experiment as soon as it evaluates it. There is no path that looks at the data, finds nothing, and looks again next week. Repeated peeking at an accumulating result is how a nominal five percent false positive rate becomes twenty or thirty percent in practice, and this loop cannot do it: one look, then the experiment closes. The cost of that virtue is that the single look happens at the earliest moment it is permitted, which is also the moment with the least evidence behind it.

Put those two facts side by side and it is worse than slow learning. A completed experiment never reopens; _complete_experiment writes status completed and nothing anywhere sets it back to active. At five per arm almost every outcome is INCONCLUSIVE by construction. So a context change that genuinely helps gets one underpowered look, comes back as no significant difference because it could hardly come back as anything else, and is closed on that basis permanently. I built the threshold to stop the system adopting noise. What it actually does is retire real improvements the test was never equipped to detect, and there is no route back to retry one.

The exploration rate decides where this lands hardest, and it picks the worst place. _get_task_uncertainty returns maximum uncertainty for any task type with fewer than ten recorded sessions, which pushes the exploration probability straight to its 0.4 ceiling, and that count is scoped per organization. A small organization therefore runs the largest share of its sessions as trials while generating the fewest sessions to finish any of them with. It experiments hardest precisely where five outcomes per arm takes longest to accumulate, and every one of those trials still closes after a single look.

The stored result hides that rather than surfacing it. effect_size goes into the results JSON as a bare difference between two success rates, with no interval around it, so a gap measured on five sessions per arm is recorded in the same shape, and reads later with the same authority, as one measured on five hundred.

Waiting costs something. Bad ideas stay alive longer, good ones take more sessions before they become default behavior, and some combinations never get tested at all because the generator cut them off early. That pressure is almost certainly how the number ended up at five: a gate that low returns verdicts fast, and returning verdicts fast is the entire problem with it.

The operational states are active, completed and cancelled. An active experiment can still receive assignments and a completed one can store results, but the useful one is cancelled: it stops shaping sessions without pretending it ever reached a conclusion. The view v_experiment_results puts treatment counts, control counts, status, and timestamps together so the operator can see whether a run is still gathering evidence or ready for judgment.

That is the trade I made for agent memory and retrieval: learn slower, record the branch, measure after work, and require enough samples before future sessions change. Three of those four are built and working. The fourth is a number I wrote on an object and never wired up, and writing this post is what made me look.

The rule I started with still holds. A learning system that changes future inputs has to earn that change with outcomes, or it will preserve coincidence as policy. Randomization is what makes an outcome mean anything. The sample threshold is what stops it being noise. I built the first one properly. The second I wrote down on a dataclass and never wired up, so the system will go on drawing conclusions from five sessions and filing them with the same confidence it would give five hundred.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

The Matte Learns Only Inside the Band

Daniel Romitelli — Tue, 18 Aug 2026 06:21:06 +0000

A bad cutout rarely announces itself as a bad cutout. The car lands on a new backdrop, the paint looks clean, then a thin piece is gone. An antenna. A tire lip. The dark seam under a rocker panel. The complaint that comes back is never technical. The vehicle looks wrong.

I wanted the last correction stage to fix fuzzy edges without handing it the whole car to rewrite. That sounds like a small distinction. It stops being small the first time a model improves one boundary and quietly damages another. So the rule is physical. Edit the uncertain strip. Leave the settled area alone.

This is Part 2. Part 1, "Negative Space Is a Label", was about supervision: what the pixels beside an object teach a model, and why a shadow touching a tire has to be labeled as evidence against foreground. This one moves from training to runtime. A mask already exists. Where is a learned stage allowed to act?

1. The contract lives in the band

CarSegNet is the research implementation here. Its pipeline module splits the route by media type, and the docstring says the design more clearly than any diagram I could draw after the fact.

Stills run SAM 3 text concept, then NSJ alpha, then composite. A detector box prompt and a depth prior are optional inputs. Video runs SAM 3.1 multiplex propagation, per-frame NSJ with temporal handling, a depth-parallax plate, composite, encode.

The list matters less than the handoff. SAM gives a semantic prior. NSJ receives a trimap band. The compositor receives a matte only after the prior and the refiner have each done bounded work.

flowchart TD
image[Vehicle Image]
segment[Concept Mask]
trimap[Trimap Band]
refiner[NSJ Alpha Refiner]
depth[Depth Prior]
composite[Showroom Composite]
frozen[Prior Frozen Outside Band]
image --> segment
segment --> trimap
trimap --> refiner
image --> depth
depth --> refiner
refiner --> composite
segment -.-> frozen
frozen --> composite

The diagram is a contract. It is not a model zoo. The refiner edits the uncertain strip. The semantic prior owns the rest of the frame.

Models build lazily. A pure recomposite run against cached mattes never pays to load a large segmentation checkpoint. The cost lands on whichever execution path needs that model first. I take that trade. Cached matte work should stay cheap and inspectable, and loading every model for every run hides an orchestration problem behind hardware capacity.

2. The band uses image disagreement

The alpha refiner lives in its own module, and the one-line file description is the entire design: trimap, band crop, NSJ, or deterministic fallback.

A plain band can be built from the prior alone. Take the hard mask, grow it, shrink it, call the ring unknown. That catches soft contour error. It fails on a prior that is confidently wrong.

The comment in the band builder names that case in capitals: WRONG AND CONFIDENT. Both halves of a morphological band are functions of the prior, so a prior with no doubt produces no band at all. Fill in a wheel opening and the hole becomes confident foreground. Drop a roof antenna and those pixels become confident background. Either way the missing area can sit nowhere near an iso-contour, and a morphology-only band never asks the refiner to look there.

So CarSegNet adds an image term. Where the photograph shows strong structure and the prior shows nothing happening, that disagreement opens the band. The photograph says edge. The mask says flat. That argument is worth examining.

It stays bounded. The search is restricted to the subject's own neighborhood, the edge criterion is relative to the image instead of a fixed number, and the band has a ceiling it cannot cross.

Those limits cost something. Widen the neighborhood and foliage, fence lines, or lot texture start lighting up the image term. Tighten it and the antenna case stays frozen. The ceiling is the one I would defend hardest. It stops a local repair path from turning into a full-frame request, which means a badly wrong prior has to be rejected upstream instead of handed to the refiner as though it were close.

That is the transferable part. A learned correction needs a declared edit domain. Here the domain is a trimap band carrying a disagreement term, so confident prior mistakes get a chance to be examined near the vehicle.

3. Full-frame alpha drifts quietly

The tempting design is simpler. Segmentation produces a rough mask, a neural refiner outputs a full alpha matte, compositing uses that alpha. Fewer moving parts. Far more hidden authority.

A full-frame output can fix a tire edge and move a roofline in the same pass. It can smooth a window halo and erase a mirror. The bad part is that such a trade can improve an averaged score, because most pixels in a vehicle photograph are easy background or easy paint. The listing still fails at the one boundary a buyer looks at.

NSJ is small. Size is not the safety property, and I want to be exact about that, because small models get described as safe all the time. A small unconstrained model can still damage broad vehicle topology. The constraint around the output does the work.

The completed checkpoint audit is what stopped me from reading boundary gain as deployment clearance. Boundary scores improved with the trained checkpoint. The same audit showed the model almost never recovered enclosed openings. The outline got better. The topology did not.

That changes what the stage is allowed to claim. NSJ is an edge repair component. It is not evidence that the system understands window holes, wheel openings, cabins, or glass ownership. Those errors are structural, so they need separate gates.

The fallback path has the same shape. With no trained checkpoint the refiner runs a deterministic guided-filter route instead of stopping the pipeline at missing weights. The system stays runnable on day one. The tradeoff is visible in what each path can actually do. Guided filtering cleans a local alpha transition. Learned interior reasoning waits for trained weights and a passing topology gate.

I cut a synthetic hard-case metric table out of this argument while drafting it. The numbers were useful during development. Without the full setup sitting next to them they were decoration, and they pulled attention away from the contract. The code path is the stronger evidence: band crop, NSJ or fallback, guided filter, prior preserved outside the declared edit area.

4. A guard catches the silent sliver

The same module that routes the models also rejects one specific mask failure before it reaches compositing. There is a dedicated error type for mattes that are fragments rather than vehicles.

The default came out of measurement rather than taste. Sliver mattes cluster far below anything a real vehicle produces under the serving prompt. Legitimate vehicles start well above that cluster and run up to most of the frame. Between the two populations sits a wide empty band, and the default lives inside it. That gap is the only reason I trust a single scalar here.

It is a different kind of trust region than the band. The trimap band limits where alpha refinement may edit. The subject-fraction check limits which priors are allowed into the rest of the route at all.

The asymmetry is what justifies it. A miss is visible and recoverable. A tiny foreground sliver composites silently and looks like a strange crop, so it gets its own exception type while still subclassing the pipeline error every caller already handles.

There is a cost. Small distant vehicles, odd crops, and dealer photos with unusual framing can trip it. The threshold moves through configuration and can be switched off entirely. I keep the default because the measured gap is wide for this serving path, and in a media workflow a silent sliver is worse than a loud miss.

5. Glass got a smaller claim

Glass is where boundary language gets overloaded. A side window is transparent, reflective, tinted, and part of the vehicle shell, depending on which question you are asking. One binary mask answering all of them is false authority.

The bootstrap model predicts a glass region of interest from a single RGB vehicle image. Its scope is deliberately narrow: find the reviewed glass region for later work. The module's constants name the capabilities it does not have. That is useful friction. Future code has to cross a named boundary before it can pretend a region mask solved reflection removal.

The follow-on model has a stricter data contract than ordinary open and closed window classification, and the pilot captures do not satisfy it yet. Screening material is not the same as clearing it to train. The tooling says so in its own docstring, and the pilot is still marked not training ready.

The part that belongs in a post about the trimap band is the decode rule. The learned model answers a narrow question inside reviewed bounds. Every pixel outside that region is copied from the input, exactly. Same design pressure, different subsystem.

6. The composite makes excess authority visible

The final judgement happens after the vehicle lands on a plate. The compositor harmonises color in LAB, matches defocus, adds a contact shadow, and applies light wrap. Those operations can make a correct cutout sit convincingly in a new scene. None of them restores a mirror that alpha refinement erased.

This is where vague model authority turns expensive. Old asphalt left under a tire means contact shadow competes with captured ground. A halo that survives around a window means light wrap makes the bad edge look intentional. An antenna dropped before composition means every downstream operation is polishing a false matte.

Video adds one more constraint. The pipeline uses a depth-parallax plate before compose and encode, so there is one still plate with local motion rather than a generated background per frame. Camera freedom drops. Repeatability goes up. Same matte, same plate, same depth map, same frames again.

The selftest keeps model availability separate from pipeline correctness. It runs on CPU with no network, no GPU, and no model downloads, exercising image and video runs against stubbed backends. Cached mattes, compositing, encoding, and quality reports can fail or pass without waiting on gated weights.

That split matters for what comes next. Part 3 goes inside the vehicle, where the opening stage stays disabled until calibration and a locked topology gate pass. The problem there is harder than fuzz at the edge. Subtract alpha inside a guarded interior region while preserving the filled outer silhouette.

A vision system gets safer when every learned edit carries a boundary condition.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

When Missing Privacy Evidence Becomes Zero

Daniel Romitelli — Sun, 16 Aug 2026 09:38:02 +0000

At the end of a training run, the privacy measurement existed. The client logged it. Then the return value dropped it.

That small omission changed the meaning of the entire system. Downstream code treated missing evidence as zero privacy cost, an accountant displayed a clean budget it had never been told to advance, and export proceeded because a file existed. Nothing crashed. Every component looked locally reasonable. The false conclusion appeared only when I traced one fact across all of them.

1. The measurement that vanished

The system uses federated learning, so each client trains locally and sends an update to an aggregator. The client applies differential privacy during training through Opacus. At the end of each epoch, data_ingest/fl_client.py asks the privacy engine for epsilon, the numerical privacy-loss bound at a chosen delta:

epsilon = privacy_engine.get_epsilon(DP_TARGET_DELTA)

The value is real enough to appear in the log. It is absent from the value returned to the server:

return self.get_parameters(config={}), n_samples, {"train_loss": avg_loss}

The aggregator in fl_aggregator/strategy.py expects a different contract. It looks for an epsilon metric and supplies a default when the key is missing:

fit_res.metrics.get("epsilon", 0.0)

That default is the decisive line. A measured value of zero and an unobserved value are different facts. The first can support a decision. The second means the decision lacks an input. Converting both to the same floating-point number is an epistemic type error: the program has collapsed what it knows with what it failed to learn.

The neighboring loss metric reveals the same boundary mismatch. The client returns train_loss; the strategy asks for loss. Both look plausible in isolation, so ordinary local review can miss the disagreement. The problem lives between modules, in the meaning of their shared record.

There is a second trap. Even if every client returned epsilon, taking the arithmetic mean of those values would be telemetry, not necessarily a valid global privacy composition. Different clients can have different sampling rates, step counts, and exposure histories. An average can hide the most exposed participant. The server needs the accounting events required by its threat model, not a comforting aggregate of already-composed answers.

2. Four correct components can imply a false system

The server also creates a PrivacyAccountant. It can use a Privacy Loss Distribution backend, called PLD in the code, fall back to a Renyi Differential Privacy accountant, called RDP, record total steps, compute epsilon, and report whether the configured limit has been reached. Its unit tests call step() and verify that the number moves.

The running application does not call that method. A source search finds PrivacyAccountant.step() in the accountant tests, but no production caller advances the server instance created in fl_aggregator/server.py. The status endpoint can therefore report an accountant value of zero even after local training has consumed privacy budget. The display is reading its object correctly. The object was never given the events that make its answer meaningful.

The export path then introduces a third independent truth. When the training server exits, a finally block invokes the post-training pipeline:

finally:
    _update_state(fl_running=False, current_round=num_rounds)
    FL_CURRENT_ROUND.set(num_rounds)
    logger.info("Flower server finished")

    _post_training_pipeline()

That pipeline reconstructs the aggregate model and writes an Open Neural Network Exchange file, called ONNX in the code. The download endpoint serves the latest model if a path exists. The zero-knowledge export endpoint likewise checks for an ONNX file before it starts compilation. Neither route asks whether privacy observations were complete, whether the accountant corresponds to this run, or whether the recorded budget was still valid at the moment of export.

flowchart TB
    A["Client computes epsilon"] --> B["Returns train_loss only"]
    B --> C["Aggregator substitutes 0.0"]
    C --> D["ONNX export"]
    D --> E["Zero-knowledge artifacts"]
    P["Server privacy accountant"] -. "not advanced" .-> D
    G["Required eligibility gate"] -. "missing" .-> D

This is why component inventories are weak architecture evidence. The codebase contains private training, an accountant, an export service, and a proof toolchain. Listing those nouns makes the design sound complete. Following one decision from observation to release shows that their authority never converges.

3. A computation proof is not a training-history proof

The exporter in fl_aggregator/zkml/exporter.py is substantial. It can produce model.onnx, calibration input, circuit settings, a compiled circuit, proving and verification keys, and a verifier contract. It uses EZKL, a zero-knowledge proof toolchain for machine-learning models.

Those outputs answer an important question: can a verifier check the statement encoded by this circuit and its public inputs? They do not automatically answer a different question: did the model enter the circuit through an observed training run whose privacy events were complete and within policy?

I call that second property export eligibility. It is deliberately narrower than general provenance. Provenance can tell me where an object came from. Eligibility must decide whether these exact bytes may cross a boundary now.

The distinction matters because proof systems are literal. A valid proof says that the encoded relation held. It does not inherit facts that were never encoded or bound to the relation. If the circuit digest is unrelated to the training round, or the privacy state is unrelated to the exported model digest, the system has several valid facts without a valid conjunction.

The export predicate I want is explicit:

exportable =
    privacy evidence is complete
    AND the authoritative accountant is within its configured limit
    AND the accountant snapshot names the completed training round
    AND the model digest names the bytes sent to circuit generation
    AND every required artifact exists and matches its recorded digest

The first clause is essential. “Not exhausted” is insufficient when the accountant was never advanced. Unknown must fail closed before the numerical comparison is even allowed to run.

4. The missing object is an atomic export manifest

The repair is not another dashboard field. It is a small, immutable manifest emitted at the only place that can see the complete decision. The current code does not implement this object; this is the contract the audit derives.

Field	What it binds	Reject export when
`schema_version`	The parser to the contract version	The version is unsupported
`training_run_id` and `federated_round`	The files to one completed aggregate	Either identity is absent or mutable
`privacy_state`	Observation status, such as measured, unobserved, or invalid	The state is not measured
`epsilon`, `epsilon_limit`, and `delta`	The measured loss to the policy used for the decision	Values are missing, non-finite, or over limit
`accounting_backend` and `accounted_steps`	The result to its composition method and event count	The backend failed or the event count disagrees with the run
`model_digest`	The decision to the exact ONNX bytes	Recomputed bytes differ
`circuit_digest`	The model export to the compiled relation	The circuit is missing or does not match
`proving_key_digest` and `verifying_key_digest`	The bundle to its cryptographic material	Either key is absent or changed
`source_commit`	The run to a reviewable source state	The source identity is unavailable
`exported_at_utc`	The snapshot to a specific release event	The timestamp is absent or malformed

The write order is part of the contract. First, the server freezes a snapshot containing the round identity and authoritative accounting state. It refuses missing client events instead of substituting zero. It composes those events according to the declared accounting model and stops if the state is unobserved, inconsistent, or over budget.

Only then does it write the ONNX model into a staging directory, compute the content digest, and give those exact bytes to circuit generation. After compilation, it digests the circuit and key material. The manifest is written last. A single rename promotes the staging directory to its final run-specific location. The staging and final directories must share a filesystem if the rename is expected to be atomic.

The serving rules become simple. “Latest” means the newest completed bundle, not the newest loose file. Download rechecks the model digest against the manifest. Proof export accepts a run identity and reads the model named by that bundle. A partial directory, stale key, missing privacy event, or mismatched digest is unavailable by construction.

This design also changes the interface between client and server. A privacy-enabled client must return structured accounting evidence, including the step count and parameters needed by the chosen composition rule. The aggregator must reject an update that claims private training but omits that evidence. Diagnostic client epsilon can still be recorded, but it cannot silently become the server’s authorization rule through an average.

5. The real novelty is the conjunction

The earlier design idea was to preserve context as it crosses a boundary. This failure is different. No individual context object was missing. The client had one truth, the aggregator inferred another, the accountant held a third, and the exporter acted on a fourth. The novel object is the conjunction that none of them could assert alone.

That is also why the bug survived superficially strong evidence. The client log showed epsilon. The accountant endpoint returned a structured report. The model file existed. The proof directory contained cryptographic artifacts. Each observation was true, yet the sentence assembled from them was false: this model is eligible for export under this privacy history.

The correction is a general systems rule. Never let absence inhabit the same value as success. Never let a dashboard object become authoritative unless the events that advance it are part of the production path. Never let file existence stand in for a completed decision. When several subsystems jointly authorize an irreversible boundary crossing, make their conjunction a first-class object and bind it to the exact output bytes.

The hardest code-review findings are often not broken functions. They are false theorems assembled from locally correct premises. Finding one requires reading the gaps between modules as carefully as the modules themselves.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

A Context Object Should Carry Its Receipt

Daniel Romitelli — Sun, 16 Aug 2026 02:58:41 +0000

A stored fact can be wrong in a quiet way. The answer still reads clean. A preference from an old exchange gets reused, the message goes out with confidence, and later nobody can tell why that detail was allowed back into the result.

That is the failure I built around. When a system returns remembered material, the caller needs the text plus the reason it passed the reuse check. A log line found after the action is weak evidence. The object that leaves the memory service has to carry the admission record with it.

1. Keep the outside surface small

This is the pattern I used in Holographic, Law-Bound Memory (HLM), a stand-alone memory brain outside application code. The README describes public Application Programming Interface (API) routes under /api/brain/*, with internal /api/v1/* services behind that layer.

The outside shape is intentionally thin: register an agent, write a fact, build a capsule. The Python Software Development Kit (SDK) in sdks/python/hlm_sdk/client.py shows the boundary without exposing table names or policy code:

import httpx

class HLMClient:
 def __init__(self, base_url: str, token: str | None = None):
 self.base_url = base_url.rstrip("/")
 self._client = httpx.AsyncClient(headers={"Authorization": f"Bearer {token}"} if token else None)

 async def register_agent(self, name: str):
 r = await self._client.post(f"{self.base_url}/api/brain/agents/register", json={"name": name})
 r.raise_for_status
 return r.json

 async def write_fact(self, text: str, tags: list[str] | None = None, selectors: list[str] | None = None):
 r = await self._client.post(f"{self.base_url}/api/brain/memory/facts",
 json={"text": text, "tags": tags or [], "selectors": selectors or []})
 r.raise_for_status
 return r.json

 async def build_capsule(self, query: str, budget_tokens: int = 2048):
 r = await self._client.post(f"{self.base_url}/api/brain/context/capsule",
 json={"query": query, "budget_tokens": budget_tokens})
 r.raise_for_status
 return r.json

The TypeScript client in sdks/node/src/index.ts exposes the same calls as registerAgent, writeFact, and buildCapsule. That costs me a compatibility surface at the gateway. I accept the cost because admission policy in every consumer becomes drift. One caller skips a selector, another copies an old threshold, a third treats a nearby match as enough. Centralizing the decision gives the service a place to say yes or rebuild before the application acts.

2. Write facts with handles the service can check

A plain text memory is easy to save. It gives retrieval very little to inspect later. HLM writes each fact with tags, selectors, and an optional tenant field so the service has decision axes before a query shows up.

The write model in services/memory/app/main.py is small:

from typing import List
from pydantic import BaseModel, Field

class FactIn(BaseModel):
 text: str
 tags: List[str] = Field(default_factory=list)
 selectors: List[str] = Field(default_factory=list)
 tenant_id: str | None = None

That class feeds create_fact. The row itself lands in brain_facts. Two more writes follow against the same fact id, one for facets and one for predicates. The response returns the new id with both sets attached.

generate_facets has two paths in the current scaffold. A known selector value produces a specific facet row. Anything empty or unmatched falls back to a general facet, built from the first 256 characters of the text with the token count capped at 64. Those are constants in the code rather than performance claims. What they show is the shape: retrieval sees more than a blob of prose.

generate_predicates turns selector strings into one predicate joined with AND, swapping the first colon for =. That is rough. It is also enough, because the fact now leaves the write path carrying handles a machine can check. The tradeoff lands on the writer. A caller that sends empty selectors can still store text, but later selection has fewer axes to test.

3. Decide admission before capsule assembly

The reuse service is Conformal-Causal Reuse (CCR). Its request model in services/ccr/app/main.py carries the cache key, artifact type, selectors, and optional numeric controls:

from fastapi import FastAPI
from pydantic import BaseModel, Field

app = FastAPI(title="HLM CCR", version="0.1.0")

class CCRRequest(BaseModel):
 tenant_id: str | None = None
 cache_key: str
 artifact_type: str = "resume_kit"
 selectors: list[str] = Field(default_factory=list)
 similarity: float | None = None
 tau: float | None = None

The rule in reuse_or_rebuild is direct: a hit requires similarity > tau and the required selector kinds stakeholder, time, and channel to be present. Missing one of those kinds makes the service return rebuild. When the request omits values, the code uses 0.9 for similarity and 0.8 for tau. Those numbers are defaults in the function. They are not measured latency, quality, or production calibration.

The response includes decision, tau, similarity, and causal_ok. All four travel with the answer. Accepted material can name the rule that admitted it, and a rejection arrives as a rebuild decision instead of a silent empty match.

Calibration stays beside the same service. CalibIn accepts selector, similarity, and span_error; update_calibration computes a rounded tau and clamps it between 0.5 and 0.95. I kept that logic near the decision endpoint because threshold repair separated from the admission rule becomes another place for drift. The cost is coupling. CCR owns both the current decision and the local adjustment path.

4. Build the capsule with provenance attached

The orchestrator joins the pieces. In services/orchestrator/app/main.py, /api/v1/capsule derives selectors from the query and posts them to CCR. What comes back shapes the capsule: content, confidence values, reasoning metadata, and a proof value.

The intended object is signed context rather than an anonymous bag of nearest neighbors. The current branch is still a scaffold. Episode writes return receipt: "merkle:demo", and the orchestrator repeats that same demo value in its local capsule response. The slot is real and the hardening is unfinished, so the caveat stays in the design.

The four-step path is the engineering pattern:

flowchart TD
 factWrite[Fact write] --> governedReuse[Reuse decision]
 governedReuse --> capsuleBuild[Capsule build]
 capsuleBuild --> proofReceipt[Proof receipt]

services/gateway/app/main.py contains merkle_root(items). It hashes the item strings and folds pairs until one hash is left, duplicating the last leaf when the count is odd. The result gets a merkle: prefix. The gateway is the right home for it, because the external object is formed there. Downstream code should receive a single object holding the selected material, the CCR decision fields, and a provenance value it can store or compare later.

The architecture document names this Proof-of-Context (PoC): Merkle roots over snapshot, version, tau, model, and ids. The label matters less than the placement. If applications learn to consume loose context first, provenance turns into a retrofit. Retrofitted evidence is usually optional. Optional evidence disappears under deadline pressure.

5. Preserve the same object across streamed updates

Memory work does not always end at the first capsule. services/orchestrator/app/main.py has a local loop that yields Server-Sent Events (SSE) through a StreamingResponse; each packet includes a generated packet_id, summary fields, next actions, and a timestamp before the loop pauses. The gateway forwards this through /api/brain/context/stream. The README and architecture notes describe the larger outbox path with leases, backoff, a dead-letter queue (DLQ), and a resume stream over SSE or WebSockets (WS). The current code handles the visible stream contract; the documented shape says lineage has to move with later packets as well. That adds overhead compared with returning an array from a nearest-neighbor endpoint, but a resumed update without the original admission data is just another loose event.

6. Own the memory lifecycle

This design buys safer reuse by moving work into the memory service. Writers must send useful selectors. The gateway becomes stricter. The service has to keep admission metadata and provenance beside the content from write, through CCR, into capsule creation and streaming. I prefer that pressure inside HLM over spreading half-copied rules through applications, because systems that remember should expose the conditions under which memory became usable.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series