DEV Community: Daniel Romitelli

A Perfect Fit Can Recover the Wrong Physics

Daniel Romitelli — Wed, 14 Oct 2026 07:42:51 +0000

AutoLensAI moves photographed vehicles into virtual studios for dealership listings. I need to change the surroundings without changing the car: its equipment, its cabin, even the damage in its paint. A reflection across a headrest puts those requirements in the same pixels.

On October 4, I had to change what a glass estimate was called. The code could fit a factor relating two photographs. It had not established that the factor measured how much light the glass transmitted. Illumination and camera processing could affect it too.

Calling the value transmission would have made downstream code treat an uncalibrated estimate as a measured property of the pane. I kept the appearance estimate and made the missing evidence explicit.

That distinction is easy to dismiss while the reconstructed image looks right. It becomes harder to dismiss when two explanations fit every observed sample, yet one predicts the wrong result as soon as the lighting changes.

The number the photographs actually identify

Rendering starts with geometry, materials, and light, then predicts an image. Inverse rendering works backward from images to those causes. My glass problem belongs to the second direction, but a good reconstruction alone does not tell me how much of the explanation I have recovered.

Consider one intensity channel with perfect alignment, a linear camera response, no reflection, no clipping, and no noise. Let a reference sample be $x_i$, the observed sample be $y_i$, transmission be $\tau$, and the illumination ratio between conditions be $g$:

$$
y_i = \tau g x_i.
$$

The observations determine the product $\tau g$. They do not, by themselves, separate its factors.

Explanation	Transmission $\tau$	Illumination ratio $g$	Product
Synthetic ground truth	0.40	1.50	0.60
Alternative	0.60	1.00	0.60

Both rows produce the same image. Assuming unchanged illumination picks the second row and makes the transmission estimate 50% too high.

These numbers are chosen for a reproducible counterexample, not taken from a vehicle. The ambiguity is exact: replacing $(\tau,g)$ with $(c\tau,g/c)$ leaves the image unchanged for any positive $c$ that keeps both parameters admissible.

An optimizer can return one answer from that family. A prior can favor one answer, and regularization can make the choice stable. None of those choices adds an observation. I need to know which part of the result came from the photograph and which part came from an assumption.

Removing reflection from this example makes the point more specific, not less relevant. Transmission and illumination are already inseparable here. Adding an unknown reflection field does not automatically supply the missing information.

Held-out pixels can have the same blind spot

The script at the end generates 64 positive reference samples. It fits the product on 32 and reserves the other 32 for evaluation. Both explanations reproduce all the reserved samples with zero error in the recorded run.

Nothing leaked from the held-out set. The split did its job. It tested reconstruction under a condition that both explanations could reproduce.

The distinction I care about appears under a different test. Set a new illumination ratio independently to $g_{\mathrm{new}}=1$, and do not allow another fit. The true transmission predicts $0.4x$. The alternative predicts $0.6x$. Their agreement on the source has become a 50% relative prediction error.

Allowing the second explanation to adjust its illumination again would hide the mistake. The changed condition has to remain fixed by evidence outside the fit.

More pixels are useful when estimating the product in noise. They do not separate the factors in this model. The accompanying tests check 8, 64, and 4,096 samples and find the same missing degree of freedom. More images have a similar qualification: if every image brings another unconstrained illumination multiplier, the absolute scale can remain ambiguous.

This is why I separate source replay from a test under changed lighting. Source replay asks whether the explanation can reproduce what it was given. The second test asks whether it predicts something it was not allowed to adjust itself to match.

Architecture of the public experiment

flowchart TD
  truth[Known synthetic parameters] --> source[Source observations]
  source --> fit[Fit on training samples]
  fit --> alternatives[Image-equivalent explanations]
  alternatives --> replay[Reconstruct held-out samples]
  source --> replay
  replay --> sourceError[Source reconstruction error]
  alternatives --> changed[Predict under known new lighting]
  lighting[Independent lighting condition] --> changed
  truth --> target[Separate target ground truth]
  lighting --> target
  changed --> compare[Compare changed-condition predictions]
  target --> compare
  compare --> parameterError[Error hidden by source agreement]

The target comes from the known synthetic scene, not the candidate estimate. Using the candidate's own parameters to build its reference would make the comparison circular. The diagram shows the evaluation; the private recovery method is outside its scope.

What would let me separate them?

The Jacobian makes the missing information visible. It describes how the predicted samples change as each parameter changes:

$$
J=\begin{bmatrix}g\mathbf{x}&\tau\mathbf{x}\end{bmatrix}.
$$

The columns are proportional. There are two parameter columns but only one independent direction of change, so the matrix has rank one. An increase in transmission can be canceled by a decrease in illumination.

Now add a separate measurement of $g$. Its derivative with respect to $(\tau,g)$ is $[0,1]$. Appending that observation raises the rank to two, provided the reference has nonzero energy and $g$ is nonzero. In this model, that is enough to separate the factors.

An illumination value computed from the same unresolved image ratio adds no independent information. Neither does another region that carries the same unknown optical factor merely because I give it a different label.

Uncertainty matters after the rank problem is resolved. Suppose the product is exactly $0.6$, but the new measurement bounds illumination between $1.425$ and $1.575$. Then

$$
\tau\in\left[\frac{0.6}{1.575},\frac{0.6}{1.425}\right]
\approx[0.381,0.421].
$$

That interval uses deliberately chosen bounds. It shows what happens when the independent reading is imperfect. On a real capture, uncertainty in the fitted product, camera response, alignment, and image model would also have to be accounted for.

Camera calibration cannot be skipped by calling the working pixels linear. Inverting an sRGB transfer function removes that transfer function; it does not undo every operation that produced a processed photograph. In Debevec and Malik's radiance-recovery method, known exposure relationships are used to recover the imaging response up to scale. That response is part of the recovery, not a name assigned to the decoded array.

The October 4 audit had not established the corresponding calibration for my capture path. Its illumination diagnostics also lacked uncertainty bounds. Those are reasons to retain an appearance estimate, even when the structural test and image fit look encouraging. They are not reasons to discard the work, but they change what I can use its output to claim.

608 images, five matched openings

I had already encountered a different way for a good image score to conceal a bad result.

In August, I evaluated a trained alpha refiner on 608 images from 38 vehicles. Alpha matting determines how much of the source to retain at each pixel. The refiner's job was to improve vehicle boundaries, and its boundary score did improve over the upstream semantic mask.

Then I counted the enclosed background openings.

Output	Region IoU	Boundary IoU	Matched openings	Extra openings
Semantic mask	0.9904	0.4233	1 of 1,193	72
Fallback refinement	0.9815	0.2596	28 of 1,193	2,492
Trained refinement	0.9908	0.4762	5 of 1,193	111

From the August evaluation record. These images were used for checkpoint selection; they were not an untouched test set.

IoU measures overlap with the reference region. Body pixels dominate it. A filled wheel opening occupies little area beside an entire door or hood, but it still carries the old background into the composite. Boundary IoU looks more closely at contours. The opening comparison checks whether enclosed background components have survived at all.

The trained model had the best boundary score and still missed 1,188 reference openings. The fallback matched more openings but created 2,492 extras. The extra count prevented me from treating that apparent improvement as useful recovery.

This failure had a different cause from the glass ambiguity. The refiner could modify only its allowed boundary region. A confidently filled opening outside that region was fixed with respect to the learned weights. Training that same stage longer could not give it control of pixels its output definition excluded.

In the glass example, different parameters explain the same observations. Here, the desired correction can sit outside the model's permitted outputs. The first problem needs evidence or assumptions that separate the explanations. The second needs a prediction target and output region capable of representing the correction, plus tests for the new false openings that freedom can introduce.

I retained the checkpoint for its boundary improvement. Its clean studio images and binary glass labels did not establish performance on dealer lots or recovery of light through glass. The opening counts were enough to stop the broader acceptance claim without denying what the model had learned.

I rejected the lowest error

On September 3, the relighting work presented another trap. I had no photograph of the actual truck under the destination lighting. Judging a correction only by how little it disturbed that truck could favor a method that barely changed anything.

I used a synthetic calculation with a known destination response to compare the alternatives. The approximate geometry used for lighting could leave its own pattern in the result. Suppressing that pattern was useful until the suppression also weakened the broad shading I was trying to add.

Candidate	Target RMS log error	Unwanted-pattern correlation	Protected-detail damage	Intended-shading correlation
Baseline	0.3869	0.9994	0.0594	0.9472
A	0.3242	0.0276	0.0042	0.9557
B	0.3257	0.0134	0.0000	0.9530
C	0.3335	0.0109	0.0000	0.8955

Recorded September 3 synthetic comparison. Candidate letters replace the private settings. Damage is a fraction of protected detail; the two correlations compare the unwanted pattern and the intended shading.

A had the lowest error against the target in log brightness, but it damaged protected structure. C removed more of the unwanted pattern than B, but it also weakened the intended shading. I selected B. A single ranking by either target error or artifact suppression would have selected something else.

The damage test needed checking too. Brightening a badge is not the same as changing its lettering. In the retained controls, smooth illumination changes registered zero structural damage. Replacing “F-150” with “F-250” registered 0.0629. That gave the rejection a specific meaning: the test was responding to an identity change, rather than any difference in RGB.

The calculation isolated the operator's behavior. It left out indirect illumination, render noise, real geometry gaps, and material boundaries. I can use it to compare those candidates, not quote its error as an expected result on a dealership photograph.

Run the optical check

The transmission example is intentionally small enough to inspect. It is standard multiplicative ambiguity, not a claim to a new theorem. The broader problem has a long history: Belhumeur, Kriegman, and Yuille's bas-relief result shows how compensating changes in shape, reflectance, and light can preserve images under different assumptions from this glass example. Modern methods such as NeRFactor make their representations and priors explicit when recovering shape, reflectance, and illumination.

Here the useful result is that even a perfect held-out image fit cannot tell the two rows apart. A known change of lighting can.

Save the following as optical_identifiability.py. Install NumPy with python -m pip install numpy, then run python optical_identifiability.py.

import numpy as np


def estimate_product(reference, observed):
    x = np.asarray(reference, dtype=np.float64)
    y = np.asarray(observed, dtype=np.float64)
    if x.ndim != 1 or y.shape != x.shape or x.size == 0:
        raise ValueError("expected matching nonempty 1D arrays")
    if not np.isfinite(x).all() or not np.isfinite(y).all():
        raise ValueError("samples must be finite")
    energy = float(x @ x)
    if not np.isfinite(energy) or energy <= 0:
        raise ValueError("reference must have finite positive energy")
    product = float(x @ y) / energy
    if not np.isfinite(product):
        raise ValueError("fitted product must be finite")
    return product


x = np.linspace(0.1, 0.8, 64)
true_tau, true_light = 0.4, 1.5
observed = true_tau * true_light * x
training = np.arange(x.size) % 2 == 0
heldout = ~training
product = estimate_product(x[training], observed[training])

for tau in (0.4, 0.6):
    fitted_light = product / tau
    prediction = tau * fitted_light * x[heldout]
    rmse = float(np.sqrt(np.mean((prediction-observed[heldout])**2)))
    assert rmse < 1e-14
    print(f"tau={tau:.1f}, light={fitted_light:.1f}, RMSE={rmse:.3g}")

# The new light is fixed independently; it cannot compensate for wrong tau.
new_light = 1.0
expected = true_tau * new_light * x[heldout]
wrong = 0.6 * new_light * x[heldout]
relative_error = np.linalg.norm(wrong-expected) / np.linalg.norm(expected)
assert np.isclose(relative_error, 0.5)
print(f"Changed-light relative error: {relative_error:.1%}")

# Check whether the observations distinguish the two parameters.
jacobian = np.column_stack((true_light*x, true_tau*x))
augmented = np.vstack((jacobian, [0.0, 1.0]))
assert np.linalg.matrix_rank(jacobian) == 1
assert np.linalg.matrix_rank(augmented) == 2

recovered_tau = product / true_light
assert np.isclose(recovered_tau, true_tau)

light_low, light_high = 1.425, 1.575
assert 0 < light_low <= light_high
interval = (product / light_high, product / light_low)
assert interval[0] < true_tau < interval[1]
print("Ranks:", np.linalg.matrix_rank(jacobian),
      np.linalg.matrix_rank(augmented))
print(f"Recovered tau with known light: {recovered_tau:.3f}")
print(f"Tau bound with uncertain light: [{interval[0]:.3f}, {interval[1]:.3f}]")

The run produced:

tau=0.4, light=1.5, RMSE=0
tau=0.6, light=1.0, RMSE=0
Changed-light relative error: 50.0%
Ranks: 1 2
Recovered tau with known light: 0.400
Tau bound with uncertain light: [0.381, 0.421]

RMSE is the root mean square difference between the held-out prediction and observation. The assertions allow floating-point roundoff. The companion ten-test suite also checks invalid inputs, unchanged input arrays, rank at different sample counts, and propagation of the illumination bounds. The example and all ten tests passed with Python 3.13.5 and NumPy 2.3.5.

The private model, capture procedure, training recipe, and recovery settings are not needed to reproduce this calculation. The August and September tables report retained experiments; the code above is the separate synthetic example run for this article.

For the actual glass estimate, the next empirical test needs an independently known change of condition and an uncertainty budget for the measurements. I need the estimate to survive that change of light, with no second fit to rescue it.

Cover: generated editorial artwork, not an image from the experiments.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

Part 2: The Window Is Not the Vehicle

Daniel Romitelli — Wed, 14 Oct 2026 06:16:22 +0000

AutoLensAI composites photographs of real vehicles into virtual studios for dealership listings. The setting changes. The vehicle's bodywork, equipment, cabin, and visible condition have to survive that change.

The windows make this harder than the outline suggests.

A side window can show a headrest, a reflection of a building behind the photographer, and another car visible through the far side of the cabin. Move the vehicle into a studio and pieces of the parking lot can come with it. Remove too much and the headrest goes too.

In Negative Space Is a Label, I wrote about teaching a matting model to reject shadows and floor reflections beside a vehicle. A dark patch touching a tire can still be background. The work in this follow-up concerns side glass, where the unwanted surroundings can overlap the vehicle detail I need to preserve.

Getting the outer edge right leaves that problem unresolved.

1. The problem moves inside the outline

A vehicle matte tells the compositor how much of the source image to retain at each location. Around the car, that helps separate a bumper from the lot behind it or preserve the fine edge of a mirror.

Inside a window, retaining the photographed color can retain several things at once.

The glass itself belongs to the vehicle. So do the seats, headrests, pillars, and other cabin surfaces visible behind it. A building reflected across those surfaces belongs to the surroundings. So does the parked car seen through an unobstructed view out the opposite window.

Those contributions don't necessarily occupy separate pixels. A reflection can cross a headrest. Calling that pixel “vehicle” keeps the reflection with it. Calling it “background” can remove part of the headrest.

Treating the whole window as opaque foreground preserves the old scene. Treating it as an empty opening can put the new backdrop where upholstery should be. Adjusting one transparency value across the pane cannot distinguish the seat from the view beyond it.

A glass mask can trace the window perfectly while the correction inside it damages the cabin. The mask locates the work. It does not establish which contributions can be separated.

2. A test passed with the correction removed

In a local glass-authoring experiment recorded on September 8, 2026, the correction was replaced with a function that returned the original image. A check was supposed to reject that result because no glass treatment had occurred.

The check passed anyway.

The operation assembling the image introduced tiny floating-point differences even when its inputs were identical. The check detected those differences and treated them as evidence that the correction had done something.

A regression test with synthetic floating-point data exposed the size of the error: 31 of 988 numeric elements differed, with a maximum absolute difference of 1.1920929e-07. Those are individual numeric values, not 31 altered vehicle pixels and not a measurement of reflection-removal accuracy.

The control had failed for a reason that would be almost impossible to establish by looking at the image. It was accepting arithmetic noise as intervention.

After the fix, unchanged inputs stayed bit-identical. The recorded local caller rejected the source-copy control and wrote no after image. The focused test run finished with 22 tests passing. That record establishes the local test behavior on that date, not coverage of the production system.

This exposed two different obligations. Protected pixels must remain unchanged, but a difference elsewhere does not establish useful work. A test that asks only whether two images differ cannot tell a repaired reflection from rounding error or damaged upholstery.

3. A cleaner window is not necessarily a more accurate one

For a dealership image, I care about whether the result still represents the vehicle being sold. A window that looks cleaner can fail that requirement.

An erased headrest might disappear into a dark cabin without drawing attention. An opening that has been made too transparent might show an attractive studio background where the car should block it.

A brighter interior might look better while giving the wrong impression of the tint. Darkness alone is no reason to erase a seat, just as brightness alone is no reason to erase a reflection crossing one.

Moving an image toward a cleaner reference does not, by itself, measure the glass's physical properties. It does not prove how much light the pane transmitted, or that the removed color came entirely from reflection.

The same applies to detail hidden by a reflection. An inferred seat can be visually convincing without matching the seat in that vehicle. Where the capture does not establish the underlying detail, generating a plausible replacement changes the nature of the photograph.

I am using this system to present an existing car. Preserving uncertainty is preferable to presenting an inferred feature as something the camera recorded.

4. The glass step has no permission to repair the rest of the car

In Part 1, the boundary refiner could work only within an uncertain strip at the edge of the car. The glass step follows the same restriction: it can modify a limited region rather than the whole photograph.

At the glass-processing step, pixels outside the permitted region are copied from the input unchanged. That includes the painted door, window trim, pillars, and other protected areas. The restriction applies to this operation, rather than making a claim that every pixel in a resized or encoded final image is numerically identical to the capture.

The architecture separates proposing a change, limiting where it can be written, and deciding whether the result is acceptable. The correction method remains inside one box.

Architecture

flowchart TD
  source[Source photograph] --> proposal[Proposed glass correction]
  source --> copy[Original protected pixels]
  proposal --> write[Restricted write]
  copy --> write
  region[Permitted edit region] --> write
  write --> check[Protected pixel check]
  source -.-> check
  check -->|Fail| reject[Reject candidate]
  check -->|Pass| review[Cabin and reflection review]
  source -.-> review
  review -->|Unsupported| reject
  review -->|Supported| accept[Accept correction]

A change to a protected pillar fails the copy check even when the window looks better. Passing that check sends the candidate to image review; it does not approve the correction.

The following is an independent example using NumPy, Python's array library. It demonstrates restricted copying, not the reflection-removal method. Both image arrays must already refer to the same pixel locations and color representation.

import numpy as np


def apply_patch(
    source: np.ndarray,
    candidate: np.ndarray,
    editable: np.ndarray,
) -> np.ndarray:
    """Demonstrate restricted copying, not glass correction."""
    if source.ndim != 3 or source.shape[-1] != 3 or source.size == 0:
        raise ValueError("source must be a nonempty H x W x 3 RGB array")
    if candidate.shape != source.shape:
        raise ValueError("source and candidate shapes must match")
    if source.dtype != np.float32 or candidate.dtype != np.float32:
        raise TypeError("both images must use float32")
    if editable.shape != source.shape[:2] or editable.dtype != np.bool_:
        raise ValueError("editable must be an H x W boolean mask")
    if not np.isfinite(source).all() or not np.isfinite(candidate).all():
        raise ValueError("image values must be finite")

    result = source.copy()
    result[editable] = candidate[editable]
    return result

The example has no blending, resizing, color conversion, or transparency handling. It copies the source and replaces only the permitted RGB samples. Shape validation prevents an incompatible array from being accepted, but matching shapes do not prove that two photographs are aligned.

The boundary itself can be wrong, though. A region drawn too tightly prevents useful work near the edge. A region drawn too loosely can admit trim or bodywork into an area intended for glass. Restricting edits does not make region selection infallible.

The consequence of a tight boundary is often a visible remnant of the original scene. That can be inspected and traced back to the region that blocked the correction. Allowing broader edits to hide it would make the system harder to judge, because the apparent improvement could come from changing the car.

I would rather correct the region that was wrong than give a later stage more freedom to conceal the mistake.

5. Four tests, and what they cannot establish

These tests use a synthetic four-by-four RGB array, not a captured vehicle or a production training sample. They check an unchanged candidate, a restricted edit, damage outside the permitted region, and damage inside it.

The byte comparison is deliberate. The requirement is an exact copy, not an image that is numerically close enough to the original.

import unittest


class CopyRuleTests(unittest.TestCase):
    def setUp(self):
        self.source = np.full((4, 4, 3), 0.25, dtype=np.float32)
        self.source[0, 0, 0] = -0.0
        self.editable = np.zeros((4, 4), dtype=bool)
        self.editable[1:3, 1:3] = True

    def check_protected(self, result):
        self.assertEqual(
            result[~self.editable].tobytes(),
            self.source[~self.editable].tobytes(),
        )

    def test_unchanged_candidate_preserves_every_byte(self):
        result = apply_patch(self.source, self.source.copy(), self.editable)
        self.assertEqual(result.tobytes(), self.source.tobytes())

    def test_only_permitted_pixels_change(self):
        before = self.source.tobytes()
        candidate = np.full_like(self.source, 0.75)
        result = apply_patch(self.source, candidate, self.editable)
        self.check_protected(result)
        np.testing.assert_array_equal(result[self.editable], candidate[self.editable])
        self.assertEqual(self.source.tobytes(), before)

    def test_one_protected_change_is_detected(self):
        result = self.source.copy()
        result[0, 0, 1] = 0.5
        with self.assertRaises(AssertionError):
            self.check_protected(result)

    def test_inside_damage_passes_the_copy_check(self):
        candidate = self.source.copy()
        candidate[1, 1] = 0.0
        result = apply_patch(self.source, candidate, self.editable)
        self.check_protected(result)
        np.testing.assert_array_equal(result[1, 1], candidate[1, 1])
        self.assertFalse(np.array_equal(result[1, 1], self.source[1, 1]))


if __name__ == "__main__":
    unittest.main()

Save both Python blocks, in order, as glass_copy_example.py. Install NumPy with python -m pip install numpy, then run python glass_copy_example.py. The example runs four tests.

The last test is the important limit. It darkens a permitted sample and still passes the protected-pixel check. That is expected: the check knows where a change occurred, not whether the changed sample belongs to a headrest. The test makes that limitation explicit rather than treating a passing preservation check as proof of image quality.

The September 8 review had a separate visual failure. The replacement scene read as a bright inserted opening beside the retained glass and cabin. A wider correction removed more of the original scene, but the preview remained rejected for photographic delivery. Fixing the no-change control did not make that image acceptable.

That is why image review still needs the source photograph. It has to establish whether the same cabin structures remain, whether they still block the view behind them, and whether old surroundings have been removed without deleting vehicle detail. Neither an accurate glass outline nor a count of changed pixels answers those questions.

6. When the evidence runs out

Part 1 separated vehicle, confirmed background, and uncertainty. That last category remains important here, but the reason for uncertainty has changed.

At the outer edge, the question was often where the vehicle ended. Inside the window, I can know where the pane is and still lack enough evidence to separate what the photograph contains.

The practical response is to limit the correction, preserve the parts that are established, and leave unresolved areas visible as unfinished work. A rule about where changes are allowed helps enforce that response. It does not solve the missing-information problem.

A remaining reflection may make the image less polished. A missing headrest makes it inaccurate. The finished listing still has to show the car I photographed.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

A Media Asset Is a Stateful Object

Daniel Romitelli — Mon, 12 Oct 2026 08:19:15 +0000

A phone can upload a sharp photo and still leave the product with nothing usable. The bytes are there. A thumbnail may exist. A background cutout may even look clean. None of that answers the question the product actually has to answer: can this image be shown, retried, reviewed, or published?

That is the failure I wanted to avoid. Vehicle listings are made out of media, yet accepting images is the easy part. The hard part is keeping the original, the expensive work, the human decision, and the publish button attached to the same object after retries and partial success.

This post is separate from the earlier provenance and approval-gate work. Those posts were about receipts and invalid final states. This one is about custody across storage, database jobs, vision workers, derived artifacts, human review, and publication. The model I use is simple: a media asset is a stateful object moving through a bounded vision pipeline. The endpoint bag is the failure mode.

The curated highlight for this build was Vehicle media intelligence pipeline with bounded multi-stage vision processing. Every expensive computer vision (CV) step has to write back to the same asset story, or the product will eventually show an image whose approval came from somewhere else.

1. Intake is proven by an event, not by a timestamp

The listing design starts with one walk-around becoming one finished listing: a hero set, social placements, description, and a spec sheet, with review before publication. Later work reserves room for 360 output and a forty-image set through SceneBundle. Product-shaped, yes. The important part sits underneath.

The trigger is a durable row in project_events named capture.media_secured. Its idempotency key starts with media-secured, includes the capture session identifier, and ends at version one. The payload is MediaSecuredEvent. It carries the project identifier, source asset identifiers, required roles satisfied, required roles missing, and processingEnqueued. Extra fields are rejected.

That last field is there because false is a real result. The phone may have secured the bytes while the next work item failed to enqueue. I wanted that outcome reported as a first-class condition, instead of being inferred later from silence.

The design also says the event row is the proof that photos are secured. The media_secured_at column alone is not enough, because another writer stamps it during egress. Two writers used the same timestamp for different meanings. The event has a name, a payload, and an idempotency key, so the capture handoff can be replayed or audited without guessing.

Only digest-verified source assets get their first processing_jobs rows. The first job key begins with preflight and includes the source asset identifier. That is the first custody boundary: work attaches to source asset identity, not to a temporary URL the interface happened to hold.

2. Separate endpoints break at the first partial retry

The tempting application programming interface (API) shape is easy to draw. Upload. Remove background. Enhance. Make thumbnail. Ask for status. Each endpoint has a clean request and a clean response, then the first uneven result arrives.

A derived image exists, while the quality verdict is missing. A retry produces a better frame, while the listing still points at the old one. A social crop finishes before the parent image is accepted. The review queue now contains outputs from different attempts of the same vehicle.

I did put bounded API surfaces around image work. The bulk per-image operations endpoint accepts image identifiers, optional transformation settings, and a priority hint. Its operation enum covers background removal, enhancement, and thumbnail creation. A separate status shape checks progress by vehicle identifier and can filter by job identifiers.

Those schemas are useful. They keep requests sane. They do not decide whether a listing is ready. Readiness lives in the persistent asset graph, because the last endpoint to run is often the wrong place to ask.

The listing build uses Postgres for that spine. It rejected an older five-queue description because the queues did not exist. The table is processing_jobs, with six check-constrained stages and states for queued, running, succeeded, failed, cancelled, and declined.

Workers claim rows with a SQL path that uses FOR UPDATE SKIP LOCKED. Stale work can be reclaimed after fifteen minutes. Attempts stop at three. The service role owns the claim path.

The drainer keeps the same shape. /api/cron/drain-processing runs from Vercel cron every five minutes. The default budget is ten, the maximum is fifty, and the route ceiling is one hundred twenty seconds. It is guarded by CRON_SECRET and PROJECT_PROCESSING_DRAIN_V1. Stage dispatch sits in STAGE_RUNNERS inside processing-drainer.ts.

That choice costs latency. A poller feels slower than a push system. I accepted that cost because the expensive part is keeping media, review, and publishability in agreement after a retry.

3. Four lanes have to agree

I draw the system as four lanes because each lane can tell a partial truth. Storage can prove bytes exist. The database can prove work was requested. A worker can say it produced something. The product can show a button as ready. The useful rule is the binding between them.

flowchart TD
  subgraph storageLane
    sourceBytes[Signed source bytes]
    derivedBytes[Derived media bytes]
  end
  subgraph databaseLane
    securedEvent[capture.media_secured event]
    jobRows[processing_jobs rows]
    reviewDecision[append-only review decision]
  end
  subgraph workerLane
    preflight[matte preflight]
    renderStage[studio render]
    listingStage[listing generation]
    marketingStage[marketing generation]
  end
  subgraph productLane
    ownerReadiness[owner visible readiness]
    reviewQueue[human review queue]
    publicationGate[publication gate]
  end
  sourceBytes --> securedEvent
  securedEvent --> jobRows
  jobRows --> preflight
  preflight --> renderStage
  renderStage --> listingStage
  listingStage --> marketingStage
  marketingStage --> derivedBytes
  derivedBytes --> reviewDecision
  reviewDecision --> reviewQueue
  reviewQueue --> publicationGate
  jobRows --> ownerReadiness```



The storage lane begins with signed phone evidence. In `src/project/services/vehicle-project-orchestrator.ts`, the orchestrator keeps digest verification narrow. Digest verification proves transport: the bytes that left the phone are the bytes in storage. It does not prove the image depicts a car. The field named `contentState` can say `not_evaluated`, pending, verified, or refused. It must never translate a digest match into content approval.

The same orchestrator signs URLs in parallel and returns them beside the asset states. That comment matters. A client cannot render a picture without receiving what the platform thinks of it at the same time. The URL is bounded surface area, never the master record.

The database lane carries job ownership and decisions. In `src/project/domain/states.ts`, every asset carries storage, quality, review, and publication as independent fields, plus `machineProduced`. `publicationBlockers` returns all blockers. `mayPublish()` is true only when that list is empty, so the answer and the explanation cannot drift apart.

The product lane has a different problem. Owners need to see work that is still blocked from publication. `src/quality/contracts/project-read-model.ts` calls this out directly: the newest produced asset can be visible to the owner while the hero claim remains reserved for cleared media. `publication_state: blocked` means unpublished. It has never meant hidden from the owner.

That distinction fixed a real product trap. If the dashboard hides every blocked output, the owner cannot review the work. If it treats every visible output as publishable, the public listing can outrun approval. The read model keeps those two surfaces separate.

## 4. Fanout is part of the product shape

The batch image processing API accepts one to twenty remote image URLs. It can request background removal, an enhancement level, responsive outputs, and watermarking. It can also associate the batch with a parent record identifier.

That upper bound controls more than cost. It keeps the work explainable. When a vehicle has a batch in flight, the status response can speak in terms of that vehicle, those image identifiers, and the job identifiers that matter. The video generation request has the same instinct in a stricter form: supplied image identifiers must be exactly three. Exact length makes downstream assembly simpler, and it rejects bad input before compute starts.

Priority needs restraint. The bulk endpoint accepts a priority hint. Elsewhere in the app, cache strategy uses levels where critical outranks high, then medium, then low. Priority helps when interactive work competes with background work. It should remain scheduling metadata. Once priority becomes hidden ownership, the product view and worker view start telling different stories.

The listing route avoids that split by keeping stage claims database-owned. Consumers claim through the remote procedure call or are invoked by cron. There is no pub/sub layer in that design. The capture flow polls.

Polling is a tradeoff, and I would not hide it. It burns time between readiness and the next drain. It also keeps claims, retries, and stale recovery in the same place as the media records. For this system, that was the cleaner failure mode.

## 5. Machines propose, humans accept

The listing generation stage has narrow authority. It takes the accepted hero set and writes proposed records for review. The design rule is blunt: machines propose, and only an actor shaped as `human:<uuid>` confirms. Condition findings follow the same line. Machines may propose findings. They do not confirm, reject, set severity, or drop them.

This matters because vision outputs look finished. A mask has pixels. A generated backdrop can look better than the source lot. A social frame can be well composed. The product still needs to separate produced from accepted.

The marketing generation stage starts from the accepted hero. It produces per-aspect re-renders for one-to-one, four-to-five, nine-to-sixteen, and sixteen-to-nine placements.

The output is letterboxed on dominant tone, so the model extends the scene while the car stays whole. It also stamps International Press Telecommunications Council (IPTC) DigitalSourceType and verifies the stamp on every produced frame. The implementation closed a measured gap where the core before-and-after publish path did not stamp.

That detail belongs inside generation. If a social frame is only an output URL, stamping becomes optional plumbing. Once publication depends on the asset carrying its receipts, verification has to happen where the frame is made.

The database gate is stricter. The project asset check makes published unrepresentable without digest verification, quality passed, and review approved or marked unnecessary. Takedown is never blocked.

I prefer a locked door to an apology in logs. A warning after an unreviewed image is public helps nobody.

## 6. Refusal has to be named

Vision workers fail in ways that ordinary endpoint errors flatten. They time out. They return partial output. A provider can accept a job and then vanish from the caller view. The operator still has to decide whether to retry, decline, or wait.

The framing drainer has one concrete answer. A slow matte becomes a per-item refusal with the note will retry next tick. It does not take the whole invocation down. That worker uses concurrency three and a per-item timeout of ninety thousand milliseconds. The comment explains the budget math: batches stay inside the route three hundred second ceiling at the maximum budget.

The same comment says the underlying fetch is not aborted. The worker may still finish. The next tick can ride a warm cache or record the result. The tradeoff is explicit: at most one orphan per slow item per tick, rather than eight.

That ugly detail belongs in the media custody model because timeout does not always mean failed image. Sometimes it means the drain cycle refused to spend all of its budget on one item. Sometimes the delayed result should be attached to the same attempt on the next pass.

The deletion worker follows the same rule from another direction. An API error or 404 alone does not prove a graphics processing unit (GPU) job stopped. Cleanup is allowed after a same-ID terminal response, a persisted terminal observation, or the provider maximum time-to-live after the fenced request. No endpoint-wide purge.

External systems can be vague. My database should name the uncertainty.

## 7. Derived media stays with its parent

The most dangerous derived asset is the one that looks better than its parent. A thumbnail can be crisp. A crop can be framed well. A studio render can look ready. None of that proves the source set passed review.

The listing work tightened this rule around parentage. Listing photos inherit their alpha at attempt one. Failed rerenders retry as their own family. A listing photo is never retried on its own. Orbit frames are reviewed as a set. A later change requires exact parent 360 approval before listing creation.

Those are product rules expressed as lineage. A child output does not declare independence because it has pixels. If the parent set lacks approval, children wait. If a rerender fails, it belongs to its attempt family. If a 360 has no ruling, the owner may ask for another render, while the system still refuses to treat absence as approval.

The 360 gate makes that concrete in `supabase/migrations/20261119000000_listing_360_approval_gate.sql`. The source-authority function returns a media delivery shape with schema version, delivery identifier, parent job, owner, actor, project, capture session, source set hash, generation, render attempt, source asset identifiers, listing batch, social selection, derived status, and failure code. The derived object carries enough parent identity to prevent a listing from being built from the wrong ring.

This is where the post differs from a pure provenance piece. Provenance tells you where an artifact came from. Custody tells the product what may happen next, who may decide, and which derived outputs must wait for their parent.

Any system that spends real money changing pixels needs ownership to move with the pixels, or the product will eventually believe the wrong thing.

---

🎧 **Listen to the audiobook** — [Spotify](https://open.spotify.com/show/4ABVd5yDVfbX9HlV5JjT7D) · [Google Play](https://play.google.com/store/audiobooks/details/How_to_Architect_an_Enterprise_AI_System_And_Why_t?id=AQAAAECafz8_tM&hl=en) · [All platforms](https://www.craftedbydaniel.com/audiobook)
🎬 [Watch the visual overviews on YouTube](https://youtube.com/playlist?list=PLRteDbGJPYDb9XNjecvHplGlgW7tIv_q6)
📖 [Read the full 13-part series](https://www.craftedbydaniel.com/blog/series/how-to-architect-an-enterprise-ai-system-and-why-the-engineer-still-matters)

I Had to Build What the Bumper Could See

Daniel Romitelli — Wed, 30 Sep 2026 07:12:07 +0000

The carrier’s chrome reflected my light card as a white rectangle. I moved the light to the wet road and rebuilt the shot around what the bumper saw.

1. The rectangle in the bumper

With the card gone, I traced the bumper’s reflection back to the apron.

A polished surface reflects along a direction set by its angle and the camera’s position. Roughness spreads that reflection. It cannot bring a lit patch of road into metal that faces somewhere else. I checked the reflected direction before touching the shader again.

The teaching plate is flat. The bumper is not. Each patch of chrome looks at a different piece of apron, so one lit mark on the road cannot serve the whole bar. I had to use the shot camera and light the ground that bar was actually returning.

The scripts below use generic geometry and no production assets. They run with Python 3.10 or newer and need no extra packages. In mirror_ray.py, a vertical plate reflects a flat floor at teaching coordinates:

"""Adapted educational example; no film assets, scene, or production settings."""
from math import sqrt


def unit(vector):
    length = sqrt(sum(v * v for v in vector))
    if length == 0:
        raise ValueError("A direction must have nonzero length")
    return tuple(v / length for v in vector)


def ground_hit(point, normal, camera):
    normal = unit(normal)
    view = unit(tuple(c - p for c, p in zip(camera, point)))
    dot = sum(n * v for n, v in zip(normal, view))
    reflected = tuple(2 * dot * n - v for n, v in zip(normal, view))
    if reflected[2] >= 0:
        return None
    distance = -point[2] / reflected[2]
    return tuple(p + distance * r for p, r in zip(point, reflected))


if __name__ == "__main__":
    point, normal = (0, 0, 0.6), (0, -1, 0)
    for height in (1.4, 2.2):
        hit = ground_hit(point, normal, (0, -4, height))
        assert hit is not None and abs(hit[2]) < 1e-9
        print(f"camera_z={height:.1f} -> ground_hit={tuple(round(v, 3) for v in hit)}")
    assert ground_hit(point, normal, (0, -4, 0.6)) is None

Run python3 mirror_ray.py. The tested output is:

camera_z=1.4 -> ground_hit=(0.0, -3.0, 0.0)
camera_z=2.2 -> ground_hit=(0.0, -1.5, 0.0)

The plate stayed still. Raising the camera by 0.8 meters moved the reflected floor point 1.5 meters, from three meters ahead of the plate to 1.5 meters ahead. In the carrier scene, I traced the bumper’s reflected directions to the actual ground and put fill there. In the later frame, the chrome held dark ground structure and the card was gone.

2. Put the road where the reflection lands

Two wet paths lead toward the service bay. Snow is visible near the camera and around the door.

I built the winter lot around the service building. Its lit door gave the camera a destination; parked rows kept the route narrow. The road starts broad in the foreground and runs straight to that door. Near the lens, snow crosses in long streaks. At the far wall it becomes flecks against the light. I judged the snowfall by where I could see it.

A flat dark stripe gave the lamp little to describe. I cut shallow depressions into the terrain and shaped uneven slush at their edges. Their shoulders caught light. The wet middle sent a different reflection toward the camera. A worn path has a cross-section. Snow can remain beside it.

Cover: snow on the upper-deck vehicles, amber markers along the carrier, dark ground in the chrome.

I bought a hauler model, then rebuilt it into Online Auto Connection truck #171 with permission from the owner, Mark Subjeck. The other vehicles also began as licensed models that I modified for the film.

The loaded carrier beside the parked rows.

I placed the overhead camera to check where the deck sat among those rows. At road level, the rows compress toward the service building. I returned to the low camera to judge the paths and the snow.

3. Build a smaller scene to test the reflection

Three renders, same plate: road light off, road light on, then a second camera with the light left alone. Look at the floor first. If the bright patch is not the point the plate is reflecting, more power only lights the wrong ground. Then look at the metal.

A short road, a box on wheels, and a polished front plate are enough. The adapted rut_strip.py writes a 10 m road strip with two shallow troughs. Run python3 rut_strip.py demo-ruts.obj with a new output filename. It reports 3,321 vertices and a lowest height of 0.007 m. The full generator is under Scripts.

Import the OBJ into Blender at one unit per meter and check that its upper face normals point up. The surface starts at 0.025 meters. Each track center drops by up to 0.018 meters. Change the track width, then inspect the shoulder from the camera.

Use a beveled cube for the body and four cylinders for wheels. Check their contact with the road. Put a thin plate at the front, facing back toward the first camera.

For the plate’s Principled shader, start with metallic 1 and roughness around 0.06. Start the exposed track around 0.18 roughness and the pale snow around 0.65, with fine Noise Texture feeding a small Bump node. These are starting values for this test.

To keep snow beside the tracks, take the X coordinate from Geometry Position through Separate XYZ. Measure the distance to track centers at -0.7 and 0.7 meters and keep the smaller distance. That drives a Mix between wet material and snow, with a starting blend from 0.14 to 0.33 meters from either center. If the tracks read as ink marks, inspect the height profile, transition width, and light direction one at a time.

Put the first camera low and aim it at the plate. Use a broad area light to illuminate the road ahead of the box. Place the second camera about 0.8 meters higher, still aimed at the plate. Compare the third render with the second before changing roughness.

For a moving test, key a straight camera push and inspect its first, middle, and last frames. Check wheel contact and whether flakes remain visible near the lens and farther into the scene. Place a recorded mechanical contact at a visible action, then listen on speakers and headphones.

4. Know which frame you judged

At thumbnail size the truck reads and the chrome does not. At 1280 × 720 I could see dark ground in the bumper, snow on the upper deck, and whether a wheel sat on the lot. That is the file I kept a receipt for.

Run python3 frame_receipt.py your-render.png to print image dimensions, byte count, and SHA-256. For the cover it returned 3,219,108 bytes and a hash matching the handoff receipt. The full script is under Scripts. Its pixel_review field reads NOT_ESTABLISHED_BY_THIS_SCRIPT. The hash is not the review.

5. Four questions from the directors

Adam Greenberg described wetting the streets for Terminator 2 to darken gray pavement, and separately described moving reflections in car windows and hoods. Where did my wet road enter the bumper’s reflection?

Which visible fixture revealed the snow? Dan Laustsen’s backlit rain in John Wick: Chapter 3 sent me to the foreground, then the service door. The overhead posed a different question after Chad Stahelski’s discussion of readable wide action: could I place the carrier among the parked rows before cutting closer?

Christopher Nolan’s account of building sound effects and music together on Dunkirk left one question: which mechanical action deserved the sound cut?

6. The cold open

The 12-second cold open below is a 390 × 220 phone-size preview with stereo sound. It starts in darkness, then reveals exhaust, falling snow, a headlamp, and red bodywork. Play it with sound on.

Your browser does not support embedded video.

Open the 12-second preview with sound.

The clip is from One Lap, my commercial for AutoLensAI.

Glass, wet asphalt, and chrome show pieces of a set that sit outside the frame.

Scripts

rut_strip.py — road mesh generator

"""Adapted educational OBJ generator; a new toy surface, not film geometry."""
import argparse
import json
from math import exp
from pathlib import Path


def surface_height(x):
    distance = min(abs(x - 0.7), abs(x + 0.7))
    return 0.025 - 0.018 * exp(-((distance / 0.19) ** 2))


def write_obj(path):
    nx, ny = 80, 40
    vertices = [(-2 + 4 * i / nx, -1 + 10 * j / ny,
                 surface_height(-2 + 4 * i / nx))
                for j in range(ny + 1) for i in range(nx + 1)]
    with path.open("x", encoding="utf-8") as stream:
        stream.write("# Adapted educational rut strip, units are meters\n")
        for x, y, z in vertices:
            stream.write(f"v {x:.6f} {y:.6f} {z:.6f}\n")
        for j in range(ny):
            for i in range(nx):
                a = j * (nx + 1) + i + 1
                stream.write(f"f {a} {a+1} {a+nx+2} {a+nx+1}\n")
    assert len(vertices) == 3321
    assert abs(min(v[2] for v in vertices) - 0.007) &lt; 1e-9
    return {"vertices": len(vertices), "quads": nx * ny,
            "minimum_z_m": round(min(v[2] for v in vertices), 6),
            "geometry_checked": True, "rendered": False}


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("output", type=Path)
    args = parser.parse_args()
    print(json.dumps(write_obj(args.output), sort_keys=True))

frame_receipt.py — PNG file receipt

"""Adapted read-only PNG receipt; integrity is not visual approval."""
import argparse
import hashlib
import json
import struct
from pathlib import Path


def inspect_png(path):
    digest = hashlib.sha256()
    with path.open("rb") as stream:
        header = stream.read(24)
        if header[:8] != b"\x89PNG\r\n\x1a\n" or header[12:16] != b"IHDR":
            raise ValueError("Expected a PNG with an IHDR header")
        digest.update(header)
        for block in iter(lambda: stream.read(1024 * 1024), b""):
            digest.update(block)
    width, height = struct.unpack("&gt;II", header[16:24])
    return {"file": path.name, "width": width, "height": height,
            "bytes": path.stat().st_size, "sha256": digest.hexdigest(),
            "pixel_review": "NOT_ESTABLISHED_BY_THIS_SCRIPT"}


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("image", type=Path)
    args = parser.parse_args()
    print(json.dumps(inspect_png(args.image), indent=2))

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

I Had to Build What the Bumper Could See

Daniel Romitelli — Wed, 30 Sep 2026 05:39:50 +0000

After I removed the light card, I checked the bumper from the camera view and worked backward to the part of the apron it reflected. The road had to carry the light now. Its surface and the camera angle had to agree.

That changed how I worked on the winter lot in One Lap. The bumper needed ground to reflect. The road needed worn paths with actual shape. Snow had to remain visible between the camera, the parked cars, and the lit service building. Each part affected what the next camera view could show.

1. The rectangle in the bumper

A polished surface responds to the direction it faces and the position of the camera. Roughness spreads a reflection; it cannot bring an unlit patch of ground into a bumper that is looking somewhere else. I had to inspect the truck from the camera position, then work on the part of the lot that appeared in its metal.

A small numerical exercise makes the camera change plain. It uses a vertical reflective plate and a flat floor. Raise its camera by 0.8 meters and the floor point appearing in the plate moves 1.5 meters. These are adapted teaching coordinates, separate from my film scene. The scripts and Blender exercise below use generic geometry and no production assets or settings.

Save this as mirror_ray.py. It runs with Python 3.10 or newer and needs no packages:

"""Adapted educational example; no film assets, scene, or production settings."""
from math import sqrt


def unit(vector):
    length = sqrt(sum(v * v for v in vector))
    if length == 0:
        raise ValueError("A direction must have nonzero length")
    return tuple(v / length for v in vector)


def ground_hit(point, normal, camera):
    normal = unit(normal)
    view = unit(tuple(c - p for c, p in zip(camera, point)))
    dot = sum(n * v for n, v in zip(normal, view))
    reflected = tuple(2 * dot * n - v for n, v in zip(normal, view))
    if reflected[2] >= 0:
        return None
    distance = -point[2] / reflected[2]
    return tuple(p + distance * r for p, r in zip(point, reflected))


if __name__ == "__main__":
    point, normal = (0, 0, 0.6), (0, -1, 0)
    for height in (1.4, 2.2):
        hit = ground_hit(point, normal, (0, -4, height))
        assert hit is not None and abs(hit[2]) < 1e-9
        print(f"camera_z={height:.1f} -> ground_hit={tuple(round(v, 3) for v in hit)}")
    assert ground_hit(point, normal, (0, -4, 0.6)) is None

Run python3 mirror_ray.py. The tested output is:

camera_z=1.4 -> ground_hit=(0.0, -3.0, 0.0)
camera_z=2.2 -> ground_hit=(0.0, -1.5, 0.0)

The plate did not move. At the lower camera height, it showed ground three meters from the plate. At the higher position, it showed ground only 1.5 meters away. If I light the first patch and then raise the camera, the highlight can vanish from the metal. The script computes an intersection; it does not render or judge an image. I still inspect the actual frame at native size.

For the carrier section, I checked reflected directions against the actual ground surfaces in the scene and put fill on the ground the camera could see through the bumper. In the later carrier view, the chrome carries a broad silver response with dark ground structure through it. I can no longer pick out the card as a rectangle.

2. Put the road where the reflection lands

The road-level approach. Two wet paths lead toward the service bay; its lamps catch snow at several distances.

The building gives this frame a destination. Parked vehicles close in from both sides. Snow near the camera crosses in long streaks; smaller flecks remain around the lit door. The exposed road pulls the eye through the middle.

I judged the snowfall by where it could be seen. The nearest flakes cut across the foreground. Around the building they become small marks against the door and work lights. That distance change matters when the storm and the road have to occupy the same space; an even screen of white would cover the scene without telling me how deep the lot is.

A dark stripe could have occupied those pixels, but a flat stripe gave the lamp little to describe. I shaped the exposed paths as shallow depressions with uneven slush at their edges. Their shoulders catch light. The wet middle sends a different reflection toward the camera. A worn path has a cross-section. Snow can remain beside it.

The carrier needed the same weather as the lot around it. The overhead view helped me check the arrangement before returning to the low camera:

Snow stays on the upper-deck roofs while amber markers trace the carrier’s length.

From above, I could see where the loaded deck sat among the parked rows. At road level, those same rows compress toward the service building. I used the two views for different checks: one for the lot’s geography, one for the path and weather that a viewer would feel while approaching it.

The selected cover let me inspect a third relationship at native size. Snow covered the hoods, roofs, and windshields of vehicles on the upper deck, while the carrier’s small markers still described its length. The weather was present on the payload as well as in the air and on the road. That gave the truck a place in the winter lot instead of treating it as a clean object dropped in front of one.

3. Build a smaller scene to test the reflection

Make three renders: the first camera with the road light off, the same camera with it on, then a second camera with the light unchanged. Keep the polished front plate material fixed. That sequence lets you see whether the road enters the reflection and what the camera changes.

The film scene used licensed vehicle models. For this exercise, make a short road and an unbranded vehicle from simple shapes. The plate at its front is the surface to watch.

Shape the road

Save this second adapted script as rut_strip.py. It writes a four-meter-wide, ten-meter-long Wavefront Object (OBJ) mesh with two shallow troughs:

"""Adapted educational OBJ generator; a new toy surface, not film geometry."""
import argparse
import json
from math import exp
from pathlib import Path


def surface_height(x):
    distance = min(abs(x - 0.7), abs(x + 0.7))
    return 0.025 - 0.018 * exp(-((distance / 0.19) ** 2))


def write_obj(path):
    nx, ny = 80, 40
    vertices = [(-2 + 4 * i / nx, -1 + 10 * j / ny,
                 surface_height(-2 + 4 * i / nx))
                for j in range(ny + 1) for i in range(nx + 1)]
    with path.open("x", encoding="utf-8") as stream:
        stream.write("# Adapted educational rut strip, units are meters\n")
        for x, y, z in vertices:
            stream.write(f"v {x:.6f} {y:.6f} {z:.6f}\n")
        for j in range(ny):
            for i in range(nx):
                a = j * (nx + 1) + i + 1
                stream.write(f"f {a} {a+1} {a+nx+2} {a+nx+1}\n")
    assert len(vertices) == 3321
    assert abs(min(v[2] for v in vertices) - 0.007) < 1e-9
    return {"vertices": len(vertices), "quads": nx * ny,
            "minimum_z_m": round(min(v[2] for v in vertices), 6),
            "geometry_checked": True, "rendered": False}


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("output", type=Path)
    args = parser.parse_args()
    print(json.dumps(write_obj(args.output), sort_keys=True))

Run it with a new filename:

python3 rut_strip.py demo-ruts.obj

The tested run produced 3,321 vertices and 3,200 faces, with a lowest height of 0.007 meters. The generator refuses to overwrite an existing file. I checked its geometry and counts.

Import the OBJ into Blender with one unit per meter. Check that the upper face normals point up. The height function starts at 0.025 meters and lowers each track center by up to 0.018 meters. Change the width value, then look from the camera again. You are changing the shoulder that catches light, not merely painting a darker line.

Give the metal a place to look

A beveled cube can stand in for a vehicle body. Put four cylinder wheels under it and check that they meet the road. Add a thin plate at the front, facing back toward the first camera. No badge, grille, or purchased asset is needed.

For the material test, use a Principled shader on each surface:

Start the front plate at metallic 1 and roughness around 0.06. Its reflection should reveal the set around it.
Make the exposed track darker and smoother, with roughness around 0.18. Give the snow a pale material at roughly 0.65 roughness, with fine Noise Texture feeding a small Bump node. Keep the wet road smoother than the snow.
To place snow beside the tracks, take the X coordinate from Geometry Position and Separate XYZ. Measure its distance to track centers at -0.7 and 0.7 meters, keep the smaller distance, and map that value into a Mix between wet material and snow. Start the blend around 0.14 to 0.33 meters from a center. The mesh supplies the dip; the shader supplies the surface response.

Those roughness and blend values are starting controls, not measured settings from One Lap. If the tracks read as two ink marks, inspect their height profile, transition width, and light direction one at a time.

Move the camera, keep the light

Put one camera low and aimed at the front plate. Place a broad area light so it illuminates the road ahead of the vehicle, rather than presenting a bright card directly to the plate. Make a second camera about 0.8 meters higher or to the side. Keep both cameras aimed at the plate.

Compare the third render with the second. Its reflection may change even though the light stayed put. Return to the ray script before adjusting roughness.

For a short moving study, key a straight camera push that clears the wheels and ground. Inspect its first, middle, and last frames before rendering the full range. A sparse Geometry Nodes field of instanced flakes can test whether snow remains legible near the lens and at the building. Record your own wind and one mechanical contact for sound. Place that contact at a visible action, then listen once on speakers and once on headphones.

4. Know which frame you judged

I inspected the selected carrier render at its native 1280 × 720 size. That let me see the upper-deck snow, marker lamps, bumper response, and dark parts of the lot. I also needed to know that I was opening the selected file rather than an earlier lighting pass or a smaller preview.

This third adapted script, frame_receipt.py, records a Portable Network Graphics (PNG) file’s dimensions, byte count, and SHA-256 digest. It reads the file without changing it:

"""Adapted read-only PNG receipt; integrity is not visual approval."""
import argparse
import hashlib
import json
import struct
from pathlib import Path


def inspect_png(path):
    digest = hashlib.sha256()
    with path.open("rb") as stream:
        header = stream.read(24)
        if header[:8] != b"\x89PNG\r\n\x1a\n" or header[12:16] != b"IHDR":
            raise ValueError("Expected a PNG with an IHDR header")
        digest.update(header)
        for block in iter(lambda: stream.read(1024 * 1024), b""):
            digest.update(block)
    width, height = struct.unpack(">II", header[16:24])
    return {"file": path.name, "width": width, "height": height,
            "bytes": path.stat().st_size, "sha256": digest.hexdigest(),
            "pixel_review": "NOT_ESTABLISHED_BY_THIS_SCRIPT"}


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("image", type=Path)
    args = parser.parse_args()
    print(json.dumps(inspect_png(args.image), indent=2))

Run python3 frame_receipt.py your-render.png. Under Python 3.10.12, the script was run against the selected cover and another reviewed image. The cover’s dimensions, byte count, and digest matched its handoff record.

The last field has a narrow meaning. Matching bytes establish which file I inspected. They say nothing about whether the snow has depth or a wheel touches the ground. Open the frame at its actual size for those decisions. For the exercise, save the three comparison renders with their camera and light choices, then make a receipt for each.

5. Four questions I brought back from the directors

I used interviews as checks on my own shots. Each source gave me a question I could ask of the lot.

Adam Greenberg: When he described wetting streets for Terminator 2, he also discussed moving lights appearing in a car’s windows and hood. Where should the road carry light, and what does the carrier reflect? Greenberg in American Cinematographer.
Dan Laustsen: His account of backlit rain in John Wick: Chapter 3 pushed me to check snow at more than one distance. Which visible fixture reveals it? Laustsen in American Cinematographer.
Chad Stahelski: His discussion of readable wide action sent me back to the overhead image. Can a viewer place the carrier among the parked rows before the cut moves closer? This is how I used the principle, not a copied John Wick setup. Stahelski in Filmmaker.
Christopher Nolan: His account of building sound effects and music together during Dunkirk raised an editing question. Which mechanical action deserves the sound cut? Nolan at Cannes.

6. Coming soon: the AutoLensAI commercial

The 12-second clip below is a phone-size preview from One Lap, the upcoming commercial for my AutoLensAI application. It has a stereo audio track. The picture starts in darkness, then reveals exhaust, falling snow, a headlamp, and red bodywork. Play it with sound on.

Your browser does not support embedded video.

Watch the 12-second preview of the One Lap commercial with sound.

The AutoLensAI commercial is coming soon. Glass, wet asphalt, and chrome can all show pieces of a set that sit outside the frame.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

You Want Engineers Using AI on the Job. Why Turn It Off in the Interview?

Daniel Romitelli — Sun, 20 Sep 2026 08:58:52 +0000

Forty-five minutes in, I'd walked through systems I built. The architecture, the integrations, the auth decisions, what broke in production and what I did about it. There are working products online with demos anyone can open. Nobody opened them.

Then came the coding exercise. Share your screen. Open the editor. Turn off AI, disable autocomplete. Write a program that calls an API and returns a picture of a bird and a fact about it.

The endpoint was supplied. So were the library, the response shape, and the expected output. The product and architecture decisions were already out of the picture. What remained was implementing a tiny, prescribed task without the tools I'd normally use.

I've shipped a lot of things. I have never once shipped a bird.

When I talk to someone who knows a field I know, I can usually hear it inside five minutes. They ask about a condition I haven't mentioned. They tell me why an approach works here and turns into a problem two systems over. Change an assumption on them and they follow it downstream without stopping to reload.

I've found that in software and I found it in a woodworking shop. People who have actually done the work hand you details you can push on. The part that took three times longer than the estimate. The shortcut they took once and will never take again. You always have somewhere left to go.

A first impression still needs checking. Keep going with the follow-ups, because somebody can stumble over a sentence and still understand the problem cold, and somebody else can be very smooth and very empty. But by the end of a real technical conversation, an interviewer should be holding evidence.

So why spend forty-five minutes asking about my experience, decline to look at the results, and then let a small unaided API call cast the deciding vote?

1. The woodworking shop

I owned a woodworking shop with expensive equipment in it. A jointer, a planer, a table saw with a fence I could trust. That equipment took rough lumber and turned it into square, four-sided, dimensioned stock faster than I could ever do it by hand. That was the entire point of buying it.

A stack of perfectly milled boards still wasn't furniture. I had to know what I was building, how the joints would carry load, where the material would move across a winter, and whether the finished piece was any good. If I couldn't put it together, the machinery didn't excuse that. The gap was mine.

Some excellent woodworkers can walk onto a site with basic tools and build something beautiful. I respect that ability. It doesn't follow that using a planer proves somebody lacks it, and making a person repeat every prep operation by hand will not tell you whether they can deliver the finished piece. It'll tell you they can push a hand plane.

I bought equipment to take back time that didn't need to be spent, and to get consistency I couldn't hold by hand at volume. What I got in exchange was attention I could spend on the design and on the parts of the job the machines hadn't solved.

AI isn't a planer. It can hand you something confidently wrong, which makes verification part of the engineering problem instead of a formality. But verification is a thing you can design, automate, and hold to a standard.

The objective is a well-built result. I don't hand out extra credit for preserving unnecessary labor on the way there.

2. Nobody panics about the teleprompter

Here's a tool that's been around since the 1950s.

An executive can deliver a prepared speech from a piece of glass. A news anchor can read a newsroom's script. We understand that the person on camera may be working from words prepared with other people. Imagine interrupting a CEO halfway through a town hall: hang on, kill the prompter, let's find out if he actually believes any of this.

I'd have questions about what he said. Whether he'd memorized it wouldn't settle them.

Apply the same test to a quantitative analyst: take away the computer and ask for long division on a legal pad. You could learn something about their arithmetic. You'd still have questions about whether they understand the risk they're being asked to model.

Spellcheck. Route planning. CAD. Compilers, for that matter. We could spend the afternoon removing assistance and congratulate ourselves on how much harder we'd made the job.

The honest objection is that these tools aren't equivalent. A teleprompter displays a prepared script; a generative model can supply reasoning the person never had. Fine. That difference is worth examining. Tell me which ability this role requires unaided, why this exercise measures it, and I'll take that seriously.

What I won't accept is the reflex. Finding a tool in someone's hand is the beginning of a question, not the answer to it.

3. Walowitz is mine

I've spent the better part of a decade building my own assistant. Walowitz is part of it, in my editor and across the other places I use it. I haven't released it publicly. I built it for myself.

Cognitive relay is one reason I built it. I can read a raw log, see the relationship in it fast, and still lose the explanation somewhere between knowing the answer and getting it out of my mouth on demand. Anxiety makes that worse. I have understood a problem and blown the explanation in front of people. It doesn't feel like a communication problem. It feels like being found out.

For recall, the material I want back is my own writing: notes, comments, ledgers, and decisions I recorded while doing the work. Retrieved passages carry references back to their source. That lets me return to the record rather than pretend a new model-generated explanation is something I wrote years ago.

Exact identifiers and semantic search do different jobs, so I run both and put exact matches first when they're available. Asking for the function where I handled a specific case isn't the same request as asking for everything related to an idea. The combined results are deduplicated before entering the context.

Walowitz cancels retrieval when a newer request overtakes it and expires temporary context on purpose. Material from the previous subject can be completely accurate and completely wrong for the conversation happening now. Skip that and you've built a very sophisticated way to remember the wrong thing at exactly the right moment.

Recall isn't permission to act. As I extend the assistant into screens for my kids, a request to trace the letter B shouldn't inherit authority to touch the router. If I ever confuse those scopes, I have considerably larger problems than a coding interview.

The practice ledger has separate states for aided, unaided, and unrecorded turns. Zero means nobody measured this. It doesn't mean unaided. Within a tracked turn, an aided label doesn't revert merely because the display later clears. The ledger still has paths that don't record aid. I won't pretend it is finished.

I chose those states because a ledger based only on the session's opening mode would record a preference and call it behavior. A flattering default becomes indistinguishable from a real measurement once somebody writes it down.

Pick one of those decisions. Keep asking. Change a condition and see whether I can follow the consequences. We can discuss those choices without distributing private source or a former employer's work.

But first, apparently, we need to establish whether I can fetch the bird unassisted.

4. Knowing what not to build

The first decision on anything is whether it should exist. Who needs it, what are they trying to do, what would make it worth using.

While one team builds a capability from scratch, somebody else may find a maintained service or an open model that already does it and spend those weeks shipping the part nobody has solved. Checking that is part of the job. License, operating cost, what you'd depend on, what happens when it changes under you.

There are real reasons to build your own. The available option might require sending customer records somewhere they're not allowed to go, or become unaffordable at the volume you're planning for. I want those limits investigated before anybody commits the team to months of work, not discovered afterward.

That judgment is invisible in a test where the build-versus-buy decision was made for me before I opened the editor.

5. Companies already changed the test

Karat surveyed 400 engineering leaders across the United States, India and China. Seventy-one percent said AI was making technical skills harder to assess, while 62% of organizations still prohibited it in technical interviews.

Karat co-founder Mo Bhende put it to IEEE-USA plainly: "the fundamental job has changed, but technical interviews, as we know them, have not."

Canva announced in June 2025 that backend, frontend and machine-learning candidates were expected to use AI tools. Its pilot found that some candidates who could code struggled anyway, because they couldn't guide the tools or recognize a bad suggestion. Canva kept code fluency and technical depth as requirements. Allowing assistance exposed a weakness the old exercise hadn't been designed to assess.

Meta's July 2025 internal memo gave two reasons for developing AI-enabled interviews: they better represent the working environment, and they make AI-based cheating less effective. 404 Media reported the memo with WIRED. Read that second one twice. Allowing visible assistance was part of the stated rationale for protecting the assessment.

CoderPad's 2026 survey found its hiring respondents split 34% prohibiting, 46% allowing broadly or with constraints, and 20% deciding case by case. Asked what shows skill when AI is allowed, 66% picked catching and fixing its mistakes and 56% picked explaining trade-offs and correctness.

Excluding AI is a choice. It isn't a settled professional requirement, and it shouldn't be presented to a candidate as one.

Anthropic's candidate guidance, updated July 2025, prohibits AI in live interviews and take-homes unless permission is given. The same page describes using Claude to develop questions, draft communications, analyze hiring metrics and source candidates. Claude doesn't make the hiring decisions.

Preparing a question and answering one are different activities, so different rules on the two sides aren't automatically hypocrisy. But the restriction still owes the applicant an explanation. Which ability does this job require unaided, and why does this exercise deserve the weight you're giving it?

6. What about cheating

The concern is legitimate and I'm not going to wave at it.

Fabric's January 2026 report analyzed 19,368 interviews on its own AI interview platform. It flagged 38.5% of candidates for cheating behavior, and roughly 61% of those flagged scored above its passing threshold. That's one vendor's detection and scoring, not an independently established industry rate, and it doesn't establish that every flagged person cheated.

If the people you believe haven't demonstrated competence are clearing your threshold at that rate, the scoring deserves investigation alongside their behavior. Adding a ban doesn't establish that the score measures the ability you're hiring for.

There's risk pointed the other way too. A 2020 NC State and Microsoft study put 48 computer science students through a whiteboard problem. The ones solving it while watched and narrating performed about half as well as the ones working privately. Observation and narration both changed at once, and they were students, not principal engineers. The size of that gap is still worth noticing before anyone insists the stressful version is the honest one.

AI can also undermine learning, and I'm not leaving that out because it complicates my case. In Anthropic's January 2026 randomized study, 52 mostly junior engineers learned an unfamiliar Python library. The AI group averaged 50% on a quiz covering concepts they'd used minutes earlier, against 67% unaided. The speed gain wasn't statistically significant.

Inside the AI group, people who asked conceptual questions or followed generated code with questions to understand it scored better than people who mostly delegated. That comparison was exploratory and can't establish cause. It still gives an interviewer a behavior worth investigating rather than treating every use of assistance as equivalent.

I'm not claiming AI always makes experienced engineers faster, either. METR's February 2026 update revisited its earlier 19% slowdown finding and reported that the follow-up couldn't reliably estimate the current effect, largely because developers declined to participate or withheld tasks they didn't want to attempt without AI. The experiment was missing people and work that could have changed the answer.

A fast, plausible mistake is still a mistake.

7. Keep the birds, then change the response

I'm not asking anybody to stop testing me. I spent part of my twenties in military aviation, where competence was checked by people qualified to check it. I have no objection to proving I can do the work.

A failure scenario can justify taking a capability away. Then the question is how the person handles that failure in the job they're trained to do. That's an explanation I can understand. "We always turn the tools off" isn't one.

So keep the bird exercise. Then make it worth the hour.

Change the response halfway through. Introduce a timeout. Hand me an implementation that looks reasonable and has a defect sitting in it, and ask what needs testing before anyone should trust it. Let me use my tools, including automated review, and watch whether my checks find the problem and what I do once they have. Make me explain why the evidence I produced addresses the changed requirement.

Then take a problem with real decisions in it. Talk about what the user needs and let me investigate the options. Ask why I'd use an existing service instead of building one. Change a constraint and follow it with me. I can use research or a model to explore it, and you can ask what made me accept one approach and abandon another.

Work I can legitimately share gives you another place to start. A demo leads into the decisions behind it without exposing a former employer's code.

Nobody needs to complete an unpaid production project for this. A bounded exercise plus a serious technical conversation gives you several kinds of evidence instead of one convenient proxy. Give candidates comparable access and consistent criteria. That addresses an important part of fairness without pretending the working environment doesn't exist.

If your process can make time to watch my screen share but not to open what I built, the priorities are backward.

And if you're using a tool to assess me, own how you interpret its output. That standard runs both directions or it isn't a standard.

8. What I'd want to know before I said yes

I wouldn't want to join a company that expects end-to-end AI engineering and treats an unaided API call as the deciding test, particularly if the people running it can't connect it to the role. I'd question whether their understanding of the job had caught up to what they're asking for. I don't need to prove their whole stack is obsolete before deciding to pass.

There's a question worth asking out loud in either direction. Am I being hired to work inside an established engineering practice, or to modernize the software and change how it gets built? The second one comes with budget decisions, approval processes, and people outside the development team who have to agree. That's a different job, and whoever takes it needs the authority and the resources to do it.

A paycheck doesn't resolve a mandate nobody defined. I'm not interested in spending my first six months arguing for the methods I was supposedly hired to bring.

I'm asking you to put the bar where the work is.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

CommandCanvas: Building Around Shared Objects

Daniel Romitelli — Sat, 05 Sep 2026 05:27:41 +0000

In CommandCanvas, a sketch can become a structured diagram beside its source. A person can rearrange the result, a collaborator can edit it, and a supported agent host can address it through WebMCP. Each operation refers to objects in the same room, with identities and versions that survive changes in how people interact with them.

I built the shared canvas mutations around one commit path. Pointer input, voice, and tools can propose changes, but the server must establish which member authorized a change and whether it still applies to the current room state. Adding another input should not introduce another definition of what counts as saved.

Objects that people and tools can both address

A note in CommandCanvas has an identity, a title, a position, dimensions, and a version. Its content follows a note schema. A task board has columns and tasks. A sketch has strokes and points. A structured diagram has its own payload and can retain a reference to the sketch it came from.

These objects share spatial behavior while keeping their different kinds of content. Moving a board should preserve its tasks. Resizing a sketch should preserve the drawing. An operation can address a particular object and check its version before changing it. The object model makes those distinctions explicit.

That gives a tool request something concrete to address: the selected sketch, a populated board, or an existing object to transform. It also constrains the result. A model response has to fit a supported object schema before the application can submit it as a canvas change.

The distinction is useful when people and an agent take turns. They can refer to the same object without reconstructing its content from a screenshot or carrying a separate copy into another conversation.

Where a canvas change commits

The browser holds the visible objects, current selection, and room state. Direct input produces canvas commands. WebMCP and optional embedded voice pass through a capability runtime that validates their arguments and current context before invoking the command adapters.

WebMCP exposes capabilities to a supported agent host. Embedded voice uses a separate OpenAI Realtime connection and a narrower set of capabilities. Both reach the application's command adapters; neither gets its own route around room membership or revision checks.

The durable room path looks like this:

flowchart TD
  subgraph browser[Browser workspace]
    direct[Pointer, touch and hand intents] --&gt; command[Canonical canvas command]
    tools[WebMCP and optional voice] --&gt; capability[Capability validation]
    capability --&gt; command
  end
  subgraph server[Server and database]
    api[Authenticated room API] --&gt; guards[Membership and revision checks]
    guards --&gt; commit[Postgres mutation transaction]
    commit --&gt; records[Objects, room revision and receipt]
  end
  command --&gt;|Command, source and base revision| api
  records --&gt;|Authoritative readback| origin[Originating canvas]
  records --&gt;|Revision notification| reload[Other clients reload room state]
  reload --&gt; peers[Collaborator canvases]
  presence[Presence and cursor Broadcast] -.-&gt; origin
  presence -.-&gt; peers

The browser proposes a command; the server establishes who may commit it. Solid arrows trace the durable command and synchronization path. Dotted arrows show participant presence and cursor traffic, which do not create mutation receipts. On narrow screens, scroll the diagram horizontally.

The request carries a command ID, its reported input source, and the room revision the client worked against. The HTTP route verifies the bearer token. The room service loads membership, derives the actor, reads the current canvas, and builds a mutation plan. The database operation checks the expected revision again when committing the change.

That second check matters because the room can change between the server reading it and attempting the write. A command prepared against revision 12 cannot commit against revision 13 by assuming its original view still applies. The stale request is rejected.

The shared revision makes the order of committed changes explicit. It also makes conflicts coarser: two people working on different objects can still contend for the same room revision. The application does not automatically merge those edits. That is the cost of using a single room revision as a commit precondition.

The transaction persists the object changes, advances the room revision, and writes the receipt. The server reloads the authoritative state and checks for the expected receipt before returning success. Other clients receive a compact revision notification and reload the room. The room service contains that commit and readback path.

Why drag previews stay out of history

Applying that commit path to every cursor sample would make ordinary movement compete with object edits for room revisions. It would also fill history with intermediate positions that are of little use when someone wants to undo a completed action.

Cursors and movement previews therefore remain ephemeral. Supabase Presence describes connected participants, and Broadcast carries the frequent updates. Stable object changes go through the mutation path and produce the receipt that other clients can verify.

The room can show motion before it has a new committed state. That distinction is necessary for responsive interaction, and it has to survive into the UI: seeing another person's cursor move is not evidence that their edit was saved.

Turning a sketch into another object

Sketch interpretation introduces a longer gap between preparing a change and trying to commit it. A provider can finish its work after the drawing it received has changed.

The transformation captures the selected sketch's ID and version, then rasterizes it to a PNG in the browser. A provider interprets that image and returns structured output. The application validates the payload, checks the source reference, and checks that the requested output kind agrees with the result.

The person can keep drawing while the provider works. Before submitting the new object, the transformation checks the source version again. If the sketch changed, interpretation may have succeeded, but the application refuses to create the diagram from that result. The provider time has already been spent; accepting its answer anyway would attach an outdated interpretation to the current work.

sequenceDiagram
  participant C as Canvas
  participant P as Interpretation service
  participant R as Room API
  C-&gt;&gt;C: Capture sketch ID and version
  C-&gt;&gt;P: Rasterized sketch and instructions
  P--&gt;&gt;C: Structured result
  C-&gt;&gt;C: Validate payload and recheck source
  alt Source unchanged
    C-&gt;&gt;R: Create linked diagram at current room revision
    R--&gt;&gt;C: Committed object and receipt
  else Source changed
    C-&gt;&gt;C: Refuse the stale transformation
  end

The generated diagram becomes a separate canvas object. The original sketch remains available, including when interpretation fails or its source has changed. On narrow screens, scroll the diagram horizontally.

Creating the result still uses the ordinary command path. After the save, the transformation looks for the new diagram and its receipt in the authoritative returned state. A provider response alone is not enough to report that the room contains the diagram.

The source-version check and the room-revision check address different changes. One checks the drawing used for interpretation; the other protects the shared commit. Neither establishes that the model understood the drawing correctly.

That is why the result appears beside the preserved source. Someone can compare them, point out a missing relationship, or try another interpretation without recovering an overwritten sketch. The output remains a structured object that later commands can address. These checks and the separate-object creation are implemented in the sketch transformation orchestrator.

Deciding when a hand is drawing

The commit checks only help after an input has produced the right command. Hand tracking has an earlier decision to make: whether a moving fingertip should create ink at all. A valid room membership and a current revision cannot answer that.

On the default local path, an opt-in camera feeds MediaPipe Hand Landmarker in a browser worker. The interaction layer receives landmarks and decides what action they may produce. The index fingertip supplies the pen position. A separate thumb-to-middle-finger clutch supplies pen-down intent.

That separation lets someone move their hand to the start of another stroke without drawing a line across the intervening space. Closing the clutch engages the pen; opening it lifts the pen. A later engagement begins another stroke with its own identity.

The clutch uses different engagement and release thresholds, along with temporal confirmation. This hysteresis prevents a small amount of motion near one threshold from repeatedly switching the pen on and off. The drawing policy also distinguishes provisional thresholds from calibrated ones.

Those rules are testable, but a state-machine test cannot tell me whether the interaction feels comfortable after ten minutes, or whether it holds up under poor lighting and partial occlusion. Physical-hand usability remains experimental. Pointer, touch, and typed controls keep the workspace usable while that work continues.

The default hand path processes camera frames locally. Asking for sketch interpretation is a separate operation: it sends an image of the selected drawing for interpretation. The distinction matters when explaining what leaves the device.

The agent is not the actor

Once several inputs could change the same room, the receipt needed to describe both the member responsible for a committed change and the path that requested it.

The earlier schema already had a separate source column. Its rule was too restrictive: a WebMCP source could only accompany an agent actor. The receipt still retained the authorizing user's ID and required room membership, but its actor classification and display name presented the action as belonging to a generic agent.

The correction binds durable WebMCP canvas mutations to the authenticated member. A host is classified as human; another room member is classified as participant. The source remains webmcp. The SQL wrapper derives the classification from stored membership:

v_effective_actor_type := case v_member_role
  when 'host' then 'human'
  when 'participant' then 'participant'
  else null
end;

This aligns the receipt's visible attribution with the identity that authorized the request. It also allows the reader to distinguish a tool-originated change from direct interaction.

The compatibility work was less visible. The wrapper accepts request shapes from both releases and canonicalizes them before persistence. The table constraint continues to permit the historical agent-plus-WebMCP shape so older rows remain valid; a stricter validator in the write path enforces the new behavior. The migration adds the broader constraint without scanning historical rows immediately, and a separate migration performs validation. The attribution migration shows both responsibilities.

There are limits to what this receipt says. The source field records an application input path; it does not authenticate a particular external agent host. The current service also normalizes most participant inputs to collaborator, preserving webmcp and system separately. It therefore cannot answer every question about whether a remote participant used touch, voice, or a pointer.

The shared write path gives those receipts a consistent meaning across supported inputs. Each refers to committed work, identifies the affected objects, and can support guarded undo.

Reviewing the work before it leaves the room

The optional participant filmstrip keeps a conversation beside the canvas. Its small peer-to-peer WebRTC connection uses Supabase signaling and remains separate from canvas persistence and embedded voice.

Sending a meeting packet introduces another boundary. A saved canvas object does not authorize an email. The packet has its own workflow: preparation creates a snapshot, and approval binds the content and recipients being reviewed. A tool can stage a send request, but the host must perform the final Send action before the configured server transport may submit it.

That requires an explicit review step after the room has produced useful work. The host approves the packet that will leave the room, with its intended recipients. Actual email delivery remains a separate result to verify.

What the recorded checks establish

I built CommandCanvas intending to submit it to the WebMCP Challenge, but didn't submit it.

The verification ledger records native Chrome WebMCP execution, a real voice-provider call that created a board, a real sketch-interpretation request that created structured output beside its source, and two browser clients converging on a durable mutation and receipt. Those are individual recorded checks, rather than proof that the whole intended experience has passed on physical devices.

The documented gaps include a ChatGPT built-in-browser Site Tools invocation, final physical-hand and microphone ergonomics, real email delivery, and cross-network TURN traversal. WebMCP's shared-page model is described in the official Site Tools documentation; host support and an observed invocation still have to be established separately for CommandCanvas.

The next useful rehearsal is a complete session with two people: draw something, explain it, create structured work from it, let the other person change it, and review the outcome together. I want to see whether revision conflicts interrupt that sequence, where selection becomes unclear, and whether the hand controls earn their place beside the pointer. Those are questions about working in the room that individual commit checks cannot settle.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

The Approval Machine That Refuses Before It Drafts

Daniel Romitelli — Wed, 26 Aug 2026 12:10:28 +0000

A message can be wrong before anyone writes it.

That sounds strange until the queue asks for a reply the system has no business drafting. If the member is under a policy flag that disables that intent, the safe answer is not a cautious paragraph. The safe answer is no draft. No retry buffer. No trace payload with forbidden text inside it. Nothing to accidentally send later.

I built an AI message review console around that shape. A card enters the day’s queue and leaves through exactly one exit: blocked before generation, held by a hard rule, or eligible for score-based release. The large language model can draft, classify, and explain. It does not get the final word.

1. The spine is the product

The whole system is readable from pipeline/run.py. That file calls five stages in order: policy, generation, hard rules, scoring, then decision. The ordering matters more than the individual model prompts.

The first gate runs before any model call. The third gate runs after generation and before scoring. The scoring stage is late on purpose, because a score is a judgement about quality. It is not permission to ignore an eligibility flag, a contraindication, missing equipment, scope drift, a stale thread state, or tone that misses a recent life event.

flowchart TD
  queue[Draft request enters queue] --> policy[Pre generation policy]
  policy -->|intent disabled| blocked[Blocked before generation]
  policy -->|allowed| generation[Generate grounded draft]
  generation --> hardRules[Deterministic hard rules]
  hardRules -->|rule fails| held[Held by hard rule]
  hardRules -->|rules pass| scoring[Probabilistic scoring]
  scoring --> decision[Eligible for score based release]

That diagram is the contract I wanted the code to enforce. The interesting part is the missing arrow: there is no route from scoring back into policy or hard rules. A high score cannot wash out a refusal, which is the same fail-closed shape Abhijat Chaturvedi argues for in Fail Closed, Not Open: Designing an AI Gateway for Regulated Enterprises.

2. Pre-generation refusal is a privacy boundary

The naive version of this system is easy to build. Generate the message, run checks, discard the message if the checks fail. It feels safe because the user never sees the rejected text.

It is not, because discarding is not the same as never creating. Once generated, text exists in process memory, logs, traces, retries, recordings, and whatever future evaluation set someone builds from today’s artifacts.

pipeline/stage1_policy.py makes that ordering explicit. The file’s own comment is blunt: if a member has an active policy flag that disables the intent the day’s queue asked for, the stage returns blocked and nothing is generated. The table has two rows, and only one suppresses. A flag can raise scrutiny without disabling generation, which keeps “this topic needs checking” separate from “this topic is not ours.”

From core/models.py, the policy row is plain data:

class PolicyRule(Strict):
    """One row of the pre-generation policy table.

    disables_intents is the whole mechanism. A flag that disables nothing is
    still recorded in the trace, because the difference between a flag that
    suppresses and one that merely raises scrutiny is exactly what stage 1 is
    demonstrating.
    """

    flag: PolicyFlag
    disables_intents: list[DraftIntent]
    route_to: list[CareRole]
    coach_explanation: str

What I like about this shape is that the refusal is auditable without being clever. disables_intents is the mechanism. A route and explanation travel with it, so the operator sees why the machine refused instead of getting a silent missing card.

The cost is that policy has to be modeled before generation. You cannot hide vague eligibility rules in a prompt and hope the model declines. If the rule is meant to prevent text from existing, it belongs before the generator. Ahmed P makes the case in An LLM is not a security boundary: he builds a moderation layer, then walks through it, because a model reads instructions and data from the same token stream.

3. Generation is allowed only after policy clears it

Stage 2 is the point where the model can do useful work. In this console, generation runs a real tool loop. The generation mode is recorded in the trace and response, because showing generated text while implying it was queued text would overstate what happened.

That detail matters because the console is reviewing AI-drafted messages, not pretending every draft has the same origin. The downstream gates do not change based on the mode. Policy has already cleared the request. Hard rules still run. Scoring still waits.

pipeline/run.py records the generation stage with the provider, model, mode, turn count, and tool call count. If the generator proposes different wording while seeded mode is in use, the trace says the queued text was kept. The review layer should not blur authorship just because both paths pass through the same adapter boundary.

The tradeoff is ceremony. A simple demo could have skipped modes, traces, and tool counts. I kept them because the system is making approval decisions, and approval decisions need receipts.

4. Hard rules are post-generation, deterministic, and score-proof

Some checks need the text. Equipment is a good example: the system cannot know whether a message asks for something impossible until there is a message to inspect. Thread state and tone also depend on the relationship between the draft and recent messages.

So hard rules live after generation.

pipeline/stage3_hard_rules.py says the rule set ignores every score and fails closed. Five rules run there. Three are pure code. Two use a model to classify something, then keep the decision in Python by combining strict enum values with an and.

The thread-state rule in pipeline/rules/thread_state.py is deterministic and carries its limit in the file comment. It reads who spoke last, not whether the last reply and the draft are about the same thing. The window is explicit:

ANSWERING_INTENTS = {DraftIntent.reschedule, DraftIntent.check_in}
WINDOW = timedelta(days=7)

I prefer that kind of visible weakness to a hidden prompt instruction. At seventeen cases the distinction between “coach spoke last” and “coach answered this exact topic” does not bite. At scale, the fix would be topic matching, not stretching the window and pretending the predicate became smarter.

The hard-rule result model also keeps the decision state concrete. From core/models.py:

class Severity(str, Enum):
    """What a failed rule costs.

    block: the message must not be sent as written.
    hold: a coach has to look at it. Cheaper to be wrong about.
    """

    hold = "hold"
    block = "block"

class Highlight(Strict):
    target: Literal["draft", "thread"]
    start: int  # located by exact substring match; never fabricated
    end: int
    label: str
    kind: Literal["movement", "claim", "tone", "message"] = "movement"

class HardRuleResult(Strict):
    rule: HardRuleName
    passed: bool
    severity: Severity | None = None  # set only when passed is False
    reason: str
    triggering_message_id: str | None = None
    highlights: list[Highlight] = Field(default_factory=list)
    detail: str = ""  # engineering detail; stays out of the coach copy

That comment on start is a rule about evidence. Where a rule has a span to point at, the offsets are found by exact substring match rather than generated, so the console cannot underline text it invented after the fact.

The thread-state rule is the exception. Its finding is about who spoke last, not about any phrase, so it has no span:

highlights=[Highlight(target="thread", start=0, end=0, label="already answered", kind="message")],

kind="message" is the field that matters, and the reference is triggering_message_id, set beside it to the id of the message that triggered the hold. The zeros mean no span. This rule points at a whole message, which is the right granularity for it and the wrong thing to call an exact offset.

The cost is that hard rules need measurements. pipeline/stage3_hard_rules.py does that measuring: it scans the draft for movements and makes the classifier calls. The rule modules receive measurements and return blockers. That shape is why the rules can be tested without a network.

5. A model can classify without becoming the judge

Two hard rules use a model because pattern matching is the wrong tool for the job. The scope rule in pipeline/rules/scope.py is about whether a draft interprets a clinical value instead of pointing at a clinician. The failure is semantic: two messages can share most words while one crosses scope and the other routes correctly.

The tone rule has the same shape. A high-energy push can be fine on most days and wrong when a recent thread contains a hardship. The problem may not be inside the draft alone. It can be in the relationship between the draft and a message from a few days earlier.

The model returns strict enums. Python decides.

That choice costs some flexibility. If the enum set is too small, the classifier has to force an edge case into a label that does not fit. But I would rather expand a typed vocabulary than let a free-form model answer become an approval predicate.

6. Typed failures are decisions, not missing values

The approval machine also has to handle broken model calls. A timeout, malformed structured output, and an unusable rubric are different failures. Treating all of them as zero would make the system look more numeric while hiding the reason it refused.

core/errors.py names the failures below the adapter boundary:

class ProviderError(RuntimeError):
    """Anything that went wrong below the adapter boundary."""

    def __init__(self, message: str, *, provider: str, model: str) -> None:
        super().__init__(message)
        self.provider = provider
        self.model = model

class ProviderUnavailable(ProviderError):
    """Timeout, connection failure, 5xx, or rate limit. Triggers failover."""

class StructuredOutputError(ProviderError):
    """The call returned, but not something that validates.

    raw_excerpt is the first 400 characters of what actually came back. It goes
    in the trace so a reader can see the malformation rather than trust the
    label on it.
    """

    def __init__(
        self, message: str, *, provider: str, model: str, raw_excerpt: str | None = None
    ) -> None:
        super().__init__(message, provider=provider, model=model)
        self.raw_excerpt = (raw_excerpt or "")[:400] or None

class TruncatedJson(StructuredOutputError):
    """Output stopped mid-token. Unparseable."""

class IncompleteObject(StructuredOutputError):
    """Parsed as JSON, but a required field is missing."""

class NullResponse(StructuredOutputError):
    """The provider returned no content at all, or an explicit null."""

class OutOfRange(StructuredOutputError):
    """A value parsed and is the right type but violates its bound."""

A provider timeout, output that stopped mid-token, an object missing a required field, and a score of 1.4 on a zero-to-one dimension are four different failures, and each gets a name instead of a fallback value. OutOfRange is not clamped: rounding that 1.4 down to 1.0 would turn a broken judge into a passing grade.

core/models.py carries the same philosophy. The file comment says absence refuses, it never defaults. A missing measurement is a blocker, not a zero. That is why rubric scores are optional on the gate decision and why judge failure has its own type instead of a fallback value.

This is less convenient than filling blank fields with zeros. It also stops a dashboard from mixing “bad answer” with “no answer.” Those are different operational problems.

7. Scoring is late because it is allowed to be uncertain

Only after policy and hard rules pass does the probabilistic part get a vote. The threshold stage clears a card only if the samples agree that it clears, using the minimum of three dimensions rather than an average.

The judge is not asked once. core/settings.py sets ensemble_judge: bool = True and judge_samples: int = 5. The stage header gives the reason: on this fixture set, five identical calls move the weakest score by up to 0.28, wider than the 0.70-to-0.80 range the threshold slider was built around.

So the decision comes from how often the samples agree, not from where one of them landed. Three or four usable samples out of five still support an agreement rate, and the failures are recorded. Below three, the stage blocks.

The numbers below come from one run. make eval sweeps the threshold across the seventeen-case fixture set in fixtures/cases.yaml and writes eval/results.json. It uses recorded provider responses rather than live calls, which is what provider: "replay" in that file means and what makes the sweep reproducible offline. Everything here is that run at its default threshold of 0.70.

Of seventeen cards, three clear automatically, nine go to the coach, and five are blocked. Nothing labelled as needing a human was auto-cleared.

Four of the fourteen that did not clear are over-holds: cards labelled as not needing a human that the gate declined to release. One card is undecidable, where the judge could not resolve it and a coach looks at it instead. eval/metrics.py excludes those from the over-hold count, because a judge that could not answer and a gate that held a card it should have released are different failures. Mean spread across the samples is 0.148.

I kept the over-hold cost in the system view and the decision notes instead of tuning it away. A fail-closed system can still be lazy if it hides that cost, because holding too much burns human attention. The point is not to make refusal free. The point is to make the refusal cost visible.

8. What the design costs

The cost lands in maintenance, not at runtime.

Every refusal here is something a person has to keep current: a policy row, an enum member, a threshold, a fixture case proving the rule still fires. That is more surface than a prompt asking a model to be careful. It buys artifacts that can be diffed, tested, and argued with in a pull request. A paragraph of instructions cannot be tested the same way.

The number I would watch in production is the over-hold count, not the block count. Four out of seventeen is survivable on a fixture day and would be a staffing problem at a thousand cards a morning.

Fail-closed systems rarely fail by letting something through. They fail by becoming more expensive without anyone writing down why, until somebody raises the threshold to make the queue manageable. That is a decision to make against a measured over-hold rate, which is why the sweep exists and why the over-held column stays on the screen.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

Confidence Procedures Are Production Code

Daniel Romitelli — Wed, 26 Aug 2026 11:48:12 +0000

A clean sample can sound stronger than it is. Someone checks a handful of items, finds no exceptions, and the sentence wants to become broader than the evidence: the process worked, the control operated, the population was clean.

That sentence is where I stopped trusting the product unless the math had the same status as the API. If a compliance audit sampling tool emits a confidence claim, the confidence procedure is production code. It needs adversarial tests. A formula in a notebook is too far away from the thing users read.

This post is about the bound in core/bounds.py, the inverse sample-size question in the same module, and the evaluation gate in eval/coverage.py and eval/run.py. The tool reports a one-sided upper confidence bound on the number of failures in a finite population. Then it tries to prove that bound covers at the confidence it claims.

1. The number is a frontier, not a fact

The bound answers a narrow question: after drawing n items from a population of N and observing k exceptions, what is the largest true failure count D still compatible with the sample at the requested confidence?

That wording matters. The tool is not estimating the true number of failures. It is stating what the sample rules out. A clean sample does not mean the population is clean. It means failure counts above the frontier are no longer supported by the observed draw at the selected confidence.

The implementation uses the hypergeometric distribution because audit sampling draws without replacement. The same item cannot be selected twice. That is a small detail in code and a large detail in interpretation when the sample is a meaningful fraction of the population, which is exactly the situation where finite-population correction starts to matter (MeasuringU).

DECISIONS.md has the blunt version of the tradeoff. At N=1,000 with n=25, the exact finite-population bound is 11.1 percent; the binomial comparison is 11.3 percent. Close enough that the distinction feels academic. At N=64, the gap opens as the sampling fraction rises: at n=5 the binomial is 7 percent high, at n=25 it is 45 percent high, at n=48 it is 94 percent high, and at a full census of 64 it reports 4.6 percent where the true answer is zero.

That last case decided the design. If I examine every item in a finite population, the upper bound on unobserved failures cannot stay above zero after a clean census. The binomial approximation has forgotten that the population can be exhausted.

def exact_upper_bound(N: int, n: int, k: int, confidence: float = 0.95) -> Bound:
    """What a sample of n from N, showing k exceptions, actually supports."""
    _validate(N, n, k, confidence)
    D = _max_failures(N, n, k, confidence)
    return Bound(
        N=N,
        n=n,
        k=k,
        confidence=confidence,
        max_failures=D,
        max_failure_rate=D / N,
        rule_of_three=(3.0 / n) if (k == 0 and n > 0) else None,
        binomial_max_failure_rate=_clopper_pearson_upper(n, k, confidence),
    )

I kept the binomial value in the returned model as a comparison, never as the answer. That made the approximation visible without letting it drive the verdict.

The other choice in this function is less obvious: the result carries N, n, k, and confidence alongside the bound. core/models.py says the rule directly: no bare float crosses a signature. A number without its inputs is exactly the failure mode this tool is trying to avoid.

2. The hypergeometric inversion is the product behavior

The core search lives in _max_failures. It inverts the probability statement instead of sampling possible worlds. For each possible true failure count D, the observed exception count K has a hypergeometric distribution. The reported bound is the largest D for which observing at most k exceptions is still probable enough.

The implementation depends on monotonicity: adding more failures to the population can only make small observed counts less likely. That turns the search into an exact binary search over integer failure counts, rather than a heuristic over percentages.

if n == 0:
    # Zero draws exclude nothing. Every item could be a failure.
    return N
if k == n:
    # Every draw was an exception. P(X <= n) == 1 for every D.
    return N

alpha = 1.0 - confidence
lo, hi = k, N  # D = k always satisfies: P(X <= k | D = k) == 1
while lo < hi:
    mid = (lo + hi + 1) // 2
    if _cdf_at_most_k(k, N, mid, n) >= alpha - _ALPHA_TOL:
        lo = mid
    else:
        hi = mid - 1
return lo

The edge cases are part of the contract. With zero draws, the sample excludes nothing. If every draw is an exception, every possible failure count remains compatible with observing at most n exceptions, so the upper bound is the whole population.

The middle is the frontier. For fixed N and n, every observed k cuts the lattice of possible true failure counts into allowed and ruled-out values. The product prints the frontier, not the hidden truth.

flowchart TD
  population[Finite Population N] --> draw[Draw n Without Replacement]
  draw[Draw n Without Replacement] --> observed[Observed Exceptions k]
  observed[Observed Exceptions k] --> inversion[Hypergeometric Inversion]
  inversion[Hypergeometric Inversion] --> frontier[Reported Upper Bound D]
  frontier[Reported Upper Bound D] --> claim[Allowed Claim]
  population[Finite Population N] --> lattice[All True Failure Counts D]
  lattice[All True Failure Counts D] -.-> inversion[Hypergeometric Inversion]
  binomial[Binomial Approximation] -.-> wrongFrontier[Forgets Finite Population]
  alwaysN[Always Return N] -.-> vacuous[Always Covers But Says Nothing]
  inversion[Hypergeometric Inversion] ==> exactPath[Exact Finite Population Bound]

I folded the bad alternatives into the same diagram because they fail for different reasons. The binomial approximation can be too loose when the sampling fraction rises. Always returning N would pass a naive coverage check, because it covers everything. The exact finite-population inversion has to satisfy both constraints: cover at the stated confidence and still say something.

3. The inverse question is where product pressure shows up

Once the tool can say what a sample supports, the next question is predictable: what sample size would support a stronger sentence?

That inverse question also belongs in core/bounds.py. It uses the same finite-population bound, only flipped around. Instead of asking, “given N, n, and k, what is the maximum supported failure rate,” it asks what draw size would be needed for a target cap under an assumed exception rate.

I am careful with that framing because this is where tools start to lie politely. A target cap is not a measurement. An assumed exception rate is an input. If those are treated as facts, the sample-size panel becomes a confidence costume for planning assumptions.

The model boundary helps here. In core/models.py, Bound validates internal consistency: k cannot exceed n, n cannot exceed N, the bound cannot fall below the observed exceptions, and the maximum failures cannot exceed the population. That is plain defensive programming, but in this domain those checks are also argument hygiene.

A product that emits statistical language has two outputs. There is the value on the screen, and there is the sentence a human will say after reading it. The inverse sample-size path exists because the second output matters. If the current sample cannot support the stronger sentence, the tool should state the cost of that sentence rather than rounding the current evidence upward.

4. Coverage is a gate, not a chart

The evaluation runner treats coverage as the first measurement. eval/run.py describes four measurements, but coverage runs first and stops the run if it fails. The target is never adjusted to match the result.

That is the right shape for a statistical product. Detection rates, draws to first exception, and prior sensitivity are useful after the confidence procedure is valid. Before that, they are decoration.

eval/coverage.py checks the confidence claim two ways. The exact path enumerates outcomes analytically. The Monte Carlo path replays the same quantity through the random draw mechanism. They are not redundant. Exact enumeration tests the math. Simulation tests that the implementation path agrees with the math.

def exact_coverage(N: int, n: int, D: int, confidence: float = NOMINAL) -> float:
    """P(true D falls inside the reported bound), summed over all outcomes."""
    ks = np.arange(0, n + 1)
    pmf = hypergeom.pmf(ks, N, D, n)
    covered = np.array(
        [exact_upper_bound(N, n, int(k), confidence).max_failures >= D for k in ks]
    )
    return float(pmf[covered].sum())

This is the cleanest test in the project. For a fixed N, n, and true failure count D, every possible observed k is finite. So the test sums the probability mass for the outcomes where the reported bound contains the true D.

There is no sampling error in that number. If it falls below the nominal confidence, the procedure does not cover. The product is wrong.

The Monte Carlo version uses NumPy’s hypergeometric draw and then runs the same bound calculation on the observed exception counts. The default is 10,000 runs with seed 0.

def mc_coverage(
    N: int,
    n: int,
    D: int,
    confidence: float = NOMINAL,
    runs: int = MC_RUNS,
    seed: int = 0,
) -> float:
    """The same quantity by simulation, through the real draw path."""
    rng = np.random.default_rng(seed)
    ks = rng.hypergeometric(ngood=D, nbad=N - D, nsample=n, size=runs)
    bounds = {
        int(k): exact_upper_bound(N, n, int(k), confidence).max_failures
        for k in np.unique(ks)
    }

The dictionary cache is a small detail, but I like it. A simulation can draw the same k many times, and the bound for that k does not change. Cache the unique observed counts, then compare repeated outcomes to the same computed frontier.

The readme states the result before anything else: verdict passed. Minimum coverage observed was 0.9500 against nominal 0.9500, across 5,943 exhaustively enumerated configurations. The worst configuration was N=64, n=25, D=16, at 0.9505. The largest disagreement between exact and simulated coverage was 0.0039.

Those numbers are not marketing. They are the release condition for every sentence that depends on the bound.

5. A passing coverage test can still be useless

A malicious implementation can pass a lower-bound coverage check by returning N for every input. It will always contain the true failure count, because the true count cannot exceed the population.

That is the trap in testing confidence procedures. “Covers at least 95 percent” is necessary. Alone, it rewards vacuity.

The suite adds an anti-vacuity ceiling. The readme states it as a test: minimum coverage must not rise above 0.96. A bound that always returned N would cover at 100 percent and say nothing, so that implementation should fail.

This is one of those tests that looks strange until you need it. Most test suites only reject undercoverage. Here, overcoverage can also be a bug because the product is supposed to say the strongest true sentence supported by the evidence. Too safe becomes uninformative.

There is a second guard: shrink the bound by one and confirm coverage breaks. That catches the opposite fake. Without it, the coverage test could be passing while failing to detect that the frontier is load-bearing. If reducing every reported upper bound by one still passed, the original test would not be sharp enough to prove the implemented boundary.

The two guards create pressure from both sides. The bound cannot be loose enough to say nothing, and it cannot be tightened without losing the claimed confidence.

6. Ground truth stays outside the estimator

The simulator needs hidden failure rates. The estimator must never see them.

That boundary is explicit. simulation/draw.py is the only module permitted to read Stratum.true_failure_rate. Its docstring says why: everything under core/ estimates, while the simulation knows the answer. If an estimator could read ground truth, the evaluation would be theatre.

The same file says the boundary is enforced three ways: parse every file under core/ and api/ for the identifier, assert that no estimator module imports the simulation package, and perturb every true rate in the fixtures while checking that estimator output does not move by a single digit.

I care about this boundary more than I expected to when I started. Statistical code has an especially ugly failure mode: it can look more correct after it cheats. If ground truth leaks into the estimator, the bounds become beautifully calibrated to information the auditor never had.

The HTTP surface follows the same rule. server/main.py says every response is a public projection. Stratum carries simulator ground truth, so nothing of that type is serialized. The browser receives projections, never the domain object that knows the answer.

That separation costs some convenience. You write more models. You pass more values around. You make tests about identifiers and imports, which feels crude until the first time it prevents a silent category error.

7. The formula was the easy part

The finite-population bound is compact. The production problem is larger: keep the approximation visible but out of the verdict, carry the inputs with every output, invert the distribution over integer counts, prove coverage by enumeration, replay it through simulation, reject vacuity, and make ground truth unreachable from estimators.

I do not think of that as “extra validation” around the math. It is the math becoming a product surface.

A confidence number is a promise about repeated behavior. Once that promise appears in software, the test suite has to attack the promise itself, because the user will build a sentence from the number whether the code earned it or not.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

A Rehearsal Is Only Cheap In Distribution

Daniel Romitelli — Mon, 24 Aug 2026 03:19:27 +0000

TL;DR In generative video pipelines, running cheap low-step sketches to pick parameters sounds like free optimization. But when prompts go out-of-distribution, surrogate scorers return noise, turning a \$0.002 check into a bad decision that triggers a \$15 compounding failure. Here's why skipping the cheap step is sometimes the cheapest option.

Three numbers run Scenematic's generation loop. A think-frame costs \$0.002. A full render costs \$0.50. A bad scene that slips through and gets built on costs about \$15.50, because the scene chain compounds it before anyone looks. The constant in lib/generation-loop.ts carries the arithmetic in a comment: 15.502, // CALIBRATION_TARGET: 0.002 + 0.50 + 15.00.

Most of the pipeline exists to keep spend at the cheap end of that ladder. One module decides when the cheap step should be skipped entirely. A hundred-contract baseline then put numbers on how often that decision was wrong.

1. The rehearsal

lib/think-frames.ts generates quick, low-inference-step sketches before committing to a full-quality keyframe. The file header credits DeepGen's think tokens as the inspiration. Each sketch tries a different preservation focus, character, environment, mood, composition, or atmosphere, with its own image-to-image strength and seed. The reward mixer scores the batch and the winner's parameters go to the full render.

The economics only work if those scores mean something. That assumption fails quietly, and it fails hardest on the prompts where a rehearsal looks most useful.

2. Where the scores stop meaning anything

Scoring a sketch of A detective leans forward across a metal table, interrogating a nervous suspect under fluorescent lights works fine. The scorer has seen a thousand shots like it. Scoring A sentient equation writes itself across a blackboard that extends infinitely in all dimensions does not fail loudly. It returns a number, and the number is noise. Both prompts are verbatim from the baseline harness.

lib/ood-detector.ts measures the distance instead of hoping. It keeps a reference corpus of 25 in-distribution prompts, roughly 8 dialogue, 9 cinematic, 8 multi-asset, and embeds them with all-MiniLM-L6-v2 through @xenova/transformers. Local model, no API call.

The embeddings mean-pool into a unit centroid, computed once per process and cached. An incoming prompt gets embedded the same way. Epistemic uncertainty is one minus the cosine similarity to that centroid, and at or above the threshold the detector sets bypass_surrogate: true.

Bypass means skipping the cheap step, so the prompt the system understands least is the one that goes straight to the expensive render. Backwards from most cost optimizations, and deliberate. Out past the corpus a rehearsal is a \$0.002 lie that steers a \$0.50 decision toward a \$15 mistake, and skipping it buys the removal of a bad witness at the price of one render. The gate recuses an unqualified judge rather than filtering bad prompts.

Every evaluation writes a row to an ood_events table: the uncertainty, the threshold that was applied, the bypass flag, and cost_incurred at either \$0.50 or \$0.002. That table is where everything else in this post comes from.

flowchart TD
  P["Compiled prompt"] --> U["Uncertainty = 1 - cosine vs corpus centroid"]
  U --> G{"At or above the category threshold?"}
  G -->|"BYPASS"| F["Straight to full render, $0.50"]
  G -->|"SURROG"| T["Think-frame rehearsal, $0.002"]
  T --> W["Full render using the winning sketch's parameters"]

3. One hundred contracts

scripts/run-phase1-baseline.ts runs 100 steering contracts through the gate. A steering contract is the typed record the compiler emits for one scene: the prompt, the chosen model and seed, the quality targets, and the audit trail of what happened to it. The batch splits into 80 in-distribution prompts across dialogue, cinematic, and multi-asset, plus 20 out-of-distribution (OOD) prompts, abstract and high-complexity. Seeds are fixed at 42 plus the contract index. The renders are real, on RunPod pods running ltx2, wan22, and hunyuan behind per-model semaphores of 3, 2, and 2. Each console line prints SURROG or BYPASS next to the measured uncertainty.

A caveat before the numbers. The reward heads, the four per-dimension scores that judge a finished render (R_smooth, R_motion, R_semantic, R_narrative), are simulated in this harness: in-distribution signals drawn around 0.65 with jitter, OOD around 0.35, because real per-head scoring of rendered video was not wired up yet. The embeddings, the routing, the renders, and the telemetry are real. The simulation still surfaced two real bugs, covered below.

The cost waterfall, straight out of fig4_cost_waterfall.csv:

Stage	Count	Cost
Contracts submitted	100
OOD bypassed	31	\$0.00 extra
GPU rendered	69	\$15.64
Passed quality floor	43
HITL flagged	26
False negatives	53	\$821.60 compound

Twenty-two of the 26 human-in-the-loop (HITL) flags blamed the narrative head. The Phase 2 commit message gives the aggregate: a 25.3% false negative rate, with R_narrative dominating 72.4% of HITL flags. Fifty-three contracts finished under the quality floor, and at \$15.50 of modeled compound damage each the ledger prices those misses at \$821.60, against \$15.64 for every render in the run combined.

4. Thresholds from the misses

Phase 2 swept the threshold from 0.30 to 0.95 in steps of 0.01 and computed a full confusion matrix at each stop. The F1-optimal global threshold came out at 0.31, which flags 97 of 100 contracts and puts the cost column at \$48.51 per hundred. Flagging nearly everything does maximize F1. It also defeats the point of having a surrogate at all.

So the shipped config ignored F1. The calibration_metadata block in quality-thresholds-v2.json records the target it optimized instead: Catch ALL known FNs per category with margin. Take the known misses, find the minimum uncertainty among them per category, set the threshold just under it. DIALOGUE's one known false negative sat at 0.631, so DIALOGUE got 0.62. SCENIC's four bottomed out at 0.506, so 0.50. ACTION's five at 0.495, so 0.49. The global fell from 0.72 to 0.55 and now serves as the fallback for categories with no data.

Per category matters because the sweep exposed an inversion a single global threshold cannot encode. The config file says it in one line: SCENIC and ACTION had LOWER uncertainty but HIGHER FN rates — category thresholds fix this.

A scenic prompt reads as familiar to the embedding space. Sweeping drone shot, golden light, the corpus is full of that texture. The renders still miss. Uncertainty and failure rate are correlated across the whole population and inverted inside two of its slices, and one knob cannot express that.

ABSTRACT kept the old 0.72, since genuinely weird prompts carry high uncertainty on their own. NARRATIVE got 0.65 with an annotation calling it a conservative estimate with no false negative data behind it. That one is a guess with a label on it, and it stays a guess until a NARRATIVE prompt fails in a logged run.

The detector loads this file at runtime and falls back to the legacy global 0.72 if it is missing. The sweep range and step size are recorded next to the thresholds they produced, along with the per-category evidence.

5. Two gates that could not fire

The baseline caught two bugs I would not have found by reading the code, because the code looked fine.

The HITL gate required the composite score to be under the floor AND at least one reward head to breach a z-score of -2.0. With simulated signals at 0.65 ± 0.1, the commit message for 20aa5ec describes the result: the z-score condition was mathematically impossible to satisfy with simulated signal distribution of 0.65 ± 0.1. Two conditions joined by AND, one of them unsatisfiable. The gate sat silent while contracts failed under it, and the dashboards looked calm the whole time.

The fix inverted the roles. The reward floor of 0.65 is now the primary trigger, and per-head z-scores only attribute which head gets blamed.

The second bug was in the false negative counter itself. It only counted SURROG contracts, on the theory that a false negative means the surrogate path trusted a prompt it should not have. That definition missed 8 contracts the gate bypassed which still rendered below the floor. A bypassed prompt that fails is still the pipeline failing, whatever path it took, so the counter now takes any contract under the floor.

6. The head that ran cooler

R_narrative took 72.4% of the HITL blame in the phase 2 baseline. The narrative scoring was doing its job. Its head just runs cooler than the others: population mean 0.55, against 0.60 for R_motion and 0.70 for R_smooth and R_semantic. Four heads held to the same absolute bar, and the coolest one took nearly all the blame.

The fix in lib/reward-mixer.ts is z-score normalization per head. normalizeHeadScore computes (raw − mean) / std against per-head population stats stored in the same versioned config, so a head only triggers when it is unusual for itself. computeSubReason then cross-references the other heads whenever R_narrative does trigger: motion also low means pacing, semantic also low means fidelity, narrative alone means coherence.

7. Infra noise is not model failure

Twenty hunyuan events in the baseline were graphics processing unit (GPU) failures, out-of-memory and handler crashes, sitting in ood_events and dragging the quality numbers around. So the table grew gpu_error and superseded columns. evaluateOOD returns the row id of the event it just logged, and the runner calls markOODEventGpuError on a GPU error so the dashboard can segment it. Re-running contracts with --rerun-contracts="81,82,100" marks the old contaminated rows superseded instead of deleting them. queryDashboards takes a view argument, clean or all, and clean excludes both flags.

Even total infrastructure collapse gets a row: ALL_MODELS_DOWN logs an OOD event with gpu_error=true. Without the segmentation, a crashed pod reads as a quality regression, and one afternoon of flaky hardware quietly recalibrates your thresholds for you.

8. What is still provisional

The detector landed at 234 lines and sits just over 300 after the Phase 2 changes. The scoring inside it is a cached centroid and a cosine. Nothing is learned and none of it touches a GPU. The cost constants carry a CALIBRATION_TARGET marker with a comment saying to recalibrate after 100 real runs or a provider switch, so the provisional numbers announce themselves and are greppable.

Plenty is still provisional. The reference corpus is 25 prompts, and every threshold in the v2 config is calibrated against the centroid those 25 produce. Adding corpus prompts moves the centroid, which shifts every uncertainty measurement, which invalidates the per-category thresholds. Corpus and thresholds have to version together or the calibration quietly stops describing anything. The reward heads are still the simulated ones, so the head_stats block needs re-deriving from real scores before the z-score triggers mean what they claim. And NARRATIVE still has no false negative data.

The part I trust is the ledger. Both dead gates were invisible in code review and obvious in the event counts, and the event counts only exist because every decision writes a row, including the decision to spend more.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

Calibration Is Bet Sizing

Daniel Romitelli — Sun, 23 Aug 2026 12:19:16 +0000

The last post was about making a number trustworthy. Leakage geometry, purge widths, de-overlap, a baseline that could not cheat. It ended with a minute-scale ceiling that held at 52% across seven configurations and a model family swap.

This one is about what happens after you trust the number. Because a probability you are going to bet on is a different object from a probability you are going to report.

The probabilities are not decorative

The path-passage classifier is a three-class LightGBM. It returns p_up, p_down, p_none. Those go straight into the expected-value score that decides whether to take a trade and how big:

long_score  = p_up * (B - C) + p_down * (-B - C) + p_none * (-C)
short_score = p_up * (-B - C) + p_down * (B - C) + p_none * (-C)

B is the barrier, C the cost. Read the arithmetic. Every term is linear in a probability. Scale p_up by 1.2 and you scale the long score by very nearly 1.2.

So miscalibration does not stay in the model. It becomes a bet-sizing error, in proportion, in the bins where the gate actually fires. A classifier that is right 70% of the time while claiming 90% is not 20 points wrong. It is sizing every position in that bin as though the edge were far larger than it is.

Boosted trees are known for uncalibrated softmax output. I had been consuming it as if it were a probability.

The audit

Seven live assets. For each one, fit an Inductive Venn-Abers wrapper on the time-ordered older 80% of that model's training data, 6,988 rows, and evaluate against a 500-row uniform-random sample of the newer 20%, seed 42. The LightGBM models are reloaded from disk and left alone. Only the wrapper is fit.

Measure Expected Calibration Error and log-loss, before and after.

Asset	ECE before → after	ECE Δ	Log-loss Δ
BTC	0.1272 → 0.0621	-51.2%	-5.5%
ETH	0.1795 → 0.0298	-83.4%	-11.5%
SOL	0.1680 → 0.0386	-77.0%	-10.6%
XRP	0.2219 → 0.0645	-70.9%	-17.7%
ADA	0.1419 → 0.0369	-74.0%	-8.0%
LINK	0.1260 → 0.0737	-41.5%	-1.2%
LTC	0.1508 → 0.0603	-60.0%	-14.1%

Every asset has a real gap. That settles the first question, which was whether this was one bad model or a property of the setup. It is systematic.

The second question is the interesting one.

The direction is asset-specific, and that rules out the easy fix

ETH fails the way boosted trees are supposed to fail. Its worst reliability bin is [0.90, 1.00]. Eleven samples. Mean stated confidence 93.8%. Empirical accuracy 45.5%.

Most certain, least reliable. That is the bin where the EV gate fires hardest and the position sizes are largest.

The other six fail in the opposite direction.

XRP, in the [0.60, 0.70] bin: 93.1% accuracy at 64.9% stated confidence, n=116. A 28-point understatement.

LTC, same bin: 90.4% accuracy at 64.7% confidence, n=115.

That is a suppressed-signal failure. The gate does not fire often enough, because the stated confidence lags what the model actually delivers. It costs money quietly, by declining trades that were good.

One asset over-confident. Six under-confident.

Which kills the convenient answer. Platt scaling and temperature scaling apply one monotone correction. They cannot pull ETH's tail down and push XRP's middle up at the same time, because those are corrections in opposite directions. A single global calibrator fits the average of two failure modes and helps neither.

Per-asset Venn-Abers works here because it fits each asset's own reliability curve and does not assume a shape.

Where the split came from

This is the part I did not expect, and it is the reason I keep the training params in the same document as the audit.

The hardened LightGBM settings were min_data_in_leaf=400, num_leaves=8, max_depth=3. I introduced those specifically to stop the ETH-style saturation, where terminal leaves go to 1.0 and the model claims certainty it has not earned.

They worked. They also worked too well.

Constraining the leaves prevented the over-confidence failure and produced a structural under-confidence pattern across the rest of the universe. The fix for one failure mode manufactured the opposite failure mode in six assets.

ETH is the lone holdout that still saturates, because in the rare cases where it is genuinely certain its leaves still reach the top of the range. Eleven samples in that bin tells the story: sparse and extreme.

That is a straight tradeoff I made without knowing I was making it. Reliability diagrams are what showed it. An accuracy score would have shown a modest improvement and nothing else, because averaging is exactly the operation that hides a bin.

The bar was set before the numbers came back

ECE has to improve by at least 50%, and log-loss by at least 5%. Both, not either.

Six assets clear it. LINK-USD misses both, at -41.5% and -1.2%.

LINK is not a data problem. It has the same 8,736 training rows as everything else, so this is model quality rather than availability. The LightGBM may simply be better calibrated for LINK already, in which case there is less for the wrapper to do and the small lift is honest. Or the calibrator needs different hyperparameters. Both are worth a follow-up and neither is resolved today.

So the rollout is selective. Six of seven, and LINK stays on raw softmax until somebody investigates it.

The mechanism is deliberately boring. The loader falls back to raw softmax for any asset with no calibrator on disk, so excluding LINK means either leaving its pickle off the deployment image or adding an allow-list:

allow = set(os.environ.get(
    "VENN_ABERS_ASSETS",
    "BTC-USD,ETH-USD,SOL-USD,XRP-USD,ADA-USD,LTC-USD",
).split(","))

One environment variable and three lines in the loader. A rollout that cannot express "these six and not that one" ends up shipping the failure with the successes.

What this audit does not prove

The status line at the top of the integration guide says prototype, not integrated into the live inference path. That is still true, and the caveats are worth having in the open.

The holdout is a 500-row uniform sample, seed 42. The 80/20 split is on the model's own training data rather than on live signals.

There has been no live-data refit, because signals_history does not yet carry realized 24-hour outcomes for the p_up/p_down/p_none rows. Migration 028 landed on 2026-05-21 to start collecting them.

BTC's barrier is 150 bps and the rest are 200, so the ECE deltas compare cleanly but the absolute ECE values across assets carry a class-balance shift. I treat those as informal.

LINK's failure in particular should be re-run once live outcomes exist. A training-data fit can hide a different live picture, and that is exactly the asset where I would expect it.

Why this is the second half of the last post

The geometry work answered whether the measurement could be trusted. Purge, embargo, de-overlap, a split that cannot leak. All of that gets you a number you can believe.

Believing the number is not the same as being able to bet on it. The ceiling post established that the minute-scale direction signal is capped near 52% and that the cap is real rather than an artifact. This audit establishes something narrower and more immediate: on the horizon where there is signal, the probability the model hands the sizer is not the probability it should act on, and the correction is different for every asset.

Accuracy averages. A reliability diagram does not. The bin where the model was most confident and least correct is worth more attention than any figure computed across the whole set, because it is the bin where the money goes.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

Experimentation Is the Missing Evaluation Layer for Agent Memory

Daniel Romitelli — Sun, 23 Aug 2026 10:43:40 +0000

An agent can fail because it missed the right material. It can also fail because I gave it the wrong material with confidence. The worse case is quieter: the wrong material sits beside a successful task, then looks useful afterward.

A skilled user may ask for architecture notes because they know where to look. Those notes can travel with a good result even when they did not cause it. If the system learns from that trace alone, the next session may get extra text that feels justified and still wastes the agent's attention.

That failure has a name and a direction. The same expertise that makes someone request the right document also makes them likelier to finish the task without it. Skill causes both the request and the outcome, so skill is a common cause sitting upstream of the thing I am trying to measure. A system that reads the trace and credits the document has attributed to the context what belonged to the person. This is not noise that averages out as traffic grows. More sessions from confident users make the estimate tighter and no less wrong, which is the property that makes it dangerous: the system becomes more certain of a relationship it never established.

There is no way to subtract that bias afterward from the trace alone, because the trace does not record why the material was requested. The only cheap instrument that removes it is deciding who gets the material by coin flip instead of by request. Randomization makes assignment independent of skill, so whatever difference survives between the two branches is attributable to the material rather than to the person holding it.

This is the problem I built the active experimentation layer for in Zero Context Loss (ZCL). ZCL is the context learning platform I use for my AI agents. This post is about one part of it: the evaluation layer that tests changes to context provisioning while the agent is doing real work. It is also, by the end, an honest account of how much that layer can currently prove, which is less than its own vocabulary suggests.

1. A context change becomes a hypothesis

The active learning code lives in zcl_core/learning/active_learning.py. The central object is Hypothesis. I made the candidate change carry its own treatment, control, task type, metric, and minimum sample threshold.

That shape is deliberate. Reordering two documents has a different target from including a document whose value is uncertain. The first can be judged by time. The second can be judged by success. If those choices stay implicit, the system can promote a vague improvement without saying what improved.

Declaring the metric before the data arrives matters more than it looks. A hypothesis that names its outcome in advance can only be judged on that outcome. A hypothesis that stays vague can be judged on whichever of success, duration or token count happens to have moved, and something almost always has. Writing the metric into the object is the cheapest available guard against choosing the comparison after seeing the result.

The hypothesis object uses a UUID, a type string, a description, the task kind, treatment and control payloads, a metric name, and min_samples set to 20. The default threshold sits on the object because I wanted the brake to travel with the proposed change. Section 6 returns to whether that brake is connected to anything.

The generator in ActiveLearning stays close to context the system can actually apply. It looks at documents already used for a task type, then builds order hypotheses from the top five documents against the next slice. It also asks for uncertain documents and creates inclusion tests for up to three of them. Each hypothesis takes its id from uuid4() — called, not referenced.

A context system with limited traffic can burn every session it has on combinations that will never gather enough evidence to matter, so the generator stops early and some pairings never become live candidates at all, and I gave that coverage up on purpose. The generator is not there to enumerate everything that could be tested. It is there to keep the number of live tests small enough that each one can actually finish.

2. Assignment happens while context is being built

The integration point is zcl_core/provision/context_provider.py. The context provider creates the session id with uuid4(), decides whether the request should explore, asks value attribution for learned document values, handles cold start, then filters the selected document ids through causal evidence before the final bundle goes back to the agent.

Assignment has to happen before the agent consumes the material, because the branch has to be decided by the coin rather than by the request, and that ordering is doing more work here than anything else in the file. A report run later can describe what happened, but by then the material has already been chosen by whatever mixture of habit, skill and retrieval score produced it, and the confound from the opening is already baked into the data. Deciding during provisioning is what converts an observation into an experiment.

The returned ContextBundle carries is_experimental, experiment_id, and experiment_group. Those fields are plain bookkeeping, and they are the difference between analysis and guesswork. If a session was in treatment, the bundle says so. If it was in control, the evaluator does not have to reconstruct that from log order.

flowchart TD
 candidate[Candidate context strategy] --> assignment[Randomized assignment]
 assignment --> treatment[Treatment context]
 assignment --> control[Control context]
 treatment --> outcome[Task outcome]
 control --> outcome
 outcome --> evaluation[Statistical evaluation]
 evaluation --> promoted[Promoted strategy]
 evaluation --> retired[Retired strategy]```



The diagram lies in one place, and it is the place most likely to matter. Randomizing sessions treats sessions as independent units. They are not. The same person returns, so their skill appears in both arms across the week rather than being held constant within one. Learned document values also update between sessions, which means the control arm is not fixed while the experiment runs; the baseline drifts under the test. Randomizing the person rather than the session would remove the first problem, and freezing learned values for the duration of an experiment would remove the second. The current design does neither, and the effect is that the estimate is noisier than a clean trial of the same size.

## 3. How much of the traffic gets spent on trials

`ActiveLearning` starts with `base_exploration_rate = 0.2`, uses `uncertainty_multiplier = 0.5`, and caps the final probability at 0.4. These are configuration values in the implementation. They are not presented as observed production rates, throughput measurements, or hardware-dependent results.

The decision is small: compute task uncertainty for the organization, add the uncertainty bonus to the base rate, cap it, then compare that probability with `random.random()`.

The uncertainty calculation is scoped by organization. That stops one organization's history from making another organization's context appear more certain than it is. In the helper, fewer than 10 sessions returns maximum uncertainty. After that, the code reads outcomes for the task, computes success-rate variance as a Bernoulli variance, and reduces the result as sample count grows using the log of the count. With no outcomes, it uses 0.25 as maximum variance.

It is a cheap signal for where to spend trials, and it makes no claim at all about why a document helped.

There are two traditions tangled together in that decision, and they do not want the same thing. Spending more trials where uncertainty is high is bandit reasoning, and a bandit's goal is to minimise regret: give as many sessions as possible the best-known context while still learning. A controlled experiment has a different goal, which is an unbiased estimate of an effect, and it is happiest with a fixed allocation decided in advance. The two are not the same discipline. Adaptive allocation is known to bias the naive difference in means, because the amount of data each arm receives depends on how the arm has been performing.

Here the two are only loosely coupled: the uncertainty rate governs whether a session explores at all, while assignment within a live experiment is a fair split. That keeps the bias small. But the honest description of this layer is that it uses a bandit to decide when to run trials and a fixed randomization to run them, and if the exploration rate ever starts responding to the results of a specific live experiment, the estimator stops being trustworthy.

Capping it at 0.4 means the areas with the least evidence still spend most of their sessions on the path already known, which slows learning down badly. I kept the cap anyway. Without it a sparse task type turns into churn, every request treated as a trial, and nothing ever settles into a default.

## 4. The schema keeps the branches separate

The experiment tables are defined in `migrations/004_experiments.sql`. The migration separates the proposed change from the per-session assignment. One table stores the hypothesis text, task type, treatment, control, status, results, and timestamps. The other stores the session, experiment id, group name, and assignment time. Both take their ids and timestamps from `uuid_generate_v4()` and `NOW()`, called rather than named.



```sql
CREATE TABLE zcl_experiments (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    hypothesis TEXT NOT NULL,
    task_type VARCHAR(100) NOT NULL,
    treatment JSONB NOT NULL,
    control JSONB NOT NULL,
    status VARCHAR(20) NOT NULL DEFAULT 'active', -- active, completed, cancelled
    results JSONB,
    started_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    completed_at TIMESTAMPTZ,

    CONSTRAINT valid_status CHECK (status IN ('active', 'completed', 'cancelled'))
);

CREATE TABLE zcl_experiment_assignments (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    session_id UUID NOT NULL REFERENCES zcl_sessions(id) ON DELETE CASCADE,
    experiment_id UUID NOT NULL REFERENCES zcl_experiments(id) ON DELETE CASCADE,
    group_name VARCHAR(20) NOT NULL, -- treatment, control
    assigned_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),

    UNIQUE(session_id, experiment_id),
    CONSTRAINT valid_group CHECK (group_name IN ('treatment', 'control'))
);

JSONB is PostgreSQL's binary JSON type. I used it here because treatment and control payloads vary by hypothesis type. Ordering two documents is shaped differently from excluding one document, and forcing those into a rigid column set would add ceremony without improving the evaluator.

The two constraints are doing real work, and they are doing the kind of work that is easy to skip and expensive to skip. UNIQUE(session_id, experiment_id) means a session is assigned once to a given experiment. Without it, a retry or a duplicated provisioning call would let one session contribute two rows to the same arm, which inflates the sample count with correlated data and makes the test more confident than the evidence warrants. The group check keeps the result set to treatment or control, so a typo cannot open a third bucket that then silently reduces the size of both real arms.

The migration also adds indexes for task type, status, active experiments, assignment lookup, session lookup, and group lookup. The write path needs to find active tests during provisioning. The read path needs to count assignments when judging results. Those are different access patterns, so the schema names both.

5. Outcomes are measured after the task

Retrieval can produce tidy scores before the agent acts. Rank, similarity, token fit, and document presence all arrive early. None of them says whether the work succeeded.

This is the gap the title is about, and it is worth stating plainly rather than implying it. The standard instruments for retrieval quality — recall at k, mean reciprocal rank, normalised discounted cumulative gain — all score a ranking against a set of documents somebody labelled relevant in advance. They answer whether retrieval found what a human said was relevant. Agent memory has to answer something else: whether the material changed what the agent did. Those questions come apart precisely where the interesting failures live. A document can be topically relevant, rank first, be judged relevant by any labeller, and still cost the agent attention it needed elsewhere. Recall at k cannot see that, because the harm is not in the ranking. It is in what happened next.

Asking a model to grade the retrieval has the same shape. It scores the plausibility of the material against the request, which is a judgement made before the work and without knowing how the work went. Both instruments measure the retrieval, and both stop exactly where the question starts.

The migration defines get_experiment_outcomes. It joins experiment assignments to recorded outcomes, grouped by branch. The function returns group name, total sessions, successful sessions, success rate, and average time in minutes.

CREATE OR REPLACE FUNCTION get_experiment_outcomes(p_experiment_id UUID)
RETURNS TABLE (
    group_name VARCHAR,
    total_sessions BIGINT,
    successful_sessions BIGINT,
    success_rate FLOAT,
    avg_time_minutes FLOAT
) AS $$
BEGIN
    RETURN QUERY
    SELECT
        ea.group_name::VARCHAR,
        COUNT(*)::BIGINT as total_sessions,
        SUM(CASE WHEN o.success THEN 1 ELSE 0 END)::BIGINT as successful_sessions,
        AVG(CASE WHEN o.success THEN 1.0 ELSE 0.0 END)::FLOAT as success_rate,
        AVG(o.time_minutes)::FLOAT as avg_time_minutes
    FROM zcl_experiment_assignments ea
    JOIN zcl_outcomes o ON ea.session_id = o.session_id
    WHERE ea.experiment_id = p_experiment_id
    GROUP BY ea.group_name;
END;
$$ LANGUAGE plpgsql;

This is where the layer stops being retrieval self-scoring. An inclusion hypothesis can be judged on whether the session succeeded; an order hypothesis is better judged on how long it took. Either way the metric was declared up front, and the outcome table supplies whatever was actually measured.

There is a floor under all of this that randomization cannot reach. zcl_outcomes.success is a bare BOOLEAN NOT NULL, and nothing in the schema says who decides it or on what evidence. Randomizing the assignment removes the confound between the user and the branch they got. It does nothing whatsoever about a mismeasured outcome. If success is set by a heuristic, or reported by the same person whose skill I was trying to control for in the first place, then the bias I built this entire layer to remove walks back in through the dependent variable — and this time the system reports it with a p-value attached, which makes it harder to argue with rather than easier.

If I want to judge some other behavior later, the outcome schema has to carry it first, or the candidate cannot be promoted inside this loop at all. It is a narrow door to walk through, and I would still rather have it than a soft win assembled from whatever trace happened to be lying nearby.

6. Minimum samples slow the system on purpose

ActiveLearning sets alpha = 0.05, with the code comment p < 0.05 for significance. The result object carries effect_size, p_value, significant, and recommendation. Evaluation compares the two arms' success counts with a chi-square test on a two-by-two contingency table, and the migration includes experiment_ready_for_analysis, which checks whether an experiment has enough samples before evaluation.

This is the guardrail I wanted most. A document can land in treatment, ride along with one successful session, and look useful if the system is hungry for reinforcement, so the threshold is supposed to make it wait.

At least that is what I had been telling myself. Writing this section I went to check the number, and the number is not wired to anything.

Hypothesis.min_samples is set to 20. Nothing reads it. It sits on the dataclass, rides into the row, and not one code path consults it before a result gets judged. The gate that actually runs is in analyze_experiment, which bails out only when an arm holds fewer than five outcomes, and its SQL counterpart experiment_ready_for_analysis takes p_min_per_group INTEGER DEFAULT 5. So the brake is five, not twenty. The twenty is decoration, and it is the kind that looks clean and fails quietly.

Five per arm cannot carry the conclusion the code is willing to draw from it. Work the table: five sessions in each branch, chi-square at alpha 0.05, and exactly two of the thirty-six possible outcomes clear significance. Five successes against zero. Zero against five. Both land at p equal to 0.011, and every other cell in that table is a null before the experiment starts. At this size the test is not measuring an effect. It is asking whether the two arms disagreed about every single session, which is not the question I meant to ask.

Chi-square is the wrong instrument here anyway. It wants expected cell counts of about five or more, and a two-by-two table built from five observations per arm cannot give it that. Fisher's exact test handles tables this small and the swap is one line. Raising the gate to the twenty already written on the object does not rescue it either — at twenty per arm the smallest difference the test can see is still around thirty percentage points.

Here is the scale I should have worked out before writing any of it. A ten point improvement in success rate, 0.50 to 0.60, at eighty percent power and the same alpha, needs roughly 194 sessions per arm. Five points needs about 782. Those are the numbers at which the words sitting in recommendation — ADOPT, REJECT — mean what they claim. Ten points is a substantial win for a context change. This layer would need forty times the evidence it currently demands to notice one.

There is a related exposure in how many tests run at once. The generator can open five order hypotheses and three inclusion hypotheses for a single task type. Eight independent tests at alpha 0.05 carry roughly a 34 percent chance that at least one clears significance by luck alone, and the one that clears is the one that gets promoted into every future session's context. A Bonferroni correction, or simply refusing to run more than one live experiment per task type at a time, is the cheap fix.

One thing the design gets right by accident of structure is worth crediting, because it is the error most A/B systems make. analyze_experiment completes the experiment as soon as it evaluates it. There is no path that looks at the data, finds nothing, and looks again next week. Repeated peeking at an accumulating result is how a nominal five percent false positive rate becomes twenty or thirty percent in practice, and this loop cannot do it: one look, then the experiment closes. The cost of that virtue is that the single look happens at the earliest moment it is permitted, which is also the moment with the least evidence behind it.

Put those two facts side by side and it is worse than slow learning. A completed experiment never reopens; _complete_experiment writes status completed and nothing anywhere sets it back to active. At five per arm almost every outcome is INCONCLUSIVE by construction. So a context change that genuinely helps gets one underpowered look, comes back as no significant difference because it could hardly come back as anything else, and is closed on that basis permanently. I built the threshold to stop the system adopting noise. What it actually does is retire real improvements the test was never equipped to detect, and there is no route back to retry one.

The exploration rate decides where this lands hardest, and it picks the worst place. _get_task_uncertainty returns maximum uncertainty for any task type with fewer than ten recorded sessions, which pushes the exploration probability straight to its 0.4 ceiling, and that count is scoped per organization. A small organization therefore runs the largest share of its sessions as trials while generating the fewest sessions to finish any of them with. It experiments hardest precisely where five outcomes per arm takes longest to accumulate, and every one of those trials still closes after a single look.

The stored result hides that rather than surfacing it. effect_size goes into the results JSON as a bare difference between two success rates, with no interval around it, so a gap measured on five sessions per arm is recorded in the same shape, and reads later with the same authority, as one measured on five hundred.

Waiting costs something. Bad ideas stay alive longer, good ones take more sessions before they become default behavior, and some combinations never get tested at all because the generator cut them off early. That pressure is almost certainly how the number ended up at five: a gate that low returns verdicts fast, and returning verdicts fast is the entire problem with it.

The operational states are active, completed and cancelled. An active experiment can still receive assignments and a completed one can store results, but the useful one is cancelled: it stops shaping sessions without pretending it ever reached a conclusion. The view v_experiment_results puts treatment counts, control counts, status, and timestamps together so the operator can see whether a run is still gathering evidence or ready for judgment.

That is the trade I made for agent memory and retrieval: learn slower, record the branch, measure after work, and require enough samples before future sessions change. Three of those four are built and working. The fourth is a number I wrote on an object and never wired up, and writing this post is what made me look.

The rule I started with still holds. A learning system that changes future inputs has to earn that change with outcomes, or it will preserve coincidence as policy. Randomization is what makes an outcome mean anything. The sample threshold is what stops it being noise. I built the first one properly. The second I wrote down on a dataclass and never wired up, so the system will go on drawing conclusions from five sessions and filing them with the same confidence it would give five hundred.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series