DEV Community: Daniel Romitelli

A Rehearsal Is Only Cheap In Distribution

Daniel Romitelli — Mon, 24 Aug 2026 03:19:27 +0000

TL;DR In generative video pipelines, running cheap low-step sketches to pick parameters sounds like free optimization. But when prompts go out-of-distribution, surrogate scorers return noise, turning a \$0.002 check into a bad decision that triggers a \$15 compounding failure. Here's why skipping the cheap step is sometimes the cheapest option.

Three numbers run Scenematic's generation loop. A think-frame costs \$0.002. A full render costs \$0.50. A bad scene that slips through and gets built on costs about \$15.50, because the scene chain compounds it before anyone looks. The constant in lib/generation-loop.ts carries the arithmetic in a comment: 15.502, // CALIBRATION_TARGET: 0.002 + 0.50 + 15.00.

Most of the pipeline exists to keep spend at the cheap end of that ladder. One module decides when the cheap step should be skipped entirely. A hundred-contract baseline then put numbers on how often that decision was wrong.

1. The rehearsal

lib/think-frames.ts generates quick, low-inference-step sketches before committing to a full-quality keyframe. The file header credits DeepGen's think tokens as the inspiration. Each sketch tries a different preservation focus, character, environment, mood, composition, or atmosphere, with its own image-to-image strength and seed. The reward mixer scores the batch and the winner's parameters go to the full render.

The economics only work if those scores mean something. That assumption fails quietly, and it fails hardest on the prompts where a rehearsal looks most useful.

2. Where the scores stop meaning anything

Scoring a sketch of A detective leans forward across a metal table, interrogating a nervous suspect under fluorescent lights works fine. The scorer has seen a thousand shots like it. Scoring A sentient equation writes itself across a blackboard that extends infinitely in all dimensions does not fail loudly. It returns a number, and the number is noise. Both prompts are verbatim from the baseline harness.

lib/ood-detector.ts measures the distance instead of hoping. It keeps a reference corpus of 25 in-distribution prompts, roughly 8 dialogue, 9 cinematic, 8 multi-asset, and embeds them with all-MiniLM-L6-v2 through @xenova/transformers. Local model, no API call.

The embeddings mean-pool into a unit centroid, computed once per process and cached. An incoming prompt gets embedded the same way. Epistemic uncertainty is one minus the cosine similarity to that centroid, and at or above the threshold the detector sets bypass_surrogate: true.

Bypass means skipping the cheap step, so the prompt the system understands least is the one that goes straight to the expensive render. Backwards from most cost optimizations, and deliberate. Out past the corpus a rehearsal is a \$0.002 lie that steers a \$0.50 decision toward a \$15 mistake, and skipping it buys the removal of a bad witness at the price of one render. The gate recuses an unqualified judge rather than filtering bad prompts.

Every evaluation writes a row to an ood_events table: the uncertainty, the threshold that was applied, the bypass flag, and cost_incurred at either \$0.50 or \$0.002. That table is where everything else in this post comes from.

flowchart TD
  P["Compiled prompt"] --> U["Uncertainty = 1 - cosine vs corpus centroid"]
  U --> G{"At or above the category threshold?"}
  G -->|"BYPASS"| F["Straight to full render, $0.50"]
  G -->|"SURROG"| T["Think-frame rehearsal, $0.002"]
  T --> W["Full render using the winning sketch's parameters"]

3. One hundred contracts

scripts/run-phase1-baseline.ts runs 100 steering contracts through the gate. A steering contract is the typed record the compiler emits for one scene: the prompt, the chosen model and seed, the quality targets, and the audit trail of what happened to it. The batch splits into 80 in-distribution prompts across dialogue, cinematic, and multi-asset, plus 20 out-of-distribution (OOD) prompts, abstract and high-complexity. Seeds are fixed at 42 plus the contract index. The renders are real, on RunPod pods running ltx2, wan22, and hunyuan behind per-model semaphores of 3, 2, and 2. Each console line prints SURROG or BYPASS next to the measured uncertainty.

A caveat before the numbers. The reward heads, the four per-dimension scores that judge a finished render (R_smooth, R_motion, R_semantic, R_narrative), are simulated in this harness: in-distribution signals drawn around 0.65 with jitter, OOD around 0.35, because real per-head scoring of rendered video was not wired up yet. The embeddings, the routing, the renders, and the telemetry are real. The simulation still surfaced two real bugs, covered below.

The cost waterfall, straight out of fig4_cost_waterfall.csv:

Stage	Count	Cost
Contracts submitted	100
OOD bypassed	31	\$0.00 extra
GPU rendered	69	\$15.64
Passed quality floor	43
HITL flagged	26
False negatives	53	\$821.60 compound

Twenty-two of the 26 human-in-the-loop (HITL) flags blamed the narrative head. The Phase 2 commit message gives the aggregate: a 25.3% false negative rate, with R_narrative dominating 72.4% of HITL flags. Fifty-three contracts finished under the quality floor, and at \$15.50 of modeled compound damage each the ledger prices those misses at \$821.60, against \$15.64 for every render in the run combined.

4. Thresholds from the misses

Phase 2 swept the threshold from 0.30 to 0.95 in steps of 0.01 and computed a full confusion matrix at each stop. The F1-optimal global threshold came out at 0.31, which flags 97 of 100 contracts and puts the cost column at \$48.51 per hundred. Flagging nearly everything does maximize F1. It also defeats the point of having a surrogate at all.

So the shipped config ignored F1. The calibration_metadata block in quality-thresholds-v2.json records the target it optimized instead: Catch ALL known FNs per category with margin. Take the known misses, find the minimum uncertainty among them per category, set the threshold just under it. DIALOGUE's one known false negative sat at 0.631, so DIALOGUE got 0.62. SCENIC's four bottomed out at 0.506, so 0.50. ACTION's five at 0.495, so 0.49. The global fell from 0.72 to 0.55 and now serves as the fallback for categories with no data.

Per category matters because the sweep exposed an inversion a single global threshold cannot encode. The config file says it in one line: SCENIC and ACTION had LOWER uncertainty but HIGHER FN rates — category thresholds fix this.

A scenic prompt reads as familiar to the embedding space. Sweeping drone shot, golden light, the corpus is full of that texture. The renders still miss. Uncertainty and failure rate are correlated across the whole population and inverted inside two of its slices, and one knob cannot express that.

ABSTRACT kept the old 0.72, since genuinely weird prompts carry high uncertainty on their own. NARRATIVE got 0.65 with an annotation calling it a conservative estimate with no false negative data behind it. That one is a guess with a label on it, and it stays a guess until a NARRATIVE prompt fails in a logged run.

The detector loads this file at runtime and falls back to the legacy global 0.72 if it is missing. The sweep range and step size are recorded next to the thresholds they produced, along with the per-category evidence.

5. Two gates that could not fire

The baseline caught two bugs I would not have found by reading the code, because the code looked fine.

The HITL gate required the composite score to be under the floor AND at least one reward head to breach a z-score of -2.0. With simulated signals at 0.65 ± 0.1, the commit message for 20aa5ec describes the result: the z-score condition was mathematically impossible to satisfy with simulated signal distribution of 0.65 ± 0.1. Two conditions joined by AND, one of them unsatisfiable. The gate sat silent while contracts failed under it, and the dashboards looked calm the whole time.

The fix inverted the roles. The reward floor of 0.65 is now the primary trigger, and per-head z-scores only attribute which head gets blamed.

The second bug was in the false negative counter itself. It only counted SURROG contracts, on the theory that a false negative means the surrogate path trusted a prompt it should not have. That definition missed 8 contracts the gate bypassed which still rendered below the floor. A bypassed prompt that fails is still the pipeline failing, whatever path it took, so the counter now takes any contract under the floor.

6. The head that ran cooler

R_narrative took 72.4% of the HITL blame in the phase 2 baseline. The narrative scoring was doing its job. Its head just runs cooler than the others: population mean 0.55, against 0.60 for R_motion and 0.70 for R_smooth and R_semantic. Four heads held to the same absolute bar, and the coolest one took nearly all the blame.

The fix in lib/reward-mixer.ts is z-score normalization per head. normalizeHeadScore computes (raw − mean) / std against per-head population stats stored in the same versioned config, so a head only triggers when it is unusual for itself. computeSubReason then cross-references the other heads whenever R_narrative does trigger: motion also low means pacing, semantic also low means fidelity, narrative alone means coherence.

7. Infra noise is not model failure

Twenty hunyuan events in the baseline were graphics processing unit (GPU) failures, out-of-memory and handler crashes, sitting in ood_events and dragging the quality numbers around. So the table grew gpu_error and superseded columns. evaluateOOD returns the row id of the event it just logged, and the runner calls markOODEventGpuError on a GPU error so the dashboard can segment it. Re-running contracts with --rerun-contracts="81,82,100" marks the old contaminated rows superseded instead of deleting them. queryDashboards takes a view argument, clean or all, and clean excludes both flags.

Even total infrastructure collapse gets a row: ALL_MODELS_DOWN logs an OOD event with gpu_error=true. Without the segmentation, a crashed pod reads as a quality regression, and one afternoon of flaky hardware quietly recalibrates your thresholds for you.

8. What is still provisional

The detector landed at 234 lines and sits just over 300 after the Phase 2 changes. The scoring inside it is a cached centroid and a cosine. Nothing is learned and none of it touches a GPU. The cost constants carry a CALIBRATION_TARGET marker with a comment saying to recalibrate after 100 real runs or a provider switch, so the provisional numbers announce themselves and are greppable.

Plenty is still provisional. The reference corpus is 25 prompts, and every threshold in the v2 config is calibrated against the centroid those 25 produce. Adding corpus prompts moves the centroid, which shifts every uncertainty measurement, which invalidates the per-category thresholds. Corpus and thresholds have to version together or the calibration quietly stops describing anything. The reward heads are still the simulated ones, so the head_stats block needs re-deriving from real scores before the z-score triggers mean what they claim. And NARRATIVE still has no false negative data.

The part I trust is the ledger. Both dead gates were invisible in code review and obvious in the event counts, and the event counts only exist because every decision writes a row, including the decision to spend more.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

Calibration Is Bet Sizing

Daniel Romitelli — Sun, 23 Aug 2026 12:19:16 +0000

The last post was about making a number trustworthy. Leakage geometry, purge widths, de-overlap, a baseline that could not cheat. It ended with a minute-scale ceiling that held at 52% across seven configurations and a model family swap.

This one is about what happens after you trust the number. Because a probability you are going to bet on is a different object from a probability you are going to report.

The probabilities are not decorative

The path-passage classifier is a three-class LightGBM. It returns p_up, p_down, p_none. Those go straight into the expected-value score that decides whether to take a trade and how big:

long_score  = p_up * (B - C) + p_down * (-B - C) + p_none * (-C)
short_score = p_up * (-B - C) + p_down * (B - C) + p_none * (-C)

B is the barrier, C the cost. Read the arithmetic. Every term is linear in a probability. Scale p_up by 1.2 and you scale the long score by very nearly 1.2.

So miscalibration does not stay in the model. It becomes a bet-sizing error, in proportion, in the bins where the gate actually fires. A classifier that is right 70% of the time while claiming 90% is not 20 points wrong. It is sizing every position in that bin as though the edge were far larger than it is.

Boosted trees are known for uncalibrated softmax output. I had been consuming it as if it were a probability.

The audit

Seven live assets. For each one, fit an Inductive Venn-Abers wrapper on the time-ordered older 80% of that model's training data, 6,988 rows, and evaluate against a 500-row uniform-random sample of the newer 20%, seed 42. The LightGBM models are reloaded from disk and left alone. Only the wrapper is fit.

Measure Expected Calibration Error and log-loss, before and after.

Asset	ECE before → after	ECE Δ	Log-loss Δ
BTC	0.1272 → 0.0621	-51.2%	-5.5%
ETH	0.1795 → 0.0298	-83.4%	-11.5%
SOL	0.1680 → 0.0386	-77.0%	-10.6%
XRP	0.2219 → 0.0645	-70.9%	-17.7%
ADA	0.1419 → 0.0369	-74.0%	-8.0%
LINK	0.1260 → 0.0737	-41.5%	-1.2%
LTC	0.1508 → 0.0603	-60.0%	-14.1%

Every asset has a real gap. That settles the first question, which was whether this was one bad model or a property of the setup. It is systematic.

The second question is the interesting one.

The direction is asset-specific, and that rules out the easy fix

ETH fails the way boosted trees are supposed to fail. Its worst reliability bin is [0.90, 1.00]. Eleven samples. Mean stated confidence 93.8%. Empirical accuracy 45.5%.

Most certain, least reliable. That is the bin where the EV gate fires hardest and the position sizes are largest.

The other six fail in the opposite direction.

XRP, in the [0.60, 0.70] bin: 93.1% accuracy at 64.9% stated confidence, n=116. A 28-point understatement.

LTC, same bin: 90.4% accuracy at 64.7% confidence, n=115.

That is a suppressed-signal failure. The gate does not fire often enough, because the stated confidence lags what the model actually delivers. It costs money quietly, by declining trades that were good.

One asset over-confident. Six under-confident.

Which kills the convenient answer. Platt scaling and temperature scaling apply one monotone correction. They cannot pull ETH's tail down and push XRP's middle up at the same time, because those are corrections in opposite directions. A single global calibrator fits the average of two failure modes and helps neither.

Per-asset Venn-Abers works here because it fits each asset's own reliability curve and does not assume a shape.

Where the split came from

This is the part I did not expect, and it is the reason I keep the training params in the same document as the audit.

The hardened LightGBM settings were min_data_in_leaf=400, num_leaves=8, max_depth=3. I introduced those specifically to stop the ETH-style saturation, where terminal leaves go to 1.0 and the model claims certainty it has not earned.

They worked. They also worked too well.

Constraining the leaves prevented the over-confidence failure and produced a structural under-confidence pattern across the rest of the universe. The fix for one failure mode manufactured the opposite failure mode in six assets.

ETH is the lone holdout that still saturates, because in the rare cases where it is genuinely certain its leaves still reach the top of the range. Eleven samples in that bin tells the story: sparse and extreme.

That is a straight tradeoff I made without knowing I was making it. Reliability diagrams are what showed it. An accuracy score would have shown a modest improvement and nothing else, because averaging is exactly the operation that hides a bin.

The bar was set before the numbers came back

ECE has to improve by at least 50%, and log-loss by at least 5%. Both, not either.

Six assets clear it. LINK-USD misses both, at -41.5% and -1.2%.

LINK is not a data problem. It has the same 8,736 training rows as everything else, so this is model quality rather than availability. The LightGBM may simply be better calibrated for LINK already, in which case there is less for the wrapper to do and the small lift is honest. Or the calibrator needs different hyperparameters. Both are worth a follow-up and neither is resolved today.

So the rollout is selective. Six of seven, and LINK stays on raw softmax until somebody investigates it.

The mechanism is deliberately boring. The loader falls back to raw softmax for any asset with no calibrator on disk, so excluding LINK means either leaving its pickle off the deployment image or adding an allow-list:

allow = set(os.environ.get(
    "VENN_ABERS_ASSETS",
    "BTC-USD,ETH-USD,SOL-USD,XRP-USD,ADA-USD,LTC-USD",
).split(","))

One environment variable and three lines in the loader. A rollout that cannot express "these six and not that one" ends up shipping the failure with the successes.

What this audit does not prove

The status line at the top of the integration guide says prototype, not integrated into the live inference path. That is still true, and the caveats are worth having in the open.

The holdout is a 500-row uniform sample, seed 42. The 80/20 split is on the model's own training data rather than on live signals.

There has been no live-data refit, because signals_history does not yet carry realized 24-hour outcomes for the p_up/p_down/p_none rows. Migration 028 landed on 2026-05-21 to start collecting them.

BTC's barrier is 150 bps and the rest are 200, so the ECE deltas compare cleanly but the absolute ECE values across assets carry a class-balance shift. I treat those as informal.

LINK's failure in particular should be re-run once live outcomes exist. A training-data fit can hide a different live picture, and that is exactly the asset where I would expect it.

Why this is the second half of the last post

The geometry work answered whether the measurement could be trusted. Purge, embargo, de-overlap, a split that cannot leak. All of that gets you a number you can believe.

Believing the number is not the same as being able to bet on it. The ceiling post established that the minute-scale direction signal is capped near 52% and that the cap is real rather than an artifact. This audit establishes something narrower and more immediate: on the horizon where there is signal, the probability the model hands the sizer is not the probability it should act on, and the correction is different for every asset.

Accuracy averages. A reliability diagram does not. The bin where the model was most confident and least correct is worth more attention than any figure computed across the whole set, because it is the bin where the money goes.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

Experimentation Is the Missing Evaluation Layer for Agent Memory

Daniel Romitelli — Sun, 23 Aug 2026 10:43:40 +0000

An agent can fail because it missed the right material. It can also fail because I gave it the wrong material with confidence. The worse case is quieter: the wrong material sits beside a successful task, then looks useful afterward.

A skilled user may ask for architecture notes because they know where to look. Those notes can travel with a good result even when they did not cause it. If the system learns from that trace alone, the next session may get extra text that feels justified and still wastes the agent's attention.

That failure has a name and a direction. The same expertise that makes someone request the right document also makes them likelier to finish the task without it. Skill causes both the request and the outcome, so skill is a common cause sitting upstream of the thing I am trying to measure. A system that reads the trace and credits the document has attributed to the context what belonged to the person. This is not noise that averages out as traffic grows. More sessions from confident users make the estimate tighter and no less wrong, which is the property that makes it dangerous: the system becomes more certain of a relationship it never established.

There is no way to subtract that bias afterward from the trace alone, because the trace does not record why the material was requested. The only cheap instrument that removes it is deciding who gets the material by coin flip instead of by request. Randomization makes assignment independent of skill, so whatever difference survives between the two branches is attributable to the material rather than to the person holding it.

This is the problem I built the active experimentation layer for in Zero Context Loss (ZCL). ZCL is the context learning platform I use for my AI agents. This post is about one part of it: the evaluation layer that tests changes to context provisioning while the agent is doing real work. It is also, by the end, an honest account of how much that layer can currently prove, which is less than its own vocabulary suggests.

1. A context change becomes a hypothesis

The active learning code lives in zcl_core/learning/active_learning.py. The central object is Hypothesis. I made the candidate change carry its own treatment, control, task type, metric, and minimum sample threshold.

That shape is deliberate. Reordering two documents has a different target from including a document whose value is uncertain. The first can be judged by time. The second can be judged by success. If those choices stay implicit, the system can promote a vague improvement without saying what improved.

Declaring the metric before the data arrives matters more than it looks. A hypothesis that names its outcome in advance can only be judged on that outcome. A hypothesis that stays vague can be judged on whichever of success, duration or token count happens to have moved, and something almost always has. Writing the metric into the object is the cheapest available guard against choosing the comparison after seeing the result.

The hypothesis object uses a UUID, a type string, a description, the task kind, treatment and control payloads, a metric name, and min_samples set to 20. The default threshold sits on the object because I wanted the brake to travel with the proposed change. Section 6 returns to whether that brake is connected to anything.

The generator in ActiveLearning stays close to context the system can actually apply. It looks at documents already used for a task type, then builds order hypotheses from the top five documents against the next slice. It also asks for uncertain documents and creates inclusion tests for up to three of them. Each hypothesis takes its id from uuid4() — called, not referenced.

A context system with limited traffic can burn every session it has on combinations that will never gather enough evidence to matter, so the generator stops early and some pairings never become live candidates at all, and I gave that coverage up on purpose. The generator is not there to enumerate everything that could be tested. It is there to keep the number of live tests small enough that each one can actually finish.

2. Assignment happens while context is being built

The integration point is zcl_core/provision/context_provider.py. The context provider creates the session id with uuid4(), decides whether the request should explore, asks value attribution for learned document values, handles cold start, then filters the selected document ids through causal evidence before the final bundle goes back to the agent.

Assignment has to happen before the agent consumes the material, because the branch has to be decided by the coin rather than by the request, and that ordering is doing more work here than anything else in the file. A report run later can describe what happened, but by then the material has already been chosen by whatever mixture of habit, skill and retrieval score produced it, and the confound from the opening is already baked into the data. Deciding during provisioning is what converts an observation into an experiment.

The returned ContextBundle carries is_experimental, experiment_id, and experiment_group. Those fields are plain bookkeeping, and they are the difference between analysis and guesswork. If a session was in treatment, the bundle says so. If it was in control, the evaluator does not have to reconstruct that from log order.

flowchart TD
 candidate[Candidate context strategy] --> assignment[Randomized assignment]
 assignment --> treatment[Treatment context]
 assignment --> control[Control context]
 treatment --> outcome[Task outcome]
 control --> outcome
 outcome --> evaluation[Statistical evaluation]
 evaluation --> promoted[Promoted strategy]
 evaluation --> retired[Retired strategy]```



The diagram lies in one place, and it is the place most likely to matter. Randomizing sessions treats sessions as independent units. They are not. The same person returns, so their skill appears in both arms across the week rather than being held constant within one. Learned document values also update between sessions, which means the control arm is not fixed while the experiment runs; the baseline drifts under the test. Randomizing the person rather than the session would remove the first problem, and freezing learned values for the duration of an experiment would remove the second. The current design does neither, and the effect is that the estimate is noisier than a clean trial of the same size.

## 3. How much of the traffic gets spent on trials

`ActiveLearning` starts with `base_exploration_rate = 0.2`, uses `uncertainty_multiplier = 0.5`, and caps the final probability at 0.4. These are configuration values in the implementation. They are not presented as observed production rates, throughput measurements, or hardware-dependent results.

The decision is small: compute task uncertainty for the organization, add the uncertainty bonus to the base rate, cap it, then compare that probability with `random.random()`.

The uncertainty calculation is scoped by organization. That stops one organization's history from making another organization's context appear more certain than it is. In the helper, fewer than 10 sessions returns maximum uncertainty. After that, the code reads outcomes for the task, computes success-rate variance as a Bernoulli variance, and reduces the result as sample count grows using the log of the count. With no outcomes, it uses 0.25 as maximum variance.

It is a cheap signal for where to spend trials, and it makes no claim at all about why a document helped.

There are two traditions tangled together in that decision, and they do not want the same thing. Spending more trials where uncertainty is high is bandit reasoning, and a bandit's goal is to minimise regret: give as many sessions as possible the best-known context while still learning. A controlled experiment has a different goal, which is an unbiased estimate of an effect, and it is happiest with a fixed allocation decided in advance. The two are not the same discipline. Adaptive allocation is known to bias the naive difference in means, because the amount of data each arm receives depends on how the arm has been performing.

Here the two are only loosely coupled: the uncertainty rate governs whether a session explores at all, while assignment within a live experiment is a fair split. That keeps the bias small. But the honest description of this layer is that it uses a bandit to decide when to run trials and a fixed randomization to run them, and if the exploration rate ever starts responding to the results of a specific live experiment, the estimator stops being trustworthy.

Capping it at 0.4 means the areas with the least evidence still spend most of their sessions on the path already known, which slows learning down badly. I kept the cap anyway. Without it a sparse task type turns into churn, every request treated as a trial, and nothing ever settles into a default.

## 4. The schema keeps the branches separate

The experiment tables are defined in `migrations/004_experiments.sql`. The migration separates the proposed change from the per-session assignment. One table stores the hypothesis text, task type, treatment, control, status, results, and timestamps. The other stores the session, experiment id, group name, and assignment time. Both take their ids and timestamps from `uuid_generate_v4()` and `NOW()`, called rather than named.



```sql
CREATE TABLE zcl_experiments (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    hypothesis TEXT NOT NULL,
    task_type VARCHAR(100) NOT NULL,
    treatment JSONB NOT NULL,
    control JSONB NOT NULL,
    status VARCHAR(20) NOT NULL DEFAULT 'active', -- active, completed, cancelled
    results JSONB,
    started_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    completed_at TIMESTAMPTZ,

    CONSTRAINT valid_status CHECK (status IN ('active', 'completed', 'cancelled'))
);

CREATE TABLE zcl_experiment_assignments (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    session_id UUID NOT NULL REFERENCES zcl_sessions(id) ON DELETE CASCADE,
    experiment_id UUID NOT NULL REFERENCES zcl_experiments(id) ON DELETE CASCADE,
    group_name VARCHAR(20) NOT NULL, -- treatment, control
    assigned_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),

    UNIQUE(session_id, experiment_id),
    CONSTRAINT valid_group CHECK (group_name IN ('treatment', 'control'))
);

JSONB is PostgreSQL's binary JSON type. I used it here because treatment and control payloads vary by hypothesis type. Ordering two documents is shaped differently from excluding one document, and forcing those into a rigid column set would add ceremony without improving the evaluator.

The two constraints are doing real work, and they are doing the kind of work that is easy to skip and expensive to skip. UNIQUE(session_id, experiment_id) means a session is assigned once to a given experiment. Without it, a retry or a duplicated provisioning call would let one session contribute two rows to the same arm, which inflates the sample count with correlated data and makes the test more confident than the evidence warrants. The group check keeps the result set to treatment or control, so a typo cannot open a third bucket that then silently reduces the size of both real arms.

The migration also adds indexes for task type, status, active experiments, assignment lookup, session lookup, and group lookup. The write path needs to find active tests during provisioning. The read path needs to count assignments when judging results. Those are different access patterns, so the schema names both.

5. Outcomes are measured after the task

Retrieval can produce tidy scores before the agent acts. Rank, similarity, token fit, and document presence all arrive early. None of them says whether the work succeeded.

This is the gap the title is about, and it is worth stating plainly rather than implying it. The standard instruments for retrieval quality — recall at k, mean reciprocal rank, normalised discounted cumulative gain — all score a ranking against a set of documents somebody labelled relevant in advance. They answer whether retrieval found what a human said was relevant. Agent memory has to answer something else: whether the material changed what the agent did. Those questions come apart precisely where the interesting failures live. A document can be topically relevant, rank first, be judged relevant by any labeller, and still cost the agent attention it needed elsewhere. Recall at k cannot see that, because the harm is not in the ranking. It is in what happened next.

Asking a model to grade the retrieval has the same shape. It scores the plausibility of the material against the request, which is a judgement made before the work and without knowing how the work went. Both instruments measure the retrieval, and both stop exactly where the question starts.

The migration defines get_experiment_outcomes. It joins experiment assignments to recorded outcomes, grouped by branch. The function returns group name, total sessions, successful sessions, success rate, and average time in minutes.

CREATE OR REPLACE FUNCTION get_experiment_outcomes(p_experiment_id UUID)
RETURNS TABLE (
    group_name VARCHAR,
    total_sessions BIGINT,
    successful_sessions BIGINT,
    success_rate FLOAT,
    avg_time_minutes FLOAT
) AS $$
BEGIN
    RETURN QUERY
    SELECT
        ea.group_name::VARCHAR,
        COUNT(*)::BIGINT as total_sessions,
        SUM(CASE WHEN o.success THEN 1 ELSE 0 END)::BIGINT as successful_sessions,
        AVG(CASE WHEN o.success THEN 1.0 ELSE 0.0 END)::FLOAT as success_rate,
        AVG(o.time_minutes)::FLOAT as avg_time_minutes
    FROM zcl_experiment_assignments ea
    JOIN zcl_outcomes o ON ea.session_id = o.session_id
    WHERE ea.experiment_id = p_experiment_id
    GROUP BY ea.group_name;
END;
$$ LANGUAGE plpgsql;

This is where the layer stops being retrieval self-scoring. An inclusion hypothesis can be judged on whether the session succeeded; an order hypothesis is better judged on how long it took. Either way the metric was declared up front, and the outcome table supplies whatever was actually measured.

There is a floor under all of this that randomization cannot reach. zcl_outcomes.success is a bare BOOLEAN NOT NULL, and nothing in the schema says who decides it or on what evidence. Randomizing the assignment removes the confound between the user and the branch they got. It does nothing whatsoever about a mismeasured outcome. If success is set by a heuristic, or reported by the same person whose skill I was trying to control for in the first place, then the bias I built this entire layer to remove walks back in through the dependent variable — and this time the system reports it with a p-value attached, which makes it harder to argue with rather than easier.

If I want to judge some other behavior later, the outcome schema has to carry it first, or the candidate cannot be promoted inside this loop at all. It is a narrow door to walk through, and I would still rather have it than a soft win assembled from whatever trace happened to be lying nearby.

6. Minimum samples slow the system on purpose

ActiveLearning sets alpha = 0.05, with the code comment p < 0.05 for significance. The result object carries effect_size, p_value, significant, and recommendation. Evaluation compares the two arms' success counts with a chi-square test on a two-by-two contingency table, and the migration includes experiment_ready_for_analysis, which checks whether an experiment has enough samples before evaluation.

This is the guardrail I wanted most. A document can land in treatment, ride along with one successful session, and look useful if the system is hungry for reinforcement, so the threshold is supposed to make it wait.

At least that is what I had been telling myself. Writing this section I went to check the number, and the number is not wired to anything.

Hypothesis.min_samples is set to 20. Nothing reads it. It sits on the dataclass, rides into the row, and not one code path consults it before a result gets judged. The gate that actually runs is in analyze_experiment, which bails out only when an arm holds fewer than five outcomes, and its SQL counterpart experiment_ready_for_analysis takes p_min_per_group INTEGER DEFAULT 5. So the brake is five, not twenty. The twenty is decoration, and it is the kind that looks clean and fails quietly.

Five per arm cannot carry the conclusion the code is willing to draw from it. Work the table: five sessions in each branch, chi-square at alpha 0.05, and exactly two of the thirty-six possible outcomes clear significance. Five successes against zero. Zero against five. Both land at p equal to 0.011, and every other cell in that table is a null before the experiment starts. At this size the test is not measuring an effect. It is asking whether the two arms disagreed about every single session, which is not the question I meant to ask.

Chi-square is the wrong instrument here anyway. It wants expected cell counts of about five or more, and a two-by-two table built from five observations per arm cannot give it that. Fisher's exact test handles tables this small and the swap is one line. Raising the gate to the twenty already written on the object does not rescue it either — at twenty per arm the smallest difference the test can see is still around thirty percentage points.

Here is the scale I should have worked out before writing any of it. A ten point improvement in success rate, 0.50 to 0.60, at eighty percent power and the same alpha, needs roughly 194 sessions per arm. Five points needs about 782. Those are the numbers at which the words sitting in recommendation — ADOPT, REJECT — mean what they claim. Ten points is a substantial win for a context change. This layer would need forty times the evidence it currently demands to notice one.

There is a related exposure in how many tests run at once. The generator can open five order hypotheses and three inclusion hypotheses for a single task type. Eight independent tests at alpha 0.05 carry roughly a 34 percent chance that at least one clears significance by luck alone, and the one that clears is the one that gets promoted into every future session's context. A Bonferroni correction, or simply refusing to run more than one live experiment per task type at a time, is the cheap fix.

One thing the design gets right by accident of structure is worth crediting, because it is the error most A/B systems make. analyze_experiment completes the experiment as soon as it evaluates it. There is no path that looks at the data, finds nothing, and looks again next week. Repeated peeking at an accumulating result is how a nominal five percent false positive rate becomes twenty or thirty percent in practice, and this loop cannot do it: one look, then the experiment closes. The cost of that virtue is that the single look happens at the earliest moment it is permitted, which is also the moment with the least evidence behind it.

Put those two facts side by side and it is worse than slow learning. A completed experiment never reopens; _complete_experiment writes status completed and nothing anywhere sets it back to active. At five per arm almost every outcome is INCONCLUSIVE by construction. So a context change that genuinely helps gets one underpowered look, comes back as no significant difference because it could hardly come back as anything else, and is closed on that basis permanently. I built the threshold to stop the system adopting noise. What it actually does is retire real improvements the test was never equipped to detect, and there is no route back to retry one.

The exploration rate decides where this lands hardest, and it picks the worst place. _get_task_uncertainty returns maximum uncertainty for any task type with fewer than ten recorded sessions, which pushes the exploration probability straight to its 0.4 ceiling, and that count is scoped per organization. A small organization therefore runs the largest share of its sessions as trials while generating the fewest sessions to finish any of them with. It experiments hardest precisely where five outcomes per arm takes longest to accumulate, and every one of those trials still closes after a single look.

The stored result hides that rather than surfacing it. effect_size goes into the results JSON as a bare difference between two success rates, with no interval around it, so a gap measured on five sessions per arm is recorded in the same shape, and reads later with the same authority, as one measured on five hundred.

Waiting costs something. Bad ideas stay alive longer, good ones take more sessions before they become default behavior, and some combinations never get tested at all because the generator cut them off early. That pressure is almost certainly how the number ended up at five: a gate that low returns verdicts fast, and returning verdicts fast is the entire problem with it.

The operational states are active, completed and cancelled. An active experiment can still receive assignments and a completed one can store results, but the useful one is cancelled: it stops shaping sessions without pretending it ever reached a conclusion. The view v_experiment_results puts treatment counts, control counts, status, and timestamps together so the operator can see whether a run is still gathering evidence or ready for judgment.

That is the trade I made for agent memory and retrieval: learn slower, record the branch, measure after work, and require enough samples before future sessions change. Three of those four are built and working. The fourth is a number I wrote on an object and never wired up, and writing this post is what made me look.

The rule I started with still holds. A learning system that changes future inputs has to earn that change with outcomes, or it will preserve coincidence as policy. Randomization is what makes an outcome mean anything. The sample threshold is what stops it being noise. I built the first one properly. The second I wrote down on a dataclass and never wired up, so the system will go on drawing conclusions from five sessions and filing them with the same confidence it would give five hundred.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

The Matte Learns Only Inside the Band

Daniel Romitelli — Tue, 18 Aug 2026 06:21:06 +0000

A bad cutout rarely announces itself as a bad cutout. The car lands on a new backdrop, the paint looks clean, then a thin piece is gone. An antenna. A tire lip. The dark seam under a rocker panel. The complaint that comes back is never technical. The vehicle looks wrong.

I wanted the last correction stage to fix fuzzy edges without handing it the whole car to rewrite. That sounds like a small distinction. It stops being small the first time a model improves one boundary and quietly damages another. So the rule is physical. Edit the uncertain strip. Leave the settled area alone.

This is Part 2. Part 1, "Negative Space Is a Label", was about supervision: what the pixels beside an object teach a model, and why a shadow touching a tire has to be labeled as evidence against foreground. This one moves from training to runtime. A mask already exists. Where is a learned stage allowed to act?

1. The contract lives in the band

CarSegNet is the research implementation here. Its pipeline module splits the route by media type, and the docstring says the design more clearly than any diagram I could draw after the fact.

Stills run SAM 3 text concept, then NSJ alpha, then composite. A detector box prompt and a depth prior are optional inputs. Video runs SAM 3.1 multiplex propagation, per-frame NSJ with temporal handling, a depth-parallax plate, composite, encode.

The list matters less than the handoff. SAM gives a semantic prior. NSJ receives a trimap band. The compositor receives a matte only after the prior and the refiner have each done bounded work.

flowchart TD
image[Vehicle Image]
segment[Concept Mask]
trimap[Trimap Band]
refiner[NSJ Alpha Refiner]
depth[Depth Prior]
composite[Showroom Composite]
frozen[Prior Frozen Outside Band]
image --> segment
segment --> trimap
trimap --> refiner
image --> depth
depth --> refiner
refiner --> composite
segment -.-> frozen
frozen --> composite

The diagram is a contract. It is not a model zoo. The refiner edits the uncertain strip. The semantic prior owns the rest of the frame.

Models build lazily. A pure recomposite run against cached mattes never pays to load a large segmentation checkpoint. The cost lands on whichever execution path needs that model first. I take that trade. Cached matte work should stay cheap and inspectable, and loading every model for every run hides an orchestration problem behind hardware capacity.

2. The band uses image disagreement

The alpha refiner lives in its own module, and the one-line file description is the entire design: trimap, band crop, NSJ, or deterministic fallback.

A plain band can be built from the prior alone. Take the hard mask, grow it, shrink it, call the ring unknown. That catches soft contour error. It fails on a prior that is confidently wrong.

The comment in the band builder names that case in capitals: WRONG AND CONFIDENT. Both halves of a morphological band are functions of the prior, so a prior with no doubt produces no band at all. Fill in a wheel opening and the hole becomes confident foreground. Drop a roof antenna and those pixels become confident background. Either way the missing area can sit nowhere near an iso-contour, and a morphology-only band never asks the refiner to look there.

So CarSegNet adds an image term. Where the photograph shows strong structure and the prior shows nothing happening, that disagreement opens the band. The photograph says edge. The mask says flat. That argument is worth examining.

It stays bounded. The search is restricted to the subject's own neighborhood, the edge criterion is relative to the image instead of a fixed number, and the band has a ceiling it cannot cross.

Those limits cost something. Widen the neighborhood and foliage, fence lines, or lot texture start lighting up the image term. Tighten it and the antenna case stays frozen. The ceiling is the one I would defend hardest. It stops a local repair path from turning into a full-frame request, which means a badly wrong prior has to be rejected upstream instead of handed to the refiner as though it were close.

That is the transferable part. A learned correction needs a declared edit domain. Here the domain is a trimap band carrying a disagreement term, so confident prior mistakes get a chance to be examined near the vehicle.

3. Full-frame alpha drifts quietly

The tempting design is simpler. Segmentation produces a rough mask, a neural refiner outputs a full alpha matte, compositing uses that alpha. Fewer moving parts. Far more hidden authority.

A full-frame output can fix a tire edge and move a roofline in the same pass. It can smooth a window halo and erase a mirror. The bad part is that such a trade can improve an averaged score, because most pixels in a vehicle photograph are easy background or easy paint. The listing still fails at the one boundary a buyer looks at.

NSJ is small. Size is not the safety property, and I want to be exact about that, because small models get described as safe all the time. A small unconstrained model can still damage broad vehicle topology. The constraint around the output does the work.

The completed checkpoint audit is what stopped me from reading boundary gain as deployment clearance. Boundary scores improved with the trained checkpoint. The same audit showed the model almost never recovered enclosed openings. The outline got better. The topology did not.

That changes what the stage is allowed to claim. NSJ is an edge repair component. It is not evidence that the system understands window holes, wheel openings, cabins, or glass ownership. Those errors are structural, so they need separate gates.

The fallback path has the same shape. With no trained checkpoint the refiner runs a deterministic guided-filter route instead of stopping the pipeline at missing weights. The system stays runnable on day one. The tradeoff is visible in what each path can actually do. Guided filtering cleans a local alpha transition. Learned interior reasoning waits for trained weights and a passing topology gate.

I cut a synthetic hard-case metric table out of this argument while drafting it. The numbers were useful during development. Without the full setup sitting next to them they were decoration, and they pulled attention away from the contract. The code path is the stronger evidence: band crop, NSJ or fallback, guided filter, prior preserved outside the declared edit area.

4. A guard catches the silent sliver

The same module that routes the models also rejects one specific mask failure before it reaches compositing. There is a dedicated error type for mattes that are fragments rather than vehicles.

The default came out of measurement rather than taste. Sliver mattes cluster far below anything a real vehicle produces under the serving prompt. Legitimate vehicles start well above that cluster and run up to most of the frame. Between the two populations sits a wide empty band, and the default lives inside it. That gap is the only reason I trust a single scalar here.

It is a different kind of trust region than the band. The trimap band limits where alpha refinement may edit. The subject-fraction check limits which priors are allowed into the rest of the route at all.

The asymmetry is what justifies it. A miss is visible and recoverable. A tiny foreground sliver composites silently and looks like a strange crop, so it gets its own exception type while still subclassing the pipeline error every caller already handles.

There is a cost. Small distant vehicles, odd crops, and dealer photos with unusual framing can trip it. The threshold moves through configuration and can be switched off entirely. I keep the default because the measured gap is wide for this serving path, and in a media workflow a silent sliver is worse than a loud miss.

5. Glass got a smaller claim

Glass is where boundary language gets overloaded. A side window is transparent, reflective, tinted, and part of the vehicle shell, depending on which question you are asking. One binary mask answering all of them is false authority.

The bootstrap model predicts a glass region of interest from a single RGB vehicle image. Its scope is deliberately narrow: find the reviewed glass region for later work. The module's constants name the capabilities it does not have. That is useful friction. Future code has to cross a named boundary before it can pretend a region mask solved reflection removal.

The follow-on model has a stricter data contract than ordinary open and closed window classification, and the pilot captures do not satisfy it yet. Screening material is not the same as clearing it to train. The tooling says so in its own docstring, and the pilot is still marked not training ready.

The part that belongs in a post about the trimap band is the decode rule. The learned model answers a narrow question inside reviewed bounds. Every pixel outside that region is copied from the input, exactly. Same design pressure, different subsystem.

6. The composite makes excess authority visible

The final judgement happens after the vehicle lands on a plate. The compositor harmonises color in LAB, matches defocus, adds a contact shadow, and applies light wrap. Those operations can make a correct cutout sit convincingly in a new scene. None of them restores a mirror that alpha refinement erased.

This is where vague model authority turns expensive. Old asphalt left under a tire means contact shadow competes with captured ground. A halo that survives around a window means light wrap makes the bad edge look intentional. An antenna dropped before composition means every downstream operation is polishing a false matte.

Video adds one more constraint. The pipeline uses a depth-parallax plate before compose and encode, so there is one still plate with local motion rather than a generated background per frame. Camera freedom drops. Repeatability goes up. Same matte, same plate, same depth map, same frames again.

The selftest keeps model availability separate from pipeline correctness. It runs on CPU with no network, no GPU, and no model downloads, exercising image and video runs against stubbed backends. Cached mattes, compositing, encoding, and quality reports can fail or pass without waiting on gated weights.

That split matters for what comes next. Part 3 goes inside the vehicle, where the opening stage stays disabled until calibration and a locked topology gate pass. The problem there is harder than fuzz at the edge. Subtract alpha inside a guarded interior region while preserving the filled outer silhouette.

A vision system gets safer when every learned edit carries a boundary condition.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

When Missing Privacy Evidence Becomes Zero

Daniel Romitelli — Sun, 16 Aug 2026 09:38:02 +0000

At the end of a training run, the privacy measurement existed. The client logged it. Then the return value dropped it.

That small omission changed the meaning of the entire system. Downstream code treated missing evidence as zero privacy cost, an accountant displayed a clean budget it had never been told to advance, and export proceeded because a file existed. Nothing crashed. Every component looked locally reasonable. The false conclusion appeared only when I traced one fact across all of them.

1. The measurement that vanished

The system uses federated learning, so each client trains locally and sends an update to an aggregator. The client applies differential privacy during training through Opacus. At the end of each epoch, data_ingest/fl_client.py asks the privacy engine for epsilon, the numerical privacy-loss bound at a chosen delta:

epsilon = privacy_engine.get_epsilon(DP_TARGET_DELTA)

The value is real enough to appear in the log. It is absent from the value returned to the server:

return self.get_parameters(config={}), n_samples, {"train_loss": avg_loss}

The aggregator in fl_aggregator/strategy.py expects a different contract. It looks for an epsilon metric and supplies a default when the key is missing:

fit_res.metrics.get("epsilon", 0.0)

That default is the decisive line. A measured value of zero and an unobserved value are different facts. The first can support a decision. The second means the decision lacks an input. Converting both to the same floating-point number is an epistemic type error: the program has collapsed what it knows with what it failed to learn.

The neighboring loss metric reveals the same boundary mismatch. The client returns train_loss; the strategy asks for loss. Both look plausible in isolation, so ordinary local review can miss the disagreement. The problem lives between modules, in the meaning of their shared record.

There is a second trap. Even if every client returned epsilon, taking the arithmetic mean of those values would be telemetry, not necessarily a valid global privacy composition. Different clients can have different sampling rates, step counts, and exposure histories. An average can hide the most exposed participant. The server needs the accounting events required by its threat model, not a comforting aggregate of already-composed answers.

2. Four correct components can imply a false system

The server also creates a PrivacyAccountant. It can use a Privacy Loss Distribution backend, called PLD in the code, fall back to a Renyi Differential Privacy accountant, called RDP, record total steps, compute epsilon, and report whether the configured limit has been reached. Its unit tests call step() and verify that the number moves.

The running application does not call that method. A source search finds PrivacyAccountant.step() in the accountant tests, but no production caller advances the server instance created in fl_aggregator/server.py. The status endpoint can therefore report an accountant value of zero even after local training has consumed privacy budget. The display is reading its object correctly. The object was never given the events that make its answer meaningful.

The export path then introduces a third independent truth. When the training server exits, a finally block invokes the post-training pipeline:

finally:
    _update_state(fl_running=False, current_round=num_rounds)
    FL_CURRENT_ROUND.set(num_rounds)
    logger.info("Flower server finished")

    _post_training_pipeline()

That pipeline reconstructs the aggregate model and writes an Open Neural Network Exchange file, called ONNX in the code. The download endpoint serves the latest model if a path exists. The zero-knowledge export endpoint likewise checks for an ONNX file before it starts compilation. Neither route asks whether privacy observations were complete, whether the accountant corresponds to this run, or whether the recorded budget was still valid at the moment of export.

flowchart TB
    A["Client computes epsilon"] --> B["Returns train_loss only"]
    B --> C["Aggregator substitutes 0.0"]
    C --> D["ONNX export"]
    D --> E["Zero-knowledge artifacts"]
    P["Server privacy accountant"] -. "not advanced" .-> D
    G["Required eligibility gate"] -. "missing" .-> D

This is why component inventories are weak architecture evidence. The codebase contains private training, an accountant, an export service, and a proof toolchain. Listing those nouns makes the design sound complete. Following one decision from observation to release shows that their authority never converges.

3. A computation proof is not a training-history proof

The exporter in fl_aggregator/zkml/exporter.py is substantial. It can produce model.onnx, calibration input, circuit settings, a compiled circuit, proving and verification keys, and a verifier contract. It uses EZKL, a zero-knowledge proof toolchain for machine-learning models.

Those outputs answer an important question: can a verifier check the statement encoded by this circuit and its public inputs? They do not automatically answer a different question: did the model enter the circuit through an observed training run whose privacy events were complete and within policy?

I call that second property export eligibility. It is deliberately narrower than general provenance. Provenance can tell me where an object came from. Eligibility must decide whether these exact bytes may cross a boundary now.

The distinction matters because proof systems are literal. A valid proof says that the encoded relation held. It does not inherit facts that were never encoded or bound to the relation. If the circuit digest is unrelated to the training round, or the privacy state is unrelated to the exported model digest, the system has several valid facts without a valid conjunction.

The export predicate I want is explicit:

exportable =
    privacy evidence is complete
    AND the authoritative accountant is within its configured limit
    AND the accountant snapshot names the completed training round
    AND the model digest names the bytes sent to circuit generation
    AND every required artifact exists and matches its recorded digest

The first clause is essential. “Not exhausted” is insufficient when the accountant was never advanced. Unknown must fail closed before the numerical comparison is even allowed to run.

4. The missing object is an atomic export manifest

The repair is not another dashboard field. It is a small, immutable manifest emitted at the only place that can see the complete decision. The current code does not implement this object; this is the contract the audit derives.

Field	What it binds	Reject export when
`schema_version`	The parser to the contract version	The version is unsupported
`training_run_id` and `federated_round`	The files to one completed aggregate	Either identity is absent or mutable
`privacy_state`	Observation status, such as measured, unobserved, or invalid	The state is not measured
`epsilon`, `epsilon_limit`, and `delta`	The measured loss to the policy used for the decision	Values are missing, non-finite, or over limit
`accounting_backend` and `accounted_steps`	The result to its composition method and event count	The backend failed or the event count disagrees with the run
`model_digest`	The decision to the exact ONNX bytes	Recomputed bytes differ
`circuit_digest`	The model export to the compiled relation	The circuit is missing or does not match
`proving_key_digest` and `verifying_key_digest`	The bundle to its cryptographic material	Either key is absent or changed
`source_commit`	The run to a reviewable source state	The source identity is unavailable
`exported_at_utc`	The snapshot to a specific release event	The timestamp is absent or malformed

The write order is part of the contract. First, the server freezes a snapshot containing the round identity and authoritative accounting state. It refuses missing client events instead of substituting zero. It composes those events according to the declared accounting model and stops if the state is unobserved, inconsistent, or over budget.

Only then does it write the ONNX model into a staging directory, compute the content digest, and give those exact bytes to circuit generation. After compilation, it digests the circuit and key material. The manifest is written last. A single rename promotes the staging directory to its final run-specific location. The staging and final directories must share a filesystem if the rename is expected to be atomic.

The serving rules become simple. “Latest” means the newest completed bundle, not the newest loose file. Download rechecks the model digest against the manifest. Proof export accepts a run identity and reads the model named by that bundle. A partial directory, stale key, missing privacy event, or mismatched digest is unavailable by construction.

This design also changes the interface between client and server. A privacy-enabled client must return structured accounting evidence, including the step count and parameters needed by the chosen composition rule. The aggregator must reject an update that claims private training but omits that evidence. Diagnostic client epsilon can still be recorded, but it cannot silently become the server’s authorization rule through an average.

5. The real novelty is the conjunction

The earlier design idea was to preserve context as it crosses a boundary. This failure is different. No individual context object was missing. The client had one truth, the aggregator inferred another, the accountant held a third, and the exporter acted on a fourth. The novel object is the conjunction that none of them could assert alone.

That is also why the bug survived superficially strong evidence. The client log showed epsilon. The accountant endpoint returned a structured report. The model file existed. The proof directory contained cryptographic artifacts. Each observation was true, yet the sentence assembled from them was false: this model is eligible for export under this privacy history.

The correction is a general systems rule. Never let absence inhabit the same value as success. Never let a dashboard object become authoritative unless the events that advance it are part of the production path. Never let file existence stand in for a completed decision. When several subsystems jointly authorize an irreversible boundary crossing, make their conjunction a first-class object and bind it to the exact output bytes.

The hardest code-review findings are often not broken functions. They are false theorems assembled from locally correct premises. Finding one requires reading the gaps between modules as carefully as the modules themselves.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

A Context Object Should Carry Its Receipt

Daniel Romitelli — Sun, 16 Aug 2026 02:58:41 +0000

A stored fact can be wrong in a quiet way. The answer still reads clean. A preference from an old exchange gets reused, the message goes out with confidence, and later nobody can tell why that detail was allowed back into the result.

That is the failure I built around. When a system returns remembered material, the caller needs the text plus the reason it passed the reuse check. A log line found after the action is weak evidence. The object that leaves the memory service has to carry the admission record with it.

1. Keep the outside surface small

This is the pattern I used in Holographic, Law-Bound Memory (HLM), a stand-alone memory brain outside application code. The README describes public Application Programming Interface (API) routes under /api/brain/*, with internal /api/v1/* services behind that layer.

The outside shape is intentionally thin: register an agent, write a fact, build a capsule. The Python Software Development Kit (SDK) in sdks/python/hlm_sdk/client.py shows the boundary without exposing table names or policy code:

import httpx

class HLMClient:
 def __init__(self, base_url: str, token: str | None = None):
 self.base_url = base_url.rstrip("/")
 self._client = httpx.AsyncClient(headers={"Authorization": f"Bearer {token}"} if token else None)

 async def register_agent(self, name: str):
 r = await self._client.post(f"{self.base_url}/api/brain/agents/register", json={"name": name})
 r.raise_for_status
 return r.json

 async def write_fact(self, text: str, tags: list[str] | None = None, selectors: list[str] | None = None):
 r = await self._client.post(f"{self.base_url}/api/brain/memory/facts",
 json={"text": text, "tags": tags or [], "selectors": selectors or []})
 r.raise_for_status
 return r.json

 async def build_capsule(self, query: str, budget_tokens: int = 2048):
 r = await self._client.post(f"{self.base_url}/api/brain/context/capsule",
 json={"query": query, "budget_tokens": budget_tokens})
 r.raise_for_status
 return r.json

The TypeScript client in sdks/node/src/index.ts exposes the same calls as registerAgent, writeFact, and buildCapsule. That costs me a compatibility surface at the gateway. I accept the cost because admission policy in every consumer becomes drift. One caller skips a selector, another copies an old threshold, a third treats a nearby match as enough. Centralizing the decision gives the service a place to say yes or rebuild before the application acts.

2. Write facts with handles the service can check

A plain text memory is easy to save. It gives retrieval very little to inspect later. HLM writes each fact with tags, selectors, and an optional tenant field so the service has decision axes before a query shows up.

The write model in services/memory/app/main.py is small:

from typing import List
from pydantic import BaseModel, Field

class FactIn(BaseModel):
 text: str
 tags: List[str] = Field(default_factory=list)
 selectors: List[str] = Field(default_factory=list)
 tenant_id: str | None = None

That class feeds create_fact. The row itself lands in brain_facts. Two more writes follow against the same fact id, one for facets and one for predicates. The response returns the new id with both sets attached.

generate_facets has two paths in the current scaffold. A known selector value produces a specific facet row. Anything empty or unmatched falls back to a general facet, built from the first 256 characters of the text with the token count capped at 64. Those are constants in the code rather than performance claims. What they show is the shape: retrieval sees more than a blob of prose.

generate_predicates turns selector strings into one predicate joined with AND, swapping the first colon for =. That is rough. It is also enough, because the fact now leaves the write path carrying handles a machine can check. The tradeoff lands on the writer. A caller that sends empty selectors can still store text, but later selection has fewer axes to test.

3. Decide admission before capsule assembly

The reuse service is Conformal-Causal Reuse (CCR). Its request model in services/ccr/app/main.py carries the cache key, artifact type, selectors, and optional numeric controls:

from fastapi import FastAPI
from pydantic import BaseModel, Field

app = FastAPI(title="HLM CCR", version="0.1.0")

class CCRRequest(BaseModel):
 tenant_id: str | None = None
 cache_key: str
 artifact_type: str = "resume_kit"
 selectors: list[str] = Field(default_factory=list)
 similarity: float | None = None
 tau: float | None = None

The rule in reuse_or_rebuild is direct: a hit requires similarity > tau and the required selector kinds stakeholder, time, and channel to be present. Missing one of those kinds makes the service return rebuild. When the request omits values, the code uses 0.9 for similarity and 0.8 for tau. Those numbers are defaults in the function. They are not measured latency, quality, or production calibration.

The response includes decision, tau, similarity, and causal_ok. All four travel with the answer. Accepted material can name the rule that admitted it, and a rejection arrives as a rebuild decision instead of a silent empty match.

Calibration stays beside the same service. CalibIn accepts selector, similarity, and span_error; update_calibration computes a rounded tau and clamps it between 0.5 and 0.95. I kept that logic near the decision endpoint because threshold repair separated from the admission rule becomes another place for drift. The cost is coupling. CCR owns both the current decision and the local adjustment path.

4. Build the capsule with provenance attached

The orchestrator joins the pieces. In services/orchestrator/app/main.py, /api/v1/capsule derives selectors from the query and posts them to CCR. What comes back shapes the capsule: content, confidence values, reasoning metadata, and a proof value.

The intended object is signed context rather than an anonymous bag of nearest neighbors. The current branch is still a scaffold. Episode writes return receipt: "merkle:demo", and the orchestrator repeats that same demo value in its local capsule response. The slot is real and the hardening is unfinished, so the caveat stays in the design.

The four-step path is the engineering pattern:

flowchart TD
 factWrite[Fact write] --> governedReuse[Reuse decision]
 governedReuse --> capsuleBuild[Capsule build]
 capsuleBuild --> proofReceipt[Proof receipt]

services/gateway/app/main.py contains merkle_root(items). It hashes the item strings and folds pairs until one hash is left, duplicating the last leaf when the count is odd. The result gets a merkle: prefix. The gateway is the right home for it, because the external object is formed there. Downstream code should receive a single object holding the selected material, the CCR decision fields, and a provenance value it can store or compare later.

The architecture document names this Proof-of-Context (PoC): Merkle roots over snapshot, version, tau, model, and ids. The label matters less than the placement. If applications learn to consume loose context first, provenance turns into a retrofit. Retrofitted evidence is usually optional. Optional evidence disappears under deadline pressure.

5. Preserve the same object across streamed updates

Memory work does not always end at the first capsule. services/orchestrator/app/main.py has a local loop that yields Server-Sent Events (SSE) through a StreamingResponse; each packet includes a generated packet_id, summary fields, next actions, and a timestamp before the loop pauses. The gateway forwards this through /api/brain/context/stream. The README and architecture notes describe the larger outbox path with leases, backoff, a dead-letter queue (DLQ), and a resume stream over SSE or WebSockets (WS). The current code handles the visible stream contract; the documented shape says lineage has to move with later packets as well. That adds overhead compared with returning an array from a nearest-neighbor endpoint, but a resumed update without the original admission data is just another loose event.

6. Own the memory lifecycle

This design buys safer reuse by moving work into the memory service. Writers must send useful selectors. The gateway becomes stricter. The service has to keep admission metadata and provenance beside the content from write, through CCR, into capsule creation and streaming. I prefer that pressure inside HLM over spreading half-copied rules through applications, because systems that remember should expose the conditions under which memory became usable.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

When an Agent Tool Call Becomes a Real Request

Daniel Romitelli — Sun, 16 Aug 2026 02:34:26 +0000

1. Nothing crashed

A recruiter asked for candidates published since a date. The assistant answered with confidence, and the count it gave looked plausible.

That was the bug. The search ran, results came back, and nothing crashed. The date constraint never survived the trip from the model’s tool call to the database query, so the assistant counted the wrong universe and spoke as if it had done the exact request.

This is production code, not a prototype. The path runs through Azure Artificial Intelligence (AI) Foundry, then a Model Context Protocol (MCP) bridge, then my application service over Hypertext Transfer Protocol (HTTP). The bridge turns a model tool call into an Application Programming Interface (API) request.

The boundary accepts whatever the model emitted and translates it into the service contract. When translation changes the meaning, it should stop.

2. The bridge is the caller now

The path is short. Foundry asks for a tool. The MCP bridge receives the call, parses JavaScript Object Notation (JSON), switches on the tool name, builds a request body, calls the application, then returns tool output through submitToolOutputs.

The model did not emit typed parameters for the date. It emitted an Open Data Protocol (OData) filter string. The application endpoint does not speak OData.

flowchart TD
recruiter[Recruiter]
agent[Azure AI Foundry Agent]
run[Foundry Run]
toolCall[Tool Call With OData Filter String]
bridge[MCP Bridge]
parsedArgs[Parsed Arguments And Extracted Date]
api[Application API]
typedRequest[HTTP Request With Typed Parameters]
results[Search Results And Count]
toolOutputs[Submit Tool Outputs]
answer[Answer With Count]
recruiter --> agent
agent --> run
run --> toolCall
toolCall --> bridge
bridge --> parsedArgs
parsedArgs --> typedRequest
typedRequest --> api
api --> results
results --> bridge
bridge --> toolOutputs
toolOutputs --> run
run --> agent
agent --> answer
answer --> recruiter

The cause sits in mcp-servers/azure-agent-mcp/src/services/azure-agent-client.ts, inside executeToolCall. For the candidate search tool, the bridge tries to rescue the date with this regex: args.filter.match(/date_published\s+ge\s+(\d{4}-\d{2}-\d{2})/). If it matches, dateFilter = dateMatch[1].

Then the bridge sends a body shaped for the application: query defaults to *, location falls back from one argument name to another, limit is capped with Math.min(Number(args.top) || Number(args.limit) || 20, 50), the raw filter is passed through, and the extracted date goes out as a typed field when parsing worked.

This is the normal shape of the bug. The regex is a parser for a query language the model was never constrained to emit correctly.

If the model phrases the filter differently, the match misses. dateFilter stays null. The request still has query text, a limit, count inclusion, maybe a raw filter string, and enough shape to look legitimate.

This is what goes over the wire. Values anonymized, structure untouched:

{
  "query": "wealth advisor",
  "location": null,
  "designations": null,
  "min_experience": null,
  "min_aum": null,
  "min_production": null,
  "remote_only": null,
  "limit": 20,
  "include_count": true,
  "filter": "date_published gt 2026-01-14T00:00:00Z",
  "date_from": null
}

Nothing about that body looks wrong. It is well formed. It validates. Most of those nulls are honest, because nobody asked for a location or a minimum book of business.

Read the last two fields together, though. filter says a date bound was requested. date_from says none was applied. The pattern accepts ge and the model wrote gt, so a single comparison operator is the entire distance between a bounded search and an unbounded one. Quoting the date, or writing date_published/gt, or putting the clause second in a compound filter, all land in the same place.

Those two fields contradict each other inside one request body, and nothing on either side compares them.

A dropped filter does not throw. It answers.

3. Partial answers have a silence cost

The same shape appears inside app/agents/orchestrator.py. AgentOrchestrator.process routes the query, runs the primary agent, starts extra agents in parallel, then combines whatever comes back. It also carries a timing dictionary with router_ms, agent_ms, and total_ms.

The parallel branch uses asyncio.gather(*secondary_tasks, return_exceptions=True). Only successful AgentResponse objects enter secondary_results through secondary_results.append(resp.to_dict).

return_exceptions=True buys a partial answer instead of no answer. It costs silence: nothing logs the exception, so an agent that fails every time looks identical to one that had nothing to add.

A secondary agent starts raising on every call. Responses keep arriving, so no alert fires. total_ms gets better, because a task that raises immediately finishes faster than one that does real work. The system looks healthier as it loses coverage.

So you go check the stats. get_agent_stats reports configured, and for each agent a name, a tools_count, and a model. That is the configuration, all of it. Nothing counts invocations. Nothing counts failures. An agent that has not returned a usable response in a week reports exactly what it reported the day it worked.

The filter hides one more distinction. isinstance(resp, AgentResponse) and resp.success discards a raised exception and a returned-but-unsuccessful response into the same place. A crash and a considered "I have nothing useful here" are the same event downstream.

That is the same bug as the dropped date. Partial execution is fine when the system can say which part was partial. Without that, the fallback hides the failure.

4. The bridge is admin surface

The MCP server in mcp-servers/azure-agent-mcp/src/index.ts registers 19 tools. They cover agent lifecycle, thread lifecycle, and index inspection. A connected client can administer agents, inspect search infrastructure, execute workflows, and ask questions through the same bridge.

That changes how I read tool schemas. They are policy, not help text.

Fallbacks make agents feel forgiving. args.location || args.state || null is a small decision about equivalence. Some of those choices are harmless. Others erase meaning.

The date case is the one that changes which rows come back. A user asked for a bounded set. If parsing fails, widening the set answers a different question from the one asked.

5. The contract the edge should enforce

Two kinds of argument arrive at that switch statement, and they deserve different treatment.

Typed fields the application already understands: location, min_experience, limit, remote_only. Each has a contract. min_experience is a number or it is absent, and both states are unambiguous.

Then the free-form string. filter is a sentence the model composed in a query language nothing on my side implements. It has no contract at all. The bridge tries to recover intent from it with a regular expression. A regular expression is a guess about what the model will write.

State the rule plainly. A constraint that changes which rows come back must arrive in a typed field the application understands. If it arrives only as a string, and that string cannot be parsed, the request stops. An error tool output works. A clarification back to the model works. Sending the query anyway, minus the constraint, does not.

What decides whether a rule applies is whether the constraint changes the result set. Misparse a sort order and the same rows come back in a worse sequence. Somebody reports that inside a day. Misparse a date bound and different rows come back under a count that still reads as reasonable, and the report never arrives, because from the outside there is nothing to report.

The bridge already has everything it needs to make that call. args.filter being present is the signal that a scoped request exists. The moment extraction fails to produce a typed value from it, the request body is known to be wrong before it is sent.

6. Reject almost-right input

The fix I take from this bug is narrow. Strings are allowed at the edge when the upstream interface allowed strings. A string that encodes a result-changing constraint needs visible parse failure.

For this date constraint, the bridge already knows a scoped request exists because args.filter exists. If extraction cannot produce the typed date, the safe behavior is an error tool output or a clarification request. That is less magical. Good.

Strict parsing annoys users sooner. It also keeps scope honest. For a date-bounded count, a visible stop beats a wide query dressed as precision.

7. Catching the next one

Start with the log line that is already there:

if (dateMatch) {
  dateFilter = dateMatch[1];
  console.error(`[AzureAgent] Extracted date filter: ${dateFilter}`);
}

There is no else. The parse that worked writes a line. The parse that failed writes nothing. That is exactly backwards, and it is why this ran in production without leaving a trace. A successful parse is the one outcome nobody needs told, and it is the only one on record.

Three things I want at that boundary.

Log the miss, and log the input that caused it. When args.filter is present and the pattern does not match, write the filter string verbatim. Do that for a week and you stop guessing at which dialects the model emits, because you have the list. My regex was built for ge and the wild data was going to include gt whether I planned for it or not.

Count the unconstrained fallbacks. One counter, incremented whenever a request goes out with a filter string present and its corresponding typed field null. Flat at zero means the parser is keeping up. Climbing means the assistant is answering a wider question than the one it was asked, and the number tells you how often. This is the signal that finds the bug before a person does.

Assert on the shape of the request. filter present with date_from null is a contradiction sitting in the body the bridge just built, catchable with one conditional before the call goes out. Checking the answer instead is hopeless here, because plausible output is the entire failure mode.

Then look for the same pattern elsewhere in the switch. A few cases down the same switch, get_recent_candidates builds a search from query, limit, and include_count, and carries this comment:

// Note: date filtering would need to be added to the backend

A tool whose name promises recency, sending no date at all. Somebody saw it, wrote it down, and shipped it anyway. That comment is a detection mechanism that only fires when a human happens to open the file.

When a model calls your API, you have a new caller that does not read documentation, does not honor types, and produces syntactically valid arguments that mean the wrong thing.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

Algorithmic Matching Without an LLM

Daniel Romitelli — Wed, 12 Aug 2026 22:00:03 +0000

A recruiter opens a search result and sees a candidate at the top. The score says the match is strong. The card has to say why: the credential matched, the location fit, a production figure landed in a high band, availability read as urgent, and one mismatch got named instead of buried.

No language model wrote any of that. Nothing generative runs in the ranking path or the explanation path. The card text falls out of the same arithmetic that produced the order, computed in one pass, and it costs no extra call.

This describes work I did on a client system. The numbers and factor names here are generic. Take the design, not the numbers.

1. A score is not a reason

Two designs fail here, and they fail in opposite directions.

A bare ranked list makes the recruiter audit the system by hand. Candidate three sits above candidate seven and nothing on screen says whether the production number, the designation, or the commute drove it. The user either trusts the order without evidence or opens both profiles and reconstructs the comparison manually, which is the work the ranking was supposed to absorb.

The second design is the one most teams reach for now. Rank first, then hand the top results to a language model and ask it to write a sentence about each. The output reads well and carries no obligation to be true. Nothing binds the sentence to the arithmetic, so the model can credit a factor that contributed nothing while omitting the one that moved the candidate twenty places. The recruiter cannot tell the difference, because a fluent wrong answer and a fluent right answer look identical on a card.

That failure is worse than a missing explanation. A blank space invites a question. A confident sentence closes one.

There is a third option, and it is older than both. The scoring code already knows why. Make it say so.

2. Evidence as a return value

The scoring path holds every fact the card needs: parsed requirements, candidate fields, numeric attributes, credentials, availability text, and the mismatch checks. It knows them at the moment it assigns weight. Asking a later stage to recover that from a float is throwing the answer away and paying a model to guess it back.

So each feature returns two things. What it contributed, and what it saw.

FeatureResult {
  name
  weight
  score_delta
  evidence
  positive_reasons
  concerns
}

final_score = sum(r.weight * r.score_delta for r in feature_results)
card_payload = {
  highlights: collect(r.positive_reasons),
  concerns:   collect(r.concerns),
  evidence:   collect(r.evidence)
}

The number and the text come off one set of objects in a single pass. There is no second interpretation layer to drift, because there is no second layer.

The cost is writing freedom. Card copy now moves only when scoring moves, and a product request to phrase something differently becomes a change inside the feature that earned it. I take that trade, because the alternative buys flexible prose by giving up any guarantee the prose is about this candidate.

3. What each feature is allowed to claim

The design earns its value by being specific about what each component may assert. A vague contract produces vague cards.

feature	compares	what it may put on the card
keyword overlap	requirement terms against location, role, employer, credentials	the terms that actually matched
percentile ranking	numeric fields such as portfolio size and recent production	that a value sits in a high band within the returned set
credential detection	required designations against held designations	exact hit, partial hit, or absent
availability parsing	timing language in candidate text	an urgency level, not a date it cannot prove
concern flags	requirements the candidate misses	the specific friction, next to the positives

Every row is set arithmetic, a comparison, or a rank. Nothing on that list needs a model, and nothing on it can hallucinate, because none of these components can produce a claim that was not computed.

Percentile framing carries most of the interpretive load, and it carries a caveat with it. A raw production figure means nothing to a reader without a reference set, and it means something different in every market. Reporting the band instead of the number keeps the claim inside what the data supports. But the band is computed against the returned set, not against the world, so it shifts as the query shifts. The card should say top of these results rather than top performer, and if the copy ever drops that qualifier the claim has quietly grown past its evidence.

Availability is the tightest constraint. A candidate writing that they are open to conversations supports an urgency level and nothing more. Turning that into a start date would invent a fact the platform never had, which is exactly the move a generative summarizer makes without noticing.

The weights behind the total are constants somebody chose, spread across a handful of factor families, and whatever they are they should be visible to the person using the tool. A recruiter who thinks credentials should not carry the share they carry for a given role needs something specific to argue with. Weights are a product decision wearing an engineering costume, and hiding them does not make them neutral.

4. Where a model runs, and where it does not

One learned component sits in this pipeline. Embeddings narrow the candidate set before any weight is applied, and the retrieval side is not naive about it. Records carry more than one vector, weighted by what the text is, so a query about experience searches the experience representation more heavily than a headline.

That placement is the whole argument. Retrieval produces a set. Scoring produces an order and the reasons for it. A retrieval model that returns a slightly different set is recoverable, because the next stage still evaluates every member on stated criteria and the recruiter still sees why each one placed where it did. A generative model writing the reason is not recoverable, because its output is the last word and nothing downstream checks it.

flowchart TD
job["Job<br/>Context"]
parsed["Parsed<br/>Requirements"]
text["Text<br/>Features"]
numeric["Numeric<br/>Features"]
payload["Card Payload<br/>score and evidence"]
event["Candidate<br/>Metadata Event"]
job --> parsed
parsed --> text
parsed --> numeric
text --> payload
numeric --> payload
payload --> event

Retrieval, evaluation, and delivery stay separate stages. One service wraps the database search, semantic retrieval, duplicate suppression, and batching, and answers a single question: who is in the set. The matching engine weighs candidates across components and answers the second: how well does each one fit. A third assembles the typed payload the browser consumes and emits it with the candidate metadata.

That last split is deliberate. The engine can collapse component weights into a sentence for internal workflows where a sentence is enough, while the response path carries named fields for an interface that renders them individually and cannot parse a paragraph.

5. The payload exists before the card does

The client receives the candidate record, the score, location, designation, and the explanation fields in one typed event.

Emitting them together is the enforcement mechanism. The explanation cannot be assembled lazily on the client, cannot be filled in by a later request, and cannot quietly become optional, because the event schema is where the two halves meet. If a scoring change stops producing keyword hits or drops concern flags, the gap shows up at the boundary rather than three weeks later in a support thread.

6. Thresholds expire, and mine did

Scores feed tiers. Above one cutoff a match fires an immediate alert, below it the match waits for a daily roundup, below that a weekly one. Those cutoffs were not guesses. I sampled the real score distribution across the live candidate and job set, found where the population actually sat, and placed the boundaries where the data justified them.

Then I changed the scoring model. New factors went in, the distribution moved upward, and the old cutoffs stopped meaning what they had meant. I re-derived them by hand and moved on.

That worked. What it did not do is leave behind a mechanism.

The tell showed up in the source. The class docstring still described the original cutoffs long after the constants had changed, and nothing caught it, because a stale comment does not fail a test. Documentation and code disagreed about the most consequential numbers in the alerting path, and the only reason I know is that I went back and read both.

A hand-set threshold is a claim about a distribution. It expires the moment the distribution moves, and it expires silently, because a number that no longer means what it meant still returns a value.

The uncomfortable part is that the fix already existed in the same codebase. I had built a calibration service for a different problem: it computes a conformal threshold per selector from a rolling window of observed errors, records every threshold change with its delta and its provenance, and monitors drift. Cache reuse decisions got statistically grounded cutoffs. Alert tiers got constants in a constructor.

Difficulty was not the difference. The cache had an error signal. Reuse a stale entry and something downstream notices, so observations accumulate and calibration has something to calibrate against. Nobody was labeling whether a match just under the cutoff was a miss. Without that, there is no error distribution to fit, and conformal machinery has nothing to work with.

Calibration is usually blocked on ground truth, not on math. If you want calibrated thresholds in a ranking system, the work is not implementing the quantile logic. It is building the feedback path that tells you when a threshold was wrong.

7. What determinism costs

Adding a signal is no longer one commit.

A generated paragraph absorbs a new factor for free, since the prose just mentions it. A typed payload does not. A new feature touches the scorer that produces it, the schema that carries it, the renderer that displays it, and the fixtures that assert it. Four places, one idea, and they have to agree.

Schema changes split cleanly by direction. Adding a field is additive and safe. Renaming or removing one is not, and a client using optional chaining will render a card missing that row without raising anything.

The sharper cost is expressive. The card can only say what the vocabulary permits. When a recruiter looks at a result and knows the real reason is something the feature set never modeled, the interface has no slot for it, and there is no honest workaround. The fix is to model the signal, which is slower than writing a sentence about it. A language model would have covered that gap immediately, with prose about anything you asked it to discuss. Giving up that coverage is the actual price here, and it is not small.

The return is concrete. Explanation costs a few field lookups instead of a network round trip and a generation, which is the difference between explaining every result in a long list and explaining the top three. A given candidate against a given job produces an identical card every time, which means a recruiter disputing a result and an engineer reproducing it are looking at one artifact. Past rankings replay from stored evidence without re-running inference. There is no per-result token cost. And there is no surface on which a hallucinated reason can appear, because nothing in the path can produce a sentence that was not derived from a comparison.

8. An explanation you can actually falsify

Post-hoc explanation has a faithfulness problem that the field has not solved. Feature attribution methods build a second model that approximates the first, and the gap between the approximation and the real decision process is exactly what you cannot measure without already knowing the answer. Asking a language model for a rationale is the same problem with worse error bars. In both cases the explanation is a claim about the computation rather than a product of it, and no experiment cleanly separates a correct explanation from a plausible one.

Derive the explanation from the arithmetic and that stops being a philosophical problem. It becomes a test.

Take a scored candidate and change one input. Remove the required credential, hold everything else fixed, and rerun.

Three assertions follow. The credential row moves from exact to absent. The total drops by that factor's weighted contribution and by no more than that. Every other row of the card is byte identical to the previous run.

That is an ablation test for explanation fidelity, and it fails loudly in the cases that matter. A feature that moves the score without emitting evidence breaks assertion one. A feature that emits evidence without moving the score breaks assertion two. Coupling between features that should be independent breaks assertion three, and that third one catches real bugs, because a shared normalization step is the kind of thing that quietly makes one factor's change ripple into another factor's text.

Run the same experiment against a generated rationale and there is nothing to assert. The prose may change, may not, may change in an unrelated place. No outcome falsifies anything, which is another way of saying no outcome confirms anything either.

9. How it breaks, and how you find out

The dangerous failure is silent and partial.

Highlights arrive empty. The score is still there, the layout still renders, and the card degrades into the bare ranked list this design was built to replace. Optional chaining on the client turns a missing array into no row instead of an error, so nothing logs and nobody files a bug. Users see a slightly emptier card and assume that candidate simply had less to say.

That failure needs a metric, not a test. Count how many emitted events carry an empty highlights array. Zero is expected. A number that starts climbing after a deploy tells you which change severed the connection between scoring and evidence, and it tells you before a recruiter learns to stop reading the card.

This design does not make the ranking better. A candidate sitting at the top for a bad reason still sits at the top, and binding the copy to the arithmetic means that mistake gets displayed with the same confidence as a correct call.

What changes is how long that survives contact with a user. A recruiter who can see that a match ranked high on a credential the role does not require has been handed the bug. Wire the explanation to the score and the ranking stops being something you defend in a meeting. It becomes something the product argues about with you, one card at a time.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

Negative Space Is a Label

Daniel Romitelli — Tue, 11 Aug 2026 06:12:03 +0000

A car mask can pass review and still teach the model to keep the wrong pixels.

The outline looks clean. The bumper is inside. The wheels are inside. Then the trained network holds onto the dark patch under the tires, because the label treated that patch as part of the vehicle's visual neighborhood. Training stays quiet. Production gets loud the first time a listing photo drags a strip of the old lot onto a new backdrop.

AutoLensAI turns dealer photography into listing-ready vehicle media. This installment follows the earlier pieces on segmentation and image provenance, then narrows to one question: how do I teach a matting model that the shadow touching a tire is evidence against foreground rather than a faint version of it?

1. The failure arrives without an error message

Vehicle matting estimates which pixels belong to the vehicle, at finer boundary resolution than segmentation gives. Tires, rocker panels, glossy showroom floors, and the halo under a lowered front lip are where a pretty binary mask does its damage.

Two cases cause most of it. A cast shadow can touch rubber and still sit outside the object. A reflection can match paint color exactly and still belong to the floor. Both look like they belong to the car in a thumbnail. Neither belongs to it in geometry.

A binary target has no vocabulary for that distinction. Every pixel is in or out, so the annotator's only lever is where to put the line. Push the line outward and shadow becomes vehicle. Pull it inward and the wheel arch loses its edge. Neither answer says the thing that matters, which is that some exterior pixels are ordinary background and some are adversarial background sitting one pixel from the object.

The model learns the difference anyway. It learns it wrong, because nothing in the supervision ever separated the two.

2. Three states, not two

The supervision contract uses three:

state	meaning	training treatment
vehicle	body, glass, wheels, trim, and visible geometry	foreground loss
hard negative	attached cast shadow, floor reflection, rejected candidates, confirmed exterior	background loss
unknown	unresolved boundary or ambiguous exterior	excluded from loss

Each state has one producer, which is what makes the target reproducible instead of negotiated.

flowchart TD
alpha["Render<br/>Alpha"]
spill["Shadow and<br/>Reflection"]
rivals["Rejected<br/>Candidates"]
uncertain["Boundary<br/>Band"]
label["Training<br/>Label"]
pos["255<br/>Vehicle"]
neg["0<br/>Hard Negative"]
ign["128<br/>Unknown"]
alpha --> label
spill --> label
rivals --> label
uncertain --> label
label --> pos
label --> neg
label --> ign

The commit note that started this was blunt: "Hard negatives by construction: the render alpha separates a car from its own shadow." That is the whole idea. The renderer already knows which pixels came from vehicle geometry and which came from scene lighting, because it computed both. Flattening those two facts into a single soft matte discards the most useful contrast in the frame, then asks a network to rediscover it from RGB.

Hard negative mining normally means sweeping a dataset for examples the model gets wrong, scoring them, and resampling. Here there is nothing to sweep. The difficulty is a byproduct of how the image was made, so the negatives arrive addressed instead of found.

3. What counts as negative

Shadow and reflection are the obvious members. The interesting ones are the other cars.

Dealer photography happens on lots. A frame usually holds more than one vehicle, and a segmentation model asked for vehicles returns all of them, correctly. Concept segmentation is not wrong when it hands back four masks. It answered the question it was asked.

AutoLensAI settles that with a small arbitration layer before matting runs. Candidate masks are ranked by a weighted combination of normalized mask area, monocular depth proximity, and image centrality. The winner is the hero. That ranking encodes the rule a photographer follows without ever stating it: the subject is the large, near, centered one.

The arbitration has a second job that is easy to miss. Once the hero is chosen, every losing candidate becomes known exterior. Not unknown. Not skipped. A sedan parked two spaces over is a vehicle-shaped, vehicle-colored, vehicle-textured region that is definitively not this vehicle, which makes it some of the strongest negative evidence available. A dataset that marks only shadow as negative leaves that signal on the floor.

The negative region therefore comes from three sources with three derivations:

source	derived from	what it teaches
cast shadow	separated lighting pass	darkness attached to the object is not the object
floor reflection	separated lighting pass	matching paint color is not membership
rejected candidates	hero arbitration output	correct class, wrong instance

The unknown band is the only place the pipeline admits it does not know.

4. Encoding, and why a label has to name its parent

The exporter writes one label image plus the identity of the artifact that produced it. Pixel values carry the training meaning directly:

value	zone	loss mask
255	vehicle	foreground target
0	hard negative	background target
128	unknown	ignored

Generation is mechanical:

Threshold rendered vehicle alpha into the positive region.
Dilate the alpha boundary by a small radius and mark that band unknown.
Rasterize separated shadow and reflection pixels as negative, except where vehicle alpha already owns the pixel.
Mark rejected hero candidates and confirmed exterior as negative.
Store the result with the matte identifier, the source render, and the review decision.

Step five came from a second commit note: "The label tool could not have told you which matte it labelled, and now it cannot avoid saying."

That reads like housekeeping and is not. A reviewer approving a boundary is approving a specific artifact from a specific render at a specific moment. Store the approval against a filename and the next re-render silently inherits a judgment made about different pixels. The decision binds to the matte, or it is a guess wearing a timestamp.

5. The ignore band is a contract, not a hedge

Training reads the three values literally. Foreground loss applies where the label is 255. Background loss applies where it is 0, including every hard negative beside tires and lower panels. Pixels at 128 leave both the numerator and the denominator, in loss and in evaluation.

valid = (label != IGNORE)
target = (label == VEHICLE).float()

per_pixel = criterion(pred, target)          # no reduction
loss = (per_pixel * valid).sum() / valid.sum().clamp(min=1)

Unknown is absence with a defined meaning. It is not a weak vote for background, and treating it as one is the exact bug this design exists to prevent.

The cost is friction, and it lands on every stage that touches the file.

A binary mask previews anywhere, compresses well, and drops into any loader. A three-zone target asks for more. The exporter, the review tool, the manifest, the dataset reader, the metric code, and the visualization layer all have to hold the same definition of 128. Six places, one number.

Two failures show what happens when one of them forgets.

A preview tool that remaps 128 to black shows the reviewer a boundary that does not exist in the file. Approval then gets recorded against an image nobody looked at.

An evaluator that counts ignored pixels as correct background improves every number while supervision quietly degrades. Consider what sits in that band. Mostly boundary, and boundary is mostly hard. Folding it into background buys a free win on the dashboard and pays for it in the product.

That failure is the worse of the two, because it looks like progress.

6. The band that hides at training time is the band that acts at inference

CarSegNet's refinement stage, NeuralSegJet, uses the same geometry from the other direction.

A learned alpha refiner may modify pixels only inside an explicit uncertainty band. Outside it, the semantic prior is frozen. Inside it, a learned gate mixes the prior with the neural decoder's output. The refiner never holds authority over the whole frame.

flowchart TD
prior["Semantic<br/>Prior"]
outside["Outside Band<br/>Frozen"]
gate["Inside Band<br/>Learned Gate"]
decoder["Decoder<br/>Output"]
alpha["Refined<br/>Alpha"]
prior --> outside
prior --> gate
decoder --> gate
outside --> alpha
gate --> alpha

The symmetry is the part worth keeping. At training time the band says do not grade me here, because membership could not be proven. At inference time the same band says you may edit here, because this is where proof was unavailable. One region, two roles, both descending from the same admission of uncertainty.

The constraint buys a specific safety property. An unconstrained refiner asked to sharpen a wheel arch is equally free to decide the vehicle has a different silhouette, since nothing stops it from moving mass far from the boundary. Freezing the prior outside the trust region lets the refiner repair an edge and forbids it from inventing topology. Mirror stalks and glass edges get their fine detail back without any freedom to grow a fender.

The trade is real. A trust region caps how much a strong refiner can fix, so a prior wrong by more than the band's width cannot be rescued downstream. That failure has to be corrected upstream in segmentation or hero selection instead. I accept it, because upstream errors are visible and diagnosable while a refiner rewriting object shape is neither.

7. Proving the contract survives the pipeline, without a GPU

A three-state target is only as good as the weakest stage that handles it, which makes this a testing problem more than a modeling one.

CarSegNet runs end to end on CPU with the segmenter, depth model, and background provider stubbed out. Everything else stays real: orchestration, compositing, encoding, and quality measurement.

The metrics computed in that mode are the production ones. IoU, boundary IoU, alpha error, and temporal stability across frames.

Separating model capability from pipeline correctness is what makes the ignore band testable at all. Does 128 survive the loader? Does the evaluator drop the band from both terms? Does compositing respect the negative region? None of those depend on weights.

They are ordinary software questions with ordinary answers, available before a GPU is warm and before any checkpoint exists.

Boundary IoU earns its seat here. Overall IoU is dominated by body pixels, so a mishandled band barely moves it.

The boundary variant measures the strip where the three states disagree. That is exactly where a folded ignore band does its damage, which makes it the number that fails loudly when someone gets this wrong. Break the band on purpose in a stubbed run and it should sink, while the overall figure stays nearly flat. If both hold steady, the test is watching the wrong thing.

8. Neighboring contracts stay separate

The same discipline shows up in parts of AutoLensAI that have nothing to do with matting.

The field-resolution contract in docs/quality/04-integration-field-resolution.md separates an absent measurement from a nearby value that could pass for it. glassVehicleShare and glassCoverage are computed against different denominators, so neither percentage substitutes for the other however close they read. Matting labels obey the same rule. Positive, hard negative, and unknown answer three different questions, and they stay distinct through measurement or the distinction was decorative.

Presentation draws a different boundary, and blending it with training truth is its own hazard. docs/pitch/align_pairs.py registers before and after gallery images so a wipe slider compares framing honestly. It fits a global similarity transform against the painted-body mask using uniform scale, translation, and a bounded rotation with abs(theta) <= 0.06. It refuses shear, local warping, and large viewpoint change, because those reshape the thing being compared instead of aligning it.

That script exists so a viewer can trust a slider. It holds no opinion about what the model should learn and it never writes a label. Keeping the two jobs in separate files is not tidiness, it is the reason a demo cannot quietly become a dataset.

9. What review looks like now

The reviewer's question changed, and that is the practical result of everything above.

The old question was whether the silhouette looks clean. The new one is whether every object-adjacent pixel carries the right role: keep, reject, or skip.

The second question is slower and harder to eyeball, and it is answerable. A reviewer can point at the shadow under a rocker panel and say that should be rejected, not skipped. The sentence has a defined meaning in the file, a defined effect on the loss, and a defined artifact it attaches to.

The payoff arrives during failures. When a model preserves a tire shadow, nothing about the diagnosis is guesswork. The renderer either separated the shadow pass or it did not. The exporter either rasterized it as negative or it did not. The label either carries 0 in that region or it carries 128. The loader either honored those values or flattened them. Four checks, each with a definite answer, ending at the stage where negative space turned back into foreground.

Negative space is not the absence of a label. It is the part of the label doing the teaching.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

Predict the Exit, Not the Close

Daniel Romitelli — Mon, 10 Aug 2026 12:41:22 +0000

Pramaana is a crypto research and trading system I built around one constraint: a model prediction is only useful when it describes the event that will actually decide the position. This article focuses on its path-passage classifier, which estimates which price barrier an asset reaches first.

A 24-hour closing-price forecast can be directionally correct and still describe the wrong trade outcome. Bracketed positions close when the take-profit level is reached, the stop-loss level is reached, or the holding window expires. Pramaana trains on that sequence directly: upper barrier first, lower barrier first, or neither within 24 hours.

The feature is interesting because the statistical target and the capital decision refer to the same event.

1. A closing price cannot recover the trade path

A common market model predicts the return between now and a fixed horizon:

$$
\text{return} = \log\left(\frac{\text{price at horizon}}{\text{price now}}\right)
$$

That target supports many research tasks. It does not describe a bracketed position.

A bracket order has two exit levels. One secures a gain and the other limits a loss. The trade ends when either level is touched, or when the holding period expires. Sequence determines the result.

Pramaana's earlier close-to-close targets discarded the high and low path inside the holding window. A later close therefore could not tell the model whether the upper or lower bracket had already been touched. The repository names the missing information directly: wicks, drawdowns, take-profit hits, and the order in which the barriers were reached.

The problem was in the training target before model choice or tuning entered the discussion.

2. First-passage classification labels the event that closes the position

First-passage is the first time a process reaches a defined level. Pramaana uses the next hourly bar's open as the reference price because that is the first point available after a signal is formed.

The current methodology places a 200 basis point threshold on each side for most assets. One basis point is 0.01 percent, so 200 basis points equals 2 percent.

The target schema is explicit in the label module:

def first_passage_col_names(horizon_bars: int, barrier_bps: float) -> tuple[str, str, str]:
    h = f"h{horizon_bars}"
    b = f"b{int(round(barrier_bps))}"
    return (
        f"target_up_first_{h}_{b}",
        f"target_down_first_{h}_{b}",
        f"target_none_{h}_{b}",
    )

The helper keeps the three outcomes explicit for each horizon and barrier width. The next function writes those outcomes into the training frame while preserving rows whose future window is incomplete:

def _add_first_passage_columns(
    df: pd.DataFrame,
    horizon_bars: int,
    barrier_bps: float,
) -> pd.DataFrame:
    up_col, dn_col, none_col = first_passage_col_names(horizon_bars, barrier_bps)
    up, dn, none, valid = _first_passage_for_barrier(df, horizon_bars, barrier_bps)
    df[up_col] = np.where(valid, up.astype(np.float32), np.nan)
    df[dn_col] = np.where(valid, dn.astype(np.float32), np.nan)
    df[none_col] = np.where(valid, none.astype(np.float32), np.nan)
    return df

Invalid trailing rows stay undefined instead of being forced into one of the three classes. The diagram shows the full path from signal to the cost-aware score used later in the trading decision.

flowchart TD
    signal[Signal at hour t] --> entry[Next-hour open]
    entry --> bracket[Set +2% and -2% barriers]
    bracket --> scan{First event in next 24 hours}
    scan -->|Upper| up[Class: up_first]
    scan -->|Lower| down[Class: down_first]
    scan -->|Neither or ambiguous| none[Class: none]
    up --> model[Estimate outcome probabilities]
    down --> model
    none --> model
    model --> score[Net long and short expected value]
    score --> gate[Rolling-percentile entry gate]

An hourly bar can cross both thresholds. Its high and low prove that both levels were touched, but hourly data cannot recover their order. Pramaana does not invent one. That observation is treated as unresolved.

Rows near the dataset tail also remain undefined when a complete 24-hour future window does not exist. Every valid label can therefore be traced back to a specific entry reference, two thresholds, and an observable first touch.

3. The model and the order use the same geometry

The training target uses the same 24-hour horizon and symmetric 2 percent levels as the trade structure.

Pramaana does not predict a broad idea of direction and then ask another component to reinterpret it. The classifier estimates the outcomes the execution rule can encounter.

up_first means the upper exit should occur before the lower exit. down_first carries the opposite ordering. none means the bracket remained unresolved when time expired.

This matters because a directional call can eventually be correct after the position has already stopped out. A long forecast has no value when the loss level is touched before the anticipated rise.

One three-class distribution supports both sides. The system does not need separate long and short models. It evaluates the same outcome probabilities under the payoff structure of each position.

4. Three probabilities become expected basis points

Pramaana trains one LightGBM model per asset. LightGBM is a tree-based classifier that estimates:

P(up_first), P(down_first), P(none)

Those probabilities are inputs, not orders. The decision layer maps the outcome distribution into a long score and a short score:

$$
\text{expected score for a side} = \sum \bigl( P(\text{outcome}) \times \text{net payoff for that side} \bigr)
$$

If the upper barrier arrives first, the long side receives the positive bracket payoff after cost and the short side receives the negative one. The signs reverse when the lower barrier arrives first. An unresolved window is valued at the horizon close, with trading cost still deducted.

The output is expected basis points, so the ranking layer receives a financial quantity rather than raw model confidence.

A large P(up_first) can still produce a weak long score. The competing outcome may remain material, the unresolved probability may be high, or fees may consume the margin. Expected value forces the whole probability distribution through the actual payoff map before a candidate can rank highly.

That distinction makes the score readable from two directions. An engineer can inspect the probability model. A portfolio reader can inspect the economic consequence.

5. Real trading cost changed the entry rule

A model can identify direction better than chance and still lose money.

Trading cost reduces every gross forecast. In Pramaana, that became decisive when the strategy was stressed against the real 35 basis point round-trip cost used by the Coinbase configuration.

Pramaana includes cost in the score instead of subtracting it after ranking. That prevents a high-confidence but low-value candidate from outranking a less dramatic signal with better net economics.

The first version admitted any score above zero. Later testing applied the real 35 basis point round-trip cost used in the Coinbase configuration. The simple threshold failed.

The response changed the decision layer. The current configuration uses a rolling-percentile gate and a smaller seven-asset universe. A candidate must rank strongly against the recent score distribution rather than remain barely positive under an easier fee assumption.

The classifier answers what may happen. The gate determines whether the estimated advantage deserves capital under current execution conditions.

6. Smaller trees produced more credible probabilities

Expected-value scoring depends on probability scale. A model that reports 99.9 percent for an event occurring roughly half the time will overstate the money attached to that branch.

An audit found that the original LightGBM settings allowed terminal leaves to become too specific. Some out-of-sample estimates exceeded 99.9 percent even though realized frequencies inside those confidence ranges were much lower.

The model was deliberately constrained:

Setting	Hardened value	Reason
Maximum leaves	8	Prevent narrow terminal regions
Maximum depth	3	Restrict interaction complexity
Minimum rows per leaf	400	Require broad empirical support
Feature sampling	50 percent	Reduce dependence on one feature subset
L1 and L2 regularization	0.1	Pull extreme leaf weights inward

After hardening, the largest observed P(down_first) across the audited cycles fell from 0.999 to about 0.83. Median expected calibration error declined from about 0.21 to 0.08 while rank order remained useful.

Lower confidence was the correct result. The revised probabilities tracked observed frequencies more closely and stopped one narrow leaf from dominating the expected-return calculation.

Accuracy measures whether the ranking contains information. Calibration determines whether its probabilities can be multiplied by money.

7. The target pipeline received its own audit

Time-series research can fail before model fitting begins. A future window placed on the wrong row can leak information from beyond the intended horizon into the target.

Pramaana's audit found such an error in vectorized path-excursion fields used for maximum favorable excursion, maximum adverse excursion, and horizon close. A double shift moved part of the calculation farther forward than intended.

The three first-passage classes were unaffected. They come from a separate forward scan that checks each subsequent bar in order. The per-asset classifiers train on those classes, so their labels remained clean.

One related value did require correction. The payoff for none used the contaminated horizon close, which inflated part of the backtest return. The field was fixed and the validation suite was rerun.

The corrected result did not collapse. Passing parameter combinations moved from 133 of 270 to 134 of 270. The best top-decile return estimate decreased from 112.6 to 87.5 basis points after the inflated portion was removed.

That outcome is more credible than preserving the larger number. The ranking survived while the reported payoff became more conservative.

8. How to read the signal without knowing the model internals

A reader who understands markets does not need to know how LightGBM constructs a tree to evaluate the decision.

The signal exposes a defined event, a 24-hour horizon, a payoff map, a cost assumption, and an admission rule. The model estimates the outcome distribution. The scoring layer converts it into net expected value for long and short positions. The rolling gate then compares that value with the recent score distribution before capital is admitted.

That structure supports direct scrutiny. Which event is being forecast? What thresholds define the position? How is an unresolved trade valued? Which fee schedule is applied? How selective is the gate? Were the probabilities calibrated outside the training sample?

Those questions also expose the main risks. Regime changes can alter the outcome distribution. Higher fees can erase small edges. Calibration can drift. Score compression can reduce the number of entries worth taking.

The prediction is useful because every number refers to a specified path, holding period, and cost model.

9. The forecast should match the action it controls

Pramaana moved from forecasting where price ends to estimating which exit occurs first.

That decision changed the labels, probability model, long and short scoring, fee test, validation process, and operator view. Each layer now refers to the same 24-hour bracket.

The principle extends beyond this system. A machine-learning target should encode the event that determines the action. When execution closes on the first threshold touch, an end-of-window label omits the part of the path that decides the trade.

Predicting exit order gives the model a question that capital can verify.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

Completion Is a Weak Memory Model

Daniel Romitelli — Fri, 07 Aug 2026 16:25:30 +0000

A learner can finish a lesson on Monday, answer the drill on Tuesday, and stare at the same idea on Friday with nothing to say. The checkbox still says done. The brain has moved on.

That mismatch is the problem I wanted Neuroloq to handle. A completion flag records that contact happened. Tutoring needs a better question: what can this learner retrieve today?

The technique is time-decayed mastery. Each concept keeps evidence about interaction, retrieval, decay, accuracy consistency, and the next useful card. The schedule comes from current memory, not yesterday's path through the course.

1. Completion misses time

Reading, drilling, and explaining aloud leave different traces, and Neuroloq already held them in attempt history, confidence checks, exercise results, voice summaries, and study context. The design change was to treat them as one learner model.

Four related changes shipped together on March 15, 2026: forgetting curve modeling, shared learner context across browser and phone sessions, knowledge stability scoring, and adaptive curriculum recommendations. They amount to one architectural move: progress became time-aware.

flowchart TD
lastSuccess[last_success] --> decay[decay score]
halfLife[half-life per learner] --> decay
accuracy[accuracy curve] --> stability[stability score]
successRate[success rate] --> stability
decay --> nextCard[recommended next card]
stability --> nextCard
caller[caller identity] --> profile[learner profile]
profile --> context[study context]
context --> nextCard
nextCard --> interaction[new interaction]
interaction --> lastSuccess
interaction --> accuracy

The diagram has two paths because Neuroloq has two study surfaces. Browser sessions can update the model in place. Phone tutoring has to identify the learner, load the profile, then continue from the same study context. If those paths diverge, the learner gets two tutors with separate memories.

2. Decay belongs to the concept

The implementation is organized around concepts rather than pages. lib/cognitiveEngine.ts handles calibration, stability, transfer, and related scoring. lib/conceptTopology.ts supplies prerequisite relationships. lib/learnerProfile.ts loads and saves the learner record. lib/forgettingCurve.ts owns the decay model.

That boundary matters. Updating memory is a transformation on a concept entry, driven by success or failure and the current time. It is not a user interface side effect.

The price is stricter instrumentation. A loose event like “finished page” is cheap to emit. A concept-level update asks every surface, lessons, drills, flashcards, and voice, to agree on which idea was touched.

3. The scheduler needs decay, not recency

A simple planner sorts by what the learner opened recently or what comes next. That helps navigation. It does not protect fragile memory.

lib/forgettingCurve.ts separates untouched material from material that has started to fade. Concepts with zero attempts belong to curriculum progression. Concepts with prior attempts have a trace worth reviewing. The review list computes decay against the current time, keeps entries below a threshold, and orders the weakest concepts first.

That split gives the curriculum engine room to choose between repair and advance. It also creates friction: a learner may want novelty while the tutor points back to a weaker prerequisite. I accept that cost because a tutor that only follows appetite becomes a playlist.

4. Stability is separate from decay

Decay answers, “how much should time have eroded this memory?” Stability answers, “how consistent has recent performance been?” They feed the same planning loop, but they are computed differently.

In lib/cognitiveEngine.ts, stability comes from recent accuracy values and variance. Classification also checks overall success rate.

export function computeStabilityScore(
 accuracyCurve: Array<{ timestamp: string; value: number }>
): number | null {
 if (accuracyCurve.length < 5) return null;
 const values = lastN(accuracyCurve, 10).map((c) => c.value);
 const m = mean(values);
 const variance =
 values.reduce((sum, v) => sum + (v - m) ** 2, 0) / values.length;
 return Math.max(0, Math.min(1, m * (1 - variance)));
}

export function classifyStability(
 stabilityScore: number | null,
 successRate: number
): StabilityTier {
 if (stabilityScore === null || successRate < 0.4) return "unstable";
 if (stabilityScore >= 0.8 && successRate >= 0.7) return "locked";
 if (stabilityScore >= 0.5) return "stable";
 if (stabilityScore >= 0.3) return "forming";
 return "unstable";
}

lastN and mean are small helpers defined in the same file.

That distinction keeps the model inspectable. A concept can be recent but shaky, old but reliable, or both stale and inconsistent. A single completion bit cannot express any of those cases. The concept entry carries five signals, each answering one question.

Signal	Question it answers
`lastInteractionDate`	When was this concept touched?
`halfLifeDays`	How quickly does it fade for this learner?
`decayScore`	How much retrieval strength remains now?
`accuracyCurve`	Has recent performance been consistent?
`successRate`	Is the learner succeeding often enough to trust the score?

The tradeoff is compression. A stability tier is legible enough for interface decisions, but it reduces messy behavior to a small label. The tutor still needs recent sessions, weak topics, and study history when speaking to the learner.

Take decorators: last touched nine days ago against a six day half-life, which leaves a decay score near 0.35, low enough for the review list. The half-life is stored per learner per concept, so the same nine day gap erodes one learner's decorators further than another's. The accuracy curve reads differently: the last ten values sit around 0.8 with little variance, which classifies as stable. Faded but stable means the memory erodes on schedule and holds up when tested. The planner queues a short retrieval rather than a reteach or new material.

5. Phone tutoring reloads the same model

Phone tutoring adds one hard requirement: the learner is outside the browser session. The tutor has to identify the caller and fetch the same profile and study context before it can teach adaptively.

The phone path lives under agents/tutor-agent/src/. The agent tools include getLearnerProfile, getStudyContext, resolveCallerIdentity, recordConceptProgress, saveDailyNote, logVoiceSummary, and endCallSummary. The matching application routes sit under app/api/agent/, including learner profile, study context, record progress, daily note, and voice summary endpoints. Phone number storage is part of the profile surface through app/api/profile/phone-number/route.ts and components/PhoneNumberSettings.tsx.

The summaries a call leaves behind land in the same record the browser reads later.

The same plumbing draws the failure boundary. A caller with no profile match cannot get adaptive tutoring, because there is no learner record to load. Guessing is the one option off the table: writing progress against the wrong identity would push one learner's evidence into another's memory, and that corruption is quieter and more expensive than a generic session. Identity resolution runs first because every later write depends on it being right.

The downside is operational weight. A static lesson bot only needs a prompt and a transcript. A memory-aware phone tutor needs identity, profile access, context loading, progress writes, and summaries. I chose the heavier path because changing channels should not reset the learner.

6. The next card is a consequence

The tempting way to build suggestions is to start with a ranked list. I prefer the reverse order: get the concept memory right, then let the next card fall out of it.

The chain is small enough to inspect. A successful retrieval updates concept evidence. Time changes decay. Recent accuracy consistency changes stability. The curriculum engine weighs review against progression. Each step has a narrow job and a clear failure mode.

Completion still has value as a navigation fact. It can say whether the learner reached the end of a lesson. It cannot say whether decorators, context managers, async, typing, or testing should come back today.

Neuroloq became more useful when I stopped asking whether a concept was done and started asking how much of it was still available right now.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

Created Is Not Ready: Terraform Ordering vs. Azure Eventual Consistency

Daniel Romitelli — Fri, 07 Aug 2026 12:48:23 +0000

The deploy finished clean. Every resource reported created. The application died on startup anyway.

The failures were never dramatic. A service asked for a secret and got told no. A database existed but its private address would not resolve yet. Nothing in the application code was wrong. The cloud had said yes to creating everything, and I had been treating yes as ready.

I built the Azure infrastructure for a recruitment platform around the gap between those two words: more than 56 resources across three environments, 12 private endpoints, 7 private DNS zones, 43 role assignments, every platform service closed to the public internet. At that size the gap between created and ready is not a corner case. It is most of the job.

1. Two different promises

Terraform makes one promise and keeps it: nothing gets created before the things it references. Output references and explicit depends_on build a dependency graph (a DAG, directed and acyclic) and the apply walks it in order. I leaned on that everywhere.

Azure's promise is smaller than it looks. When the API accepts a resource, the control plane has written it down. That is all. Role-based access control (RBAC) assignments take time to reach the services that enforce them. A private endpoint can exist while the Domain Name System (DNS) record pointing at it has not settled. A vault can be live while the identity that needs it is still a stranger to the authorization checks.

depends_on buys you creation order. Nothing else. Missing that distinction is how a green apply produces a dead platform.

2. The build spine

I treated the root infrastructure as a spine: identity, networking, observability, secrets, data services, artificial intelligence model deployments, compute, edge routing, and policy. Each part produced outputs the next one consumed.

flowchart TD
 identity[Managed identities] --> rbac[Key Vault role assignments]
 identity --> network[Virtual network and subnets]
 network --> dns[Private DNS zones]
 rbac --> vault[Key Vault]
 network --> data[PostgreSQL, Redis, and Service Bus]
 dns --> privatelink[Private endpoints and DNS zone groups]
 data --> privatelink
 vault --> compute[Container Apps and workers]
 privatelink --> compute
 identity --> compute
 compute --> edge[Front Door and Web Application Firewall]
 edge --> policy[Policy enforcement]

The diagram matters because plenty of Azure resources are valid as declarations long before they are useful as capabilities.

Identity went first. Managed identities have to exist before RBAC can grant them vault access, and vault reads sit directly on the startup path: the service reads a vault URL from the environment, builds a Key Vault client with the default Azure credential chain, and fills in missing variables from named secrets. Values already present in the environment still win, which keeps local development plain while production centralizes credentials. The price is a stricter startup. If cloud identity is wrong, the service dies early instead of limping forward half-configured.

Networking went in early but finished late. Virtual network, subnets, network security groups, and the private DNS zones came up front. The private endpoints could not, because they attach to services that did not exist yet. So the working arrangement ran: network and DNS foundations, then PostgreSQL, Redis, and Service Bus, then each service's endpoint and zone records, and only then Container Apps. A private DNS zone does not permit a database to be created. It lets consumers resolve the private address once the endpoint is attached (a distinction that sounds pedantic right up until a hostname will not resolve). Giving name resolution its own place in the spine made those failures easy to isolate.

Observability came before anything noisy: Application Insights, Log Analytics, and alert rules ahead of secrets, data, models, and compute. That delayed the first visible milestone. It also meant that when a container could not read a secret or a connection pool exhausted, the event landed in shared telemetry from the very first attempt. Retrofitted monitoring leaves gaps exactly where a new system teaches you the most.

Compute and the edge closed it out. Container Apps and workers started only after identities, secrets, private data access, messaging, caching, model endpoints, and telemetry could hold their weight. Front Door and the Web Application Firewall (WAF) sat behind that, since routing only matters once there is a stable backend, and policy enforcement came last, once the resource graph was visible enough to govern. The first public URL shows up late this way. I will take that delay over debugging application, network, identity, and routing faults at the same time.

3. What the references actually enforce

The implementation stayed boring on purpose. Root modules followed the spine, and later modules consumed outputs from earlier ones instead of rediscovering names or IDs. Identity outputs fed vault access assignments and compute identity settings. Network outputs fed endpoint subnet placement. Data service outputs fed connection settings. Pass capabilities forward, not strings sideways.

Those references, plus depends_on at module boundaries, enforced creation order. Full stop. They did not make RBAC propagation finish. They did not make a fresh role assignment visible to the vault's authorization checks, and they did not make a private hostname resolvable.

4. What readiness required on top

Readiness needed its own machinery, and none of it lives in the graph.

The bootstrap treats an authorization denial on a fresh deployment as possibly temporary. A role assignment can be accepted and not yet enforced, so the service retries with backoff before giving up. When configuration genuinely cannot be assembled, the container exits. Running half-configured is worse than not running, and the platform restart policy becomes a cheap outer retry loop while propagation catches up.

Between stages, the pipeline measured instead of assumed: can a secret actually be read, does the private hostname actually resolve. Only then did the next stage apply. And nothing became routable at the edge until the backend had proven it could reach its own dependencies.

None of this is exotic. It is the admission that the apply finishing is a checkpoint, and the platform being alive is something you verify afterward.

5. Cost by environment

The monthly figures are infrastructure forecasts from the design baseline, covering core platform services rather than labor or third-party application fees.

Environment	Monthly infrastructure forecast	Intent
Development	Around $200	Keep the full shape affordable
Staging	Around $400	Exercise integration paths before release
Production	Around $870	Run with high availability

Parity is useful until it becomes waste. I kept the shape consistent across environments, then varied scale and availability.

6. The ledger and the measurement

Terraform state is a ledger. It records what has been requested and written down, and it holds no opinion about whether any of it works yet.

Every system I have built since runs on the same rule: creation is a request, readiness is a measurement, and nothing downstream starts until the measurement says so.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series