TL;DR In generative video pipelines, running cheap low-step sketches to pick parameters sounds like free optimization. But when prompts go out-of-distribution, surrogate scorers return noise, turning a \$0.002 check into a bad decision that triggers a \$15 compounding failure. Here's why skipping the cheap step is sometimes the cheapest option.
Three numbers run Scenematic's generation loop. A think-frame costs \$0.002. A full render costs \$0.50. A bad scene that slips through and gets built on costs about \$15.50, because the scene chain compounds it before anyone looks. The constant in lib/generation-loop.ts carries the arithmetic in a comment: 15.502, // CALIBRATION_TARGET: 0.002 + 0.50 + 15.00.
Most of the pipeline exists to keep spend at the cheap end of that ladder. One module decides when the cheap step should be skipped entirely. A hundred-contract baseline then put numbers on how often that decision was wrong.
1. The rehearsal
lib/think-frames.ts generates quick, low-inference-step sketches before committing to a full-quality keyframe. The file header credits DeepGen's think tokens as the inspiration. Each sketch tries a different preservation focus, character, environment, mood, composition, or atmosphere, with its own image-to-image strength and seed. The reward mixer scores the batch and the winner's parameters go to the full render.
The economics only work if those scores mean something. That assumption fails quietly, and it fails hardest on the prompts where a rehearsal looks most useful.
2. Where the scores stop meaning anything
Scoring a sketch of A detective leans forward across a metal table, interrogating a nervous suspect under fluorescent lights works fine. The scorer has seen a thousand shots like it. Scoring A sentient equation writes itself across a blackboard that extends infinitely in all dimensions does not fail loudly. It returns a number, and the number is noise. Both prompts are verbatim from the baseline harness.
lib/ood-detector.ts measures the distance instead of hoping. It keeps a reference corpus of 25 in-distribution prompts, roughly 8 dialogue, 9 cinematic, 8 multi-asset, and embeds them with all-MiniLM-L6-v2 through @xenova/transformers. Local model, no API call.
The embeddings mean-pool into a unit centroid, computed once per process and cached. An incoming prompt gets embedded the same way. Epistemic uncertainty is one minus the cosine similarity to that centroid, and at or above the threshold the detector sets bypass_surrogate: true.
Bypass means skipping the cheap step, so the prompt the system understands least is the one that goes straight to the expensive render. Backwards from most cost optimizations, and deliberate. Out past the corpus a rehearsal is a \$0.002 lie that steers a \$0.50 decision toward a \$15 mistake, and skipping it buys the removal of a bad witness at the price of one render. The gate recuses an unqualified judge rather than filtering bad prompts.
Every evaluation writes a row to an ood_events table: the uncertainty, the threshold that was applied, the bypass flag, and cost_incurred at either \$0.50 or \$0.002. That table is where everything else in this post comes from.
flowchart TD
P["Compiled prompt"] --> U["Uncertainty = 1 - cosine vs corpus centroid"]
U --> G{"At or above the category threshold?"}
G -->|"BYPASS"| F["Straight to full render, $0.50"]
G -->|"SURROG"| T["Think-frame rehearsal, $0.002"]
T --> W["Full render using the winning sketch's parameters"]
3. One hundred contracts
scripts/run-phase1-baseline.ts runs 100 steering contracts through the gate. A steering contract is the typed record the compiler emits for one scene: the prompt, the chosen model and seed, the quality targets, and the audit trail of what happened to it. The batch splits into 80 in-distribution prompts across dialogue, cinematic, and multi-asset, plus 20 out-of-distribution (OOD) prompts, abstract and high-complexity. Seeds are fixed at 42 plus the contract index. The renders are real, on RunPod pods running ltx2, wan22, and hunyuan behind per-model semaphores of 3, 2, and 2. Each console line prints SURROG or BYPASS next to the measured uncertainty.
A caveat before the numbers. The reward heads, the four per-dimension scores that judge a finished render (R_smooth, R_motion, R_semantic, R_narrative), are simulated in this harness: in-distribution signals drawn around 0.65 with jitter, OOD around 0.35, because real per-head scoring of rendered video was not wired up yet. The embeddings, the routing, the renders, and the telemetry are real. The simulation still surfaced two real bugs, covered below.
The cost waterfall, straight out of fig4_cost_waterfall.csv:
| Stage | Count | Cost |
|---|---|---|
| Contracts submitted | 100 | |
| OOD bypassed | 31 | \$0.00 extra |
| GPU rendered | 69 | \$15.64 |
| Passed quality floor | 43 | |
| HITL flagged | 26 | |
| False negatives | 53 | \$821.60 compound |
Twenty-two of the 26 human-in-the-loop (HITL) flags blamed the narrative head. The Phase 2 commit message gives the aggregate: a 25.3% false negative rate, with R_narrative dominating 72.4% of HITL flags. Fifty-three contracts finished under the quality floor, and at \$15.50 of modeled compound damage each the ledger prices those misses at \$821.60, against \$15.64 for every render in the run combined.
4. Thresholds from the misses
Phase 2 swept the threshold from 0.30 to 0.95 in steps of 0.01 and computed a full confusion matrix at each stop. The F1-optimal global threshold came out at 0.31, which flags 97 of 100 contracts and puts the cost column at \$48.51 per hundred. Flagging nearly everything does maximize F1. It also defeats the point of having a surrogate at all.
So the shipped config ignored F1. The calibration_metadata block in quality-thresholds-v2.json records the target it optimized instead: Catch ALL known FNs per category with margin. Take the known misses, find the minimum uncertainty among them per category, set the threshold just under it. DIALOGUE's one known false negative sat at 0.631, so DIALOGUE got 0.62. SCENIC's four bottomed out at 0.506, so 0.50. ACTION's five at 0.495, so 0.49. The global fell from 0.72 to 0.55 and now serves as the fallback for categories with no data.
Per category matters because the sweep exposed an inversion a single global threshold cannot encode. The config file says it in one line: SCENIC and ACTION had LOWER uncertainty but HIGHER FN rates — category thresholds fix this.
A scenic prompt reads as familiar to the embedding space. Sweeping drone shot, golden light, the corpus is full of that texture. The renders still miss. Uncertainty and failure rate are correlated across the whole population and inverted inside two of its slices, and one knob cannot express that.
ABSTRACT kept the old 0.72, since genuinely weird prompts carry high uncertainty on their own. NARRATIVE got 0.65 with an annotation calling it a conservative estimate with no false negative data behind it. That one is a guess with a label on it, and it stays a guess until a NARRATIVE prompt fails in a logged run.
The detector loads this file at runtime and falls back to the legacy global 0.72 if it is missing. The sweep range and step size are recorded next to the thresholds they produced, along with the per-category evidence.
5. Two gates that could not fire
The baseline caught two bugs I would not have found by reading the code, because the code looked fine.
The HITL gate required the composite score to be under the floor AND at least one reward head to breach a z-score of -2.0. With simulated signals at 0.65 ± 0.1, the commit message for 20aa5ec describes the result: the z-score condition was mathematically impossible to satisfy with simulated signal distribution of 0.65 ± 0.1. Two conditions joined by AND, one of them unsatisfiable. The gate sat silent while contracts failed under it, and the dashboards looked calm the whole time.
The fix inverted the roles. The reward floor of 0.65 is now the primary trigger, and per-head z-scores only attribute which head gets blamed.
The second bug was in the false negative counter itself. It only counted SURROG contracts, on the theory that a false negative means the surrogate path trusted a prompt it should not have. That definition missed 8 contracts the gate bypassed which still rendered below the floor. A bypassed prompt that fails is still the pipeline failing, whatever path it took, so the counter now takes any contract under the floor.
6. The head that ran cooler
R_narrative took 72.4% of the HITL blame in the phase 2 baseline. The narrative scoring was doing its job. Its head just runs cooler than the others: population mean 0.55, against 0.60 for R_motion and 0.70 for R_smooth and R_semantic. Four heads held to the same absolute bar, and the coolest one took nearly all the blame.
The fix in lib/reward-mixer.ts is z-score normalization per head. normalizeHeadScore computes (raw − mean) / std against per-head population stats stored in the same versioned config, so a head only triggers when it is unusual for itself. computeSubReason then cross-references the other heads whenever R_narrative does trigger: motion also low means pacing, semantic also low means fidelity, narrative alone means coherence.
7. Infra noise is not model failure
Twenty hunyuan events in the baseline were graphics processing unit (GPU) failures, out-of-memory and handler crashes, sitting in ood_events and dragging the quality numbers around. So the table grew gpu_error and superseded columns. evaluateOOD returns the row id of the event it just logged, and the runner calls markOODEventGpuError on a GPU error so the dashboard can segment it. Re-running contracts with --rerun-contracts="81,82,100" marks the old contaminated rows superseded instead of deleting them. queryDashboards takes a view argument, clean or all, and clean excludes both flags.
Even total infrastructure collapse gets a row: ALL_MODELS_DOWN logs an OOD event with gpu_error=true. Without the segmentation, a crashed pod reads as a quality regression, and one afternoon of flaky hardware quietly recalibrates your thresholds for you.
8. What is still provisional
The detector landed at 234 lines and sits just over 300 after the Phase 2 changes. The scoring inside it is a cached centroid and a cosine. Nothing is learned and none of it touches a GPU. The cost constants carry a CALIBRATION_TARGET marker with a comment saying to recalibrate after 100 real runs or a provider switch, so the provisional numbers announce themselves and are greppable.
Plenty is still provisional. The reference corpus is 25 prompts, and every threshold in the v2 config is calibrated against the centroid those 25 produce. Adding corpus prompts moves the centroid, which shifts every uncertainty measurement, which invalidates the per-category thresholds. Corpus and thresholds have to version together or the calibration quietly stops describing anything. The reward heads are still the simulated ones, so the head_stats block needs re-deriving from real scores before the z-score triggers mean what they claim. And NARRATIVE still has no false negative data.
The part I trust is the ledger. Both dead gates were invisible in code review and obvious in the event counts, and the event counts only exist because every decision writes a row, including the decision to spend more.
🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series
Top comments (0)