DEV Community: Daniel Romitelli

Voice Input Is an Intent Language

Daniel Romitelli — Mon, 03 Aug 2026 21:23:33 +0000

If you dictate into an editor, sometimes you mean to type, sometimes you mean to command, and the app has to tell the difference before it acts.

That is the real problem. A transcript like “close tab” can be harmless prose or a destructive instruction. Treating every spoken phrase as synthetic keystrokes gives the editor a ghost typist with too much confidence.

I built the voice-control path in a desktop IDE as a small compiler pipeline: microphone event, Python bridge process, transcript event, Node.js command parser, then either an IDE action or text insertion.

The split

Python listens and transcribes. Node.js decides. The renderer displays state and inserts text at the chosen destination.

That division is the design. The Python bridge owns audio capture, voice activity detection, transcription, settings, and a line-oriented command protocol. The Electron main process owns Inter-Process Communication (IPC), privileged actions, and the parser that turns a final transcript into an editor operation.

The bridge speaks JavaScript Object Notation (JSON) lines over standard input and standard output. It accepts commands such as start, stop, status, and quit. It emits events such as ready, listening, final transcript, idle, error, and status. That protocol keeps the speech runtime replaceable while keeping workspace mutation inside the Electron side.

flowchart LR
 subgraph PythonBridgeProcess
  mic[Microphone Input] --> recorder[Audio Recorder]
  recorder --> vad[Voice Activity Detection]
  vad --> skip[Skip Transcription]
  vad --> stt[Speech To Text]
  stt --> final[Final Transcript Event]
 end
 subgraph ElectronMainProcess
  manager[Voice Manager] --> parser[Command Parser]
  parser --> action[Privileged IDE Action]
  parser --> insert[Insertion Event]
 end
 subgraph RendererProcess
  status[Status And Ghost Text]
  editor[Editor Or Terminal Target]
 end
 final --> jsonLine[JSON Line Over Stdout]
 jsonLine --> manager
 action --> ipcStatus[IPC Status Update]
 ipcStatus --> status
 insert --> ipcInsert[IPC Insertion Update]
 ipcInsert --> editor

The cost is operational shape. A child process has lifecycle, readiness, stderr handling, and environment discovery. I accepted that because the alternative was worse: audio code spread through the renderer, or transcription code gaining accidental access to editor commands.

Silence is a billable input unless it is filtered early

Voice activity detection (VAD) is the first classifier in the path. Silence wastes transcription calls and can still produce accidental text if it reaches speech-to-text, so the bridge checks for speech before sending audio onward.

That choice also keeps failure behavior crisp. When the host has no capture device, the bridge can still start and respond to commands, but the final result is empty. The renderer surfaces that as a “No speech detected” condition instead of quietly pretending dictation succeeded.

The tradeoff is dependency coupling. The bridge has to find and import the existing speech package, including recorder, transcriber, VAD, and settings modules. That is less tidy than a packaged library boundary, but it reuses the proven speech stack rather than cloning it inside the desktop app.

The parser decides between action and text

The main process never treats a final transcript as immediate typing. It passes the string into the command parser.

The current implementation has three practical outcomes:

Transcript shape	Parser result	Example behavior
Empty transcript	Reject	Show no-speech feedback, do not edit
Exact command phrase	Execute command	Save file, close tab, undo, redo, format
Payload command	Execute command with argument	Find text, jump to a line, send terminal input
Any other non-empty phrase	Insert text	Dictation lands at the active editor or terminal target

That table is the intent language. It is small on purpose. Exact phrases get command privileges. Payload commands must have a recognizable verb and argument shape. Everything else becomes content.

This means the implemented no-op path belongs to empty or silent input, not fuzzy ambiguity. I would rather keep that rule visible than pretend the parser has a confidence model it does not have. If a future version adds ambiguous rejection, it should be an explicit parser result, not a side effect of a missed match.

flowchart TD
start[Start]
transcript[Transcript]
emptySilence[Empty or Silence]
exactCommand[Exact Phrase or Payload Command]
unmatchedText[Unmatched Non Empty Text]
noOp[No Op]
command[Command]
dictation[Dictation]
ideAction[IDE Action]
textInsertion[Text Insertion]
endState[End]
start --> transcript
transcript --> emptySilence
transcript --> exactCommand
transcript --> unmatchedText
emptySilence --> noOp
exactCommand --> command
unmatchedText --> dictation
command --> ideAction
dictation --> textInsertion
noOp --> endState
ideAction --> endState
textInsertion --> endState

Collision policy is where this pattern earns its keep. “Find” by itself should not become a search without a target. “Find database” can become a command because it has a verb and payload. “Close tab” can execute because it is an exact phrase. “The phrase close tab appears in the docs” inserts as text unless the grammar says otherwise.

That creates maintenance work. Each new command adds another phrase that can collide with normal prose. The parser has to stay conservative: short destructive commands need exact matches, payload commands need argument validation, and ordinary sentences need a predictable insertion path.

The renderer is only the destination

The renderer integration handles status, provisional text while transcription is pending, insertion at the cursor, command display, and terminal forwarding. It knows whether focus is in the editor or terminal. It does not decide whether a transcript is allowed to save a file or close a tab.

The global voice trigger follows the same pattern. The hotkey path routes through the application menu event into the renderer-facing control flow, rather than letting a shortcut handler become a second command executor. The manager still owns start, stop, status, and bridge messaging.

That keeps one chain of custody for spoken input. Audio becomes a transcript. A transcript becomes a parser result. Only then does the desktop IDE apply the effect.

Treat speech like code

The useful model is a compiler. The microphone produces raw input. The bridge turns sound into a token stream. The parser classifies intent. The execution layer applies a constrained result.

Voice input becomes safer when spoken words cross that language boundary before they touch the workspace. The machinery is ordinary: a Python child process, JSON lines, an Electron manager, IPC, and a command parser. The discipline is in refusing to treat speech as typing with a microphone attached.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

Provenance Belongs in the Image Table

Daniel Romitelli — Mon, 03 Aug 2026 21:23:23 +0000

A generated image looks finished until review starts.

Someone approves the first version. Someone else crops it. A branded copy goes out. Another edit changes the prompt. A week later, the useful question is simple: which prompt, model, seed, size, parent image, and publishing settings produced the version on screen?

In a content studio, I put those answers in the PostgreSQL row that stores the image. Logs explain what happened during a run, then rotate away. Object storage keeps the bytes and forgets why they exist. The row is the only one of the three that survives edits, review, and publishing.

1. The row is the receipt

The table in apps/api/src/database/init-ai-images-table.js treats generated and edited images as one record type. An original image gets its own row. An edit gets another row, with original_image_id pointing back to the parent.

CREATE TABLE IF NOT EXISTS ai_generated_images (
  id SERIAL PRIMARY KEY,
  image_url TEXT NOT NULL,

  -- what produced it
  prompt TEXT NOT NULL,
  model VARCHAR(100) DEFAULT 'fal-ai/imagen4',
  model_version VARCHAR(100),
  seed BIGINT,
  width INTEGER,
  height INTEGER,

  -- how it derives from another row
  is_edited BOOLEAN DEFAULT FALSE,
  original_image_id INTEGER REFERENCES ai_generated_images(id),
  edit_prompt TEXT,
  edit_strength DECIMAL(3,2),

  -- what actually shipped
  branded_url TEXT,
  branding_options JSONB,

  metadata JSONB,
  tags TEXT[],
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

That self-reference is the design choice. It makes the image table append-only-ish: new variants are inserted as new rows instead of overwriting the earlier state. The cost is more rows and more discipline at write time. The benefit is editable history that product screens and debugging queries can follow.

flowchart TD
  original["Original row: prompt, model, seed, dimensions"]
  editA["Edited child: edit_prompt, edit_strength, edit_steps"]
  editB["Edited child: edit_prompt, edit_guidance_scale"]
  brandedA["Branded output: branded_url, branding_options"]
  brandedB["Branded output: branded_url, branding_options"]
  original --> editA
  original --> editB
  editA --> brandedA
  editB --> brandedB

2. Why not a separate audit table

An audit table is the obvious alternative. Write the image row, append an event per generation and edit somewhere else, reconstruct history when asked.

I did not do that, and the reason is where the cost lands. An audit table answers questions about history. The product asks questions about the current thing. Every screen that shows an image wants the prompt next to it. A gallery filters by model. A review queue sorts by whether a row is a branded variant. Each of those becomes a join against a log that grows faster than the images do.

The self-reference costs a recursive query when someone wants the whole branch, which is rare. The audit table costs a join on every render, which is constant. I moved the expensive case onto the rare one.

3. Columns carry facts the product asks about

A value gets its own column when the application filters, sorts, joins, or explains by it. metadata and branding_options stay JSONB because those shapes change more often than the core record, and tags is an array so classification can be indexed.

Field group	Question it answers
`prompt`, `negative_prompt`	What text guided the run
`model`, `seed`	Which settings identify it
`width`, `height`, `aspect_ratio`, `resolution`	What shape came back
`is_edited`, `original_image_id`	Whether it derives from another row
`edit_prompt`, `edit_strength`, `edit_guidance_scale`, `edit_steps`	Which transform produced the child
`branded_url`, `branding_options`	Which publishable variant was created
`metadata`, `tags`	Extra details and classification

Seven indexes cover the access patterns: owner and session for scoping, creation time for the default sort, saved and public flags for the two filters the gallery exposes, and model for the question that only comes up during review. Six of those are ordinary B-tree indexes on scalars. The seventh is not:

CREATE INDEX idx_ai_images_tags ON ai_generated_images USING GIN(tags);

tags is an array, so a B-tree cannot help: a B-tree indexes a value, and the query asks whether a value sits inside a collection. A Generalized Inverted Index (GIN) inverts that. It stores one entry per distinct tag pointing at every row carrying it, which turns containment into a lookup instead of a scan.

Every index is a tax on writes. Seven of them on a table that grows a row per edit is a real cost, and I took it because these images are written once and then browsed, filtered, revisited and argued about for weeks.

4. Lineage becomes a query

Once parent links live in the same table as the model parameters, review can ask for a whole branch without reading logs. This recursive Common Table Expression (CTE) starts from one original row and returns every descendant with the fields needed to reproduce or debug the result:

WITH RECURSIVE image_lineage AS (
  -- anchor: the row you are asking about, at depth 0
  SELECT id, original_image_id, image_url, prompt, model, seed,
         edit_prompt, edit_strength, branded_url, created_at, 0 AS depth
  FROM ai_generated_images
  WHERE id = $1

  UNION ALL

  -- recursive arm: anything whose parent is already in the result
  SELECT child.id, child.original_image_id, child.image_url, child.prompt,
         child.model, child.seed, child.edit_prompt, child.edit_strength,
         child.branded_url, child.created_at, parent.depth + 1
  FROM ai_generated_images child
  JOIN image_lineage parent ON child.original_image_id = parent.id
)
SELECT * FROM image_lineage ORDER BY depth, created_at;

The two arms are doing different jobs. The anchor selects one row and calls it depth zero. The recursive arm joins the table back onto the results so far, so each pass picks up the children of everything found in the pass before it. Postgres repeats that until a pass returns nothing.

depth is the column that makes the output readable. Without it the result is an unordered pile of rows; with it, ordering by depth then time reproduces the order the edits actually happened in. One original branches into many children, and every node carries enough state to explain why it exists.

The recursion has no depth limit, which is fine while edits are made by people. A loop would hang it, and a row cannot become its own ancestor through the normal write path, so I have not added a guard. If edits ever get generated in a batch, that assumption is the first thing I would revisit.

The insert path in apps/api/src/database/azure-postgres.js follows the same contract: explicit columns for the core settings and edit parameters, JSON for the irregular details.

5. What the row cannot tell you

Two gaps, and both are visible in the schema above.

model stores a slug like fal-ai/imagen4, not a version. A provider can change weights behind that name without changing the string, and the row will keep claiming a reproducibility it no longer has. Same prompt, same seed, same model value, different image. The fix is a pinned version in its own column rather than buried in metadata, because reproducibility is a question the product asks directly.

The foreign key carries no ON DELETE clause, so Postgres refuses to remove a parent that still has children. Cleanup becomes a deliberate walk down the tree instead of one statement. That is the right default here, and it is worth stating plainly: anyone who reaches for ON DELETE CASCADE to make a purge easier is deleting the receipts.

Branding obeys the same rule. image_url records what the model returned, branded_url the version intended for use, branding_options the publishing configuration. A reviewer questioning the final visual can separate model output, edit choice, and branding treatment without opening a log.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

CarSegNet v2: One Lot Photo to a Showroom Composite

Daniel Romitelli — Sun, 02 Aug 2026 01:57:31 +0000

A buyer does not open a listing and ask what the mask IoU was. They look at the mirror stalk, the antenna, the gaps between wheel spokes, the glass, and the tire line where black rubber meets whatever floor you claim the car is sitting on. If asphalt from the old lot still glows through the spokes, or the car floats a finger-width above a showroom floor that never saw those tires, they feel it before they can name it. The click goes cold. The unit sits.

That is the sales problem in one frame. CarSegNet v2 is the still-image system I built so a dealership can take one usable lot photo and walk away with showroom-grade listing media: a composite, a cutout, and an alpha matte written as first-class artifacts. The buyer stakes are real. So are the ops stakes that never show up in a segmentation paper.

The ops case: same-day media without inventing ROI

A car that arrives on a Tuesday and waits until Thursday for a photo package is inventory that cannot earn an inquiry online. It is on the lot. It is paid for. It is not sellable as a listing until someone produces media. Same-day listing media turns delivery into something a shopper can find the same afternoon. That is money saved by shortening unsellable days, and money earned by putting the unit in front of demand sooner. I am not going to invent a dollar figure or an ROI percentage for that. CarSegNet and AutoLens measure matting, topology, and pipeline plumbing. They do not publish a controlled study of dollars per unit. The category of loss is still obvious to anyone who has watched a delivered car sit dark on the website while the lot fills with snow.

Production crews and staging are the other spend. Not every unit needs a full crew day, a swept bay, and a lighting package to go live. When the ask is routine inventory media, paying that setup for every VIN is a recurring cost you feel in the calendar as much as in the invoice. A path that starts from one lot photo and lands on a cached showroom plate cuts how often that crew has to turn up for ordinary listings. Special cars can still get special shoots. The rest of the row should not wait on them.

I am writing this from Buffalo winters, not from a coasting climate. Outdoor lot shoots fail when snow is falling sideways and the windshields are white. Not every dealer can stage indoor photoshoots for every unit all season. When weather kills traditional production, inventory media still has to move. One photo to a showroom composite is the ops path for those weeks: keep listings alive when the lot is unusable as a set.

What one photo produces

One still goes in. The pipeline builds a vehicle matte, places the car on a cached or prompted showroom plate, and writes three artifacts:

*_composite.png for the listing
*_cutout.png for reuse
*_alpha.png as evidence, not a hidden intermediate

The alpha file is part of the product. If a merchandising manager rejects the frame, you can see what the matte actually decided. The stack behind that path is three named pieces, each with a clear honesty boundary:

CarSegNet still path: semantic prior, NeuralSegJet (NSJ) alpha refine, optional interior openings, plate, composite.
WindowSweep: paired glass supervision so cabin stays while reflected lot scenery can leave. Capture contract and audit tooling are real; a production glass model is not shipped.
CCR (Counterfactual Correctability Routing): product name for propose-then-verify repair routing. Openings safety and the metrics review gate are the live machinery today; a standalone CCR module is not.

Cached plates are the reliable production path: same floor, same lighting language, no provider drift between units. Prompted plates are optional when the plate library needs a new look, not something you should spend on every VIN.

Topology is the hard part, and the buyer feels it first

Most car cutout failures are edge failures: halo on the roofline, jagged mirror stalks, weak tire contact. Cars add a second class of error that region IoU barely notices: enclosed background. Wheel openings, spokes, grilles, roof rails, and glass are holes inside the object. A model can score well on the outer silhouette while painting those openings as solid vehicle. The lot then shines through a "solid" wheel, or the glass turns into painted metal.

Think of it like cutting a paper stencil of a car for a spray booth. Cutting the outer outline is one job. Cutting the windows and the gaps between spokes so paint can pass through is another. A perfect outline with the windows left filled is still a bad stencil. Boundary refinement can sharpen the contour. It cannot reopen every interior gap once the prior marked it foreground. That is why CarSegNet keeps those jobs in separate stages: each error has one place to inspect.

flowchart LR
  photo(["Dealer Lot Photo"]) --> prior["Semantic Vehicle Prior"] --> alpha["NSJ Alpha Refiner"]
  alpha --> holes{{"Opening Repair Optional"}} --> plate[("Cached Or Generated Plate")]
  plate --> comp["Composite"] --> booth(["TypeScript Booth Client"])

  classDef input fill:#1c2e28,stroke:#4a7a68,color:#d5ebe3
  classDef model fill:#1a2740,stroke:#4a6d9a,color:#d0dced
  classDef gate fill:#2a2436,stroke:#6e5f8c,color:#e2d8f0
  classDef plateNode fill:#2a261a,stroke:#8a7848,color:#efe6c8
  classDef output fill:#1a2c36,stroke:#4a7a8e,color:#cde4ee
  classDef client fill:#1e2a24,stroke:#5a8a6e,color:#d4ebe0

  class photo input
  class prior,alpha model
  class holes gate
  class plate plateNode
  class comp output
  class booth client

The openings stage is real code in carsegnet/openings.py, but configs/default.yaml ships with openings.enabled: false. Keep it off until the locked 608-frame topology acceptance gate passes. The topology thesis still drives the design when that stage is disabled, because region IoU alone will green-light a filled wheel, and a filled wheel is exactly what a shopper rejects.

Carvana topology: strong outer, almost no hole recovery

The loudest evidence for that split is the Carvana topology audit on a 608-frame, 38-vehicle validation split with the corrected still path (semantic text prompt, no detector box, NSJ on the training band).

Outer metrics for the trained NSJ checkpoint look strong: IoU 0.9908, boundary IoU 0.4762, SAD 5.4460, zero frames below 0.90 IoU. Topology tells a different story. Against 1,193 ground-truth enclosed holes, trained NSJ matched 5, left 1,188 missing, and introduced 111 extras. Raw semantic SAM matched 1. The fallback refiner matched 28 and exploded extras.

Preserve the trained checkpoint as a boundary ablation, not a production model that "solved openings." Those numbers are checkpoint validation on studio backgrounds with binary glass masks. They are not an untouched holdout and not a dealer-lot benchmark. Real lot photos and WindowSweep need separate evaluation. That contrast is why openings is a separate stage, not a footnote under IoU.

Glass, correctability, and what the booth actually does

Glass is where cheap background tools lie. They treat a windshield as keep-or-remove, and either erase the cabin or leave the lot reflection welded into the glass. What you want on a listing is different: keep the seats, the headrests, the cabin volume; replace the scene that is only sitting in the reflection.

WindowSweep is the supervision contract for that job. Same vehicle, same camera pose, same cabin rays, once with the target side glass raised and once lowered. Similar framing is not enough. If the camera moves, the pair is not teaching glass. It is teaching camera shake. The F-150 pilot under data/windowsweep/f150_pilot_001/ holds five declared pairs. Audit accepts both close pairs at the coarse geometry gate and rejects all three wide pairs because the camera moved. Those accepted pairs are enough to test registration and a first glass-loss prototype. They are not a training corpus. AutoLens product copy describes WindowSweep as in training for the same reason: the capture contract and audit tooling are real; a production glass model is not being claimed as shipped.

CCR names the dry-fit habit before you glue a repair: try a local detail fix in a controlled way, keep it only when the result earns its keep. There is no ccr.py in the CarSegNet tree today. What ships is the same discipline on interior openings and batch QA. openings.py proposes a hole cut, then refuses it if the proposal adds foreground, leaves the interior trust region, or changes the outer silhouette when verify_outer=True. metrics.py reports region accuracy, boundary accuracy, matting error, enclosed-hole topology, and a pass/fail gate so a batch job can route only questionable frames to a human. AutoLens currently lists CCR as coming soon. The openings safety contract and the metrics review gate are already how CarSegNet refuses to "fix" a unit into a worse listing.

After the matte lands on a plate, compose does four deterministic jobs in one pass: shift car colour toward the plate (global LAB, gain clamped), blur a plate that is sharper than the subject, drop a contact shadow under the tire band, and wrap a little plate colour into the vehicle edge. That stack can make a good matte believable. It cannot hide a broken one, and global LAB will not fix mixed lot lighting or reflective chrome that still mirrors the original lot.

The booth itself is thin on purpose. A TypeScript client talks to a FastAPI service over a single CUDA worker so concurrent jobs do not fight one 24 GB card. Progress rides a WebSocket with HTTP polling as fallback. Voice is a small control layer for overrides, not a second vision stack. CLI and booth share the same CarSegNet class: booth for interactive unit work, CLI for batch folders and matte-only runs.

Failure modes I keep in the open

A few limits are load-bearing, not footnotes:

If the segmenter fills a wheel and supplies no interior contour, NSJ cannot reopen it. That sentence is in PAPER.md and matches the Carvana topology table.
Openings stay disabled until acceptance evidence locks decoder thresholds. Enabling them without a readable weights checkpoint is a hard PipelineError.
Openings are still-image only; enabling them on video raises PipelineError.
WindowSweep's F-150 pilot is a geometry and glass-loss test bed, not a multi-vehicle training corpus.
CCR as a full product routing layer is named in the AutoLens stack and not yet a standalone shipped module; openings safety and the metrics review gate are the live correctability machinery today.
Harmonisation is global LAB. Mixed lot lighting will not suddenly look like a single softbox.
Reflective bodywork that mirrors the original lot is out of scope; the pipeline does not invent occluded geometry or rewrite chrome reflections.
Plate parallax on video is capped (about 3.5 percent of frame width in config). It cannot invent occluded geometry.
Isolated 1080p wall-clock timing is still an open benchmark in the README. This post does not invent an end-to-end minute count.
AGPL YOLO26 weights matter for commercial packaging; the detector is optional, depth is harder to replace.
Two external integrations still need live confirmation per the README: SAM 3.1 handle_stream_request on a real clip, and one live gpt-image-2 call. carsegnet doctor is the first stop before spending GPU hours or image credits.

The product ask sounds simple: one lot photo, one showroom composite, list the car today. The engineering ask is different: keep outer silhouette quality, interior hole honesty, glass supervision, plate style stability, and booth UX from collapsing into a single opaque model call that you cannot debug when a merchandising manager rejects the frame. A showroom composite is only as strong as the mask topology, the glass contract, and the refusal to publish a bad repair. That is also the sales case: same-day listing media without waiting on crews, staging bays, or a clear Buffalo sky, with named components you can point to when a buyer, a GM, or a skeptic asks what happens when the wheel spokes and the windshield disagree with the lot behind them.

For implementers

Repo evidence for the claims above lives in named files: carsegnet/pipeline.py, carsegnet/openings.py, carsegnet/metrics.py, carsegnet/compose.py, carsegnet/background.py, scripts/audit_windowsweep.py, and the config flags in configs/default.yaml. The rest of this section is the CLI, cascade, and booth detail a sales reader does not need in the first third.

CLI and booth entry

CLI still path with a cached plate:

carsegnet run photos/unit.jpg --bg-file plates/showroom.png

Prompted plate generation (optional path; needs OPENAI_API_KEY):

carsegnet run photos/unit.jpg --bg-prompt "wet city street at night, neon reflections, no cars"

Booth / service entry:

python server/app.py --host 0.0.0.0 --port 8080 --config configs/default.yaml

-c is the short form of --config in server/app.py. --warm exists and starts a background thread that imports carsegnet.pipeline; it does not yet call the worker's full model warm path, so treat it as a light import warm, not a guarantee that every weight is resident.

The repo targets a single 24 GB GPU worker (the README calls out an RTX 3090 Ti class card). Video uses the same service shape and runs longer. Still images are the center of the sellable path.

Still path cascade

carsegnet/pipeline.py owns CarSegNet.process_image and CarSegNet.matte_image. The still flow is linear: read the image, build the matte, load or create the plate, composite, write artifacts. Models load lazily on first use so a cold CLI call pays import cost once per process, not at module import time.

Default segmentation uses the text prompt car from configs/default.yaml. The YOLO26 box path in carsegnet/backends.py is opt-in via segmenter.use_box_prompt. Depth from yolo26x-depth.pt is on by default and feeds hero selection, contact shadow attenuation, and video plate parallax. If depth fails, the pipeline logs a warning and continues without it; shadows and parallax degrade, the matte path does not hard-stop. On a crowded lot that matters: you want the near car in the frame, not the largest blob two rows back.

Alpha refinement lives in carsegnet/nsj.py and carsegnet/refine.py. NeuralSegJet (NSJ) replaced dense Conditional Random Field (CRF) cleanup because pydensecrf does not build cleanly on the Python 3.12 stack the rest of the repo requires. NSJ works inside a trimap: confident foreground, confident background, and a narrow unknown band. A colour guided filter uses image colour so local smoothing follows visible edges. That band is good for hairline silhouette geometry: antenna wire, mirror stem, tire contact. Enclosed holes are left to carsegnet/openings.py when enabled, which is the right separation of jobs for a product you intend to sell and support.

Seed, extent, and the interior trust region

carsegnet/openings.py treats each candidate hole as a connected-component problem with two maps and one safety region.

Seed map: strict. High-confidence enclosed background only (wheel wells, floor through spokes, grille voids, glass). Seeds authorize a cut; they do not invent support alone.
Extent map: looser. Grows from a seed through nearby pixels that look like the same opening, capped by growth distance and area limits.
Interior trust region: prior-derived vehicle interior after excluding an outer silhouette guard band. Accepted pixels must stay inside it so the stage cannot eat fenders, bumpers, mirrors, or tire edges.

Decode rejects components that lack a seed, fall outside size limits, leave the trust region, or would change the exterior silhouette after subtraction (apply_opening_mask). Application is subtractive only inside the trust region. It raises OpeningSafetyError if the proposal adds foreground alpha, edits pixels outside trust, or changes the outer silhouette when verify_outer=True.

The craft analogy is a gasket, not a paint roller. The outer seal has to stay true. The bolt holes inside the gasket are separate cuts. If you enlarge a bolt hole until it breaches the outer seal, the part is scrap. Openings is built to prefer a missed hole over a breached fender, because a merchandising manager can send a filled wheel back for review, and cannot unsell a chewed quarter panel that already went live.

When openings stay disabled (the current default), the topology gate in carsegnet/metrics.py still reports the problem. Enclosed background components are matched one-to-one only when both directional coverages clear 50 percent. QA can flag every missing or extra hole even if region IoU looks excellent.

WindowSweep audit path

python scripts/audit_windowsweep.py --self-test
python scripts/audit_windowsweep.py \
  data/windowsweep/f150_pilot_001/windowsweep_manifest.json \
  --report runs/windowsweep/f150_pilot_001.audit.json

scripts/audit_windowsweep.py verifies the manifest, source SHA-256 hashes, and full-frame camera geometry before a pair is allowed near training. All eleven F-150 pilot images come from one vehicle and one session, and the manifest does not yet seal local glass regions, capture controls, split assignments, or training-rights evidence. The README says that plainly.

For new captures, mount the camera, then lock focus, exposure, and white balance before moving the window. Keep every frame from a vehicle and capture session in one split.

Plates and compositing detail

carsegnet/background.py owns plates. The prompt path calls OpenAI Images via client.images.generate, reads data[0].b64_json, and defaults to the pinned snapshot gpt-image-2-2026-04-21 in configs/default.yaml and that module. Pinning costs migration work when the provider changes. It also stops a cached matte from pairing with a drifting plate style. Retired DALL-E aliases raise BackgroundError rather than silently falling through. Prompt results land in ./.bgcache keyed by SHA-256 of prompt, model, size, and quality.

Default background size is 2048x1152 at quality: high, with retries and a 180 second timeout. That is why a live plate call is called out separately from matte work: it is money and latency outside the GPU path.

carsegnet/compose.py applies four deterministic adjustments after fitting the plate to the frame:

Step	Job	Cost
Harmonisation	Shift the car LAB statistics toward the plate (Lightness, A green-red, B blue-yellow), gain clamped.	Unusual paint and tinted glass can move too far. Global LAB will not fix mixed illuminants.
Defocus matching	Blur the plate when it is sharper than the subject (Laplacian variance, capped sigma).	Missing subject sharpness cannot be recovered.
Contact shadow	Ground the tires on the new floor from the lowest band of the silhouette, depth-attenuated when depth exists.	A bad footprint gives a weak shadow, and the car starts to float.
Light wrap	Mix a little plate colour into the vehicle edge.	Excess wrap reads as haze on the body line.

Server, WebSocket, and voice

There are two booth surfaces in the repo. server/app.py serves the FastAPI API and the static web/index.html client at /. web-next/ is a separate Next.js booth client that talks to the same API. Neither hosts the models in the browser.

GPU work runs behind FastAPI on CUDA. One worker thread serializes jobs so concurrent model loads do not fight a single 24 GB card. Job states move queued → running → done | error. Progress publishes on WebSocket /api/jobs/{job_id}/events. web-next/lib/useJob.ts follows that socket and falls back to HTTP polling every second (up to about 15 minutes) when sockets are blocked. Uploads are POST /api/jobs with multipart file plus JSON overrides; outputs are path-confined under runtime/outputs/{job_id}/.

Voice is a small control layer, not a second vision stack. web-next/lib/useVoice.ts uses browser speech-to-text; audio stays on the machine. The server parses the transcript in carsegnet/voice.py into a typed action (with a regex fallback when no API key is present) and clamps overrides before they hit the worker.

flowchart LR
  browser(["TypeScript Booth Client"]) --> api["FastAPI Service"] --> worker["Single GPU Worker"]
  worker --> models[["Prior NSJ Openings"]]
  worker --> outputs[("Composite Cutout Alpha")]

  classDef client fill:#1e2a24,stroke:#5a8a6e,color:#d4ebe0
  classDef service fill:#1a2740,stroke:#4a6d9a,color:#d0dced
  classDef compute fill:#241e30,stroke:#6e5f8c,color:#e2d8f0
  classDef model fill:#1a2740,stroke:#4a6d9a,color:#d0dced
  classDef artifact fill:#1a2c36,stroke:#4a7a8e,color:#cde4ee

  class browser client
  class api service
  class worker compute
  class models model
  class outputs artifact

CLI supports batch folders, config overrides (--set refine.band_px=18), and matte-only runs (carsegnet matte). Pick CLI or booth based on who is watching the screen and how many VINs are in the queue, not based on which path sounds more advanced.

Measurements that prove plumbing vs topology

python tests/selftest.py exercises orchestration, encoding, compositing, openings safety contracts, and quality-assurance reporting on CPU with stubbed model backends. Forty checks, no network, no GPU, no model downloads. That proves plumbing, not dealer-lot matte quality.

The synthetic selftest hard case (thin antenna, dilated and blurred prior) shows the fallback refiner can recover boundary IoU on that scene; it does not measure wheel openings or glass transmission. Upstream SAM 3.1 speed claims and config comments about video.compile are not local dealer-lot timings either. If you are selling this system, sell the measured parts as measured and the gated parts as gated.

Separating SAM prior, NSJ band refine, optional openings repair, WindowSweep glass supervision, CCR-style propose-then-verify routing, pinned or cached plates, and deterministic compose means each failure has a file. carsegnet/openings.py for seed or trust violations. carsegnet/compose.py for a shadow that does not sit under the tires. carsegnet/background.py for a plate model that drifted. scripts/audit_windowsweep.py for a glass pair whose camera moved. web-next/lib/useJob.ts for a socket that died and should have fallen back to polling.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

Messages Need a Protocol Before They Need a Chat UI

Daniel Romitelli — Fri, 31 Jul 2026 20:07:15 +0000

A patient taps a kiosk and asks for help. A staff member replies from another screen. The same message may appear in the app, trigger a push notification, fall back to Short Message Service (SMS), and later show as read.

If each screen decides what those events mean, the thread turns into a rumor mill. One client marks the message as sent. Another treats a push attempt as delivery. A third retries after a network pause. In a clinic, that ambiguity creates audit gaps, extra staff work, and a patient expectation problem: sent must mean something actionable.

The rule I wanted was simple: every message has a durable state, every transition has an owner, and urgent communication can leave the app channel when policy allows it.

That rule is why I built BidirectionalMessagingService as the coordination layer for kiosk messaging, rather than treating chat as a screen-level feature.

1. The invariant comes before the interface

The kiosk already had services for push, SMS, email, and templates. Those services deliver through channels. They do not decide the truth of the thread.

BidirectionalMessagingService owns the message lifecycle: creation, delivery policy, acknowledgement handling, read receipts, fallback decisions, and teardown. Push, SMS, and email remain transports.

flowchart TD
patient[Patient Kiosk] --> service[BidirectionalMessagingService]
staff[Staff Client] --> service
service --> thread[Message Record]
service --> push[Push Notification]
service --> sms[SMS Fallback]
service --> email[Email Delivery]
service --> receipt[Read Receipt]

The cost is ceremony. A basic widget can append text and look finished. This version asks callers to provide role, priority, permitted channels, and acknowledgement behavior. That extra shape buys one authority for changing the record.

2. The row carries the user-facing truth

The message model stores identity, direction, status, timestamps, and attachments. The durable status set is intentionally small:

status: 'sent' | 'delivered' | 'read' | 'failed';
sentAt: Date;
deliveredAt?: Date;
readAt?: Date;

Four states, and a timestamp for three of them. There is no queued, no pending, no retrying. Every one of those describes what a worker is doing rather than what happened to the patient, and a status a patient cannot act on has no business in a column they can see.

Queue membership stays in memory while work is active. It never becomes a database value. A queued item is a worker condition, not a patient-visible fact. Persisting it would force every client to explain whether the item is safe, blocked, or retrying.

Delivery options are policy, rather than a single flag:

Field	What it decides	Cost
`channels`	Which transports may carry the message	More combinations to test
`urgency`	How aggressively the system alerts users	Greater risk of alert fatigue
`requireDeliveryConfirmation`	Whether delivery needs acknowledgement	More bookkeeping
`fallbackToSMS`	Whether the message may leave the app channel	Higher external dependency surface
`translationLanguage`	Whether content needs language adaptation	More transformation risk

Timeouts live in the same policy layer. The clock starts with the active delivery attempt, and every permitted transport races against it:

const results = await Promise.allSettled(deliveryPromises);
const successfulDelivery = results.some(r => r.status === 'fulfilled' && r.value.success);

await supabase.from('messages').update({
  status: successfulDelivery ? 'delivered' : 'failed',
  deliveredAt: successfulDelivery ? new Date() : null,
}).eq('id', message.id);

allSettled rather than all is the whole point. One transport failing is not the message failing. Push can time out while SMS gets through, and the row should say delivered because a human received it. The status collapses many attempts into the single fact a patient and a nurse both need. Presence can influence routing, but it never creates another stored status.

3. Receipts and teardown are side effects

A read action moves the message to read. Notifying the sender is a side effect of that transition. If the sender notification fails, the patient still read the message. Mixing those facts would make the history less reliable.

The same ownership applies to runtime resources. Realtime subscriptions and pending work need an explicit end, so the service owns cleanup instead of scattering it across screens:

cleanup(): void {
  for (const [_, channel] of this.activeChannels) {
    channel.unsubscribe();
  }
  this.activeChannels.clear();
  this.messageQueue.clear();
}

Three lines of teardown, and the queue is cleared alongside the subscriptions on purpose. Anything still waiting to send belonged to the session that just ended. Draining it into the next one would deliver a previous patient's message to whoever is standing at the kiosk now.

That matters in a shared device flow. One patient can walk away, another can begin check-in, and an old subscription should have no vote in the new session.

4. The review table

Once the lifecycle is explicit, each operation has one test: it changes the durable record, triggers a side effect, or both.

Event	Owner	Durable status outcome	Side effects
Staff sends message	Messaging service	`sent`	Channel delivery attempts
Transport confirms delivery	Channel adapter	`delivered`	Optional confirmation notice
Patient opens message	Messaging service	`read`	Read receipt notification
Active attempt expires	Messaging service	`failed`	Optional fallback path before failure
Sender receives receipt notice	Receipt handler	No new message status	User interface acknowledgement

Consumer chat can tolerate fuzzy indicators. A clinic kiosk has less room for soft meaning because staff coordinate care through the thread and patients expect a reply to reach someone. With explicit states and owners, one status record survives channel retries, fallback routing, and client differences. The interface can vary; the thread still tells one story.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

A Cache Key Is an Equivalence Relation

Daniel Romitelli — Tue, 28 Jul 2026 08:54:10 +0000

A retry should not become a new creative decision.

When a video job fails halfway through and runs again, the user still asked for the same scene. If a timestamp or retry count changes the lookup, the system pays for another clip. If the lookup ignores a model route or seed, it can return the wrong artifact with a perfectly valid URL.

I built the video generation pipeline’s cache around that split. The hash is the system’s definition of sameness.

1. The request definition gets trimmed before hashing

The compiler produces a GenerationContract: prompt inputs, selected model route, generation mode, constraints, seed, and execution metadata. Only the artifact-defining fields enter the digest.

The implementation lives in lib/scene-compiler/ast-cache.ts:

/**
 * Hash the compiled AST to a 64-char hex string (SHA-256).
 * Identical contracts produce identical hashes regardless of
 * volatile per-request fields like timestamps or scene IDs.
 */
export function hashCompiledAST(contract: GenerationContract): string {
  const content = extractHashableContent(contract)
  return createHash("sha256").update(content).digest("hex")
}

/**
 * Build a cache key with the `ast:` prefix for Supabase lookup.
 */
export function buildCacheKey(contract: GenerationContract): string {
  return `ast:${hashCompiledAST(contract)}`
}

Abstract Syntax Tree (AST) hashing is the mechanism; classification is the design. The hashable content includes sourceInputs.prompt, sourceInputs.imageUrl, sorted referenceImageUrls, chosenModel, chosenEndpoint, generationMode, sorted constraints, and seed ?? null. Volatile fields such as compiledAt, sceneId, idempotencyKey, and retries stay out.

flowchart TD
  contract[GenerationContract] --> split{Classify each field}
  split -->|artifact identity| keep["prompt, imageUrl, referenceImageUrls,<br/>chosenModel, chosenEndpoint,<br/>generationMode, constraints, seed"]
  split -->|run history| drop["compiledAt, sceneId,<br/>idempotencyKey, retries"]
  keep --> norm["Sort constraints by type, then target"]
  norm --> sha["SHA-256 over the serialized fields"]
  sha --> key["Cache key, ast prefix plus 64 hex"]
  drop --> excluded["Never reaches the digest"]

The cost is ongoing ownership. Every new field in the request shape needs a decision: artifact identity or run history. Ambiguity becomes either wasted generation or false reuse.

2. Normalization removes accidental difference

Constraints are structured values, and array order can reflect construction path rather than meaning. I normalize them before serialization:

function sortConstraints(
  constraints: SceneConstraint[]
): Array<{ type: string; target: string; value: unknown }> {
  return constraints
    .map((c) => ({ type: c.type, target: c.target, value: c.value }))
    .sort((a, b) => {
      const typeCompare = a.type.localeCompare(b.type)
      if (typeCompare !== 0) return typeCompare
      return a.target.localeCompare(b.target)
    })
}

That rule makes two request definitions with the same constraint set hash together even if they were assembled in a different order. The tradeoff is explicit: ordering cannot carry semantic weight here. If priority later depends on position, this function has to change before the data model does.

3. The two bugs sit on opposite sides

Hash too much, and every retry misses. compiledAt lets the clock break reuse. retries turns attempt two into a separate artifact. idempotencyKey belongs to transport safety, so it should not alter creative identity.

Hash too little, and stale output looks correct. Prompt-only reuse ignores chosenModel, chosenEndpoint, generationMode, and seed. That failure is worse than a miss: the pipeline receives a real video URL and continues with the wrong clip.

Tests pin the promises: metadata changes such as compiledAt, sceneId, and idempotencyKey preserve hashCompiledAST, while buildCacheKey must match ^ast:[a-f0-9]{64}$. The same file also defines provenance-aware steering hashes with a separate steer: prefix, so namespacing is part of the storage contract.

Retries keep a stable artifact identity, and different routes and seeds stay isolated. A content-addressable lookup is only as correct as the equality rule behind it; write that rule in the language of the generated thing, then let the hash enforce it.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

Workflow JSON Is Generated Code

Daniel Romitelli — Mon, 27 Jul 2026 22:51:11 +0000

A screen recording can show a whole job without explaining it. Someone opens an inbox, checks a sender, copies a value into a customer record, compares it with a spreadsheet, sends a summary, and moves on.

You can see the work. Automation still has to survive a harder test: can the system put the job back together without dropping a step, wiring the wrong action, or importing something that looks right and fails later?

I built the n8n side of the screen-analysis project around that question. The generator does not treat the final file as a bag of text. It turns discovered automations into N8NWorkflow, N8NNode, and connection objects, then emits n8n-compatible JavaScript Object Notation (JSON). That's more ceremony than gluing strings together, but it catches a class of mistakes string assembly invites.

1. Keep the platform vocabulary narrow

Every emitted node type comes from NodeType in n8n_workflow_generator.py, an enumeration covering the triggers, language model nodes, and application integrations the generator knows how to create. Need another n8n node? I add it there before generation can use it.

That costs editing speed. A one-off node can't slip through by spelling a new identifier in a prompt. The gain is sharper failure: unsupported platform names fail in Python instead of hiding inside an importable file.

Agent configurations in n8n_agent_templates.py work the same way. AgentTemplate names the available patterns; AgentConfig carries the prompt, tools, integrations, trigger preferences, model choice, temperature, and iteration limit. The prompt gets one field. It doesn't double as the container for everything else.

from __future__ import annotations

from dataclasses import dataclass, field
from enum import Enum
from typing import List


class AgentTemplate(Enum):
    EMAIL_TRIAGE = "email_triage"
    CRM_DATA_SYNC = "crm_data_sync"
    CALENDAR_ASSISTANT = "calendar_assistant"
    DOCUMENT_PROCESSOR = "document_processor"
    COMMUNICATION_ROUTER = "communication_router"
    REPORT_GENERATOR = "report_generator"
    LEAD_QUALIFIER = "lead_qualifier"
    TASK_MANAGER = "task_manager"
    VOICE_ASSISTANT = "voice_assistant"
    MULTI_AGENT_ORCHESTRATOR = "multi_agent_orchestrator"


@dataclass
class AgentConfig:
    name: str
    description: str
    template: AgentTemplate
    system_prompt: str
    tools: List[str] = field(default_factory=list)
    integrations: List[str] = field(default_factory=list)
    triggers: List[str] = field(default_factory=list)
    llm_model: str = "gemini-2.5-flash"
    temperature: float = 0.7
    max_iterations: int = 10

The schema is also a constraint. If a new automation needs a concept AgentConfig cannot express, I extend the model first. Experiments get slower, and the export path stays honest.

2. Build objects before JSON

The generator's structured form is the n8n graph itself: nodes plus named connections. There's no second private graph format sitting behind it. N8NWorkflow, N8NNode, and connection records are the representation between analysis results and the saved JSON file.

flowchart TD
  analysis[Discovered automation] --> workflow[N8NWorkflow]
  workflow --> nodes[N8NNode objects]
  workflow --> connections[Connection records]
  nodes --> json[n8n JSON export]
  connections --> json
  json --> importer[REST importer]
  importer --> status[Import status]

That distinction matters. A detected step such as “classify this email” is mapped to concrete n8n nodes only when the generator has enough context to choose a trigger, model, integration, and connection order. Positioning is computed separately from identity, so the canvas stays readable without tying layout to node IDs.

The tradeoff is flexibility. Deterministic placement cannot match a hand-arranged canvas, and typed construction is heavier than editing a JSON file directly. For generated automations, I prefer predictable inspection over perfect visual layout.

The core object shape is simple:

from __future__ import annotations

from dataclasses import dataclass, field
from typing import Any, Dict, List


@dataclass
class N8NNode:
    id: str
    name: str
    type: str
    position: List[int]
    parameters: Dict[str, Any] = field(default_factory=dict)
    credentials: Dict[str, Any] = field(default_factory=dict)
    type_version: float = 1.0

    def to_dict(self) -> Dict[str, Any]:
        node_dict = {
            "id": self.id,
            "name": self.name,
            "type": self.type,
            "position": self.position,
            "parameters": self.parameters,
            "typeVersion": self.type_version,
        }
        if self.credentials:
            node_dict["credentials"] = self.credentials
        return node_dict

This is the part that makes the JSON feel like generated code. The object owns identity, type, parameters, credentials, version, and position before serialization happens. By the time the file exists, the important decisions have already passed through inspectable Python structures.

3. Treat import as deployment state

Generation ends at a file; operation begins when that file reaches n8n through the Representational State Transfer (REST) API. In n8n_importer.py, importer failures have a named exception, and import progress has explicit states.

from enum import Enum


class N8NError(Exception):
    """Custom exception for n8n API errors."""


class ImportStatus(Enum):
    PENDING = "pending"
    IMPORTING = "importing"
    SUCCESS = "success"
    FAILED = "failed"
    REQUIRES_CREDENTIALS = "requires_credentials"

A credential problem and a failed import need different recovery paths, so they get different labels. The deploy flow can generate automations, create supporting agent files, import them, and leave activation switched off while credentials are handled.

That separation removes convenience. One button that generates, imports, credentials, and activates would be faster for a demo. In production, splitting those actions makes partial failure recoverable.

4. The table I test against

Layer	Question it answers
`AgentTemplate`	Which automation pattern is being built?
`AgentConfig`	Which prompt, tools, integrations, trigger, and model settings describe it?
`NodeType`	Which n8n identifiers may be emitted?
`N8NWorkflow` / `N8NNode`	Which graph becomes JSON?
`ImportStatus` / `N8NError`	What happened when the artifact reached n8n?

This costs more than string interpolation: extra enums, dataclasses, object construction, save steps, and importer reports. It also makes schema changes explicit. I pay the cost because automations inferred from screen recordings already start with uncertainty; the export path should reduce it.

Workflow JSON is code the moment it can move data, call models, and route work. Treating it as generated code is how I keep a discovered process from becoming an imported accident.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

Why I Kept Search Scope Inside a Single Supabase RPC

Daniel Romitelli — Mon, 27 Jul 2026 16:03:04 +0000

The embedding was right. The similarity score was right. The answer was completely wrong.

The chunk that came back shared vocabulary with the query, shared naming conventions, shared architectural patterns. It also came from the wrong repository. Nothing crashed, nothing timed out, and I went two days without catching it.

Plausible neighbors from the wrong scope is the failure mode I'd rank as the most dangerous one in vector search. The embedding math was fine. The rows were valid. The trouble was that search decisions were getting assembled in too many places, so by the time the request reached PostgreSQL, the database had to guess which parts belonged together.

So I made the RPC the single source of truth. The caller sends the query embedding, the candidate count, and the JSONB filter together. The SQL function receives those same values together. The index is built for the same embedding column the function reads. Once those pieces move as one unit, the path stops wandering.

flowchart TD
  query[Query embedding] --> payload[RPC payload]
  count[match_count] --> payload
  filter[JSONB filter] --> payload
  payload --> rpc[Supabase RPC]
  rpc --> sql[search_embeddings]
  sql --> rows[Ranked rows]

The bug was scope, not similarity math

My first mistake was treating search as a handful of knobs instead of one request object. Build the embedding in one place, the metadata filter in another, decide candidate depth a layer above that, and the call boundary turns to mush. The function still runs. But nobody can point at one object and say: this is the exact search intent.

That matters most when retrieval is scoped, and in this codebase the filter is no afterthought. It can describe a repo, a file path, a language, a type, or any combination of metadata fields that belong in the same search slice. If I'm looking for TypeScript files in a specific repository, I want that expressed as part of the same request that carries the vector. I don't want the application inferring scope from session state, hidden defaults, or whatever the previous call happened to do.

The symptom was plausible-but-wrong neighbors, because vector similarity is happy to rank related text from the wrong place. Retrieval bugs are slippery for exactly that reason. The results don't look random. They look close enough to distract you. A chunk from the wrong repo can still share concepts, terminology, or naming conventions with the query, and if the filter is applied too late, that wrong row can look like a good answer right up until you inspect the metadata closely.

The fix was to make the request shape dull and explicit. The caller decides the search scope. The database enforces that scope. The index serves that same scope. No second pass trying to patch over a weaker request after the fact.

The caller sends one object

The wrapper I use is intentionally small. It doesn't hide the request shape, and it doesn't smuggle in extra search behavior. Vector, count, filter, passed straight into search_embeddings.

import { SupabaseClient } from '@supabase/supabase-js';

type SearchFilter = Record<string, unknown>;

interface SearchResult {
  id: string;
  document_id: string;
  chunk_text: string;
  similarity: number;
  metadata: Record<string, unknown>;
}

export async function runSearch(
  client: SupabaseClient,
  queryEmbedding: number[],
  matchCount = 5,
  filter: SearchFilter = {}
): Promise<SearchResult[]> {
  const { data, error } = await client.rpc<SearchResult>('search_embeddings', {
    query_embedding: queryEmbedding,
    match_count: matchCount,
    filter,
  });

  if (error) {
    throw error;
  }

  return data ?? [];
}

I like this shape because it's hard to misunderstand. At call time only three inputs matter: the embedding, the number of rows to return, and the structured scope filter. Searching inside one repo means passing a repo filter. Narrowing by language means adding a language key. Restricting by file type or path, same deal. The caller isn't building a query plan. It's declaring intent.

The same function can be called with a narrow filter or an empty one. An empty filter means search the whole corpus. A populated one means search the subset that matches the metadata predicate. Very different outcomes, and I want that difference visible right where the request gets created.

A typical call site stays just as clear:

const results = await runSearch(supabase, embedding, 8, {
  repo: 'the author/portfolio',
  language: 'typescript',
});

That's the point. The search boundary should be obvious at a glance. When I'm debugging a bad answer later, I want to inspect one payload and know exactly what the database was asked to do.

The database function matches the same interface

On the PostgreSQL side, search_embeddings accepts the same three inputs the caller sends. The metadata filter stays inside SQL, where it belongs. Rows get filtered first, then ranked by vector distance.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE OR REPLACE FUNCTION search_embeddings(
  query_embedding vector,
  match_count integer DEFAULT 5,
  filter jsonb DEFAULT '{}'::jsonb
)
RETURNS TABLE (
  id uuid,
  document_id uuid,
  chunk_text text,
  similarity double precision,
  metadata jsonb
)
LANGUAGE sql
STABLE
AS $$
  SELECT
    e.id,
    e.document_id,
    e.chunk_text,
    1 - (e.embedding <=> query_embedding) AS similarity,
    e.metadata
  FROM embeddings e
  WHERE filter = '{}'::jsonb OR e.metadata @> filter
  ORDER BY e.embedding <=> query_embedding
  LIMIT match_count;
$$;

CREATE INDEX IF NOT EXISTS idx_embeddings_embedding
  ON embeddings
  USING hnsw (embedding vector_cosine_ops);

The line carrying the weight is the metadata predicate: e.metadata @> filter. That isn't a cleanup step after ranking. It's part of the search itself. Rows outside the requested scope never enter the ranked candidate set.

The design matters because the database is the only place that can apply the filter consistently at the same moment it applies similarity. Filter in the application after ranking, and the query can still surface neighbors from the wrong scope first. Put the filter inside SQL, and ranking only happens across rows that already belong to the same metadata neighborhood as the request.

The similarity field is there for the caller's benefit. Cosine distance still drives the ordering internally. Externally, I want a score where larger reads as better. Returning both that score and the raw chunk text gives the next stage enough context to render, inspect, or rerank without another round trip.

The STABLE marker fits the way I use the function too. For a fixed snapshot and a fixed input payload, this is a deterministic retrieval step. It isn't a side-effect machine. It's a search function.

Why JSONB belongs in the request, not outside it

The filter is JSONB because the scope is structured rather than free-form. A metadata filter can say more than one thing at once, and it needs to do that without collapsing into a pile of ad hoc parameters. One object can describe a repository slice, a file path constraint, a language constraint, or a file-type constraint.

So there's a single place to express the search boundary. To retrieve only TypeScript chunks from a particular repository, I don't want to assemble a special query for that case. I want to build a JSONB object like { repo: 'the author/portfolio', language: 'typescript' } and pass it straight through. The SQL predicate then enforces exactly that constraint.

It's also why I keep the filter visible at the RPC boundary rather than burying it in a helper that rewrites inputs behind my back. Hidden rewrite logic is how search calls become hard to reason about. A JSONB object is simple enough to inspect, easy to log, and unambiguous in SQL.

Debugging gets a second benefit out of this. When a query returns too much, I can loosen the filter and watch the effect immediately. When it returns too little, I can inspect which metadata keys are actually present in the table. Because the filter is part of the request, there's no mystery about which layer decided the corpus was too broad or too narrow.

The same holds when I expand the metadata model. If I start attaching more structure to a chunk, file type or path segments say, the RPC signature doesn't need to change. I update the JSONB shape, then let the same search_embeddings function enforce the new predicate. The request boundary holds still while the metadata vocabulary grows.

Why the index is part of the same story

I don't think about the index as a separate optimization pass. It's part of the same guarantee the RPC makes. If the function says it'll search the embeddings table with cosine distance, the index should be built for that exact access path.

Which is why the HNSW index sits right beside the function in my mental model. The function defines the agreement. The index makes that agreement fast enough to lean on all the time. vector_cosine_ops matches the ranking strategy, so the storage layer isn't fighting the retrieval layer.

What's nice about HNSW here is that it matches the shape of the workload I care about: lots of dense vector searches, with a metadata filter that keeps the working set scoped before anything gets ranked. I'm not asking the index to do the filter's job. Each piece does the job it's good at. The metadata predicate narrows the rows, and the vector index ranks whatever remains.

That separation is what keeps the system predictable. If I ever need to inspect performance, I know where to look. Wrong neighbors showing up, I inspect the filter. Right neighbors arriving slowly, I inspect the index and the shape of the vector column. The responsibilities don't blur together.

What went wrong the first time

The first broken version of this path made the request feel more flexible than it really was. The caller knew one thing, the search function inferred another, and the database was left to reconcile them later. That's accidental complexity of exactly the sort that makes retrieval bugs hard to pin down.

The visible symptom was a result set that looked sane at a glance. The hidden problem was a request that didn't fully describe the scope of the search. Which is why the bug survived long enough to matter. Nothing crashed. Nothing timed out. The system answered the wrong question with confidence.

Once I stopped spreading that decision across layers, the failure mode went away. The request object became the single place where I could answer three questions at once:

What text or embedding is this search about?
How deep should the candidate set be?
Which metadata fields are allowed to participate?

Much better debugging surface than a trail of local variables and implicit defaults. When I have to reason about a bad answer, I want to reason about one request object and one SQL function. That's enough.

Why I trust the boundary now

The version I trust is the one where the search request is explicit enough that the database never has to guess. The caller passes the vector, the candidate count, and the JSONB filter in one place. The SQL function applies the filter inside the ranking query. The HNSW index is built on the same embedding column the function reads. Every step agrees on the same shape.

That's the whole reason search scope lives inside a single Supabase RPC. Not because it's fashionable, and not because it makes the code shorter, but because it keeps the search intent attached to the request that asked for it. The RPC boundary becomes the line where scope is declared and enforced.

Once I made that change, retrieval stopped feeling like a chain of guesses and started feeling like a reliable interface again. That matters in a system where a correct answer is only useful if it comes from the right slice of data. In the next pass, I'm pushing that same discipline further into the ingestion side, because retrieval only stays trustworthy when the embeddings and metadata that feed it are just as deliberate.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

The AgentGroupChat Pattern That Keeps the Mapper from Drifting

Daniel Romitelli — Mon, 27 Jul 2026 16:02:52 +0000

The first version of this orchestration failed in a very specific way: the mapper kept producing templates that looked plausible to a human and were wrong for the platform. The failure was not that the model could not reason. What I had built was one broad conversation carrying too many jobs at once, so a wrong structure got accepted early and then rode all the way to the end. By the time the validator complained, the chain had already lost the distinction between analysis, mapping, generation, and approval.

So I stopped treating the system like a prompt stack and started treating it like a state machine. In the workflow analyzer SaaS, runner/azure_foundry/src/orchestrator.py does the important work. It creates the kernel, registers the plugins, assembles the agents, defines who speaks next, and decides when the run is done. Wrapped around that orchestration is the state manager in runner/azure_foundry/src/state_manager.py, which lets a run resume from the last durable message history instead of inventing a fresh conversation every time. One design choice, and the behavior of the whole pipeline changed.

The architecture I wanted was easy to describe and annoying to get right. Each agent gets one contract. Each contract produces one kind of output. The validator is the gate that decides whether that output can move forward. The analyzer looks at the workflow analysis and identifies automation opportunity. The mapper turns that opportunity into abstract integration steps. The generator turns the mapped steps into a platform-shaped template. And the validator checks the structure, then either approves the run or sends it back to the generator.

Splitting the roles was part of what made this work. The other part was making the next speaker explicit, which is what AgentGroupChat gives me here: a selection strategy, a termination strategy, and a history buffer that can be rehydrated before the chat starts. With those pieces in place, the system stops behaving like a free-form exchange and starts behaving like a controlled pipeline.

What broke first

The first architecture looked tidy on paper and failed under repetition. I had an analysis step, a mapping step, a generation step, and a validation step, but the conversation around them was too loose. The mapper started inventing platform syntax because nothing in the orchestration made that impossible. The generator then built on top of that invented syntax. The validator only saw the problem after the wrong shape had already been repeated several times.

Retries were worse. If the run timed out or the validator rejected the output, the next attempt often lost the conversation history that explained why the output had failed, so the system retried from a weak starting point and repeated the same mistake. What it needed was not more creativity. It needed a strict memory of what had already happened, plus a strict rule about which agent was allowed to correct which class of mistake.

Hence the narrow contract chain with validator-led convergence. The validator does not try to repair everything. It does one job: decide whether the generated template is acceptable. No, and the generator gets another pass. Yes, and the run terminates. Correction stays local instead of every agent piling into every mistake.

The core cycle in `AgentGroupChat`

The orchestration in runner/azure_foundry/src/orchestrator.py is built around AgentGroupChat with two strategies attached: a selection strategy that decides the next speaker, and a termination strategy that decides when to stop. That pairing is the heart of the pattern.

The selection strategy reads the conversation history and follows a fixed pipeline:

Analyzer speaks first.
Mapper speaks second.
Generator speaks third.
Validator speaks fourth.
If the validator returns INVALID, the next speaker is the Generator again.
If the validator returns WORKFLOW_APPROVED, the chat ends.

Compare that to a single model answering everything inside one giant instruction block. Different shape entirely. The selection strategy makes the pipeline visible to the runtime, and it gives me somewhere to encode the retry rule explicitly instead of hiding it inside a paragraph of instructions.

The termination strategy carries just as much weight. The validator is the only agent whose result can end the run, and the run still has a maximum iteration cap of 10 so a bad cycle cannot spin forever. That cap earns its place because agent cycles do not usually fail in dramatic ways. They fail by wobbling. Repeat a small wobble enough times and you get wasted tokens, delayed jobs, and outputs that never settle.

Here is the shape of that control flow in a standalone Python example that mirrors the same idea. It does not depend on Azure or Semantic Kernel, but it shows the exact contract I want the orchestrator to enforce:

from dataclasses import dataclass
from enum import Enum
from typing import Any, Dict, List, Optional


class AgentName(str, Enum):
    ANALYZER = 'Analyzer'
    MAPPER = 'Mapper'
    GENERATOR = 'Generator'
    VALIDATOR = 'Validator'


@dataclass
class Message:
    role: str
    content: str
    name: Optional[str] = None


def rehydrate_history(history_state: List[Dict[str, Any]]) -> List[Message]:
    return [
        Message(
            role=item.get('role', 'user'),
            content=item.get('content', ''),
            name=item.get('name')
        )
        for item in history_state
    ]


def next_speaker(history: List[Message]) -> Optional[AgentName]:
    if not history:
        return AgentName.ANALYZER

    last = history[-1]
    if last.role == AgentName.VALIDATOR.value:
        if 'WORKFLOW_APPROVED' in last.content:
            return None
        if 'INVALID' in last.content:
            return AgentName.GENERATOR

    order = [AgentName.ANALYZER, AgentName.MAPPER, AgentName.GENERATOR, AgentName.VALIDATOR]
    seen = [msg.role for msg in history if msg.role in {agent.value for agent in order}]
    if not seen:
        return AgentName.ANALYZER

    last_agent = AgentName(seen[-1])
    idx = order.index(last_agent)
    return order[min(idx + 1, len(order) - 1)]


if __name__ == '__main__':
    restored = rehydrate_history([
        {'role': 'Analyzer', 'content': 'Detected CRM handoff', 'name': 'Analyzer'},
        {'role': 'Mapper', 'content': 'Trigger + action pair identified', 'name': 'Mapper'},
        {'role': 'Generator', 'content': '{"triggers": [], "actions": []}', 'name': 'Generator'},
        {'role': 'Validator', 'content': 'INVALID: missing triggers and actions', 'name': 'Validator'},
    ])
    print(next_speaker(restored))

Small as it is, that example captures the behavior I care about most: the validator does not restart the chain, and the chain does not forget where it left off. If the validator rejects the output, the next speaker is the Generator, not the Analyzer, because the analysis step already did its job.

How history gets rehydrated before the run

Resumability matters here because the orchestration runs inside Prompt Flow, not inside a single in-memory toy conversation. In the real flow, state_manager.py loads the prior session history before the orchestrator node runs. That history arrives as history_state, and the orchestrator reconstructs ChatMessageContent objects from each saved message so AgentGroupChat can continue the conversation instead of starting over.

Easy detail to miss. Expensive one to ignore. Without rehydration, every timeout becomes a reset. With it, the system can pick up from the last durable state and keep going. If the validator has already rejected a malformed template, that rejection stays in history. If the mapper already established the target platform and the apps involved, that context is still available. If the previous run ended halfway through generation, the next run does not have to relearn the same facts.

The lifecycle is straightforward:

load session history from state_manager.py
reconstitute the saved messages into the chat history
run the orchestrator with the restored history
persist the resulting execution state after the run

Durability comes from exactly that. The system continues the same conversation under the same constraints, rather than pretending the previous attempt never happened.

The template library earns its keep

The template library in runner/template_library hands the generator a starting point that already matches the platform family. The generator should not be inventing a whole import shape from memory. It should be filling a known skeleton with detected apps, triggers, and actions.

Starting from a template base, the generator can spend its effort on the parts that actually need reasoning: mapping the workflow into the right platform structure, inserting the right application names, filling the right fields. The library keeps it from wandering into a malformed top-level structure.

Which is also why the validator can stay narrow. If the generator is working from a known shape, the validator does not need to be a general-purpose critic. The validator only has to check the contract that matters for the chosen platform: do the expected keys exist, is the structure complete, and does the result satisfy the platform rules well enough to be accepted?

Every piece leans on the others. The template library constrains the starting point, the generator fills it, the validator checks it, and the selection strategy decides whether the system should move forward or send the work back to the generator.

The real runtime wiring in the orchestrator

The orchestrator itself is where the system becomes code-first instead of prompt-first. The kernel gets initialized with Azure OpenAI configuration from the environment, the validation plugin is registered, and the template library plugin is registered, all before the conversation begins. Then the agents are assembled and the selection and termination strategies are attached.

Ordering is not incidental here. The model cannot speak before the kernel has its service. The generator cannot rely on the template library unless the plugin is present. The validator cannot enforce the schema unless the validation plugin is loaded. Writing the orchestration as code makes those dependencies explicit, where a prose prompt would leave them implicit.

The runtime wiring also keeps the Azure configuration outside the orchestration logic itself. The environment supplies the deployment name, endpoint, and API key, and the kernel uses those values when it constructs the Azure chat completion service. Orchestration stays focused on behavior rather than connection plumbing.

In practice, the structure looks like this:

flowchart TD
  load[Load state by session_id] --> rehydrate[Rebuild chat history]
  rehydrate --> analyzer[Analyzer]
  analyzer --> mapper[Mapper]
  mapper --> generator[Generator]
  generator --> validator[Validator]
  validator -- WORKFLOW_APPROVED --> stop[Terminate]
  validator -- INVALID --> generator
  validator -- other --> generator
  stop --> persist[Persist state]

The important arrow is the one that returns from Validator to Generator. That is the local correction path. Invalid output does not reset the analysis, erase the mapping, or jump back to the start. It goes back to the one agent responsible for producing the template.

A runnable validator that actually checks structure

The validator must reject malformed output or it is just another polite participant in the conversation. Here is a minimal, runnable version that checks a template shape and returns a real verdict:

import json
from typing import Any, Dict, List


SUPPORTED_PLATFORMS = {'zapier', 'make', 'n8n'}


def select_template(platform: str) -> Dict[str, Any]:
    platform_key = platform.lower()
    if platform_key not in SUPPORTED_PLATFORMS:
        raise ValueError(f'Unsupported platform: {platform}')

    base = {
        'name': f'{platform_key}_automation',
        'triggers': [],
        'actions': [],
    }
    return json.loads(json.dumps(base))


def validate_template(template_json: str, platform: str) -> Dict[str, Any]:
    errors: List[str] = []

    try:
        template = json.loads(template_json)
    except json.JSONDecodeError as exc:
        return {'status': 'invalid', 'errors': [f'Invalid JSON: {exc}']}

    for key in ('name', 'triggers', 'actions'):
        if key not in template:
            errors.append(f'Missing key: {key}')

    if platform.lower() not in SUPPORTED_PLATFORMS:
        errors.append(f'Unsupported platform: {platform}')

    if not isinstance(template.get('triggers'), list) or not template.get('triggers'):
        errors.append('At least one trigger is required')

    if not isinstance(template.get('actions'), list) or not template.get('actions'):
        errors.append('At least one action is required')

    return {
        'status': 'valid' if not errors else 'invalid',
        'errors': errors,
    }


if __name__ == '__main__':
    bad = json.dumps({'name': 'demo', 'triggers': [], 'actions': []})
    good = json.dumps({'name': 'demo', 'triggers': ['new_event'], 'actions': ['create_record']})
    print(validate_template(bad, 'Zapier'))
    print(validate_template(good, 'Zapier'))

That is the behavior I want the real validator path to approximate: reject missing structure, reject empty structure, and hand back a verdict the selection strategy can use to decide the next speaker. The exact shape of this standalone example matters far less than the discipline behind it. Validation is not a suggestion. It is a gate.

Why this pattern is easier to trust

This design works because every failure has one owner. Weak analysis, and the Analyzer is the problem. Invented structure in the mapping, and the Mapper is the problem. A template that does not fit the platform, and the Generator is the problem. A template that violates the contract, and the Validator says so. Debugging gets much easier when I am not trying to infer which layer went wrong from a cloud of blended instructions.

The retry logic gets saner too. When the validator rejects the output, I do not want the entire conversation to restart. I want the generator to take another pass with the same context still in memory, and that is what the selection strategy enforces: the orchestrator reads the last message, sees the invalid verdict, and routes execution back to the generator. Nobody else has to renegotiate the earlier steps.

State rehydration matters here because a retry is a different animal from a fresh run. A fresh run throws away the exact information I need most: what failed, what was already agreed, and which platform shape was already chosen. Rehydration preserves that state so the next attempt can repair the actual fault instead of replaying the whole conversation.

What comes out of it is a system that can fail locally without collapsing globally. Sounds like a small thing until you watch a long run survive a rejection, recover from persisted history, and land on a valid template without re-deriving the entire workflow from scratch.

What actually settled it

Using multiple agents is not the strongest part of this orchestration. What settled things was the constraint: agents bounded by code, the next speaker chosen by strategy, the validator owning the final gate, and session history that survives retries. Once those pieces were in place, the mapper stopped wandering and the generator stopped improvising against the wrong shape.

That gave me exactly what I wanted from the workflow analyzer SaaS: a chain that can analyze, map, generate, validate, and resume without pretending a failed attempt never happened. Next I am interested in pushing more of the template shape into the library itself, so the generator begins with an even tighter platform skeleton and the validator has less to reject in the first place.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

Validation Geometry Is Part of the Model

Daniel Romitelli — Mon, 27 Jul 2026 16:02:41 +0000

The dangerous part was not the classifier. The label file was, sitting there looking convenient.

A saved train_y.npy artifact existed, and for this baseline it was a trap. It contained magnitude-filtered positive targets only, which made it unusable for a directional classifier. Train on it anyway and the result would look like a model comparison while actually being a dataset-artifact comparison.

So I made the LightGBM minute baseline read Pramaana's per-asset feature parquet files directly. I wasn't trying to tune trees until they confessed. I was trying to close the model-class capacity objection in the minute-level ceiling experiment without letting label construction, overlapping windows, or temporal bleed sneak into the room wearing a lab coat.

The research question behind the ICAIF 2026 paper is deliberately narrow: minute-scale cryptocurrency direction from OHLCV candles appears reproducibly capped near 52% across a broad set of model and feature configurations, while the same research stack recovers materially more directional information at the hourly horizon. Across seven minute configurations and approximately 36 million rows of minute-scale OHLCV history, the observed range is 51.4% to 52.3%. The LightGBM baseline is the seventh configuration: 46 microstructure-proxy features, a 15-minute forward return target in basis points, a stride-15 de-overlap, a no-trade filter at |target| > 10 BPS, and a per-asset 85/15 temporal split with a 15-row purge before validation.

The transferable idea is simple and annoyingly easy to violate: in time-series ML, validation geometry isn't bookkeeping. It's part of the model.

The baseline was a geometry test, not a tuning contest

A naive capacity comparison asks, “Does a different model class beat the neural setup?” That sounds reasonable until the labels are temporal, overlapping, filtered, and asset-scoped. The better question is, “Can I compare model classes without changing the target semantics or leaking nearby time into validation?”

So the script documents the protocol before it imports anything. That docstring is doing more than explaining a file. It pins down the shape of the experiment.

#!/usr/bin/env python3
"""Run a LightGBM directional baseline on the frozen M6 sniper matrix.

This closes the model-class capacity objection for the paper's minute-level
ceiling section. The script reads Pramaana's per-asset
``data/tmp_sniper_feat_*.parquet`` files directly because the saved
``train_y.npy`` artifact currently contains magnitude-filtered positive targets
only and is therefore not usable for a directional classifier.

Protocol matched to ``scripts/preprocess_sniper.py``:
  - 46 engineered microstructure-proxy features
  - 15-minute forward return target in BPS
  - stride-15 de-overlap
  - |target| > 10 BPS no-trade filter
  - per-asset 85/15 temporal split with a 15-row purge before validation
"""

from __future__ import annotations

import json
import time
from pathlib import Path

import lightgbm as lgb
import numpy as np
import polars as pl
from scipy import stats
from sklearn.metrics import balanced_accuracy_score, roc_auc_score


TRAIN_RATIO = 0.85

I like this kind of comment because you can falsify it. Every important experimental choice is named: feature count, target horizon, de-overlap, no-trade filtering, split ratio, and purge width. If the number later changes, the protocol has to change with it. (The real script also exposes these knobs as argparse CLI flags; I trimmed the flag-parsing boilerplate from this excerpt to keep the protocol in focus.)

The saved label artifact failed the most basic requirement for this comparison. It did not represent both directions, and directional classification needs the sign of the target. Magnitude-filtered positive targets only? That's a different task, not a neutral shortcut. So the baseline reconstructs the binary label from the 15-minute forward return in basis points inside the feature-parquet path, rather than inheriting a label artifact whose semantics were already wrong for this purpose.

There's a useful mental model here. A time-series split is less like cutting a deck of cards and more like cutting wet paint. The boundary smears unless you leave room for it to dry. In this baseline, the 15-row purge is that dry strip between training and validation.

The timeline is the experiment

The pipeline has only a few stages, but the order matters. The baseline starts from per-asset parquet files, reconstructs the directional target from the forward return, filters out the no-trade zone, de-overlaps with stride 15, applies a per-asset temporal split, inserts a purge gap, and only then fits and validates the model.

flowchart TD
  parquet[Per asset feature parquet] --> labels[Reconstruct sign label]
  labels --> filter[No trade filter]
  filter --> stride[Stride 15 de overlap]
  stride --> split[Per asset temporal split]
  split --> train[Training interval]
  split --> purge[15 row purge gap]
  purge --> validate[Validation interval]
  validate --> metrics[Accuracy and AUC]

The diagram looks almost too ordinary, which is exactly the trap. The ordinary-looking arrows are where most of the statistical damage would happen if they were skipped or reordered.

For each asset, the geometry is anchored by time rather than random assignment. The training interval comes first. The validation interval comes last. The purge gap sits between them. The overlapping-window hazard is caused by the target construction itself: a 15-minute forward return means nearby rows can share future information unless the split respects the horizon. The stride-15 de-overlap reduces that hazard, and the purge gap protects the split boundary.

Geometry choice	What it protects against	Concrete value in this baseline
Per-asset temporal split	Cross-time contamination within each asset	85/15 train/validation
Purge gap	Boundary bleed from nearby rows	15 rows before validation
De-overlap	Repeated labels from overlapping horizons	stride-15
No-trade filter	Tiny targets treated as tradable direction	`
Label reconstruction	Wrong task inherited from saved labels	sign of 15-minute forward return in BPS
Raw tree features	Scaling mismatch for tree baseline	tree model does not require RobustScaler

The naive version would be shorter. Load {% raw %}X, load y, fit classifier, report accuracy. It would also be wrong in exactly the way that makes a result hard to debug: the code would run, the metrics would print, and the comparison would look scientific. The error would live in the meaning of y and the geometry of the split, not in a stack trace.

That distinction matters because most bad financial ML baselines do not fail loudly. They often fail by making the wrong thing convenient. A cached label array, a random split helper, a global shuffle, a validation set reused for early stopping, a forward-return target created before de-overlap: none of these choices necessarily creates an obvious programming error. They create an evidentiary error. The model may be implemented correctly while the experiment answers a question I did not intend to ask.

Why the saved label file had to be rejected

A directional classifier is only as honest as its labels. In this baseline, the target is “15-minute forward return in BPS; binary label is sign(target).” That gives the model a two-sided problem: up versus down after filtering out the no-trade zone.

The existing saved train_y.npy artifact did not satisfy that contract. It contained magnitude-filtered positive targets only. That makes it unsuitable for a directional classifier because it no longer represents the binary sign task the baseline is supposed to measure. There is no clever model-side fix for that. Once the target artifact encodes the wrong task, using a different classifier just gives the wrong task a new costume.

The baseline therefore goes back to data/tmp_sniper_feat_*.parquet. That choice matters because the feature parquet is upstream of the bad shortcut. It lets the script reconstruct labels under the same protocol used by the sniper preprocessing path: 46 engineered microstructure-proxy features, 15-minute forward return in basis points, stride-15 de-overlap, the no-trade filter, and the per-asset temporal split with purge.

This is the part of baseline design that feels unglamorous but decides whether the result means anything. A model-class objection says, “Maybe the neural family is the reason the minute result clusters near 52%.” A contaminated label artifact would make the answer meaningless. Reconstructing labels from feature parquet keeps the comparison focused on capacity instead of accidentally comparing two different tasks.

The same principle applies outside this specific paper. If an intermediate artifact was built for a different objective, it is not a neutral cache. It is an encoded research decision. A saved target array can carry filtering, horizon, class definition, censoring, asset selection, and split assumptions. If those assumptions no longer match the experiment, downstream code should not pretend the file is just bytes on disk. It is a contract, and in this case the contract was wrong for the classifier I needed to run.

The capacity objection needed a clean target

The minute-ceiling section reports seven configurations. The first three use tens of millions of labels and increasingly rich feature sets. The fourth and fifth reduce sample size but de-overlap and alter the loss. The sixth uses a different 15-minute basis-point target and compact microstructure-proxy features. The seventh replaces the neural CQR family with a classical tree model on the frozen microstructure-proxy matrix.

That seventh row is the key capacity-control move. If the ceiling were merely an artifact of the neural setup, a different model class on the same frozen feature/target construction should have had a chance to break away. Instead, the LightGBM baseline reports 52.231% accuracy with CI [52.042, 52.420], AUC 0.53046, balanced accuracy 52.21%, and n=268342 validation samples. In the minute configurations table, that appears as 52.23% ± 0.189 for row 7.

The ceiling holds: 52.231% accuracy, 95% CI [52.042, 52.420].
Swap the neural CQR family for a classical tree on the same frozen feature and target matrix, and the result lands in the same 51.4–52.3% band. The minute ceiling is not an artifact of the neural setup.

For metric provenance, this was a CPU-only LightGBM run from scripts/run_lightgbm_sniper_baseline.py using the Python scientific stack in the paper environment: Polars for parquet loading, NumPy for arrays, SciPy for the binomial interval and significance calculation, scikit-learn metrics for balanced accuracy and AUC, and the LightGBM scikit-learn API for the classifier. The run writes data/lightgbm_sniper_baseline.json and data/lightgbm_sniper_baseline_model.txt; the reported metric is taken from that JSON artifact, not copied by hand into the manuscript.

The important part is not that LightGBM has a particular personality. It is that the baseline swapped model class while holding the validation geometry and target construction in place. Without that, “LightGBM versus neural” would be a noisy argument about everything except the model.

The training block reflects that narrow purpose. It uses a tree classifier with raw engineered features, reserves the final 15% of the training side for early stopping, and computes class weighting from the fit subset. The held-out validation interval remains untouched by early stopping.

X_train, y_train, X_val, y_val, feature_names, asset_stats = _load_sniper_arrays(pramaana)

fit_end = int(len(X_train) * 0.85)
X_fit, X_es = X_train[:fit_end], X_train[fit_end:]
y_fit, y_es = y_train[:fit_end], y_train[fit_end:]

n_pos = int(y_fit.sum())
n_neg = int(len(y_fit) - n_pos)
scale_pos_weight = n_neg / max(n_pos, 1)

clf = lgb.LGBMClassifier(
    n_estimators=2000,
    learning_rate=0.03,
    num_leaves=31,
    max_depth=6,
    min_child_samples=200,
    subsample=0.85,
    colsample_bytree=0.85,
    reg_alpha=0.5,
    reg_lambda=1.0,
    scale_pos_weight=scale_pos_weight,
    objective="binary",
    metric="binary_logloss",
    n_jobs=-1,
    random_state=42,
    force_col_wise=True,
    verbosity=-1,
)

clf.fit(
    X_fit,
    y_fit,
    eval_set=[(X_es, y_es)],
    eval_metric="binary_logloss",
    callbacks=[
        lgb.early_stopping(stopping_rounds=100),
        lgb.log_evaluation(period=100),
    ],
)

The hyperparameters are not the story I care about here. The non-obvious detail is the split inside the training side: the final 15% of train is used only for early stopping, which keeps validation as the final held-out interval rather than letting it become a tuning surface.

That one choice prevents a common baseline failure. If I had passed the paper validation interval as the LightGBM eval_set, early stopping would have made the validation set part of the training procedure. The model would not directly fit labels from validation, but the selected number of boosting rounds would be chosen by validation performance. That is enough to contaminate the final metric. The point of the baseline is not to squeeze the last basis point out of LightGBM; it is to answer whether a classical tree classifier breaks the minute ceiling under the same target and split discipline.

The results JSON captures the protocol in a compact form. This is the artifact I want beside the paper because it records not only the metric, but also the target, preprocessing, split, and scaling assumptions that make the metric interpretable.

{
  "label": "LightGBM M7 sniper baseline",
  "created_utc": "2026-05-04T02:58:24Z",
  "source": "/home/the author/Development/Python/crypto-fl-v2/data/tmp_sniper_feat_*.parquet",
  "protocol": {
    "feature_set": "46 microstructure proxy features",
    "target": "15-minute forward return in BPS; binary label is sign(target)",
    "preprocessing": "stride-15 de-overlap; |target| > 10 BPS no-trade filter",
    "split": "Per-asset 85/15 temporal split with 15-row purge before validation; final 15% of train used only for early stopping",
    "scaling": "Raw engineered features; tree model does not require RobustScaler"
  },
  "train_samples": 1520254,
  "early_stop_samples": 228039,
  "validation_samples": 268342,
  "n_features": 46
}

I care more about the protocol object than the timestamp. A metric without this surrounding geometry is just a number looking for a story, and the protocol is what stops the wrong story from attaching itself.

Temporal bleed arrives without a stack trace

The hard part about temporal bleed is that it rarely announces itself. No exception fires to tell you, “Your validation rows are too close to your training rows.” What you get is a better-looking number, and that number is seductive because it can be explained as model quality.

The minute experiments are especially exposed to this because the target horizon is short and overlapping labels are easy to create accidentally. A 15-minute forward return target means row t and row t+1 can be describing heavily shared future intervals. If the split boundary cuts through those neighborhoods without a purge, the validation side can remain too close to what the training side has already seen.

So the geometry is engineered at multiple levels rather than resting on one protective trick.

Hazard	Naive baseline failure	Geometry response
Saved label artifact has wrong semantics	Classifier trains on a target that is not the directional task	Reconstruct labels from feature parquet
Adjacent windows share future information	Validation resembles training near the boundary	Insert 15-row purge before validation
Overlapping targets inflate sample familiarity	Many rows encode nearly the same horizon	Apply stride-15 de-overlap
Asset timelines differ	Global shuffle mixes unrelated time positions	Split per asset temporally
Early stopping erodes validation independence	Held-out set becomes part of model selection	Use final 15% of train only for early stopping

This is also why I resist treating train/test split as an afterthought in financial ML. For IID tabular data, the split is often a convenience. For time series, it's a claim about causality: what information was available before the prediction time, and what was not. If that claim is false, the model can be perfectly implemented and still be scientifically useless.

The per-asset split matters for a second reason. Cryptocurrency pairs don't all have identical listing histories, liquidity regimes, or missing-data structure. A global split by row count would mix assets at unrelated calendar positions. A global random split would be worse. The baseline needs each asset’s validation samples to come from the end of that asset’s own history, with the purge applied at that asset boundary. That preserves the meaning of “future held out” even when the panel is irregular.

The no-trade filter is a different kind of guardrail. It doesn't prevent leakage. What it prevents is the classifier being evaluated on economically tiny moves as if every infinitesimal return were a meaningful directional event. The target is still a research abstraction rather than a full trading simulator with costs and execution, but the |target| > 10 BPS filter keeps the sign label from being dominated by noise around zero. That matters when the entire empirical question is about a narrow 51.4% to 52.3% band. At that scale, sloppy target construction can easily masquerade as something real.

The minute ceiling becomes more credible when the baseline cannot cheat

The paper's data-and-methods section frames the smaller de-overlapped and microstructure-proxy runs as controls against cleaner labels, different short-horizon targets, and a classical tree classifier. The LightGBM row belongs to that logic. No new headline result. A stress test aimed at one specific objection.

The minute ceiling is not “models cannot beat chance.” The reported range sits above chance in a statistical sense. The issue is effect size. Across the seven configurations, directional accuracy remains inside a 0.9 percentage-point band, from 51.4% to 52.3%. With sample sizes in the millions for the larger configurations, the open question isn't whether the models detect something. It's why substantial changes in features, scaling, loss functions, target construction, model class, and sample count don't move the result into a stronger range.

The hourly positive control is what keeps this from becoming nihilism. At the hourly horizon, the FT-Transformer/CQR walk-forward run reached 54.69% directional accuracy on 386,056 future-held-out hourly samples, with a 95% interval of 54.53–54.84 and 80.53% conformal coverage. That result comes from the five-fold expanding walk-forward CQR run recorded in backtest_results/walkforward_cqr_hourly.json and the corresponding resume log, executed with the PyTorch FT-Transformer/CQR training stack on the project’s CUDA workstation environment. One runtime distinction is worth stating plainly: the hourly result is not a LightGBM CPU run. It's a neural FT-Transformer/CQR evaluation under expanding walk-forward temporal validation, with the metric emitted by the walk-forward script and then pulled into data/results_manifest.json for paper table generation.

That contrast matters because it shows the research stack does not mechanically inflate every task. Minute OHLCV direction clusters near the ceiling. Hourly direction recovers more directional information under a stricter temporal protocol.

But the contrast only means something if the minute baseline is clean. Had the LightGBM run used a broken label artifact, the row would not close the capacity objection. It would open a new hole. Reconstructing labels from parquet and enforcing split geometry is what lets the baseline answer the narrow question it was built to answer.

The evidence-bundle design reinforces this. The paper repository is separate from the trading system because the paper is the manuscript and reproducibility layer, while Pramaana is the experiment apparatus. The paper artifacts point back to the scripts and result files that generated each table row: scripts/run_lightgbm_sniper_baseline.py, data/lightgbm_sniper_baseline.json, data/lightgbm_sniper_baseline_model.txt, and the per-asset parquet inputs for the LightGBM capacity control; backtest_results/walkforward_cqr_hourly.json, the walk-forward logs, and scripts/walkforward_cqr_hourly.py for the hourly positive control. That's the level at which a result becomes inspectable. A table row should not be a manually typed number; it should be the visible tip of an artifact chain.

What the feature importances can and cannot say

The LightGBM run also writes a model file and feature importances. The top features include return_60m, z_effort_60m, price_vs_vwap_5m, and z_effort_15m. I don't treat those importances as a market theory. Tree feature importance is useful for checking that the model is not obviously broken. It isn't a causal explanation of minute returns.

What it can say is more modest: the classifier found most of its splits in recent return, effort, and VWAP-relative features, which is consistent with the feature family the baseline was meant to test. That helps catch the opposite failure mode, a model that reports a plausible metric while accidentally training on an ID column, timestamp encoding, or mislabeled target. Feature inspection is not proof. It's a useful diagnostic once the validation geometry is already correct.

This is also why I report AUC alongside accuracy. Accuracy answers the sign-decision question at the default threshold. AUC checks whether the probability ranking carries directional information independent of that threshold. In the LightGBM run, AUC is 0.53046, which aligns with the story told by 52.231% accuracy: the model is picking up something small, not discovering a dramatically separable classification boundary.

The high-confidence slice is similarly restrained. The JSON records a threshold of p >= 0.60 or p <= 0.40, with 2,305 samples, 0.859% coverage, and 57.007% accuracy. Interesting as a calibration and selectivity diagnostic. It does not rescue the minute problem. A tiny high-confidence region with better accuracy is not the same as a broad tradable edge, especially before costs, slippage, and execution constraints. For the paper’s central claim, the full validation metric is the right number to emphasize.

The discipline is to make shortcuts impossible

The small engineering choice I'd generalize is this: when a saved artifact has ambiguous or wrong semantics, don't patch around it downstream. Rebuild from the nearest artifact whose meaning is still compatible with the experiment.

In this case, the nearest compatible artifact was the per-asset feature parquet. That forced the script to carry the target definition, filtering rule, de-overlap policy, split geometry, purge width, and scaling assumption in the same place as the model run. Less convenient than loading train_y.npy. More honest.

There's a tradeoff. Reconstructing labels from parquet couples the baseline to the upstream feature files and requires the paper repo to reference Pramaana's data path. The evidence bundle handles that by recording local evidence anchors rather than pretending the manuscript repository contains every large matrix. Compact artifacts get tracked directly with metadata and hashes; larger upstream matrices are referenced by path and size unless full hashing is requested. An engineering compromise, and one that keeps the paper repository focused on reproducibility rather than becoming a warehouse for heavyweight experiment outputs.

The payoff is that the baseline result has a shape I can defend. It says: on the frozen M6 sniper matrix, with 46 microstructure-proxy features, 15-minute forward-return sign labels, stride-15 de-overlap, a no-trade filter, per-asset temporal splitting, a 15-row purge, and a held-out validation interval, a classical tree model lands at 52.231% accuracy. That's a capacity-control statement, not a vague leaderboard entry.

It also changes how I evaluate future baselines. I want the experiment script to make the invalid path difficult. If the wrong label file is easy to load, someone eventually will. If early stopping on validation is a one-line convenience, it will creep in. If a random split helper is available in the same file as time-series code, it will eventually be called in the wrong place. Good research code does not merely implement the correct protocol; it removes temptations that produce attractive but meaningless numbers.

What I changed in how I think about baselines

I used to think of baselines as simpler models. I now think of them as simpler claims. A good baseline should remove one objection at a time. This LightGBM run removes the objection that the minute ceiling is merely a neural architecture artifact. It doesn't claim to solve execution, transaction costs, order-book dynamics, or richer market state. It says only that replacing the model family on this engineered short-horizon matrix does not break the ceiling.

That restraint is the virtue. A baseline that tries to answer every objection simultaneously becomes another opaque system. Answer one objection under carefully preserved geometry and it becomes useful evidence.

For time-series ML, the geometry is the evidence. The split, purge, stride, and label source aren't clerical details that come after the “real” modeling work. They're the rails that keep the model from learning yesterday's shadow of tomorrow.

A classifier trained on the wrong artifact can look competent. A classifier trained inside the wrong timeline can look brilliant. I'd rather have the modest number I can trust than the impressive one produced by a boundary I forgot to draw. The next credible test won't be another minute-candle model with a larger parameter count. It will be the one that adds the state variables the minute horizon is missing (order-book pressure, queue dynamics, spread formation, and execution-aware microstructure) while keeping the same discipline about labels, time, and evidence.

In time-series work, validation geometry is not an implementation detail. It is the contract that makes a model's number mean something. The same discipline that protects a live trading gate or a self-indexing memory write protects the scientific claim.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

Vector Split by Chunk: Why My Retrieval Stops at the Boundary I Drew

Daniel Romitelli — Mon, 27 Jul 2026 16:02:31 +0000

A draft of mine missed the exact file span it needed. The vector was "close." But the chunk I wanted was buried inside a larger blob, and the miss was clean enough to be embarrassing: the sort of thing that passes in a demo and grates in production, because the answer is almost right the way a blurry photo is almost a portrait.

Dense retrieval people have a name for the mechanic underneath it. Representational collapse. Ask one embedding to summarize an entire document and the model picks the dominant mode, then discards the long tail. The precise sentence you wanted lives in that tail. So you get back the right neighborhood and the wrong house, topically coherent and operationally useless.

The fix is to stop asking one vector to do that work. Split the text before embedding, embed each slice on its own, and let retrieval search the slices instead of the document. Everything below falls out of that one decision.

The boundary I chose on purpose

I split embeddings by chunk because the retrieval layer needed a smaller unit than a document. A whole-file vector is too coarse when the system has to answer from a specific path, a specific code block, or a specific rewrite. A chunk sits in between: small enough to isolate one semantic mode, large enough to keep the local context wrapped around it.

The implementation starts with a windowing function in lib/supabaseVectorStorage.ts that does one thing, slice text into overlapping fixed-size chunks. It's the dumbest viable splitter, and that's deliberate. Smarter ones exist. Sentence-aware splitters, recursive structural splitters, semantic-similarity splitters like LangChain's SemanticChunker. Each brings its own failure modes and its own tuning surface. A naive windowed splitter has exactly two knobs and no hidden behavior, which is what you want from a baseline before you let cleverness in.

/**
 * Split text into chunks for embedding
 */
export function splitTextIntoChunks(text: string, chunkSize: number = 1000, overlap: number = 200): string[] {
  const chunks: string[] = [];
  let start = 0;

  while (start < text.length) {
    const end = start + chunkSize;
    const chunk = text.slice(start, end);
    chunks.push(chunk);
    start = end - overlap;
  }

  return chunks;
}

The overlap does more work than it looks like it does. It encodes a locality assumption: an important sentence often straddles a chunk boundary, and when it does, the embedding for either neighbor still anchors near that sentence in vector space. Drop the overlap and coherent thoughts become disjoint halves with weaker mutual similarity. Common starting values in production run 10 to 20%. This one runs at 20% (200 of 1000), the upper end, trading a bit of storage for resilience at the boundaries. The Pinecone team's chunking strategies guide is the canonical writeup if you want the longer comparison.

Why the naive version fails

Embedding the whole document and calling it done works right up until retrieval needs a narrow answer. Then the vector starts behaving like a resume summary. It remembers the broad shape and forgets the sentence where the detail actually lives. That's the precision side of the precision/recall tradeoff giving way: you keep finding related documents and missing the exact one.

Chunk-level embedding changes the unit of truth. Instead of one embedding representing everything, many embeddings represent adjacent slices, and cosine similarity decides which slice the query sits closest to. More embeddings per document means more candidate matches, which means a much better chance of landing on the specific span that answers the question rather than the document that happens to contain it.

How the chunk becomes a vector

Once the text is split, each chunk gets embedded independently with OpenAI. The call is tied to the chunk text, not the parent document.

/**
 * Generate embedding for a given text using OpenAI
 */
export async function generateEmbedding(text: string): Promise<number[]> {
  try {
    const response = await openai.embeddings.create({
      model: 'text-embedding-3-large',
      input: text,
      dimensions: 2000,
    });

    return response.data[0].embedding;
  } catch (error) {
    console.error('Error generating embedding:', error);
    throw error;
  }
}

Two things in that call are worth naming. input: text receives the chunk and not the document, which is the whole purpose of the split. And dimensions: 2000 leans on OpenAI's Matryoshka representation learning: text-embedding-3-large is natively 3072-dimensional, but it's trained so that any leading prefix of the vector is itself a usable embedding. Truncating to 2000 dims keeps roughly 99% of the retrieval quality at two-thirds of the storage cost and roughly two-thirds of the cosine computation per query. Across tens of thousands of chunks behind an HNSW index, that compounds fast.

The storage path follows the same logic when chunks get inserted. Each one becomes its own record with its own index and metadata.

const chunkPromises = chunks.map(async (chunkText, index) => {
  const embedding = await generateEmbedding(chunkText);

  return {
    document_id: documentId,
    chunk_index: index,
    chunk_text: chunkText,
    embedding: embedding, // Pass as array, Supabase will handle vector conversion
    metadata: {
      title: document.title,
      chunk_index: index,
      total_chunks: chunks.length,
    },
  };
});

const chunksWithEmbeddings = await Promise.all(chunkPromises);

// Insert chunks with embeddings
const { error: embError } = await supabase
  .from('embeddings')
  .insert(chunksWithEmbeddings);

The non-obvious part is the metadata. It carries chunk_index and total_chunks alongside the vector, so a chunk is usable on its own and still locatable inside the original document. That's what enables a downstream pattern called context expansion: when a chunk matches, you can also fetch its neighbors by index and widen the window the LLM sees without polluting the similarity score. Retrieval stays narrow. Presentation gets to be generous.

Why this pattern beats one vector per file

A single file vector is cheap to reason about and expensive to trust, because it compresses too much. Split first, embed second, and you preserve the local neighborhood around each idea, which is what lets the search layer return the exact span instead of the general topic. The honest cost is more rows, more embeddings, more storage. But in retrieval, precision is usually the thing that keeps the rest of the system from sounding vaguely confident and slightly wrong.

The retrieval path that makes the split worth it

The strongest evidence that the split matters is a retrieval helper that bypasses vector similarity entirely when I already know the file path. In supabase/functions/_shared/blog-utils.ts, it queries embeddings by metadata->>'file_path' LIKE '%<path>%' so I can guarantee the accuracy agent sees chunks from the file the draft explicitly mentions.

That's hybrid retrieval in its simplest form. Lexical exact-match (the LIKE on file path) and dense vector search (the cosine query) live side by side and compose by union. Vector search is what I want for semantic recall, "find me the chunk that talks about this idea, even if it uses different words." Path-based retrieval is what I reach for when the system needs to stop being poetic and start being literal, "find me chunks from this exact file." Production retrieval stacks at scale usually go further and combine BM25 lexical scores with dense vectors via reciprocal rank fusion. This is the minimal version of that pattern, sized to the problem.

And the chunk-level split is what makes the path-based path useful, because a file can now return multiple precise spans instead of one oversized blob. If I'm asking about a function, I want the function's neighborhood, not the whole neighborhood's autobiography.

The shape of the storage model

The storage row carries the document_id, chunk_index, chunk_text, embedding, and metadata. That shape is what keeps downstream retrieval and debugging sane. When I inspect a result, I can tell where it came from, where it sits in the source, and how many sibling chunks exist around it. Simple enough to survive maintenance, specific enough to survive scrutiny, which is a rarer combination than it should be.

export interface DocumentChunk {
  id?: string;
  document_id: string;
  chunk_index: number;
  chunk_text: string;
  embedding?: number[];
  metadata?: Record<string, unknown>;
}

The practical tradeoff: granularity buys control, not magic

Chunking doesn't make retrieval smart by itself. It gives the search layer a better surface to work with, and that surface has a goldilocks zone. Too large and chunks blur together into mid-density topical clouds. Too small and they lose enough context to become syntactic fragments whose embeddings collapse into the average. The 1000-character window here is roughly two paragraphs of prose, or one tight function: large enough to carry meaning, small enough to admit only one dominant idea per row.

The overlap is the design's safety net. It encodes the assumption that meaning is local but not aligned, since important sentences refuse to respect window boundaries, and it pays a storage tax to keep adjacent chunks neighbors in vector space. The same tradeoff shows up in sliding-window attention, which preserves locality at the cost of duplicated work, and in n-gram lexical indexes, which overlap at the boundary to avoid losing cross-token matches.

The other cost is operational. More chunks, more embeddings. More embeddings, more insert work and more bytes in the HNSW graph. I accept that, because the alternative is a retrieval system that keeps returning the right topic and the wrong answer, which is a very expensive way to be unhelpful.

The retrieval story, drawn plainly

flowchart TD
  A["Source text"] --> B["Chunk split"]
  B --> C["OpenAI embed"]
  C --> D["Embeddings table"]
  D --> E["File-path query"]
  D --> F["Vector search"]
  E --> G["Chunk result"]
  F --> G

Why I kept the split at the chunk boundary

I could have pushed more logic into the embedding step. Late-interaction models like ColBERT preserve per-token embeddings and aggregate at query time, which gives you precision without choosing a chunk size up front. I could have tried to recover precision later with cross-encoder reranking. But both of those move the problem instead of solving it. They paper over a representation that's wrong at the unit of storage.

So I split by chunk, embed by chunk, store by chunk, and retrieve by chunk. Every stage speaks the same language, which makes the system easier to inspect. Better recall is part of the payoff. The bigger part is a retrieval stack whose granularity matches the way I actually ask questions.

When the answer lives inside a file, I want the search layer to arrive with a scalpel, not a shovel.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

Four Vectors, One Record: How I Split Embeddings Before They Hit Search

Daniel Romitelli — Mon, 27 Jul 2026 16:02:20 +0000

The failure that pushed me into this design wasn't subtle. My blended embedding kept returning candidates who looked fine at a glance and wrong on inspection. Ask about a very specific certification and you'd get people with the right industry background and no credential. Ask about deep experience in a narrow domain and you'd get candidates whose skills summary happened to mention the right keywords, with a work history that didn't support the match. The vector was doing exactly what I'd asked it to do: compressing an entire record into one semantic point. A candidate record just isn't one thing.

Dense-retrieval people have a name for this. Representational collapse: ask one embedding to boil down a heterogeneous object and the model gravitates to the dominant mode, discarding the long tail. For a document, that tail is a paragraph. For a candidate record, the tail is an entire field. Credentials get smoothed over by industry text. Career trajectory gets smoothed over by a polished summary. One vector turns a structured record into a compromise the search layer then has to live with.

A profile has several different centers of meaning, several semantic modes that don't collapse cleanly into one. Work history answers one class of search intent. Skills and designations answer another. A broad profile summary is useful for discovery, but it's a poor substitute for the details that actually separate one candidate from the next. So stop asking one embedding to carry all of them.

That's what app/agents/embedding_agent.py is for in this system. It generates four parallel embeddings for the same logical record: profile_vector, experience_vector, skills_vector, and general_vector. In retrieval terms that's late fusion, meaning you embed each view independently, defer the "which view matched" decision until query time, and let the search layer pick the best-matching vector per query. Then app/jobs/embedding_generator.py takes over the production concerns. It validates the input, applies retry logic, and routes terminal failures to the dead-letter queue. The split is deliberate. The agent handles semantic decomposition and caching; the job handles durability and failure control. And the reason for two layers is the bulkhead pattern, keeping the failure modes of one concern from contaminating the other, the way watertight compartments work on a ship.

The shape of the pipeline

I like a system that's honest about its steps.

flowchart TD
  sourceRecord[Source record] --> fieldSplit[Split into 4 field views]
  fieldSplit --> embeddingAgent[Embedding agent generates 4 vectors]
  embeddingAgent --> redisCache[Redis cache with 24h TTL]
  embeddingAgent --> embeddingJob[Embedding job]
  embeddingJob --> validate[Validate length and dimension]
  validate --> retryFailures[Retry transient failures]
  validate --> deadLetter[Route terminal failures to DLQ]

The agent, its constants, and the cache key

The embedding agent is specialized on purpose. Its whole assignment: produce four specific vectors, keep them cached, and run fast enough that repeated calls don't punish the API.

The contract sits in four constants at the top of the file.

EMBEDDING_MODEL = "text-embedding-3-large"
EMBEDDING_DIMENSIONS = 3072  # text-embedding-3-large native dimensions
EMBEDDING_TTL_SECONDS = 86400  # 24 hours
CACHE_KEY_PREFIX = "emb:"

Those four lines are the boundary conditions for the whole embedding pipeline, what a working systems engineer would call a contract module. The model choice defines the embedding family. The dimension count defines what the rest of the system has to expect, and what the HNSW or IVF index was sized for; if the two ever fall out of step, the mismatch shows up immediately. The TTL is the literal floor on how stale the cache can get, which makes it a direct knob on the freshness/cost tradeoff. And the prefix scopes the embedding namespace inside Redis, because a multi-tenant cache without prefixes becomes archaeology the moment anything goes wrong.

The cache key itself is where the multi-vector output stays safe.

def _generate_cache_key(self, query: str, vector_type: str = "all") -> str:
    """
    Generate a deterministic cache key from query text.

    Returns: emb:{md5_hash}:{vector_type}
    """
    normalized = query.lower().strip()
    query_hash = hashlib.md5(normalized.encode()).hexdigest()
    return f"{CACHE_KEY_PREFIX}{query_hash}:{vector_type}"

The vector_type token is the piece that makes this cache safe for a four-vector bundle. Call it key-space scoping in the small. Each view gets its own deterministic slot, so a lookup for profile_vector can't come back holding the value cached for skills_vector. Without the token the keys would collide and a lookup would return whichever vector wrote last. With it, the cache stays correct under concurrent writes from all four views. The lower().strip() normalization is what gives the key its deterministic property: semantically identical queries that differ only in whitespace or case land in the same slot. Repeated content returns quickly. New content pays the model cost once. And the cache stays legible because nothing else in Redis uses the emb: prefix.

On a miss, the agent fires all four model calls in parallel. The vectors are independent semantic views, so there's no reason to serialize them. asyncio.gather runs the four embedding requests concurrently, and the worst case is fan-out latency rather than four-times-sequential. In practice the four calls finish in roughly the wall-clock time of a single one. The agent also tracks cache hits, cache misses, embeddings generated, errors, tokens used, and total latency. Those metrics are what let me tell a slow cache miss from a model issue from a bad keying strategy once the system gets noisy. Observability has to be richer than the failure mode you're trying to debug, or you end up guessing.

The job that makes the system durable

The agent solves the semantic problem. The job in app/jobs/embedding_generator.py solves the production problem.

It doesn't invent vectors. It makes sure the vectors entering the index are valid, and that failures get classified correctly when something goes wrong. Three responsibilities: content-length validation, embedding-dimension validation, and retry/DLQ routing.

The retry policy is exponential backoff with capped intervals, 30 seconds, then 2 minutes, then 10 minutes for transient failures. That growth pattern (~4x between steps) gives a temporary upstream issue room to clear without flooding the pipeline with immediate retries, and the cap keeps the worst-case retry budget bounded. The shape is older than queue infrastructure itself, since TCP's RTO calculation does the same thing, and it works because most transient failures recover on a timescale that doesn't require sub-second polling.

But not every failure deserves another attempt. Some are terminal by definition. Content too long? Then the input has to change. Dimension count doesn't match the expected shape? Then the embedding isn't fit for indexing. Those cases go straight to the DLQ as poison messages, input that no amount of retrying will fix, because the retry policy operates on time and not on the cause of the failure.

That's the right line to draw. A retryable failure says "try again later." A terminal one says "this record needs intervention or upstream correction." Confusing the two is how queues turn into retry-storm amplifiers, with the same bad input cycling through the worker pool, burning capacity that healthy traffic needs.

Why terminal failures go straight to the DLQ

I treat the dead-letter queue as an operational control surface, not a waste bin. When a record lands there because its content is too long or its dimensions are wrong, that's worth knowing. It tells me exactly what kind of correction is needed.

It also keeps the normal path clean. Retryable failures get another chance. Invalid records get isolated. The DLQ becomes the place where malformed input, schema changes upstream, or data quality problems surface, instead of hiding under repeated attempts that read as noise on a dashboard. Which is why the terminal list is explicit: content_too_long and dimension_mismatch. Neither one is a transient condition. Retrying them would only burn time and make the failure harder to interpret.

The job's logic protects downstream search from poison data. Unglamorous sentence. It's also the entire point of the job.

The real benefit of four vectors

The payoff here has nothing to do with multi-vector being better in the abstract. What it buys is a candidate record that can answer multiple search intents without being flattened into a single compromise representation. Classical information retrieval solved this exact problem with BM25F, field-weighted BM25, which lets a search over "title, body, anchor text" weight each field separately rather than concatenating them into one bag of words. The four-view embedding is the dense-retrieval analog. Each field gets its own representation, and the search layer composes them at query time instead of leaning on a pre-flattened average.

The work-history view preserves sequence and career motion. The skills view preserves credentials, designations, and specific capabilities. The profile view keeps the higher-level picture intact. And the general view gives broad coverage when a query isn't narrowly about one field, the safety net for anything that doesn't map cleanly onto a single mode.

The record keeps its internal structure and still becomes searchable. Had I stayed with one blended embedding, queries about certification would have kept sliding toward domain-adjacent candidates, and queries about career history would have kept overvaluing polished summary language. Splitting the record before embedding makes those tradeoffs explicit instead of accidental. The search layer can now choose which view drives the score, and which one only gets to break ties.

How this changes the way I debug search

One of the best side effects of the split is that debugging got a lot cleaner. When the embedding layer is monolithic, every retrieval complaint feels like one problem. With four vectors and a separate job layer, each failure mode has a coordinate, so I can point at which vector, which stage, and which agent owns the problem instead of triangulating.

Latency high? I look at cache misses and generation time. Burning API calls? Cache hit rate. Records failing to persist? Validation and DLQ counts. Embedding shapes wrong? Then I know the issue lives in the contract, not in the ranking layer. That mapping from symptom to layer is what makes a system observable in the precise sense. It doesn't mean "we have dashboards." It means every reasonable question has a corresponding metric to read.

That division of responsibility is what makes the whole thing tractable. The embedding agent is about generating the right semantic inputs. The job is about making sure those inputs survive production constraints. Each layer stays narrow enough to inspect without guessing.

Why I kept the implementation narrow

I didn't want the embedding path turning into a catch-all orchestration layer. Plenty of systems get hard to reason about because one component starts doing semantic prep, retry control, persistence, error handling, and cache management all at once. Designers call that god-object accretion, where convenience eats coherence one feature at a time.

This one stays focused. The agent prepares and caches the vectors. The job validates and routes outcomes. The search layer consumes the resulting embeddings through the normal indexing path. That separation keeps each piece easier to test, easier to replace, and free of accidental coupling. Change the retry behavior and semantic preparation needs no rewrite. Adjust the vector split and dead-letter routing never moves. Same logic as single-responsibility design at the class level, applied at the service boundary.

What the split preserved

Accuracy was part of what I wanted to protect. The bigger thing was the shape of the candidate record itself. A blended embedding makes a record searchable while smoothing away the distinctions that separate one candidate from the next: experience isn't the same as skills, and a polished title can hide a work history that tells a different story. Four semantic views keep those distinctions somewhere search can still reach them.

That's the real win. The record stays one record, but it no longer has to pretend it only means one thing.

Next is connecting this multi-vector representation to the retrieval side with the same discipline: explicit field intent, explicit score handling, and no hidden magic between the index and the ranking layer. Late-interaction models like ColBERT push the idea further still, preserving per-token vectors and aggregating at query time, which buys precision without committing to a field schema up front. That's a much bigger lift, though, and the four-view split is the right step from where I started. Once the record isn't flattened anymore, the search layer has to earn its keep.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

Coverage Before Creativity: The RAG Gate That Keeps My Blog Pipeline Honest

Daniel Romitelli — Mon, 27 Jul 2026 16:02:09 +0000

The first failure I had to get rid of in the blog pipeline wasn't a bad paragraph. It was a bad evidence set. The system would find a few nearby chunks, mistake density for coverage, and then draft as if that narrow slice stood in for the whole repository. Text like that sounds confident right up until you put it next to the code. So I stopped treating topic selection as a writing problem and started treating it as a retrieval coverage problem.

That distinction does real work. If the upstream evidence is thin, no amount of prompt polish rescues the result, and the draft will still overfit whichever cluster of files happened to match the query first. I wanted the pipeline disciplined about breadth before it got creative about prose. So the gate moved earlier: query fan-out, file-path-aware dedupe, breadth validation, pinned excerpts, and only then the writing pass.

The gate lives before the writing step

In my pipeline the important work happens before generation starts. The dispatcher is where the topic search fans out through multiple lanes: curated highlights, a fixed RAG query pool, and recent commit-derived queries. That's deliberate. A single semantic search tends to collapse into the same dense corners of the codebase, which is exactly where a system gets persuasive and shallow at once.

The dispatcher doesn't need to know how the post will read yet. Its job is to prove that the candidate topic has enough distinct evidence behind it to deserve a draft. Which means the retrieval layer has to do more than collect relevant chunks. It has to show spread, and it has to show that the match didn't come from one file, one subsystem, or one repetitive cluster of adjacent chunks.

That's why the shared blog utilities matter. The generator path imports checkRagSufficiency and fetchFailureEvidence from supabase/functions/_shared/blog-utils.ts, and that's the right place for it: the part of the system that decides whether retrieval is good enough should sit close to the code that evaluates it.

flowchart TD
  A[Dispatcher: curated highlights] --> B[Query fan-out]
  C[Dispatcher: fixed RAG query pool] --> B
  D[Dispatcher: recent commit queries] --> B
  B --> E[RAG retrieval]
  E --> F[Dedupe by repo and file path]
  F --> G[Breadth check]
  G -->|passes| H[Pin excerpts]
  G -->|fails| I[Reject candidate]
  H --> J[Stage-compose merge]
  J --> K[Generator / drafting]

That flow is the real control surface. By the time the generator runs, a decision has already happened: is the evidence wide enough to trust?

Three query lanes, three different jobs

The fan-out isn't random, and it isn't a single blended prompt pretending to be a strategy. I built it as three separate lanes, because each one catches a different failure mode.

Curated highlight queries keep the system anchored in the kinds of features and systems I already know are worth revisiting. Those posts usually come out of places I've touched repeatedly: workflow orchestration, retrieval, caching, parsing, state management, security boundaries, or data transformation. They help the pipeline remember what's already interesting in the repository family.

The fixed query pool is the broadest lane. It's there to force coverage across architectural themes and implementation patterns instead of letting one topic family take over. This is the lane that goes looking for general system shape: event-driven flows, retrieval logic, prompt construction, orchestration, caching, retry paths, auth boundaries, model inference, ETL, and state machines. A selector living only inside the curated highlights would turn too self-referential, and one living only inside the fixed pool would turn too generic. Running both is what keeps the output grounded and varied.

Recent commit queries add the temporal dimension. They bias the selector toward what actually changed recently rather than letting the system settle into evergreen topics that no longer reflect the repository's current shape. That matters because the most obvious topic is often the wrong one once recent work has moved the architecture. A topic can be semantically relevant and still be stale in practice.

The reason for splitting the lanes is simple. No single lane is trusted to decide the topic alone. They feed the same retrieval pass, but for different reasons: one preserves editorial continuity, one broadens the architectural search, one keeps the system current. Merged, the three give me a candidate set that's much harder to fool with local similarity alone.

Why the dedupe key is repo plus file path

Once retrieval returns a pile of chunks, the next problem is repetition. Similarity search loves repetition. A single file can dominate a result set by surfacing multiple overlapping excerpts, especially when the file is dense or when several queries land in the same section of code. Let that through and the draft starts building itself around one artifact instead of one system.

Which is why file-path-aware dedupe carries so much weight here. I want repeated hits from the same repo and file path to collapse early. Five adjacent chunks all sounding relevant doesn't impress me if they're pointing at the same paragraph of the same file. What I care about is whether the sample spans distinct parts of the codebase.

Repo identity belongs in the key too. In a multi-repo setup, two chunks can look similar for completely unrelated reasons. Both might describe retrieval logic, or orchestration, or prompt shaping, while living in different systems that shouldn't count as interchangeable evidence. Repo plus file path tells me whether the sampling is broad or just rediscovering the same neighborhood under different search terms.

The practical effect is a retrieval layer that's less greedy. The first obvious cluster stops earning extra representation, and repeated evidence stops counting as coverage. Smaller candidate set. Considerably more trustworthy one.

There's a second-order benefit. Dedupe reduces the risk that a single implementation detail becomes the skeleton of the whole post. Without it, a draft can end up over-explaining one helper, one file, one branch of logic, purely because retrieval happened to hit it several times. That bias gets broken before the generator ever sees the prompt.

Measuring breadth instead of eyeballing it

After dedupe I don't ask whether the chunks feel diverse. I measure whether the sample is wide enough to support a post. That's the entire point of the breadth gate: reject candidate sets that are semantically plausible but structurally weak.

This is where checkRagSufficiency fits into the pipeline. The name is exactly what the behavior needs to be, a sufficiency check. If the retrieved set can't prove enough spread across the repository, it should not advance. The system should fail closed rather than guess.

What I like about a threshold is that it changes what retrieval means. Retrieval stops being a convenience layer that gathers whatever sits closest and becomes a gate that has to establish evidence quality before writing begins. The failure mode shifts from "draft written from a narrow slice" to "candidate rejected because the sample is too narrow." I'll take the second one every time.

That rejection path earns its keep. It catches the query pool landing too hard in one subsystem, recent changes dominating the semantic neighborhood, one file throwing off too many overlapping hits. Plenty of retrieval bugs look like success right until the prompt is assembled. The breadth check is the thing that stops them from becoming published text.

I like that the failures can produce something concrete, too. fetchFailureEvidence belongs in the same shared utility layer because it gives me a way to inspect why a candidate was rejected, which is useful during tuning. If a topic keeps failing breadth, I can see whether the cause is query bias, inadequate file-path diversity, or a retrieval window too small for the amount of material I want to cover.

The stage-compose step adds a second guardrail

File-path retrieval and semantic retrieval aren't competing systems. File-path retrieval gives me boundary-aware evidence; semantic retrieval gives me breadth across related concepts. In blog-stage-compose I merge both and dedupe a second time, since the two strategies can land on the same excerpt from different angles, and the evidence set should not regress back into repetition right before drafting.

Pinned excerpts hold the evidence in place

Once a candidate survives the coverage gate, I pin the excerpts that explain why the topic is worth writing about. Not a cosmetic step. It buys stability, because without pinned evidence the drafting stage has too much freedom to wander away from the exact chunks that earned the topic in the first place.

Pinned excerpts act like an anchor for the generation pass. They preserve the evidence trail, and they hold the prompt to the source material the draft was built from. That matters because the strongest failure mode in a retrieval-driven blog system isn't outright hallucination. It's the gradual slide: the draft starts from real evidence and then generalizes past what was actually retrieved.

Pinning the survivors makes that harder. The generation stage has to stay connected to the specific implementation details that passed the gate. Review gets easier too, since I can inspect exactly which chunks were considered important enough to carry forward.

Pinning and breadth checking together are what give the pipeline its shape. Breadth says the sample is wide enough. Pinning says these are the exact pieces that justify the topic. Between them, they stop the generator from inventing confidence the retrieval layer didn't earn.

Why I prefer rejection over a weak draft

I'm completely comfortable with a pipeline that says no. I want it to say no when the evidence is bad. A weak candidate shouldn't get rescued by a polished prompt, and if the retrieval set is narrow, rejection is the honest response.

That discipline keeps the blog output specific. It ties the writing to actual systems instead of generic patterns, and it saves me from editing around a draft that was born from a bad evidence shape. A rejected topic costs less time than a published post that looks right but misses the structure of the thing it claims to describe.

This matters more in a multi-repo environment. Once you have a handful of systems with overlapping concepts, retrieval can get too eager to collapse them into one theme. A good gate has to resist that collapse: respect repository boundaries, file boundaries, evidence density boundaries. If those boundaries aren't visible in the sample, the draft should not happen yet.

That's the philosophical change I stopped treating as optional. The system doesn't owe me a draft. It owes me a trustworthy sample, and that sample has to prove the topic is broad enough, current enough, and distinct enough to justify writing.

The actual win isn't creativity; it's control

What changed here wasn't my ability to generate prose. What changed was the quality of the evidence stack the prose starts from. The dispatcher fans out through curated highlights, a fixed query pool, and recent commit queries. The retrieval layer dedupes by stable keys, the sufficiency gate checks for breadth, and stage-compose merges file-path and semantic evidence before deduping a second time. Pinned excerpts hold the final prompt to what survived.

That sequence turns retrieval into a control system rather than a suggestion engine. It enforces a minimum standard before a draft is allowed to exist, and that's the kind of discipline a blog pipeline needs if it's going to write about real systems with real precision.

The result is fewer bad drafts, but also a pipeline that knows what it doesn't know early enough to stop itself. When the evidence is wide, the writing stage does the part it's good at. When the evidence is narrow, the system does the part I'm more grateful for. It refuses to pretend it knows.

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series

DEV Community: Daniel Romitelli

Voice Input Is an Intent Language

The split

Silence is a billable input unless it is filtered early

The parser decides between action and text

The renderer is only the destination

Treat speech like code

Provenance Belongs in the Image Table

1. The row is the receipt

2. Why not a separate audit table

3. Columns carry facts the product asks about

4. Lineage becomes a query

5. What the row cannot tell you

CarSegNet v2: One Lot Photo to a Showroom Composite

The ops case: same-day media without inventing ROI

What one photo produces

Topology is the hard part, and the buyer feels it first

Carvana topology: strong outer, almost no hole recovery

Glass, correctability, and what the booth actually does

Failure modes I keep in the open

For implementers

CLI and booth entry

Still path cascade

Seed, extent, and the interior trust region

WindowSweep audit path

Plates and compositing detail

Server, WebSocket, and voice

Measurements that prove plumbing vs topology

Messages Need a Protocol Before They Need a Chat UI

1. The invariant comes before the interface

2. The row carries the user-facing truth

3. Receipts and teardown are side effects

4. The review table

A Cache Key Is an Equivalence Relation

1. The request definition gets trimmed before hashing

2. Normalization removes accidental difference

3. The two bugs sit on opposite sides

Workflow JSON Is Generated Code

1. Keep the platform vocabulary narrow

2. Build objects before JSON

3. Treat import as deployment state

4. The table I test against

Why I Kept Search Scope Inside a Single Supabase RPC

The bug was scope, not similarity math

The caller sends one object

The database function matches the same interface

Why JSONB belongs in the request, not outside it

Why the index is part of the same story

What went wrong the first time

Why I trust the boundary now

The AgentGroupChat Pattern That Keeps the Mapper from Drifting

What broke first

The core cycle in AgentGroupChat

How history gets rehydrated before the run

The template library earns its keep

The real runtime wiring in the orchestrator

A runnable validator that actually checks structure

Why this pattern is easier to trust

What actually settled it

Validation Geometry Is Part of the Model

The baseline was a geometry test, not a tuning contest

The timeline is the experiment

Why the saved label file had to be rejected

The capacity objection needed a clean target

Temporal bleed arrives without a stack trace

The minute ceiling becomes more credible when the baseline cannot cheat

What the feature importances can and cannot say

The discipline is to make shortcuts impossible

What I changed in how I think about baselines

Vector Split by Chunk: Why My Retrieval Stops at the Boundary I Drew

The boundary I chose on purpose

Why the naive version fails

How the chunk becomes a vector

Why this pattern beats one vector per file

The retrieval path that makes the split worth it

The shape of the storage model

The practical tradeoff: granularity buys control, not magic

The retrieval story, drawn plainly

Why I kept the split at the chunk boundary

Four Vectors, One Record: How I Split Embeddings Before They Hit Search

The core cycle in `AgentGroupChat`