DEV Community: Humza Tareen

The Connective Tissue of an AI Platform: Workflow, Taxonomy, Auth, and Memory

Humza Tareen — Tue, 25 Aug 2026 21:27:58 +0000

When you're building an AI evaluation platform with multiple microservices, the "core" services get all the attention — the evaluation engine, the scoring system, the RAG pipeline. But a platform doesn't work without the connective tissue: the workflow orchestration that keeps humans in the loop, the taxonomy engine that classifies tasks intelligently, the platform service that ties authentication together, and the evaluation suites that ensure models actually remember context.

These four services don't make headlines, but they're what turned a collection of microservices into an actual platform. Here's what went into each one and why the engineering decisions mattered.

Workflow Orchestration: The Human-in-the-Loop Engine

AI evaluation is not fully automated — and it shouldn't be. Certain decisions require human judgment: Is this model response harmful? Does this evaluation rubric make sense for this domain? Is this edge case a genuine failure or acceptable behavior?

The workflow orchestrator manages these decision points. It coordinates multi-step evaluation workflows where some steps are automated (LLM scoring, data validation) and others require human approval before the pipeline continues.

The Architecture

The core is a state machine built on FastAPI and PostgreSQL. Each workflow is a DAG (directed acyclic graph) of tasks, where each node can be:

Automated: Runs immediately, calls another service (scoring, data enrichment), stores the result
Human gate: Pauses the workflow, notifies the assigned reviewer via the notification service, waits for approval/rejection
Conditional: Routes to different branches based on previous step outcomes (e.g., if confidence score < threshold, escalate to senior reviewer)

State transitions are persisted in PostgreSQL with Alembic-managed migrations. Every transition is logged — who approved what, when, and with what context. This audit trail turned out to be critical for client reporting.

Real-Time Updates with WebSocket

The original system polled the API every 5 seconds to check workflow status. With dozens of reviewers working concurrently, this created unnecessary load and a poor user experience — you'd approve a task and see nothing happen for up to 5 seconds.

I replaced this with WebSocket connections that push state changes in real-time. When a reviewer approves a step, every connected client watching that workflow sees the update instantly. The implementation uses FastAPI's WebSocket support with Redis Pub/Sub as the message broker, so it works across multiple Cloud Run instances.

# Simplified WebSocket broadcast pattern
async def broadcast_workflow_update(workflow_id: str, event: dict):
    channel = f"workflow:{workflow_id}"
    await redis.publish(channel, json.dumps({
        "type": "state_change",
        "workflow_id": workflow_id,
        "step": event["step"],
        "status": event["new_status"],
        "actor": event["actor_email"],
        "timestamp": datetime.utcnow().isoformat()
    }))

Production Logging Overhaul

The existing codebase used print() statements everywhere. In production on Cloud Run, these were effectively invisible — they'd show up as unstructured text in Cloud Logging with no way to filter, search, or correlate them.

I replaced the entire logging infrastructure with structured JSON logging. Every log entry includes a correlation ID that traces a request across the workflow orchestrator, the notification service, and whatever downstream service is involved. When a workflow fails at step 4 of 7, you can now trace exactly what happened at each step, in each service, with a single query.

Taxonomy Workflow Engine: Intelligent Task Classification

Not all evaluation tasks are the same. A code generation task requires different rubrics, different evaluators, and different tooling than a conversational AI task. The taxonomy engine is the routing layer that classifies incoming tasks and determines which evaluation workflow to apply.

The Problem It Solves

Before this service existed, task classification was manual. A project manager would look at incoming evaluation requests, decide which team should handle them, and assign the appropriate rubric. This worked at 50 tasks per day. It didn't work at thousands.

How It Works

The engine uses a combination of keyword matching, metadata analysis, and configurable rule sets to classify tasks. Each classification determines:

Which evaluation rubric to apply
Which reviewer pool to draw from (by expertise)
Whether the task requires single or multi-reviewer consensus
SLA targets for completion time

The file upload system allows clients to submit evaluation tasks in bulk via CSV/JSON uploads to GCS. The engine parses, validates, classifies each row, and enqueues them into the appropriate workflow — all asynchronously via Cloud Tasks.

Infrastructure: Cloud SQL for taxonomy rules and classification history, GCS for bulk file uploads, Cloud Run for the API layer, Cloud Tasks for async processing.

Core Platform Service: The Authentication Backbone

Every microservice in the platform needs to answer two questions: "Who is making this request?" and "Are they allowed to do this?" The core platform service provides those answers.

JWT Authentication Fixes

The existing JWT implementation had a subtle but critical bug: token validation was checking expiration time against the server's local time rather than UTC. Cloud Run instances can have slight clock drift, and this meant tokens would occasionally be rejected as "expired" when they were still valid, or accepted when they should have been rejected.

The fix was straightforward — normalize all time comparisons to UTC — but finding it required tracing sporadic 401 errors across multiple services to realize the pattern correlated with specific Cloud Run instances, not specific users.

# Before: clock-sensitive comparison
if token_exp < datetime.now():  # Local time — unreliable on Cloud Run
    raise HTTPException(401, "Token expired")

# After: UTC-normalized comparison
if token_exp < datetime.now(timezone.utc):  # Always consistent
    raise HTTPException(401, "Token expired")

User Management and GDPR Compliance

Built the user deletion endpoint — which sounds simple until you realize that "deleting a user" in a system with audit trails, evaluation history, and cross-service references means carefully cascading the deletion while preserving anonymized audit records. The implementation soft-deletes the user profile, anonymizes their evaluation history (replacing PII with hashed identifiers), and propagates the deletion event to downstream services via Pub/Sub.

Developer Experience

Improved the local development workflow by rewriting start_dev.sh to properly handle Docker container lifecycle. The previous script would silently fail if the PostgreSQL container was already running from a previous session, leading to "connection refused" errors that wasted 10-15 minutes of debugging time per developer, multiple times per week. The new script checks for existing containers, handles cleanup, and provides clear status messages.

Memory Evaluation Suite: Does the Model Remember?

One of the harder problems in LLM evaluation is measuring context retention. When you give a model a long conversation or a complex document, does it actually use information from the beginning when answering questions at the end? Or does it "forget" earlier context?

The memory evaluation suite provides structured tests for this. It generates conversations with deliberate information planted at various positions (beginning, middle, end), then asks questions that require recalling that information. The scoring tracks not just accuracy, but where in the context window the model starts losing information — which is critical data for the teams training these models.

Code Quality as a Feature

This was also where I implemented the team's first pre-commit workflow using Ruff for linting and formatting, enforced via GitHub Actions. The motivation wasn't just code aesthetics — inconsistent formatting was causing unnecessary merge conflicts across the team. Two developers would change the same file, both would reformat it differently, and the merge conflict had nothing to do with the actual logic.

After rolling out the pre-commit pipeline on this service and proving it reduced merge conflicts, we adopted it across every service in the platform.

The Engineering Pattern

What ties these four services together isn't the domain logic — it's the systematic engineering discipline I applied to each one. Every service I touched got the same treatment:

Pattern	Why It Matters
Structured JSON logging with correlation IDs	One query to trace a request across all services. Reduced mean time to diagnosis from hours to minutes.
Pre-commit hooks (Ruff, type checking)	Eliminated formatting merge conflicts. Caught type errors before they hit production.
Custom exception hierarchies	Consistent error responses across services. Clients can programmatically handle errors instead of parsing strings.
Alembic migrations	Version-controlled schema changes. Zero-downtime deployments with reversible migrations.
Security audit per service	Found hardcoded credentials, missing auth checks, and SQL injection vectors before they became incidents.

Why This Work Matters

It's easy to dismiss "I worked on four more services" as a breadth play. But the reality is that platform engineering requires breadth. The workflow orchestrator doesn't exist without the platform service providing authentication. The taxonomy engine doesn't work without the workflow orchestrator to route tasks into. The memory evaluation suite's code quality pipeline became the template for every other service.

These aren't four independent projects. They're four layers of a system that only works because someone cared enough to apply the same engineering rigor to the "boring" services that they applied to the "interesting" ones.

Whole-Ad Product Swap: Deterministic Planning First, Model Only Where Forced

Humza Tareen — Tue, 25 Aug 2026 21:27:23 +0000

Variant Multiplier already let an editor swap one section of a winning ad and keep the rest. The next request from a real production job — replacing product SL-603 with SL-808, a different hearing-aid SKU, across an entire finished ad — was a different shape of problem. It's not "change one section," it's "change every mention of the product, everywhere it appears, while keeping literally everything else the same." Two direct quotes from the editor drove the whole five-PR arc: the transcript editing was too rigid for word-by-word changes, and separately, "the music, voice, etc. should retain the same, we should keep the quality the same, and not make it do a lot of changes."

If a re-render can degrade something the editor explicitly asked to keep untouched, the render path is wrong for the job — no matter how good the model is.

The cheap fix first: let editors actually edit

PR #67 shipped before any product-swap work started, because it was the cheap, high-value half of the same feedback: "I am just able to select word by word here but I am not really able to change the whole sentence a lot easier," and separately, "I'm able to double click on these words and then just type it in." Both were UI gaps in the transcript editor, not pipeline gaps — selecting by sentence or scene instead of only by word, and retyping a line verbatim instead of only substituting individual words. Shipping this first, standalone, meant the harder product-swap work that followed didn't also have to carry an unrelated UX fix in its diff.

A product catalog the tool never had

PR #69, stacked directly on top of the transcript work, is pure groundwork with no user-visible feature of its own: a product catalog, because Variant Multiplier had no concept of "a product" at all before this. The editor's own framing made the requirement explicit: "have a product selection right here, for Pro Bluetooth, for [the other SKU], and maybe other tons of products" going forward. The catalog data itself is maintained in the main video-generation service and synced into this tool rather than duplicated and drifted — one canonical source for product identity, read by whichever tool needs it.

Planning the swap before touching a single frame

PR #70 is the center of the whole arc, and its title states the engineering decision directly: plan the swap deterministically first, and only fall back to a model where a deterministic rule genuinely can't decide. The editor's own words framed the danger of the naive approach: "if I click here Pro 3.0 → Pro Bluetooth, the AI will be able to change everything" — technically true, and exactly the failure mode to avoid, because "change everything" is also how you accidentally change the music, the pacing, or the voice quality nobody asked to touch.

type SwapPlanLine = {
  lineIndex: number;
  originalText: string;
  matchKind: 'deterministic' | 'model-required';
  newText: string;
};

function planLine(line: TranscriptLine, from: Product, to: Product): SwapPlanLine {
  // exact product-name substitution: deterministic, no model call
  if (line.text.includes(from.name)) {
    return {
      lineIndex: line.index,
      originalText: line.text,
      matchKind: 'deterministic',
      newText: line.text.replaceAll(from.name, to.name),
    };
  }
  // an indirect reference ("this device", a spec number tied to the old product)
  // has no deterministic rule — fall to the model, scoped to this one line
  return {
    lineIndex: line.index,
    originalText: line.text,
    matchKind: 'model-required',
    newText: null, // resolved by a scoped model call, not a blanket rewrite
  };
}

The output of this PR is a plan and a per-line diff the editor reviews before anything renders — not a video. Deciding what changes is separated cleanly from rendering the change, which is what makes the next PR possible without redoing this one.

Audio-only render: a byte-identical picture

PR #71 renders the plan, and it's the PR that actually delivers on "keep the quality the same, don't make a lot of changes" — taken literally rather than as a vague aspiration. The existing render path, built for section swaps, re-encodes every segment through libx264 because it has to concatenate generated b-roll with master footage. Running a pure audio change through that path would re-encode, and therefore degrade, 100% of a picture that never needed to change at all. The fix instead reissues the ad with new audio and a picture that is byte-identical to the original master — no re-encode, because the video stream simply isn't touched. That's the difference between "we tried to preserve quality" and "there is no quality to lose because the bits didn't move."

Captions were the gap nobody had asked about yet

PR #73 came from a single follow-up question, not a bug report: if the ad has captions, do we update them? The team hadn't — because the swap copies the picture untouched, captions baked into the master's pixels survived exactly as they were, now describing the wrong product on screen while the new audio said something else. On the real SL-603 job, one caption line still read the old product's brand name on screen while the new voiceover said the new SKU's name — a rejection risk on an actual client deliverable, not a cosmetic gap. The fix regenerates captions from the finished swapped audio, so the on-screen text and the spoken audio agree again.

Verifying the fix against the in-house reference, not just testing it

PR #74 is a short, disciplined close-out: before considering #73 done, review its caption approach against the equivalent module in the main video-generation service, which both tools treat as the in-house reference implementation for captions. The review confirmed the approach was already right — both use subtitle-track rendering rather than burned-in draw commands, and Variant Multiplier's captions module carries forward the sibling's own hard-won caption-timing fixes rather than reinventing them. Checking a new implementation against the team's own established reference, rather than shipping the first version that passed its own tests, is what caught that this one didn't need a second round.

The pattern: separate the decision from the render

The core lesson across five PRs is the same one #70 states outright in its title: plan deterministically, and reach for a model only for the specific lines a deterministic rule genuinely can't resolve. A full-ad product swap sounds like it demands a single large generative pass. In practice, the vast majority of a script is exact product-name substitution — free, instant, and impossible to get subtly wrong the way a model rewrite can be. Reserving the model for the handful of lines that reference the product indirectly, and reserving a re-encode for the rare case that genuinely needs one, is what let this feature ship with a render path that provably can't degrade the 95% of the ad that was never supposed to change.

Voice Pipeline Economics: Double-Billing, a Backwards Ladder, and a Lexicon That Never Reached the Voice

Humza Tareen — Tue, 25 Aug 2026 21:27:18 +0000

Every AI video pipeline eventually has to answer an unglamorous question: what did we actually pay for that clip? On the main video-generation service, the answer for months had been "a hardcoded constant." That's fine until the vendor changes its own pricing, or a code path pays for the same synthesis twice, or a voice engine mints a clone, bills for it, and never sends it downstream. Over a ten-PR run I audited and rebuilt the voice and lip-sync pipeline from the billing layer up, then used the vendor's own SKU tiers to cut cost 7x without touching output quality.

A cost model built from hardcoded constants isn't a cost model. It's a guess that happens to compile.

Billing what the vendor actually charges

PR #224 was workstream one of three from a sibling-tool audit: port the cost-accounting fixes that Presenter Generation and Variant Multiplier had already found, verifying each one against this repo's own code rather than assuming the same defect existed in the same place. Anthropic returns exact token counts on every response. Nothing in the pipeline read them — every charge was a hardcoded per-call constant, so the ledger and the vendor invoice diverged the moment usage drifted from whatever number had been typed in at launch.

The same PR closed a second gap: two editor-facing routes could spend money — kicking off a generation, retrying a step — outside any run. A run is the unit everything else (budgets, audit trail, the cost ledger) is keyed to. A spend with no run attached is a spend the ledger can't even see, which is worse than a wrong number.

Paying twice for a take the model returns unchanged

PR #225 found the sibling bug's twin: some vendor calls return the exact same asset on a retry — no new synthesis happened — and the pipeline billed a second time anyway because "call succeeded" and "call did new work" were treated as the same fact. The fix is the boring, correct kind: hash the output, and only charge when the hash changes from the take you already paid for.

A budget/quality SKU ladder that opened backwards

PR #232 replaced one hardcoded lip-sync SKU pair with two explicit fallback chains:

Chain	Ladder	Rate
budget (default)	1.9.0-beta → v2	$0.0117/s → $0.05/s
quality (rollback)	v2/pro → react-1	$0.0833/s → $0.167/s

That's a 7x cut against v2/pro on the pass, but the ladder direction is what makes it safe rather than reckless. Both chains fall back upward — to a better SKU than they opened with, never a worse one. A fallback fires because the first choice failed; degrading further would trade an outage for a silent quality drop nobody chose. react-1 only appears in the quality chain, because landing there by accident from the budget chain costs 14x the cheapest SKU with nobody having decided to pay for it.

A cache keyed on the wrong thing

PR #229 is the pattern this whole cluster keeps rediscovering in different clothes: a cache key must move when its content does. The voiceover cache was keyed on scene index — vo-scenes/<idx>.mp3 either existed or it didn't. Changing the voice, the scene's emotion, the TTS model, or any of three delivery flags left the old recording in place, generated under settings that no longer applied. Only a text edit invalidated it, because invalidateSceneVo was called from exactly two places in the whole codebase.

The fix content-keys the cache via a sidecar file rather than renaming the asset itself — the filename can't move because it's served over HTTP by path — and makes the warp chain that depends on it optional rather than assumed-present.

Minted, billed, and never sent

PR #258 is the sharpest failure in the batch. A Kling voice clone was minted from the source audio, billed to the run's ledger, and then never actually sent to the clip-generation call that was supposed to use it — the clip rendered with Kling's default voice while the ledger recorded a custom-clone charge. The bug produced a video that sounded fine, which is exactly why nobody caught it by watching output: the defect was invisible in the artifact and only visible in the diff between the request built and the request sent.

// before: clone minted, billed, and dropped on the floor
const clone = await mintKlingVoiceClone(audioBuffer);
await ledger.charge(runId, 'voice_clone', clone.cost);
// clip request never referenced clone.voiceId — Kling used its own default

// after: the clone id is a required field on the clip request,
// not an optional one the caller can forget to attach
const clip = await requestKlingClip({
  ...clipParams,
  voiceId: clone.voiceId, // TypeScript now refuses to compile without this
});

A voice bind that can 422 a whole run

PR #260 found that the voice-bind call — attaching a cloned voice to a specific Kling model invocation — could 422 and take down every clip in the run with it, not just the one clip that needed the voice. A probe surfaced the failure mode: bind failures need to degrade one clip, not cascade to the batch. PR #271 found the same class of bug from the other direction — every scene ended up with a different voice because the clone was minted from a source file that didn't exist yet at mint time, a race between file-write and clone-request that only showed up under real concurrency.

A pronunciation dictionary nobody could hear

The scripts for this pipeline are built on product-specific vocabulary — anatomical terms, brand names — and a TTS voice guesses at pronunciation differently take to take, so the same word could be said two different ways inside a single video. PR #230 built a pronunciation dictionary and, more importantly, a test that could actually fail against it, because "the audio came back different" proves nothing against a non-deterministic voice model on its own.

That dictionary shipped and did nothing for two more PRs. PR #262, on the sibling repo's own pipeline, found the lexicon was attached to a voice synthesis call nobody actually listened to downstream — the dictionary was correct, the wiring wasn't. PR #261 refactored the rule shape first, so a pronunciation rule could carry either a phoneme or a plain-text alias, because some mispronunciations aren't phonetic at all — they're the model reading a brand name as an acronym.

The pattern across all ten PRs

Every fix in this cluster is a variant of the same root cause: a value computed correctly in one place and consumed incorrectly — or not at all — somewhere downstream. The billing constant was correct until the vendor's pricing moved. The cache key was correct until content changed without the key changing. The voice clone was correctly minted and billed and then silently dropped before the request that needed it. None of these are algorithmically hard bugs. They're wiring bugs, and the only defense against wiring bugs is measuring the actual request sent, not the intent that produced it — which is why almost every PR here ends with a test or a probe that reads the real payload, not the code that built it.

Vision-in-the-Loop: When the AI Rewrites Its Own Prompts from the Generated Frame

Humza Tareen — Tue, 25 Aug 2026 21:26:43 +0000

On the AI video ad platform I work on, every scene goes through the same painful loop: write a prompt, send it to an AI video model provider, wait two minutes, open the result, squint at the frame, and decide what went wrong. Camera too wide. Product missing from the hero shot. Color palette drifted warm when the brand brief says cool neutrals. Avatar looks like a different person than scene three.

That loop was manual, slow, and expensive. Each regeneration burns GPU credits. Operators were becoming prompt engineers by accident — and still missing subtle failures until stitch time, when fixing scene four means re-rendering everything downstream.

The insight behind vision-in-the-loop prompt authoring is simple: the model that wrote the prompt can also look at its own output and rewrite the prompt with surgical fixes. Not a full replan — a per-scene correction grounded in the actual generated frame, not the operator's memory of what they hoped would appear.

The manual loop we were trying to kill

Before this work shipped, the swipe iteration flow looked like this:

Plan — Claude generates a scene-by-scene script with visual prompts
Generate — each scene renders independently through an AI video model provider
Review — operator opens the portal, compares frames to the reference ad
Rewrite — operator edits prompts in a text field, often guessing at what the model misread
Regenerate — repeat until acceptable or budget exhausted

Steps three and four are where throughput dies. An experienced operator can spot "product not visible" in three seconds, but translating that into prompt language — "medium close-up, product centered in lower third, shallow depth of field" — takes another minute per scene. Multiply by twelve scenes and three swipe iterations, and a single ad creative consumes an hour of human attention that should be spent on brand strategy, not frame inspection.

The generated frame is ground truth. The original prompt is a hypothesis. Vision-in-the-loop closes the gap between them automatically.

Vision-in-the-loop: iteration 3 architecture

The third swipe iteration introduced a per-scene feedback loop that runs immediately after the first frame of each scene is generated — before the operator ever opens the portal.

Stage	Input	Output
Frame capture	Generated clip (first keyframe extracted)	PNG at native resolution
Vision critique	Frame + original prompt + scene intent + brand constraints	Structured issue list (severity, category, evidence)
Prompt rewrite	Original prompt + issue list + reference frame (optional)	Revised prompt with targeted deltas only
Regenerate	Revised prompt + anchored persona image	New clip (single retry by default)

The vision model receives the generated frame alongside the scene's intent metadata — shot type, product placement rules, palette constraints, and whether this is a person scene or product B-roll. It returns a structured critique, not free-form prose. That structure matters: downstream rewrite logic keys off issue categories like framing_too_wide, product_absent, palette_drift, and identity_mismatch.

interface VisionCritique {
  sceneId: string;
  frameUrl: string;
  issues: Array<{
    category: VisionIssueCategory;
    severity: 'blocker' | 'warning' | 'nit';
    evidence: string;       // what the model sees in the frame
    suggestedFix: string;   // natural-language correction hint
  }>;
  passThreshold: boolean;   // true if no blockers
}

async function visionInLoopRewrite(
  scene: ScenePlan,
  generatedFrame: Buffer,
): Promise<ScenePromptRevision | null> {
  const critique = await visionCritique({
    frame: generatedFrame,
    originalPrompt: scene.visualPrompt,
    intent: scene.intent,
    brandRules: scene.brandConstraints,
  });

  if (critique.passThreshold) return null;

  const blockers = critique.issues.filter(i => i.severity === 'blocker');
  if (blockers.length === 0) return null;

  return promptRewriter.revise({
    original: scene.visualPrompt,
    issues: blockers,
    preserveLocked: scene.lockedElements, // persona anchor, product lock, etc.
  });
}

The rewrite step is constrained: it may not change locked elements (avatar reference, product SKU imagery, mandated taglines). It only adjusts the visual prompt fields that the vision critique flagged. This prevents the common failure mode where an aggressive rewrite "fixes" the framing but drops the brand voice or swaps the protagonist.

What the vision model actually catches

In production testing across roughly 200 scene generations, the vision critique surfaced issues that text-only plan QA missed entirely:

Framing drift — wide establishing shot when the plan called for medium close-up on the spokesperson
Product absence — hero product not visible despite explicit product-lock instructions in the prompt
Palette mismatch — warm golden-hour tones on a brand that specifies cool clinical whites
Background clutter — competing visual elements that dilute the focal subject
Motion mismatch — static hold when the reference scene had subtle camera push-in

Each issue maps to a prompt delta. "Framing too wide" becomes tighter focal length language and explicit subject placement. "Product absent" triggers a product-lock reinforcement clause and shot-type downgrade to guarantee visibility. The rewrite is incremental — we append and refine, not replace wholesale.

Motion-transfer defaults for person scenes

Vision-in-the-loop solves prompt accuracy. A parallel iteration — swipe iteration 2 — solved a different problem: person scenes that felt dead on arrival.

Generic text-to-video generation treats every scene the same. For product B-roll, that works — a slow pan across a bottle on marble is fine with standard generation. For person scenes, the reference ad almost always has subtle body motion, micro-expressions, and natural idle movement that pure text-to-video renders as uncanny mannequin holds.

The fix: make motion-transfer the default generation mode for any scene classified as a person scene. Motion-transfer takes a reference clip (from the source ad or a prior good take) and transfers the subject's motion onto the generated avatar, preserving liveliness while swapping identity.

function resolveGenerationMode(scene: ScenePlan): GenerationMode {
  if (scene.classification === 'person') {
    return {
      mode: 'motion-transfer',
      referenceClip: scene.referenceMotionClip ?? scene.referenceKeyframe,
      holdPolicy: 'lively', // shorter static holds, subtle idle motion
      brollCentering: 'person', // B-roll cuts stay person-centered, not product
    };
  }
  return { mode: 'standard', holdPolicy: 'default' };
}

Two supporting changes shipped alongside the default:

Livelier holds — reduced minimum hold duration on person scenes so the model doesn't freeze the subject into a portrait pose for three seconds
Person-centered B-roll — when a person scene cuts to supplementary footage, the B-roll framing centers the human subject rather than defaulting to product hero shots that break narrative continuity

The impact was immediate: first-pass acceptance rate on person scenes climbed because the output moved like the reference, not because the prompt was more poetic.

Avatar persona anchoring: stopping identity drift

Even with better prompts and motion-transfer, multi-scene ads had a persistent quality problem: the avatar looked like one person in scene two and a cousin in scene seven. Generative models don't maintain identity across independent renders — each scene is a fresh diffusion run with no memory of prior frames.

The persona anchoring fix injects the same reference image — the approved avatar headshot or prior best frame — into every person-scene generation call, not just the first one.

function buildPersonScenePayload(
  scene: ScenePlan,
  personaAnchor: PersonaAnchor,
): VideoGenerationRequest {
  return {
    prompt: scene.visualPrompt,
    imageConditioning: {
      primary: personaAnchor.referenceImageUrl,
      weight: personaAnchor.conditioningStrength, // typically 0.7–0.85
      lockFace: true,
    },
    motionTransfer: scene.motionReference,
    negativePrompt: personaAnchor.antiDriftNegatives,
  };
}

Before anchoring, operators reported "face drift" on roughly 30% of multi-scene runs. After anchoring on every person scene — not just scene one — drift dropped to single digits. The remaining failures were usually extreme angle changes (profile shots) where conditioning weight needed per-scene tuning, not absent anchoring.

Vision-in-the-loop and persona anchoring compose cleanly: if the vision critique flags identity_mismatch, the rewrite step increases conditioning weight before regeneration rather than rewriting the entire character description.

Pitch-preserving VO time-compression

A separate but related pacing fix shipped in the same arc: verbatim voiceover scripts that ran longer than the reference ad's runtime.

Advertisers often provide exact copy — legal claims, mandated disclaimers, taglines that cannot be paraphrased. When the script exceeds the reference duration, the naive fix is TTS at 1.3× speed, which sounds rushed and chipmunk-adjacent. The production fix applies pitch-preserving time-compression via FFmpeg's atempo filter chain, keeping the speaker's natural pitch while fitting the audio bed to the reference length.

# Pitch-preserving compression: 45s VO -> 38s target
ffmpeg -i vo_raw.wav \
  -filter:a "atempo=1.05,atempo=1.05,atempo=1.05" \
  -y vo_compressed.wav
# atempo caps at 2.0 per filter instance; chain for ratios > 2.0

This runs after TTS render and before stitch, only when the pacing module detects over-budget duration on a verbatim (non-rewritable) script. Rewritable scripts still go through Claude pacing auto-fit first — compression is the fallback when the words are fixed.

The feedback loop as a system pattern

Vision-in-the-loop is not just a feature — it's a reusable pattern for any generative pipeline where the output is inspectable:

Generate with current parameters
Extract a inspectable artifact (frame, audio segment, JSON output)
Critique with a multimodal model against intent metadata
Revise inputs surgically, respecting locked constraints
Regenerate once (with a per-scene cap to control cost)

Metric	Before vision-in-the-loop	After (first 2 weeks prod)
Operator prompt edits per swipe	4.2 avg across 12 scenes	0.8 avg (mostly edge cases)
First-pass scene acceptance	61%	84%
Regenerations per completed ad	2.7	1.4
Person-scene identity drift reports	~30% of multi-scene runs	<8%

The cost tradeoff is one extra vision API call per scene — cheap relative to a full video regeneration. We cap at one automatic rewrite per scene per swipe iteration; if the second render still fails vision critique, it surfaces to the operator with the structured issue list attached, so their manual edit starts from diagnosis rather than guesswork.

What I'd do differently

I'd instrument the vision critique categories earlier. We added logging after the first week and immediately saw that product_absent clustered on a specific model provider's aspect-ratio handling — a vendor issue masquerading as a prompt problem. Earlier telemetry would have shortened the debugging cycle.

I'd also batch vision critiques where scenes share a generation queue, rather than serializing them. The critique is I/O-bound; parallelizing across scenes in the same job saves 30–40 seconds on a twelve-scene run without increasing regeneration cost.

How vision-in-the-loop fits the swipe iteration arc

The feature shipped across three swipe iterations, each building on the last:

Iteration	Focus	Impact
Iteration 1	Manual swipe UI — operator-driven prompt edits	Proved the rewrite loop; too slow for production volume
Iteration 2	Motion-transfer defaults + persona-centered B-roll	Person scenes felt alive; reduced "uncanny hold" complaints
Iteration 3	Vision-in-the-loop — automated per-scene rewrite from frame	Closed the loop without operator intervention

Iteration 3 only worked because iterations 1 and 2 had already established the data model for scene prompts, locked elements, and generation modes. Vision-in-the-loop is a feedback mechanism layered on a stable generation contract — not a standalone feature.

The broader lesson: in generative systems, the model's output is the best input for the next iteration. Closing that loop inside the pipeline — instead of exporting it to a human reviewer — is how you scale from demo to production without hiring an army of prompt engineers.

Viral Feel Parity: Making AI-Generated Ads Feel Like the Original

Humza Tareen — Tue, 25 Aug 2026 21:26:38 +0000

The video generation platform I work on takes a reference viral ad and produces a new version with swapped product copy, fresh voice-over, and AI-generated scenes. Technically the output was correct — scenes rendered, VO synthesized, timeline stitched. But the editor's feedback was blunt: "the avatar keeps changing, random boxes in the video, no bed sound, doesn't feel the same."

That last phrase — doesn't feel the same — is the whole problem. Viral ads work because of accumulated micro-decisions: one consistent face, ambient room tone under the narration, captions that sit naturally on screen, pacing that breathes between beats. Our pipeline was optimizing for structural fidelity to the blueprint while ignoring perceptual fidelity to the reference. Over roughly 2,800 lines across several PRs, I closed that gap. This post walks through eight concrete failure modes and the fixes that made generated ads feel like they belonged to the same family as the original.

1. Avatar consistency

The protagonist changed appearance between scenes. Scene one showed a woman with short dark hair; scene three had a different face entirely. Root cause: avatar selection was random per scene. Each scene generation call picked from the avatar pool independently, the same way we might pick a background variant — except viewers experience the protagonist as a continuous character, not a per-shot casting decision.

The fix mirrored how we already handled voice and product: pick once, propagate everywhere. The start route and every runner resolve site now pass a single avatarSlug through the job context. Scene generators read that slug instead of rolling dice.

interface JobContext {
  avatarSlug: string;
  voiceId: string;
  productSlug: string;
}

function resolveAvatarForJob(
  blueprint: VideoBlueprint,
  overrides?: Partial<JobContext>,
): string {
  if (overrides?.avatarSlug) return overrides.avatarSlug;
  if (blueprint.avatar?.defaultSlug) return blueprint.avatar.defaultSlug;
  return pickDefaultAvatar(blueprint.demographics);
}

// Start route + all runner resolve sites
const ctx: JobContext = {
  avatarSlug: resolveAvatarForJob(blueprint, req.body),
  voiceId: resolveVoice(blueprint, req.body),
  productSlug: resolveProduct(blueprint, req.body),
};

One slug, one face, every scene. Simple invariant, large perceptual payoff.

2. Caption rendering

Editors reported "random white boxes" floating over the video. The caption overlay tried to reproduce the reference's on-screen text by drawing rectangles wherever the analyzer detected text regions. When the analyzer flagged an overlay it could not describe — a stylized graphic, a motion-blurred lower-third — the renderer still drew the pill background with no text inside. Blank boxes.

I rewrote caption rendering to draw real text on a pill background, driven by blueprint.captions.style: caps vs sentence case, accent color, light-vs-dark pill, vertical position. Undescribable overlays now draw nothing — not a placeholder rectangle. Emoji are stripped before render because our bundled font cannot glyph them reliably. And because Railway containers have no system fonts, I bundled DejaVuSans-Bold.ttf into the asset pipeline so caption typography is deterministic in production.

interface CaptionStyle {
  uppercase: boolean;
  accentColor: string;
  pillVariant: 'light' | 'dark';
  position: 'top' | 'center' | 'bottom';
}

function renderCaptionOverlay(
  scene: SceneBlueprint,
  style: CaptionStyle,
): OverlayCommand[] {
  const text = scene.caption?.text;
  if (!text || scene.caption?.undescribable) return [];

  const sanitized = stripEmoji(text);
  const display = style.uppercase ? sanitized.toUpperCase() : sanitized;

  return [{
    type: 'text-pill',
    text: display,
    fontPath: bundledFont('DejaVuSans-Bold.ttf'),
    accentColor: style.accentColor,
    pillVariant: style.pillVariant,
    position: style.position,
  }];
}

Captions went from broken rectangles to readable, styled text that matched the reference's visual language.

3. Audio bed and background sound

Output audio was clean VO only — technically pristine, perceptually sterile. The reference viral had ambient room tone, subtle music bed, the sense of a real environment. Our mix sounded like someone recorded voice-over in a vacuum.

I added an ambience bed generated via the ElevenLabs Sound Effects API, themed from the blueprint's mood and setting descriptors. The bed mixes under the VO with sidechain ducking: the voice-over keys a compressor on the bed track, so speech dips the ambience and gaps between lines let it swell back. Tunable constants evolved through editor feedback — DEFAULT_BED_VOLUME went from 0.18 to 0.30 to 0.40, duck ratio softened from 8:1 to 4:1 to 3:1, threshold adjusted so ducking felt natural rather than pumping.

interface AudioBedProvider {
  generateBed(prompt: string, durationSec: number): Promise<Buffer>;
}

const DEFAULT_BED_VOLUME = 0.40;
const DUCK_RATIO = 3;
const DUCK_THRESHOLD_DB = -24;

async function mixVoWithBed(
  voTrack: AudioBuffer,
  bedProvider: AudioBedProvider,
  blueprint: VideoBlueprint,
  bedVolume = DEFAULT_BED_VOLUME,
): Promise<Buffer> {
  const bed = await bedProvider.generateBed(
    buildBedPrompt(blueprint.style, blueprint.setting),
    voTrack.durationSec,
  );

  return ffmpegMix([
    { input: voTrack, filter: 'anull' },
    {
      input: bed,
      filter: `volume=${bedVolume},acompressor=threshold=${DUCK_THRESHOLD_DB}dB:ratio=${DUCK_RATIO}:sidechain=0`,
      sidechainFrom: voTrack,
    },
  ]);
}

The AudioBedProvider interface keeps the ElevenLabs implementation swappable. Editors can also tune audioBedVolume (0..1) per job through the edit API without redeploying.

4. Voice-over pacing

Early versions sped up the VO to fit the reference's scene timing. When synthesized speech ran longer than the reference clip, the pipeline applied atempo to compress it. The result was chipmunk narration — technically on-beat, obviously wrong.

The rule is now absolute: never speed the VO. Instead, the stitcher sizes the timeline to max(reference total duration, VO length) and stretches the picture to fill. Audio is only ever padded — silence at the tail or scene boundaries — never time-compressed. I also reverted from a single continuous VO track back to per-scene audio segments. Continuous VO sounded like one flat overlay and killed the scene-to-scene rhythm the reference relied on.

function computeSceneDuration(
  referenceDurationSec: number,
  voDurationSec: number,
): number {
  // Picture stretches; audio never speeds up
  return Math.max(referenceDurationSec, voDurationSec);
}

function buildSceneAudio(segment: VoSegment): AudioFilterGraph {
  const targetSec = computeSceneDuration(segment.refDurationSec, segment.voDurationSec);
  if (segment.voDurationSec >= targetSec) {
    return padSilence(segment.voBuffer, targetSec);
  }
  return padSilence(segment.voBuffer, targetSec); // always pad, never atempo
}

5. Emotion-driven audio

Flat delivery was another subtle tell. The reference shifted energy scene by scene — urgent hook, calm product explanation, excited CTA. Our analyzer already extracted per-scene emotion from the blueprint; we just were not using it downstream.

Each scene's emotion now drives ElevenLabs audio tags and voice settings. An "excited" scene gets higher stability variance and expressive tags; "calm" scenes get steadier settings. The mapping is declarative in the synthesis config rather than hard-coded per scene type.

const EMOTION_VOICE_SETTINGS: Record<SceneEmotion, VoiceSettings> = {
  excited: { stability: 0.35, similarityBoost: 0.75, style: 0.6 },
  calm:    { stability: 0.70, similarityBoost: 0.85, style: 0.2 },
  urgent:  { stability: 0.45, similarityBoost: 0.80, style: 0.5 },
};

function synthesizeSceneVo(
  scene: SceneBlueprint,
  voiceId: string,
): Promise<Buffer> {
  const settings = EMOTION_VOICE_SETTINGS[scene.emotion] ?? EMOTION_VOICE_SETTINGS.calm;
  return elevenLabsTts({
    text: scene.script,
    voiceId,
    voiceSettings: settings,
    audioTags: [`[${scene.emotion}]`],
  });
}

6. Transitions

Every scene joined with a hard cut. For references that used soft dissolves or morphs between beats — common in lifestyle and beauty virals — our output felt jarring. The blueprint already flagged joins as soft where the analyzer detected them; the stitcher ignored that flag and concatenated with concat.

Soft transitions now route through FFmpeg's xfade filter when the blueprint marks a join as soft. The feature is env-gated with SOFT_TRANSITIONS=1 — default off preserves the validated hard-concat path in production until a job explicitly opts in.

function buildTransitionFilter(
  leftClip: string,
  rightClip: string,
  join: SceneJoin,
): string {
  if (join.type === 'soft' && process.env.SOFT_TRANSITIONS === '1') {
    return `xfade=transition=fade:duration=${join.crossfadeSec}:offset=${join.offsetSec}`;
  }
  return 'concat';
}

7. Color grading

Scene-to-scene color drift was subtle but real — AI-generated clips from different prompts carried slightly different white balance and contrast. I added a light color-grade pass via FFmpeg's eq filter, targeting the tone described in blueprint.style (warm/cool, lifted shadows, muted saturation). It is not cinema-grade grading; it is enough to pull disparate clips toward a unified look so the final ad reads as one piece rather than a montage of unrelated generations.

8. Viral hook audio capture

The opening hook is where virals earn their name — a specific sound, a verbal tic, ambient texture that stops the scroll. We were regenerating everything from scratch and losing the reference's own hook audio entirely. The generated ad had new VO from frame zero; the reference had that opening gasp, music sting, or ambient clip.

The fix extracts and preserves the viral's hook audio segment from the reference file and splices it into the generated output for the hook window defined in the blueprint. New VO picks up after the preserved hook. The opening "feel" — the thing editors recognize instantly — survives the remix.

async function buildFinalAudio(
  referenceVideo: Buffer,
  generatedSegments: VoSegment[],
  blueprint: VideoBlueprint,
): Promise<Buffer> {
  const hookAudio = await extractAudioSlice(
    referenceVideo,
    blueprint.hook.startSec,
    blueprint.hook.endSec,
  );

  const bodyVo = stitchSegments(generatedSegments.filter(s => !s.isHook));
  return concatAudio([hookAudio, bodyVo]);
}

What changed in the edit review

After these changes shipped, the same editor who flagged avatar drift and blank boxes signed off without requesting a revision pass on feel. The fixes are not glamorous — font bundling, sidechain ratios, refusing to call atempo — but they address what viewers actually notice. Structural parity (correct scenes, correct script) is necessary. Viral feel parity (one face, real captions, ambient bed, natural pacing, preserved hook) is what makes someone believe the ad belongs in the same feed as the original.

"Doesn't feel the same" is a systems problem. Every perceptual mismatch — avatar drift, blank overlays, sterile mix, sped-up VO — is a bug as real as a crash.

If you are building AI video pipelines, measure success by editor reaction and scroll-stop rate, not just render success. The last 10% of feel is where the other 90% of the engineering lives.

Validate Before Building: A Technical Discovery Framework for Startup Clients

Humza Tareen — Tue, 25 Aug 2026 21:26:02 +0000

A startup client reached out with an eight-phase fixed-price MVP proposal — $23–27K to build an organizational intelligence platform. The founder had twenty-six years of enterprise experience across defense and telecom. They had already purchased a Dell R740 server with 512GB RAM and planned on-prem infrastructure before writing a line of application code.

They wanted me to be the implementer. I refused to start designing. That refusal — and the discovery framework behind it — changed the engagement from a build contract into an advisory relationship where the founder deprioritized the server, agreed to rethink the mission statement, and redirected budget toward validating whether anyone actually needed what they planned to build.

What they thought they wanted

The initial brief described an organizational intelligence platform: ingest documents from enterprise clients, apply AI analysis, surface organizational insights, and provide a versioning layer for collaborative intelligence work. The architecture vision included:

On-prem Dell R740 as primary compute (already purchased)
512GB RAM for in-memory document processing and model inference
Multi-tenant isolation for defense and telecom clients
Compliance-first design — no data leaves the client's control perimeter
Eight-phase delivery over approximately six months

The fixed-price SOW broke the work into phases: infrastructure setup, document ingestion pipeline, embedding and search, AI analysis layer, versioning system, access control, admin portal, and deployment hardening. Each phase had deliverables, acceptance criteria, and a price tag.

On paper, it was a well-structured enterprise project plan. The kind of document this founder had written dozens of times in a career spent inside large organizations. That was precisely the problem.

What they actually needed

During the discovery call, I mapped their stated requirements against the underlying problem. Three reframes emerged:

Stated requirement	Underlying need	Validation status
On-prem Dell R740 infrastructure	Data sovereignty for compliance-sensitive clients	Unvalidated — no clients signed
Organizational intelligence platform	Document versioning with AI-assisted analysis	USP unclear — many existing tools
Multi-tenant defense/telecom deployment	Revenue from enterprise contracts	No LOIs, no pilot agreements
512GB in-memory processing	Fast document analysis at scale	Premature — no volume data

The core insight from the discovery call: "You essentially want a document versioning system." The AI analysis layer, the organizational intelligence framing, and the on-prem data center were layers of complexity wrapped around a problem that existing tools — Git, SharePoint with versioning, Notion, Confluence — already solve in various forms. The unique value proposition hadn't been articulated, let alone validated with potential customers.

Five assumptions I challenged

1. Why on-prem?

The server was already bought. Sunk cost is a powerful psychological anchor — the founder wanted to justify the purchase by building on it. But the stated reason for on-prem was compliance: enterprise clients in defense and telecom won't send data to an external startup's cloud.

That's a real constraint — for signed enterprise clients with existing security review processes. This startup had zero clients. Building on-prem infrastructure before the first pilot means spending six weeks on rack mounting, network configuration, and backup policies instead of six weeks talking to potential customers.

Compliance requirements should be validated with a specific client's security team, not assumed from industry generalizations.

2. Compliance as a launch blocker

Defense and telecom enterprises do have strict data handling requirements. But early-stage startups don't sell to them on day one. The typical path: build an MVP, get five to ten non-defense customers, prove the product works, then pursue FedRAMP or equivalent certifications when a specific deal requires it.

Designing for IL4 compliance before validating product-market fit is building a bridge to an island you haven't confirmed exists.

3. The hardware use case

I asked directly: "What's the maximum realistic use case for this server in the next six months?" After some discussion, the answer was internal testing with dummy data. A $15,000+ server for internal testing with dummy data.

A $50/month cloud VM handles internal testing. The R740 becomes relevant when you have a signed client who requires on-prem deployment — at which point the client's contract pays for the infrastructure, not the founder's savings account.

4. Refusing to design before validation

The founder expected me to start Phase 1 — infrastructure setup — within two weeks. I said explicitly: "I'm not going to start designing until we validate the problem statement."

This is uncomfortable. The client is paying (or about to pay) and wants to see progress. Wireframes and architecture diagrams feel like progress. But designing a system for an unvalidated problem produces exactly one outcome: a well-engineered product nobody wants, delivered on schedule and on budget.

5. Counter-proposal: validate USP and problem-market fit first

Instead of the eight-phase build, I proposed a four-week discovery engagement:

Week 1: Problem statement workshop — articulate the USP in one sentence a target customer would repeat back
Week 2: Competitive landscape — map existing tools, identify genuine gaps, not assumed gaps
Week 3: Customer discovery — five to ten interviews with potential users in target industries
Week 4: Go/no-go recommendation with a revised technical approach if go

The discovery framework

This engagement crystallized a reusable framework I now apply to every early-stage consulting lead:

Step	Question	Output
1. Understand what they think they want	What did they put in the RFP/SOW/brief?	Requirements inventory (their words)
2. Map to what they actually need	What's the simplest product that solves the core problem?	Problem statement (one sentence)
3. Identify riskiest assumptions	What must be true for this to succeed?	Ranked assumption list with validation methods
4. Refuse to build until validated	Which assumptions are unvalidated blockers?	Go/no-go gate with explicit criteria

The framework is deliberately sequential. Skipping step 2 and jumping to architecture produces over-engineered systems. Skipping step 3 produces confident builds on hidden assumptions. Step 4 is the one most consultants avoid because it risks losing the engagement.

Retainer over fixed-price for early-stage work

The original proposal was fixed-price: $23–27K for eight phases with defined deliverables. Fixed-price works when requirements are stable and validated. For early-stage startups, requirements are hypotheses.

I counter-proposed a monthly retainer for the discovery phase, with a clear scope (four weeks, four deliverables) and an explicit exit: at the end of discovery, either we proceed to a build phase with validated requirements, or we part ways with the founder having spent four weeks and a fraction of the budget learning their market doesn't need what they planned.

Model	When it works	When it fails
Fixed-price MVP	Validated problem, known scope, repeat builds	Unvalidated USP, moving requirements, first product
Time-and-materials retainer	Discovery, advisory, evolving scope	No scope boundaries (scope creep without accountability)
Fixed discovery + optional build	Early-stage with go/no-go gate	Founder wants to skip discovery (red flag)

The retainer model also repositioned the relationship. Fixed-price makes you a vendor delivering to spec. Retainer makes you an advisor whose incentive is to get the problem right, not to maximize billable build hours.

Why saying no builds more trust than compliance

The instinct for a consultant taking a new engagement is to agree, start designing, and deliver something tangible quickly. The client feels momentum. You feel employed. Everyone wins — until month four when the product launches to silence.

Saying "I'm not going to start designing" in the first call is a trust signal. It communicates:

I've seen this pattern before (enterprise veteran builds startup, over-engineers before validating)
I'm not going to take your money to build the wrong thing
My incentive is aligned with your success, not my billable hours

The founder's initial reaction was surprise — they had spoken to three other consultants who were ready to start Phase 1 immediately. Within a week, they came back and said the discovery counter-proposal was the first response that felt like someone understood what they were actually trying to do, not just what they wrote in the SOW.

The enterprise-veteran startup pattern

This is a pattern I see repeatedly: experienced enterprise operators starting their first startup bring process discipline that's genuinely valuable — structured requirements, phased delivery, compliance awareness — but also habits that are actively harmful at pre-seed stage:

Infrastructure before customers — buying servers, planning data centers, designing multi-tenant isolation before a single user exists
Compliance as identity — "we're the secure option" without a specific client's security requirements driving the design
Phase-gate thinking — eight phases with acceptance criteria works for a DoD contract; it kills iteration speed for a product searching for fit
Feature completeness over learning — the SOW described a finished platform, not a hypothesis to test

The founder's twenty-six years of enterprise experience was an asset — they understood document versioning, access control, and audit trails at a depth most startup founders don't. The challenge was channeling that expertise toward a minimum validated product instead of a minimum viable datacenter.

Outcome

By the end of the discovery call and a follow-up session:

The founder deprioritized the Dell R740 — acknowledged it was sunk cost, not a launch requirement
Agreed to rewrite the mission statement from "organizational intelligence platform" to a specific, testable USP
Shifted budget from the eight-phase build to a four-week customer discovery sprint
Repositioned my role from implementer to trusted advisor with a retainer structure

No code was written. No architecture diagrams were produced. No phases were kicked off. That was the correct deliverable for where the project actually was.

When to use this framework

Apply validate-before-building when:

The client has infrastructure commitments before customer commitments
The SOW describes a platform, not a feature
Compliance requirements are assumed, not client-specific
No one can articulate the USP in one sentence a customer would agree with
Other consultants are ready to start building immediately (that's a red flag, not a competitive threat)

Don't apply it when the client has validated demand (LOIs, paying pilots, repeat customers), stable requirements, and a clear build-versus-buy decision already made. In that case, fixed-price implementation is appropriate and the discovery framework would slow them down.

Discovery deliverables that replace architecture diagrams

When you refuse to design, you still owe the client tangible output. These deliverables from the four-week discovery sprint replace wireframes and ER diagrams:

One-sentence USP — tested against five customer interviews; revised until at least three interviewees repeat it back unprompted
Assumption register — ranked list of beliefs that must be true, each tagged validated/unvalidated/killed
Competitive gap analysis — not a feature matrix, but an honest answer to "why wouldn't they use SharePoint/Notion/Git?"
Go/no-go recommendation — explicit criteria, explicit answer, explicit next steps if go

These artifacts take less time than Phase 1 infrastructure setup and produce more useful information. The founder who deprioritized the server didn't need a rack diagram — they needed five conversations that confirmed or denied whether defense contractors would pay for document versioning from a startup.

The cost of building the wrong thing well

A $27K fixed-price MVP delivered on schedule to unvalidated requirements isn't a success — it's a well-engineered product with no market, plus the opportunity cost of six months the founder could have spent learning. The four-week discovery engagement costs a fraction of that and produces either a validated path forward or an early exit that saves the remaining budget.

The hardest part of consulting isn't the engineering. It's telling a client with twenty-six years of experience and a purchased server that the most valuable thing you can do for them this month is not write code. That's the job.

A UX Polish Sprint: 8 PRs in 48 Hours

Humza Tareen — Tue, 25 Aug 2026 21:25:58 +0000

Over a weekend I shipped eight focused pull requests to polish an admin dashboard for a task seeding platform. None of them fixed a broken feature. Every one removed friction that had accumulated while the product grew: crowded toolbars, hardcoded filter lists, search that fired on every keystroke, internal jargon on buttons, missing navigation cues. Core functionality was already solid. The dashboard was just harder to use than it needed to be.

This post walks through each PR—what changed, why it mattered, and the patterns worth reusing. The through-line is simple: polish is not a big redesign. It is a series of small, intentional improvements that respect the user's time and compound into a dramatically better experience.

The context

The admin panel had grown organically over several weeks. Operators could list tasks, filter by status, drill into batches, and run bulk actions. But the UI showed its age. Five separate filter dropdowns sat inline on the toolbar next to sort controls and a search box. Filter options were hardcoded in component files, so adding a new status meant touching multiple places and hoping string literals stayed in sync. One action button still used an internal slot code instead of the human label the rest of the app used. Task tables showed a redundant arrow column even though the whole row was clickable. Search re-filtered the list on every character, including single-letter queries that matched half the database.

None of these were bugs. All of them made daily work slower and more error-prone. A 48-hour polish sprint was the right response: ship small PRs, review quickly, and let the improvements stack.

PR 1 — Filter registry (208 lines)

The status filter chips were driven by a hardcoded statusChipOptions array. Labels, colors, icons, and matching logic lived in different files. Adding "Completed" or a virtual "Duplicate" filter meant hunting magic strings across the UI and the filter reducer.

I replaced that with a single ordered registry: LISTING_FILTER_REGISTRY. Each entry is a ListingFilterDef—key, label, icon, color, and a matches() predicate. The chip row and the filter logic both derive from the same array. Adding a filter is one registry entry; removing one is deleting a line.

interface ListingFilterDef {
  key: string;
  label: string;
  icon: React.ComponentType<{ className?: string }>;
  color: string;
  matches: (task: TaskRow) => boolean;
}

const DEDUP_MARKER = "duplicate_detected";

export const LISTING_FILTER_REGISTRY: ListingFilterDef[] = [
  {
    key: "all",
    label: "All",
    icon: ListIcon,
    color: "muted",
    matches: () => true,
  },
  {
    key: "completed",
    label: "Completed",
    icon: CheckCircleIcon,
    color: "success",
    matches: (task) => task.status === "completed",
  },
  {
    key: "duplicate",
    label: "Duplicate",
    icon: CopyIcon,
    color: "warning",
    matches: (task) =>
      task.errorReason?.includes(DEDUP_MARKER) ?? false,
  },
  // ...other status filters
];

"Duplicate" is a virtual filter: it does not map to a database status. It matches tasks whose errorReason contains a deduplication marker—useful when operators need to triage collisions without learning internal status enums. "Completed" is a straight status match. Both shipped because the registry made them trivial.

PR 2 — Search minimum length (60 lines)

Free-text search ran on every keystroke. Typing a re-filtered thousands of rows and made the list feel broken. Users had no feedback about why results looked random.

I introduced MIN_SEARCH_QUERY_LENGTH = 3. Queries shorter than three characters are treated as empty—no filter applied. An inline hint appears below the input when the field is non-empty but too short.

const MIN_SEARCH_QUERY_LENGTH = 3;

function effectiveSearchQuery(raw: string): string {
  const trimmed = raw.trim();
  return trimmed.length >= MIN_SEARCH_QUERY_LENGTH ? trimmed : "";
}

// In the listing toolbar component:
{searchQuery.length > 0 && searchQuery.length < MIN_SEARCH_QUERY_LENGTH && (
  <p className="text-sm text-muted-foreground" role="status">
    Enter at least {MIN_SEARCH_QUERY_LENGTH} characters to search.
  </p>
)}

The list stops thrashing on single-character input, and users get a clear explanation instead of silence.

PR 3 — Consolidated filters panel (291 lines)

The toolbar had grown to eight dense inline controls: step, version, progress, updated, task state, plus sort field, sort direction, and clear. On a laptop screen it wrapped awkwardly and competed with the search box for attention.

Five categorical filters moved into one Filters dropdown. Inside, a DropdownMenuRadioGroup gives native single-select semantics—only one secondary filter active at a time, which matches how operators actually think ("show me tasks stuck on this step"). The trigger shows an active-count badge. Sort controls stayed as separate compact dropdowns because they are used constantly and deserve one click.

<DropdownMenu>
  <DropdownMenuTrigger asChild>
    <Button variant="outline" size="sm">
      Filters
      {activeFilterCount > 0 && (
        <Badge variant="secondary" className="ml-2">
          {activeFilterCount}
        </Badge>
      )}
    </Button>
  </DropdownMenuTrigger>
  <DropdownMenuContent align="start" className="w-56">
    <DropdownMenuLabel>Filter by</DropdownMenuLabel>
    <DropdownMenuRadioGroup
      value={secondaryFilter ?? "none"}
      onValueChange={(v) => setSecondaryFilter(v === "none" ? null : v)}
    >
      {SECONDARY_FILTER_OPTIONS.map((opt) => (
        <DropdownMenuRadioItem key={opt.value} value={opt.value}>
          {opt.label}
        </DropdownMenuRadioItem>
      ))}
    </DropdownMenuRadioGroup>
  </DropdownMenuContent>
</DropdownMenu>

Before: [Step ▼] [Version ▼] [Progress ▼] [Updated ▼] [Task state ▼] [Sort: Updated ▼] [Newest ▼] [Clear]

After: [Filters (2)] [Sort: Updated ▼] [Newest first ▼] [Clear]

Same capability, half the visual noise. Operators who live in this screen immediately noticed the calmer header.

PR 4 — Conditional task state filter

The task state filter (paused / cancelled / active) always rendered three options even when every visible task was active. Empty filters teach users that the product is unfinished.

The dropdown now mounts only when at least one loaded task has a paused or cancelled overlay state. When data refreshes and no overlay tasks remain, the filter resets to "all" and the control hides itself. No dead options, no stale filter state pointing at zero rows.

PR 5 — Back button for batch detail (10 lines)

Users could open a bulk job detail view from the jobs list but had no in-app way back except the browser chrome. Ten lines: a back button in the detail header that navigates to the list route. Obvious in hindsight; absent for weeks because nobody filed it as a bug.

PR 6 — Label cleanup (14 lines)

One row action still said "Open A1"—internal jargon for the author assignment slot. The rest of the admin panel already said "Trainer" via shared ASSIGNMENT_ROLE_LABELS. This button was the last holdout. Renamed to "Assign Trainer" with a matching aria-label. Fourteen lines, zero functional change, meaningful clarity for new operators.

PR 7 — Remove arrow column (1 line)

Task tables included a trailing arrow icon implying "click to open." The entire row was already clickable. The column added visual clutter without affordance the row did not already provide. One line deleted from the column definition.

PR 8 — Stale cache on detail route

The batch detail view could show pipeline state from a server-side cache that lagged behind the live job. That fix is documented in a separate post; it belongs in the same sprint because stale detail pages feel like broken navigation even when the list view is correct. Removing the cache from the detail route aligned what operators see with what the job runner actually did.

The compounding effect

No single PR in this sprint is impressive on a slide deck. One line here, sixty there, two hundred in the registry. Together they change how the dashboard feels:

The toolbar is scannable instead of crowded.
Filters are extensible without archaeology through components.
Search explains itself instead of failing silently.
Navigation has explicit back affordances.
Labels match the language the rest of the product uses.
Dead UI—empty filters, redundant arrows— is gone.

The lesson for me was pacing: polish sprints work when each PR is reviewable in minutes and shippable without a feature flag. Big redesigns get deferred; small improvements ship.

The pattern — registry-driven UI

The filter registry is the most reusable artifact from the sprint. Instead of scattering option arrays and switch statements across components, declare each option as data with rendering metadata and a matching function. The UI maps over the registry; tests can assert matches() in isolation.

When product asks for another chip—"Blocked on review," "Stuck in export," a virtual bucket keyed off metadata—you add one object to LISTING_FILTER_REGISTRY. No new prop drilling, no duplicated color maps, no risk that the chip label and the filter predicate disagree.

That pattern generalizes beyond status chips: assignment roles, bulk action types, export formats—anywhere the admin panel presents a fixed set of choices with per-option behavior. Registries trade a few dozen lines upfront for predictable extension later.

What I would do again

I would schedule polish sprints earlier, before friction becomes "how the product works." I would default new listing filters to registry entries on day one rather than hardcoded arrays. I would treat toolbar density as a metric: if inline controls exceed four, consolidate.

Users rarely ask for a redesign. They ask why search feels broken, why they need the browser back button, why a button says "A1." Eight small PRs answered those questions. That is UX engineering: respect people's time, ship the obvious fix, and let the compound effect do the rest.

Universal Vendor Polling: One Heartbeat Template for Every AI Model Provider

Humza Tareen — Tue, 25 Aug 2026 21:25:22 +0000

The AI video generation platform I work on does not bet on a single model vendor. Clip generation might route through Replicate one week and a direct Kling integration the next. Voiceover runs through ElevenLabs. Image conditioning hits fal. Planning and QA call Claude. Each vendor exposes a different async contract — webhooks, REST polling, SSE streams, queue IDs with opaque status enums.

When we had three vendors, each adapter shipped its own poll loop. Copy-paste with minor tweaks. When we hit five, the duplication became a reliability hazard: inconsistent timeout handling, no shared failure classification, and — most painfully — a watchdog process that killed jobs the poll loop claimed were hung but were actually still rendering on Replicate's GPU cluster.

I refactored all vendor polling into a single universal template with automatic heartbeats, configurable timeouts, exponential backoff, and structured failure classification. One abstraction, every model.

Five vendors, five polling dialects

Before the refactor, each vendor adapter owned its own waiting logic. The surface area looked like this:

Vendor	Async model	Status API	Typical render time	Pre-refactor pain
Replicate	Prediction ID + poll	GET /predictions/:id	90s–8min	No heartbeat → watchdog kill
Kling (direct)	Task ID + poll	POST status endpoint	2–6min	Hard-coded 5min timeout too short
fal	Queue + webhook optional	GET queue status	30s–3min	Webhook fallback never tested
ElevenLabs	Synchronous-ish TTS	Streaming response	5–30s	Treated as sync, no poll wrapper
Claude	SSE stream	Stream events	10–60s	Different error shape entirely

Each row was a separate while loop with different sleep intervals, different terminal states, and different ideas of what "failed" meant. Replicate returns status: "failed" with an error string. Kling returns numeric error codes. fal returns HTTP 503 on queue saturation that should retry, not abort.

The watchdog kill incident

The production incident that forced this refactor: a background watchdog monitors long-running generation jobs. If a job doesn't emit a heartbeat within 120 seconds, the watchdog assumes the worker crashed and marks the job failed, releasing resources and notifying the portal.

Replicate clip renders routinely take three to eight minutes. The Replicate adapter's poll loop looked like this:

// BEFORE — no heartbeat, fixed 3s sleep, silent hang
async function pollReplicate(predictionId: string): Promise<ReplicateOutput> {
  while (true) {
    const prediction = await replicate.predictions.get(predictionId);
    if (prediction.status === 'succeeded') return prediction.output;
    if (prediction.status === 'failed') throw new Error(prediction.error);
    if (prediction.status === 'canceled') throw new CancelledError();
    await sleep(3000);
  }
}

The loop was alive — hitting Replicate's API every three seconds — but it never told the watchdog. After 120 seconds of silence, the watchdog killed a job whose clip was 60% rendered on Replicate's side. The operator saw "failed" in the portal. Replicate still charged for the completed render we never retrieved.

The fix was not "increase the watchdog timeout globally." That would mask actual worker crashes on fast vendors. The fix was: every poll loop must emit a heartbeat on every iteration, automatically, via shared infrastructure.

The universal poll template

The refactor extracted a generic pollUntilTerminal function that wraps any vendor-specific status check:

type PollStatus = 'pending' | 'running' | 'succeeded' | 'failed' | 'cancelled';

interface PollConfig<T> {
  vendorName: string;
  jobId: string;
  checkStatus: () => Promise<{ status: PollStatus; result?: T; error?: VendorError }>;
  timeoutMs: number;
  initialIntervalMs: number;
  maxIntervalMs: number;
  backoffMultiplier: number;
  onHeartbeat: () => void;
  classifyError: (error: VendorError) => FailureClass;
}

type FailureClass = 'retryable' | 'fatal' | 'vendor_outage' | 'quota_exceeded';

async function pollUntilTerminal<T>(config: PollConfig<T>): Promise<T> {
  const deadline = Date.now() + config.timeoutMs;
  let interval = config.initialIntervalMs;

  while (Date.now() < deadline) {
    config.onHeartbeat(); // EVERY iteration, before the vendor call

    const { status, result, error } = await config.checkStatus();

    if (status === 'succeeded' && result !== undefined) return result;
    if (status === 'cancelled') throw new CancelledError(config.jobId);
    if (status === 'failed' && error) {
      throw classifyAsPollError(config.classifyError(error), error);
    }

    await sleep(interval);
    interval = Math.min(interval * config.backoffMultiplier, config.maxIntervalMs);
  }

  throw new PollTimeoutError(config.vendorName, config.jobId, config.timeoutMs);
}

Vendor adapters shrink to two functions: one that maps vendor-specific API responses to the normalized PollStatus, and one that classifies vendor errors into retryable vs fatal. All timing, backoff, heartbeat, and timeout logic lives in the template.

After: Replicate adapter

// AFTER — 12 lines, heartbeat automatic, timeout configurable per model
async function pollReplicate(
  predictionId: string,
  ctx: JobContext,
): Promise<ReplicateOutput> {
  return pollUntilTerminal({
    vendorName: 'replicate',
    jobId: predictionId,
    checkStatus: async () => {
      const p = await replicate.predictions.get(predictionId);
      return {
        status: mapReplicateStatus(p.status),
        result: p.status === 'succeeded' ? p.output : undefined,
        error: p.status === 'failed' ? { message: p.error, code: 'REPLICATE_FAIL' } : undefined,
      };
    },
    timeoutMs: ctx.modelConfig.maxWaitMs ?? 600_000, // 10min for video models
    initialIntervalMs: 2_000,
    maxIntervalMs: 10_000,
    backoffMultiplier: 1.5,
    onHeartbeat: () => ctx.heartbeat.emit('poll:replicate'),
    classifyError: classifyReplicateError,
  });
}

Every vendor adapter follows the same shape. Adding a sixth vendor means writing checkStatus and classifyError — not reinventing backoff math and heartbeat wiring.

Failure classification: retry vs abort vs escalate

Normalized polling is only half the value. The template also centralizes failure classification so upstream retry logic makes consistent decisions:

Failure class	Examples	Pipeline behavior
`retryable`	HTTP 503, queue full, transient GPU unavailable	Retry with backoff, same vendor
`vendor_outage`	5xx sustained, status endpoint down	Fail over to alternate vendor adapter
`quota_exceeded`	Rate limit, billing cap, concurrency limit	Queue for later, notify operator
`fatal`	Invalid input, content policy violation, corrupt output	Abort scene, surface to operator with context

Before classification was centralized, the Kling adapter retried content-policy failures three times (burning credits), while the Replicate adapter aborted immediately on the same class of error. Operators saw inconsistent behavior depending on which vendor routed the scene.

Exponential backoff without over-polling

Fixed-interval polling is either too aggressive (API rate limits on fast jobs) or too slow (added latency on jobs that finish between polls). The template uses exponential backoff with a cap:

Initial interval: 2 seconds — catch fast TTS and image jobs quickly
Backoff multiplier: 1.5× per iteration
Max interval: 10 seconds — never go silent long enough to worry operators
Heartbeat: every iteration regardless of interval — watchdog stays satisfied

For a six-minute video render, the poll cadence ramps from 2s → 3s → 4.5s → 6.75s → 10s (held). Total API calls: roughly 40, down from 120 at fixed 3-second intervals, with zero watchdog false positives.

Capping feel-regeneration runs

A related reliability fix shipped in the same arc: the "feel regeneration" path — which re-renders challenger clips to match reference pacing and energy — was regenerating every challenger on every quality pass with no cap.

A twelve-scene ad with three challengers per scene could trigger 36 extra video renders on a single feel-regen pass. Most of those rerenders targeted challengers that were already within quality threshold — the regen was running primary-only checks against all challengers indiscriminately.

interface FeelRegenPolicy {
  scope: 'primary-only';       // don't regen challengers unless primary fails
  maxRegensPerRun: number;       // hard cap, default 3
  qualityThreshold: number;      // skip regen if score >= threshold
}

async function runFeelRegen(
  scenes: SceneResult[],
  policy: FeelRegenPolicy,
): Promise<SceneResult[]> {
  let regenCount = 0;
  const updated: SceneResult[] = [];

  for (const scene of scenes) {
    const primary = scene.challengers.find(c => c.role === 'primary');
    if (!primary || primary.feelScore >= policy.qualityThreshold) {
      updated.push(scene);
      continue;
    }
    if (regenCount >= policy.maxRegensPerRun) {
      updated.push(scene); // cap hit — ship best available
      continue;
    }
    const regen = await regenPrimary(scene, primary);
    regenCount++;
    updated.push(applyRegen(scene, regen));
  }

  return updated;
}

Primary-only scope plus a per-run cap of three regenerations cut feel-regen GPU spend by roughly 70% without measurable quality regression — because most challenger rerenders were fixing problems the primary didn't have.

Before and after: what changed in production

Metric	Before universal template	After
Watchdog false kills (Replicate jobs)	~4 per day	0
Poll loop code (lines across vendors)	~340 duplicated	~80 template + ~25 per vendor adapter
Mean time to diagnose vendor timeout	45 min (which adapter?)	5 min (structured logs with vendor + jobId)
Feel-regen GPU calls per run	Up to 36	Max 3 (primary-only)

Design principles for multi-vendor async

If you're integrating multiple AI model providers into one pipeline, these rules saved us repeated incidents:

Normalize early. Map vendor status enums to your own terminal states at the adapter boundary, not in business logic.
Heartbeat is not optional. Any loop that can run longer than your watchdog interval must emit heartbeats — bake it into the template so adapter authors can't forget.
Timeout per model, not per vendor. Video generation gets ten minutes. TTS gets two. Image gen gets five. Store timeouts in model config, not hard-coded in adapters.
Classify failures once. Upstream retry and failover logic should consume FailureClass, not parse vendor error strings.
Cap expensive recovery paths. Regeneration, failover, and retry loops all need per-run budgets or costs compound silently.

The universal poll template isn't glamorous infrastructure. Nobody demos it to investors. But it's the difference between a pipeline that survives vendor outages gracefully and one that burns credits on phantom failures while the watchdog murders live jobs.

When Display Labels Break Sorting

Humza Tareen — Tue, 25 Aug 2026 21:25:17 +0000

On the submissions page in the admin panel, operators see a unified “Status” column. Instead of exposing raw pipeline stage codes, the UI shows human-readable labels—“Checking Quality,” “Quality Check Failed,” “Generating Output,” “Expert Review Active,” and so on—so reviewers can scan the table quickly. Sorting seemed like an obvious enhancement: click the header, group rows by that same notion of progress. What shipped first looked plausible: each cell showed the right label. The sort arrow toggled. And yet the rows did not stabilize into meaningful bands. Identical-looking statuses interleaved. Newest tasks did not float to the top within a band. Occasionally, refreshing the page reshuffled who appeared in the viewport at all. This post walks through three separate bugs that compounded into that behavior—and how seventy-eight new tests and a tight set of code changes unwound it.

The symptom: a column that “worked” but did not sort

The task system persists canonical stage and status fields (plus optional failure metadata). Presentation maps those primitives into unified labels shared between list views and detail screens. Separately, sort mode for the status column derives a numeric rank per row so the table can order consistently in memory without pushing presentation strings into SQL. On paper that split is sound: keep storage normalized, keep display friendly, keep sort keys comparable. In practice the rank function and the label function diverged. The label path consulted review outcomes and background job state; the rank path did not. Users experienced the worst possible failure mode: every affordance suggested the feature worked until someone actually tried to use sort to triage work.

Root cause one: rank ignored derived state

The original computeUnifiedRank accepted (stage, status, failureReason). That is sufficient only if each (stage, status) pair maps to exactly one user-visible status. It does not. The same database tuple can mean different things depending on whether an automated review passed, failed, or is still running, and on whether worker runs exist for output generation or expert-quality checks. For example, a pre-quality-assurance stage marked ACTIVE might display as “Checking Quality” while a review is in flight, “Quality Check Failed” after a failed check, or “Quality Check Passed” once the gate clears. All three rows shared one rank, so when sorted they collided and interleaved arbitrarily relative to one another—exactly the shuffle operators reported.

The fix was to thread the same derived inputs the label layer already used. computeUnifiedRank now takes reviewPassed (a boolean or tri-state derived from review records) and workerRuns (minimal info about active or completed background runs). With those in place we assigned fourteen distinct ranks, 0 through 13, so every pipeline status the UI can show maps to a unique ordering key. Pre-QA splits into three separate ranks instead of one. Output generation and expert-quality review each get dedicated ranks driven by whether relevant worker runs are present, so “waiting on automation” and “idle at the same stage” no longer compare equal.

// Before: same rank for every variant of a stage/status pair
function computeUnifiedRankBroken(
  stage: PipelineStage,
  status: TaskStatus,
  failureReason: string | null
): number {
  // ...
  // PRE_QA + ACTIVE always returned one rank, regardless of review outcome
}

// After: rank matches the same derived state as the display label
function computeUnifiedRank(
  stage: PipelineStage,
  status: TaskStatus,
  failureReason: string | null,
  reviewPassed: boolean | null,
  workerRuns: { hasOutputJob: boolean; hasExpertCheckJob: boolean }
): number {
  // ...
  // PRE_QA + ACTIVE branches on reviewPassed → three distinct ranks
  // Output generation vs idle uses workerRuns.hasOutputJob
}

Backward compatibility mattered: internal scripts and older API layers still call the rank helper without the new parameters. Optional arguments with conservative defaults preserve reasonable ordering for those callers while the submissions page passes the full derived context.

Root cause two: the tiebreaker was a CUID

After ranks separate coarse buckets, rows in the same bucket should sub-sort by recency—typically newest task first, matching how operators scan “what just moved.” The comparator’s secondary key was accidentally the string task primary key. Those identifiers are opaque CUIDs: lexicographic order has no relationship to creation time. Within a status group, the table looked random even when rank was finally unique. The fix keeps rank as the primary key, uses the task’s createdAt (or domain-equivalent timestamp) as the secondary key—newest first—and falls back to task id only as a stable tiebreaker when two rows share an identical timestamp.

function compareByUnifiedStatus(
  a: TaskSortRow,
  b: TaskSortRow,
  direction: "asc" | "desc"
): number {
  const ra = computeUnifiedRank(
    a.stage,
    a.status,
    a.failureReason,
    a.reviewPassed,
    a.workerRuns
  );
  const rb = computeUnifiedRank(
    b.stage,
    b.status,
    b.failureReason,
    b.reviewPassed,
    b.workerRuns
  );
  const rankCmp = direction === "asc" ? ra - rb : rb - ra;
  if (rankCmp !== 0) return rankCmp;

  const da = a.createdAt?.getTime() ?? 0;
  const db = b.createdAt?.getTime() ?? 0;
  const dateCmp = direction === "asc" ? da - db : db - da;
  if (dateCmp !== 0) return dateCmp;

  return a.id.localeCompare(b.id);
}

Null-dated rows need an explicit policy: we treat missing timestamps as 0 so they sort predictably to one end of the list and tests lock that behavior in.

Root cause three: non-deterministic windowing

Unified status sort is applied in memory with a hard cap—five thousand rows fetched from the database before ranking. That keeps admin queries bounded. The subtle bug was the initial window query: without an orderBy, the database is free to return any five thousand rows when the table is larger than the cap. Different plans, cache pressure, or concurrent writes could change which slice you see, so two refreshes of the same filter produced different apparent orderings at the tail. Adding a deterministic orderBy to the window query—aligned with the product’s default list ordering—makes the cap honest: you always process the same prefix of rows for a given filter, then sort that fixed set by unified status.

Feeding the rank: a targeted Prisma select

Correct ranks require data that was not on the original DTO. We expanded the Prisma select for the admin list load to pull only what rank computation needs: a narrow shape on related reviews (enough to derive reviewPassed) and a lightweight list of workerRuns flags or counts—not full relation graphs, not heavy JSON blobs. That keeps the query cheap while making the sort key honest. The rule is simple: if the label can branch on a field, the rank function must see it or intentionally document a different ordering contract.

const rows = await prisma.task.findMany({
  where: listWhere,
  take: 5000,
  orderBy: { createdAt: "desc" },
  select: {
    id: true,
    stage: true,
    status: true,
    failureReason: true,
    createdAt: true,
    reviews: {
      select: { passed: true, completedAt: true },
      orderBy: { completedAt: "desc" },
      take: 1,
    },
    workerRuns: {
      select: { kind: true, state: true },
      where: { state: { in: ["PENDING", "RUNNING", "SUCCEEDED"] } },
    },
  },
});

Application code maps reviews and workerRuns into the compact reviewPassed and workerRuns struct the rank helper expects, mirroring the label pipeline so the two cannot drift silently.

Tests: seventy-eight new guards

Regressions in sort logic are easy to reintroduce because they look like small refactors. The test suite now covers: rank splits for all fourteen unified statuses; backward compatibility for callers that omit new parameters; an invariant that no two distinct visible statuses share a rank; ascending and descending direction; date sub-sort within the same rank; and null handling for missing dates and missing reviews. One representative case encodes the pre-QA split explicitly—two tasks with identical stage and status but opposite review outcomes must produce different ranks and order predictably under the date tiebreaker. Property-style checks iterate every enumerated pair of unified labels to prove the rank injective for display purposes—if two ranks match, they must correspond to the same user-visible bucket. Separate cases flip sort direction end to end so we never regress to “works only descending.” Snapshot tests stayed deliberately out of scope: numeric ranks and comparator outputs are asserted directly so refactors rename strings without rewriting brittle golden files.

it("splits PRE_QA ACTIVE by review outcome", () => {
  const base = {
    stage: "PRE_QA" as const,
    status: "ACTIVE" as const,
    failureReason: null,
    createdAt: new Date("2026-05-01T12:00:00Z"),
    workerRuns: { hasOutputJob: false, hasExpertCheckJob: false },
  };

  const inProgress = {
    ...base,
    id: "task_in_progress",
    reviewPassed: null,
  };
  const failed = {
    ...base,
    id: "task_failed",
    reviewPassed: false,
  };

  expect(computeUnifiedRankForRow(inProgress)).not.toBe(
    computeUnifiedRankForRow(failed)
  );
});

After deploy, we asked operators to spot-check a few saved filters; no schema migration was required because the fix is read-path only. Going forward, any pull request that changes how a unified label reads should show the matching diff to computeUnifiedRank or an explicit note that sort semantics are intentionally unchanged—that cheap review habit keeps presentation and ordering from drifting apart again.

The lesson

Display labels and sort order are separate concerns, but they must consume the same facts whenever a label is computed from derived state. If the UI shows “Quality Check Failed” because a review row says so, the sort key must consult that same review signal. When it does not, you get the most insidious class of bug: the feature appears implemented—headers click, arrows flip, cells read well—while the ordering is meaningless. Investing in deterministic data windows, intentional tiebreakers, and exhaustive tests turns that ambiguity into ordinary, reviewable correctness work.

When Two Cache Layers Serve Stale Data

Humza Tareen — Tue, 25 Aug 2026 21:24:42 +0000

Users on the dashboard reported a confusing pattern: when they clicked a completed task in the listing, the detail page briefly showed an intermediate In Progress state—step 1 or step 2 still running—before snapping to the correct completed view. The listing page always showed the right status. Only the detail page lied, and only for a moment. That asymmetry made the bug feel like a rendering glitch until we traced two independent cache layers, each serving stale data for different reasons. Fixing one was not enough; both had to change.

This post walks through that investigation: the architecture, two root causes, the fixes, and why sometimes the right answer is to remove a cache instead of perfecting its invalidation.

The bug report

The symptom was consistent and reproducible under the right timing. Open a task while it is still executing, leave, wait for the worker pipeline to finish, then open the same task from the listing where it now appears completed. For a second or two the detail page showed an old pipeline step—In Progress—then updated. No hard refresh was required; the UI eventually corrected itself. That “eventually” pointed at caching or polling, not at permanently wrong data in object storage.

Because the listing was correct, engineers first suspected the detail page’s client state or a race in the UI. Network tab inspection showed something subtler: the first response body for the detail API sometimes described an earlier step, even though manifests in storage had already moved on. Reproducing the issue required deliberate timing—visit during execution, navigate away, wait for completion, return from the listing—but once you had that sequence, the wrong first paint was reliable enough to rule out a one-off race.

Two data paths on the detail page

The detail page does not read storage directly in the browser. It has two stacked paths:

Server: An API route loads task details from object storage via a TaskDetailService and returns JSON. That handler was wrapped in Next.js unstable_cache with a 15-second TTL and tag-based revalidation.
Client: A React Query hook calls that API route on an interval—every 10 seconds—with background refetching while the tab is focused.

So a single user session could hit a cached server response and reuse a cached client query result. The listing page followed a different path: it read manifests fresh from storage on each revalidation, without the same server cache in front of the detail route. Same task, two pages, two truths—that is what made diagnosis slow.

  Browser (detail page)
       |
       |  React Query (client cache, default gcTime 5 min)
       |  poll every 10s
       v
  GET /api/tasks/[id]  ---->  unstable_cache (server, TTL 15s, tags)
       |                           |
       |                           v
       +------------------>  TaskDetailService  ---->  object storage

  Listing page (separate path)
       |
       v
  manifests read fresh from storage (no detail-route server cache)

Root cause 1: server cache never invalidated

The API route used unstable_cache with a revalidate tag so that, in theory, writers could bust the cache when task state changed. In practice, the worker pipeline that advances steps never called revalidateTag. Workers update manifests directly in object storage. They have no knowledge of a Next.js cache sitting in front of the detail route.

What happened in production looked like this: a user opened the detail page while the task was on step 1. The server cached that JSON for up to 15 seconds. The worker finished step 2 and step 3 in storage, but nothing told the web layer to drop the entry. Another visit inside the TTL window got the cached “step 1 running” payload. React Query’s polling would eventually fetch fresher data—but the first paint could still be wrong if the server cache responded first.

That is the classic “cache invalidation only works if every writer knows about the cache” problem. Your mental model might assume tags connect storage writes to HTTP responses automatically. In a split architecture—workers in one runtime, Next.js in another—the tag is invisible unless you build an explicit bridge. We had defined the bridge on paper (tags on the cache entry) but never wired the pipeline to call it.

The tempting fix was to sprinkle revalidateTag calls through the worker pipeline whenever a step completes. That couples pipeline code to a specific web framework’s cache API—tight coupling across system boundaries, and easy to forget on the next new writer.

The better fix for this route: remove the server-side cache entirely. Client polling already bounds how often the browser hits the API. A 15-second server cache bought little latency and added a stale-serving failure mode whenever invalidation was incomplete.

Before: cached API handler

import { unstable_cache } from "next/cache";

const TASK_DETAIL_TAG = "task-detail";

export async function GET(
  _req: Request,
  { params }: { params: { id: string } }
) {
  const getCachedDetail = unstable_cache(
    async () => TaskDetailService.getById(params.id),
    [`task-detail-${params.id}`],
    { revalidate: 15, tags: [TASK_DETAIL_TAG, `task-${params.id}`] }
  );

  const detail = await getCachedDetail();
  return Response.json(detail);
}

After: direct service call

export async function GET(
  _req: Request,
  { params }: { params: { id: string } }
) {
  const detail = await TaskDetailService.getById(params.id);
  return Response.json(detail);
}

After deploying that change, server responses tracked storage on every request. The flash did not disappear.

Root cause 2: client gcTime replayed old data

Even with a fresh server, users still saw a brief stale state. The second layer was React Query’s default gcTime (formerly cacheTime): five minutes. While a user stayed on the detail page during execution, React Query stored the “step 1 in progress” result. When they navigated away and returned after completion, React Query immediately showed that cached entry—instant wrong UI—while a background refetch ran. Fresh data replaced it a moment later. Correct outcome, jarring path.

For a detail view where correctness on entry matters more than instant back-navigation, we set gcTime: 0. Leaving the page discards cached data immediately. The next visit shows a loading state, then the current truth. No stale flash. The tradeoff is intentional: users who bounce between list and detail more often pay an extra spinner, but they never see a completed task masquerading as mid-pipeline work.

Note that staleTime and gcTime solve different problems. staleTime controls how long data is considered fresh before a refetch; gcTime controls how long inactive query data stays in memory after the last subscriber unmounts. Our bug was almost entirely the latter—old data resurrected on remount, not a refusal to refetch while the page was open.

Before: default garbage collection

export function useTaskDetail(taskId: string) {
  return useQuery({
    queryKey: ["task-detail", taskId],
    queryFn: () => fetchTaskDetail(taskId),
    refetchInterval: 10_000,
    refetchIntervalInBackground: true,
    // gcTime defaults to 5 minutes — stale snapshot survives navigation
  });
}

After: discard cache on unmount

export function useTaskDetail(taskId: string) {
  return useQuery({
    queryKey: ["task-detail", taskId],
    queryFn: () => fetchTaskDetail(taskId),
    refetchInterval: 10_000,
    refetchIntervalInBackground: true,
    gcTime: 0,
  });
}

Why both fixes were required

These layers are independent. Removing only the server cache still left React Query serving a five-minute-old snapshot on remount. Setting only gcTime: 0 still allowed unstable_cache to return 15-second-old JSON from the API. Each layer could be “correct” in isolation while the combined system was wrong.

Cache invalidation is only as reliable as the least-invalidated layer. Here, the worker never participated in server invalidation, and the client deliberately kept data warm for UX on other pages—defaults that made sense globally but hurt this screen.

Layer	Symptom if unfixed	Fix
Server `unstable_cache`	Stale JSON within TTL; listing vs detail mismatch	Remove cache on detail route
React Query client cache	Instant replay of old step on return visit	`gcTime: 0` on detail query

Listing vs detail asymmetry

The listing page worked because it never depended on the cached detail handler. It aggregated status from manifests refreshed on a separate revalidation path. So operators trusted the grid and distrusted the detail page—a classic split-brain symptom when two features read the same domain through different pipelines.

When debugging “page A is right, page B is wrong,” map the full path for each screen before assuming a shared bug in storage or workers. Here, storage was fine; the caches in front of the detail route were not.

The lesson

Every cache you add is a potential source of stale data. Multiple layers multiply that risk: a bug or omission in either layer can serve old state even if the other is perfect. Tag-based revalidation only works when every writer that mutates underlying data knows about the tag—and workers that touch object storage often do not.

Sometimes the right fix is not better invalidation but fewer caches. Server caching on a route that is already polled every ten seconds was redundant. Client caching with a five-minute gcTime was wrong for a view where users care about the first paint after navigation. Together, those choices produced a bug that looked like a UI flicker but was really two systems each doing what they were told—just not coordinated with how tasks actually change in the platform.

If you are seeing a brief wrong state that self-corrects, check the stack twice: once at the edge of your framework on the server, and once in your data library on the client. Fix one, verify, then fix the other. Stale data rarely respects how neatly you drew the architecture diagram.

Trainer Gate QC: Integrating Quality Checks into a Multi-Stage Review Pipeline

Humza Tareen — Tue, 25 Aug 2026 21:24:37 +0000

AI training data pipelines do not end at model output. Human reviewers — trainers — validate, correct, and approve data before it enters the training set. The trainer gate is that checkpoint. Integrating automated QC into it means checks run at the right moment in the review lifecycle: not so early that reviewers cannot fix issues, not so late that bad data ships.

On the enterprise workflow platform, I shipped QC orchestration into the trainer review gate across a cluster of PRs — from the core integration (AGT-770) through spinner scoping, manifest transitions, claim lifecycle edge cases, and force-pass guardrails. This post covers what that integration looks like and why the edge cases matter more than the happy path.

Multi-stage pipeline UIs look simple in diagrams. They are hard because every stage has its own claim, timeout, and release semantics — and they interact.

The trainer gate in context

A typical task on the platform moves through stages:

Intake — task created, assigned to queue
Model output — AI generates initial training data
Trainer gate — human reviewer claims, edits, approves
QC orchestration — automated checks validate the reviewer's output
Release — approved data enters the training corpus

Stage	Actor	Manifest state	QC involvement
Model output finalize	System	`model_output → trainer_gate/pending`	None
Trainer claims task	Human reviewer	`trainer_gate/claimed`	None
Trainer submits review	Human reviewer	`trainer_gate/submitted`	Auto-QC triggered
QC passes	QC agent	`trainer_gate/qc_passed`	Checks complete
QC fails	QC agent	`trainer_gate/qc_failed`	Returned to reviewer
Release	System	`released`	Final validation

AGT-770: Integrating QC orchestration

The core feature wired the existing QC orchestration service into the trainer gate submission flow. Before AGT-770, QC ran as a separate post-release step — too late to catch issues reviewers could fix. After AGT-770, submitting a trainer review triggers QC automatically:

async function onTrainerSubmit(
  taskId: string,
  review: TrainerReview,
  manifest: TaskManifest,
): Promise<SubmitResult> {
  // Advance manifest to submitted
  await manifest.transition('trainer_gate', 'submitted', {
    reviewerId: review.reviewerId,
    submittedAt: Date.now(),
  });

  // Dispatch QC orchestration
  const qcJob = await qcOrchestrator.dispatch({
    taskId,
    stage: 'trainer_gate',
    payload: review.output,
    checks: manifest.qcConfig.trainerGateChecks,
  });

  return { status: 'qc_pending', qcJobId: qcJob.id };
}

QC orchestration runs configured checks — format validation, golden data comparison, LLM adjudication — asynchronously via Cloud Tasks workers. The trainer sees results in the Task Details UI when checks complete.

Manifest state transitions

AGT-865 ensured manifest advances to trainer_gate/pending when model output finalizes — the handoff point where tasks become claimable by trainers. Without this transition, tasks sat in limbo between model completion and trainer visibility.

async function onModelOutputFinalize(
  taskId: string,
  output: ModelOutput,
): Promise<void> {
  const manifest = await manifestStore.get(taskId);
  await manifest.transition('model_output', 'finalized', { outputHash: hash(output) });
  await manifest.transition('trainer_gate', 'pending', {
    claimableAt: Date.now(),
    autoSeedStatus: manifest.autoSeed?.status ?? null,
  });
  await eventBus.emit('task.trainer_gate.pending', { taskId });
}

Spinner scoping: progress only when QC runs

AGT-1082 fixed a UX bug where the auto-seed status spinner appeared for every manifest state change — not just when Auto-QC was actually in progress. Reviewers saw spinning indicators during unrelated transitions (claim, release, draft save) and could not tell whether QC was running.

function shouldShowQcSpinner(manifest: TaskManifest): boolean {
  const stage = manifest.currentStage;
  const status = manifest.currentStatus;
  // Only show spinner when Auto-QC is actively running
  return stage === 'trainer_gate'
    && status === 'submitted'
    && manifest.qcJob?.state === 'in_progress';
}

// Before AGT-1082: spinner on ANY trainer_gate status change
// After AGT-1082: spinner ONLY on submitted + qc in_progress

Spinner scoping sounds trivial. In a pipeline where status changes fire every few seconds (claim timeout checks, webhook callbacks, draft auto-save), unscoped spinners erode trust in the UI — reviewers stop believing any spinner means something is actually happening.

Claim lifecycle edge cases

The hardest part of multi-stage pipeline UIs is not the happy path. It is what happens when things go sideways:

Claim timeout and release

When a trainer claims a task but does not submit within the timeout window, the claim expires. AGT-1103 (in progress) addresses draft and manifest reset at release time — both manual release and auto-timeout:

async function onClaimRelease(
  taskId: string,
  reason: 'manual' | 'timeout',
  manifest: TaskManifest,
): Promise<void> {
  // Reset in-progress draft
  await draftStore.delete(taskId, manifest.claimedBy);

  // Reset manifest to pending (reclaimable)
  await manifest.transition('trainer_gate', 'pending', {
    releasedAt: Date.now(),
    releaseReason: reason,
    previousClaimer: manifest.claimedBy,
  });

  // Preserve submission history for audit
  await auditLog.record('claim.released', { taskId, reason });
}

Released tasks remain in My Tasks

AGT-1145 fixes a redirect bug: after releasing a task, it disappeared from the reviewer's "My Tasks" list even though the release was intentional (not a completion). Reviewers lost visibility into tasks they had started but chose to return to the queue.

Force-pass with incomplete golden data

AGT-898 blocks force-pass QC when the golden data form is incomplete. Force-pass is an admin escape hatch — bypass QC checks when they are known to be wrong for a specific task. Without the guard, admins could force-pass tasks missing required golden data fields, shipping incomplete training examples.

Edge case	Ticket	Behavior
Claim times out	AGT-1103	Reset draft + manifest to pending
Manual release	AGT-1103	Same reset, preserve audit trail
Post-release visibility	AGT-1145	Task stays in My Tasks after redirect
Force-pass without golden data	AGT-898	Block with validation error
QC spinner on wrong state	AGT-1082	Spinner only during active QC

Admin-only pipeline observability

AGT-886 restricted Task Details pipeline and events tabs to admin users. Trainers saw raw manifest JSON and webhook event logs that created confusion ("why is my task stuck?") without providing actionable information. Admins need pipeline observability for debugging; trainers need a simplified status view.

function TaskDetailsTabs({ task, user }: Props) {
  const tabs = [
    { id: 'review', label: 'Review', visible: true },
    { id: 'qc-results', label: 'QC Results', visible: true },
    { id: 'pipeline', label: 'Pipeline', visible: user.role === 'admin' },
    { id: 'events', label: 'Events', visible: user.role === 'admin' },
  ];
  return <TabBar tabs={tabs.filter(t => t.visible)} />;
}

QC orchestration interaction model

The trainer gate integration follows a request-async-respond pattern:

Trainer submits review → manifest transitions to submitted
QC orchestrator dispatches checks to configured agents
Cloud Tasks worker polls agent webhooks for results
On pass → manifest transitions to qc_passed, task proceeds to release
On fail → manifest transitions to qc_failed, task returns to reviewer with failure details

Reviewers never wait synchronously for QC. The spinner (scoped by AGT-1082) indicates progress; results appear when ready. This decoupling is essential — QC checks involving LLM adjudication can take 30–90 seconds.

Why edge cases dominate pipeline UI work

The happy path — claim, review, submit, QC pass, release — is straightforward to implement. It accounts for maybe 60% of task volume. The remaining 40% is edge cases:

Trainer claims, gets distracted, claim times out
Trainer submits, QC fails, trainer reclaims and resubmits
Admin force-passes a task with known QC false positive
Auto-seed generates draft data while trainer is mid-review
Two trainers attempt to claim the same task simultaneously

Each edge case requires manifest state consistency, draft cleanup, audit trail preservation, and UI feedback that matches the actual system state. Getting any one wrong produces "ghost tasks" — tasks that appear claimed but are not, or appear released but still hold a draft lock.

What I'd do differently

I would define claim lifecycle as an explicit state machine diagram in the repo before implementing AGT-770. We discovered edge cases incrementally through production tickets rather than modeling them upfront. A formal state machine with documented transitions would have caught AGT-1145 (post-release visibility) during design.

I would also emit structured events for every manifest transition from day one, not as a follow-up (AGT-886's events tab). Pipeline observability should be built into the transition layer, not bolted onto the UI.

The pattern: integrate QC at the gate, not after release

QC orchestration existed before the trainer gate integration. The value of AGT-770 was placement — running checks when the reviewer submits, while they can still fix issues, rather than after release when bad data is already in the corpus. Spinner scoping, claim lifecycle resets, and force-pass guardrails are the unglamorous work that makes the integration trustworthy in production.

Designing a Task Lifecycle V1 from Scratch

Humza Tareen — Tue, 25 Aug 2026 21:24:01 +0000

On the workflow platform I contribute to, tasks used to live in a flat mental model: a single status column, metadata scattered across ad-hoc fields, and no durable notion of pipeline stages beyond whatever the UI happened to imply. Evaluations kicked off scripts and dashboards reacted, but the database could not answer simple forensic questions—which model revision produced which artifact, at what point in the lifecycle, with what lineage through each gate. Operators and engineers shared the burden of Slack archaeology and spreadsheet glue.

This post is about designing and shipping a replacement from the ground up: a ten-table relational schema keyed around immutable versions, dual append-only event streams, explicit step-level state machines, and a typed REST surface—landed not as one heroic diff but as six stacked pull requests so each layer stayed reviewable and test-backed.

The problem in practice

The legacy task system modeled “where is this?” as a lone enum-ish string plus timestamps. Version history lived in brittle copies or nowhere. When an evaluation pipeline ran, there was no first-class structure for recording trials, models, or per-stage outcomes in a queryable shape. Debugging meant inferring causality from logs and praying the deploy tag matched reality.

The goal for V1 was deliberately narrow but strict: persist a task’s lifecycle as data you can traverse, correlate prompts and payloads to specific steps and evaluations, separate linear task progression from parallel eval work, and expose every mutation path through validators and handlers that behave the same whether called from cron, admin tools, or a future dashboard.

Ten tables, one coherent story

At the root sits TaskV1, the outward-facing aggregate. Rather than caching a duplicated status enum on the row, the task holds a foreign key pointer currentStepId to whichever StepV1 represents the active stage. “Status” becomes a projection: read the attached step’s state machine value, optionally join for display—but never drift from truth by forgetting to update two places.

TaskVersionV1 is the authoritative snapshot envelope. Writes that mean “the task advanced” mutate the active version inside a transaction—always paired with events—so dashboards can diff “what changed between version n and n + 1?” without relying on brittle before/after JSON blobs scattered in application memory.

Within a version, StepV1 rows implement a thirteen-state lifecycle (ordering enforced in TypeScript constants, no fragile position columns that invite reordering accidents). Events land in StepEventV1: append-only, ordered, representing the linear task pipeline—the constraint that exactly one operational step dominates at a time maps naturally to sequential history.

Prompt payloads and lineage need fast listing without exploding joins. PromptV1 carries the text-ish content alongside denormalized model / agent identifiers so dashboards can filter and sort without parsing JSON blobs. Artifact storage stays out of Postgres except for pointers: DataV1 unifies golden, source-side, calibrated, generated, and delivery-shaped blobs behind a typed discriminator and filePath into object storage.

Evaluation crosses a separate axis. EvalV1 anchors a run with trial count k and timestamps. Each concrete attempt is TrialV1, with a composite uniqueness rule on (evalId, k, model) so parallel model entries cannot collide silently. Trials emit into EvalStepEventV1—structurally akin to StepEvent—but owned by the eval subgraph so churn in one pipeline never poisons timelines meant for the other.

Finally, long-running inference style jobs persist as InsightV1: model, agent, provider as first-class varchar columns again (not tucked into JSON), surviving even when a downstream trial row is retracted—so cancellations and rework do not erase the record that work was attempted.

Relationships in Prisma (excerpt)

The centerpiece relationship is triangular: Task points at the live step and “hangs off” exactly one conceptual version envelope; steps belong to versions but carry duplicated taskId for hot paths.

model TaskV1 {
  id               String           @id @default(cuid())
  currentStepId    String?          @unique
  activeVersionId  String           @unique
  activeVersion    TaskVersionV1    @relation("ActiveEnvelope", fields: [activeVersionId], references: [id])
  currentStep      StepV1?          @relation("TaskCurrentStep", fields: [currentStepId], references: [id])
  ownedVersions    TaskVersionV1[]  @relation("TaskVersions")
  steps            StepV1[]        @relation("StepOwnership")
}

model TaskVersionV1 {
  id                 String    @id @default(cuid())
  taskId             String
  task               TaskV1    @relation("TaskVersions", fields: [taskId], references: [id])
  activeForTask      TaskV1?   @relation("ActiveEnvelope")
  steps              StepV1[]
}

model StepV1 {
  id                     String           @id @default(cuid())
  taskId                 String
  versionId              String
  task                   TaskV1           @relation("StepOwnership", fields: [taskId], references: [id])
  version                TaskVersionV1    @relation(fields: [versionId], references: [id])
  pinnedAsCurrentForTask TaskV1?         @relation("TaskCurrentStep")
  status                 String           // thirteen-state lifecycle, varchar deliberately
}

Prisma receives string fields instead of enums at the persistence layer—we traded compile-time tightening for velocity while the ontology still moved weekly.

Key design decisions

Separate event pipelines. Task steps admit a single authoritative timeline; evaluations fan out concurrent trials with their own branching. Merging both into one *_events table would have forced awkward polymorphism and invited cross-namespace bugs. Parallel eval noise stays out of operational step history.

Transactional write order. On a validated transition we always append history first—only then mutate TaskVersionV1.currentStepId (conceptually aligning the envelope) and finally repoint TaskV1.currentStepId. If any leg fails, the database rolls back; there is never a task pointing at a step whose transition never logged.

Denormalization for read paths. taskId repeats on Step, StepEvent, Prompt, and Data so common API responses avoid mandatory hops through versions for OLTP workloads that only care “what belongs to this task.”

Queryable columns instead of blobs. Model, agent, provider live as discrete columns—the extra width pays for indexing, equality filters, and human-readable CSV exports without brittle JSON path queries.

Transition validation before touch

Every PATCH-style route resolves the targeted step’s current status through a centralized map keyed by semantic step kind. Invalid attempts return HTTP 409 with the list of permissible next tokens so clients can recover without scraping error strings.

const STEP_TRANSITIONS: Record<StepKind, Partial<Record<StepStatus, StepStatus[]>>> = {
  ingestion: {
    queued: ["running", "failed"],
    running: ["awaiting_review", "failed"],
    awaiting_review: ["approved", "rejected"],
    // ...
  },
  // ...
};

export function isValidTransition(
  kind: StepKind,
  from: StepStatus,
  to: StepStatus
): boolean {
  return STEP_TRANSITIONS[kind]?.[from]?.includes(to) ?? false;
}

export function allowedNextStates(kind: StepKind, from: StepStatus): StepStatus[] {
  return STEP_TRANSITIONS[kind]?.[from] ?? [];
}

Handlers call isValidTransition before opening side effects; the 409 body serializes { error: "invalid_transition", allowed: allowedNextStates(...) } for predictable tooling.

Transactional choreography

The following sketch mirrors production shape: nested inside $transaction, inserts and pointer updates succeed atomically.

async function advanceStep(
  prisma: PrismaClient,
  input: { taskId: string; versionId: string; nextStepId: string; payload: Record<string, unknown> }
) {
  return prisma.$transaction(async (tx) => {
    await tx.stepEventV1.create({
      data: {
        taskId: input.taskId,
        type: "step_transition_requested",
        payload: input.payload,
      },
    });

    await tx.taskVersionV1.update({
      where: { id: input.versionId },
      data: { currentStepId: input.nextStepId },
    });

    return tx.taskV1.update({
      where: { id: input.taskId },
      data: { currentStepId: input.nextStepId },
    });
  });
}

In the shipped code the active version resolves from the task envelope before the transaction; the salient guarantee is ordering and single-transaction scope.

Handler template pattern

Repeated CRUD-ish routes funnel through createHandler: Zod parses, domain checks run, then Prisma executes. Middleware injects tenancy and actor context uniformly.

import { z } from "zod";

type HandlerCtx = { prisma: PrismaClient; actor: Actor };

export function createHandler<T extends z.ZodType>(opts: {
  schema: T;
  run: (args: z.infer<T>, ctx: HandlerCtx) => Promise<unknown>;
}) {
  return async (rawBody: unknown, ctx: HandlerCtx) => {
    const parsed = opts.schema.safeParse(rawBody);
    if (!parsed.success) return { ok: false, status: 400, body: parsed.error.flatten() };

    try {
      const result = await opts.run(parsed.data, ctx);
      return { ok: true, status: 200, body: result };
    } catch (err) {
      if (err instanceof ConflictError)
        return { ok: false, status: 409, body: { error: err.message, allowed: err.meta?.allowed } };
      throw err;
    }
  };
}

Golden tests instantiate handlers with ephemeral databases; the schema layer stays the choke point for regressions when new endpoints appear.

Six stacked PRs, forty-plus routes

Rather than one eighteen-thousand-line branch, implementation landed mechanically:

PR 1 — Pipeline infrastructure: registries for thirteen step states and ten gate states, typed transition matrices, shared Zod building blocks, the createHandler scaffolding, plus 119 harness tests guarding invariants.
PR 2 — Tasks + Versions: seven REST routes spanning create/read/update patterns for aggregates, nineteen tests.
PR 3 — Steps + Events: six routes with transition gates returning 409 when maps disagree, fourteen tests validating edge cases.
PR 4 — Prompts + Data: four routes for payload CRUD wired to typed paths, nine tests.
PR 5 — Evals + Trials + Insights: ten routes covering orchestration subgraph concerns, fifteen tests.
PR 6 — Auto-seed API: six routes bridging demo environments: taxonomy tables, guarded object-storage stubs, and runtime simulation clocks so integration tests approximate production cadence.

Combined: more than forty public routes and north of 176 automated tests—not counting migrations and fixtures. Each PR depended only on predecessors, enabling parallel review chatter even when sequencing merged strictly linearly.

Why stacking beat the mega-merge

A single rollout would have mixed foundational mistakes with superficial handler typos—reviewers skim, critical feedback arrives late. Small PRs anchored discussion: infra first (wrong map shapes break everything downstream), CRUD afterward (mostly boring once patterns exist). Blocking on PR 3 did not prevent colleagues from commenting on schema docs introduced in PR 1. Automated suites stayed green per slice, shrinking bisection pain when regressions surfaced.

Auto-seed integration

Demo tenants needed believable timelines without manual click-through. Auto-seed registers stage identifiers mapping to canned object-storage prefixes, upload ceilings, and legacy alias strings left over from early pilots. Runtime simulation jitter adds realistic pacing for dashboards; optimistic review semantics include an explicit rework path so “accepted then revised” narratives remain consistent with transactional rules.

The registry plugs into the same transition validators as production—demo data is not a special code path bolted sideways.

Closing

Good lifecycle design is boring on paper and dramatic in hindsight: immutable versions explain the past, FK-derived status avoids double-bookkeeping, split event logs respect genuinely different concurrency modes, and a thin REST layer enforces discipline at the boundary. Breaking delivery into stacked PRs cost a little upfront planning and saved weeks of rework when reviews caught mismatched assumptions early.

If you are contemplating a similar reset, sketch the timelines you need separately for linear workflows versus parallel experimentation first—the shape of your event tables—and only then chase CRUD ergonomics. Once the narratives fit the schema, handlers become assembly.