DEV Community: Humza Tareen

Log Levels That Lie: One Sweep, Two Repos, the Same Bug

Humza Tareen — Mon, 19 Oct 2026 10:42:08 +0000

A routine log sweep across a family of internal tools on October 12th turned up the same false-alarm pattern twice, in two codebases that share no code: a service's own startup messages were being filed as errors, purely because of which output stream they happened to be written to. Alongside it, a smaller but more interesting access-control story — opening a manager-only weekly dashboard to every editor, and discovering that "remove the UI restriction" was the easy 10% of that change.

A log aggregator usually can't read your message. It reads your stream. If your routine output goes to stderr, it's an error now, whether anything is actually wrong or not.

Eighteen stack traces that were all the same expected event

On an internal ops platform, a log sweep found its error bucket full of AuthApiError: Invalid Refresh Token: Session Expired stack traces — eighteen of them in seven days, each about eleven lines long. Every single one was a session Supabase had ended exactly as designed, on its own 48-hour time-box. Nothing was actually failing. The cause traces to how the Supabase SSR client is wired up: createServerClient registers an onAuthStateChange handler, and when the underlying auth library's initial-session load hits a refused refresh, it throws a bare error up through that handler rather than treating an expired session as a normal, expected outcome. The fix files the event as a single warning line instead of letting an expected state transition print an eleven-line stack trace under @level:error, which is exactly the kind of noise that trains a team to stop reading its own error bucket.

The same root cause, in a completely different tool

The genuinely interesting part is that the same sweep, on the same day, found the identical class of bug in an unrelated ad-variant export tool. Every deploy filed five error-level lines — Starting gunicorn, Listening at, Using worker, Booting worker, and Control socket — none of which indicate any problem whatsoever. The mechanism is almost comically simple once you see it: Railway assigns a log line's severity based on which stream it arrived on, not its content, and gunicorn writes its entire startup log to stderr by convention. Meanwhile, the two print statements in the actual application code that do report a real failure had been going to stdout, and were being filed as mere info.

So the error bucket had it exactly backwards: completely routine infrastructure chatter marked as errors, and the one place something could legitimately go wrong marked as unremarkable info. The fix routes gunicorn's routine lines to stdout and keeps the two genuine failure prints distinct. Finding this pattern twice in one sweep, in tools that don't share a line of code, says less about either codebase specifically and more about how easy it is for a hosting platform's stream-based log-level heuristic to quietly misclassify an entire category of output — worth checking for deliberately rather than assuming it only happens once.

Opening a manager-only dashboard without leaking other people's numbers

The access-control story this week came from direct editor feedback: a request to show one ad-network's performance section inside the broader Week Overview dashboard rather than as a separate page, and a follow-up request a few days later to open the entire Week Overview to every editor, not just managers, "so they are also aware of the stats."

The first request had a real structural reason behind the original split: Week Overview had been manager-only specifically because editors work the backlog, not the aggregate view, so the ad-network section had been pulled out into its own page to be reachable by editors at all. Folding it back in meant the overview itself had to open up — for managers, as a new tab alongside the existing views; for editors, as the single section they're allowed to see, with the manager-only tabs hidden rather than merely disabled.

The second request — opening the whole overview, not just that one section — is where the real engineering work was. Un-hiding a UI tab is trivial. The actual blockers were two routes (/overview/trend and /evaluations/dashboard) that explicitly refused any non-manager caller, and a row-level security policy on the underlying tables scoped to is_admin() or editor_id = my_editor_id() — meaning an editor's own database session could only ever read their own rows. Had the UI restriction simply been removed without addressing either of those, "This week" would have rendered correctly for a manager and silently shown an editor their own individual numbers mislabeled as the whole team's aggregate — a data-correctness bug disguised as a successful feature launch, and a worse outcome than just leaving the page locked.

Making two comparison views agree with each other first

A smaller companion fix tightened the Four-Week Comparison to show only completed weeks, anchoring the comparison window on the selected week once it's finished, or on the last fully-completed week otherwise, with the same clamp applied to a week selected in the future. Applied identically server-side across both the category comparison and the ad-network's own week-on-week table, so the two views can no longer silently disagree about which weeks count — the same "two dashboards reading slightly different windows" failure mode that showed up on the enterprise platform's analytics rebuild this same week, just on a smaller surface.

The pattern: infrastructure defaults and permission removals both hide a second problem

Two unrelated threads, one lesson each. A hosting platform's log-level heuristic is a default nobody chose and most teams never audit — it's worth treating "is my error bucket actually full of errors" as a question to ask periodically, not an assumption. And removing an access restriction is never just a UI change when the restriction was load-bearing for more than the interface — the route guards and the row-level security policy underneath this week's dashboard change were doing real work that a hidden tab had been quietly relying on.

Assignee Dimension, Fleet Guards, and a Lineage Fix for an Unclaimable Task

Humza Tareen — Mon, 19 Oct 2026 10:42:05 +0000

A quiet week on the enterprise workflow platform still produced one genuinely tricky lineage bug, two access-control gaps worth understanding precisely, and the kickoff of a real analytics rebuild. None of these individually needed a deep dive — together they're a good snapshot of what week-to-week maintenance on a mature review pipeline actually looks like once the big features are built.

A permission check that's technically correct but inconsistent with its sibling checks is still a bug — the inconsistency itself is what a user experiences.

A task that couldn't be claimed because its own history was valid

The sharpest bug this week was a task that became permanently unclaimable after passing through External QC. The root cause was almost paradoxical: the lineage walk that reconstructs a task version's ancestry explicitly skips over External QC markers when building the chain — by design, since those markers aren't part of the content lineage — but then required the node where the walk stopped to be markerless. When the walk legitimately stopped at a committed External QC marker, the code treated a valid, expected state as an error and threw. That turned GET /api/v1/tasks/<id>/versions into a 500, which failed the claim pre-check, which meant the claim button simply never submitted.

The fix adds a dedicated helper that recognizes this specific case — stopping at a committed External QC marker that's an ancestor, not the target — and returns the marker's own source version as the chain base instead of throwing. The bug is a good reminder that "skip X when building the chain" and "X should never appear in the chain" are two different rules, and conflating them turns a deliberately-ignored case into an unhandled one.

A fleet guard that was stricter than the page it was feeding

Separately, ADMIN-tier users had been granted access to the platform's /analytics page, but the Talent Performance tab and the Overview's "Active now" list both depend on an underlying fleet-status endpoint that was still locked to PLATFORM_ADMIN only. Because the Next.js route simply forwards the caller's bearer token straight through to that backend, an ADMIN user's otherwise-valid request got a 403 from the layer underneath the page they were already allowed to view. On screen, that surfaced as a generic "Failed to load analytics data" message on one tab and a silently empty "Active now" list on another — neither of which points a reader back to the actual cause.

The fix widens the fleet guard to accept both ADMIN and PLATFORM_ADMIN, mirroring a pattern the task-archive endpoints already used elsewhere in the platform. The lesson is narrow but recurs constantly in systems with more than one permission tier: granting access to a page is not the same as granting access to every endpoint that page calls, and the gap between the two shows up as a confusing error far from its actual cause.

A status chip that disagreed with the badge right next to it

A cancelled production run displayed "Cancelled" correctly in both the task list and the detail page's header badge — but the External QC step's own chip, sitting in the same view, still read "QC Pending." The cause: External QC is modeled as a document-driven step, so the general status-overlay logic deliberately skips it, deferring to a separate resolver that reads the gate document directly. That resolver never checked whether the overall task had been cancelled. The per-step chip rendering path had simply never been taught that "cancelled" was a state it needed to represent. The fix teaches it. It's a small bug, but a visible one — nothing erodes trust in a status dashboard faster than two labels on the same screen disagreeing with each other.

A retry budget charging the wrong party

External QC's retry-budget logic had the actors backwards relative to every other gate on the platform. Where Trainer and Final Review charge retries against the reviewer's actions, External QC had been charging the budget against the QC team's own feedback upload instead. The fix inverts it: a feedback upload with non-pass results and an exhausted retry budget is now accepted regardless, with the reviewer locked out from sending it back further; the retry is consumed specifically on a reviewer's own send-back action, not on the QC team doing their job. Getting this backwards meant the party being held accountable for a limited resource was the one least able to control whether it got consumed.

Rebuilding analytics on one shared catalog

The largest piece of work this week was the first PR in a three-PR stack building out a real analytics layer: an assignee dimension, a per-task talent drill-down, time-and-iterations tracking, and CSV export, covering four separate ticketed requirements plus a handful of audit-driven fixes found along the way. The architecturally important decision sits underneath those features rather than in them: Overview, Delivery, and Distribution are rewritten to read from the exact same shared task catalog that the All Tasks page already uses, with the server-side overview listing removed entirely. Counts match the Tasks page by construction now, computed from one status-counting function rather than from two independently-maintained queries that happened to usually agree.

That's the real fix here, even more than the new dimensions themselves. Two views of the same underlying data drifting slightly apart is one of the most common and hardest-to-notice classes of dashboard bug — nothing throws an error, numbers just quietly don't match, and users lose confidence in whichever number they saw first. Collapsing both consumers onto one source eliminates the category of bug rather than patching each instance of it.

The pattern: consistency bugs hide in the seams between correct pieces

Every fix this week was individually "correct code doing the wrong thing in context" rather than a straightforward logic error: a lineage rule that was right for most cases and wrong for one explicitly-designed one; a permission check that was internally consistent but inconsistent with a sibling page; a status resolver that was right about its own domain and silent about a domain it didn't own; a retry budget that charged a real cost to the wrong actor. None of these would show up in a unit test of the function where the bug lived — they only show up when you check that function against its neighbors.

Stylizing Every Cast Member: Hardening an Animated-Cast Feature After It Shipped

Humza Tareen — Mon, 12 Oct 2026 10:31:28 +0000

The main video-generation service's animated lane shipped and worked — for a solo presenter. The moment a real multi-person cast ran through it, the feature's own single-presenter assumption surfaced as a visible, embarrassing defect: three people in one frame, one prompt, and only one of them actually drawn as a cartoon. Getting from "works for one person" to "works for a cast" touched the stylization pipeline, the identity references handed to the clip model, the planner's own vocabulary, and a caching layer that was quietly re-buying the most expensive image a run makes every time the container restarted.

A bail-out guard written for "this isn't supported yet" and a bail-out guard written for "this is intentionally out of scope" look identical in a diff. Only one of them should still be there once the feature ships.

A guard that quietly made multi-person impossible

PR #1067 found the root cause sitting in a single early return. ensureStylisedCharacter bailed immediately whenever a job's cast had more than one member, with a comment explaining the deferral: a multi-member animated cast would render from raw photographs "exactly as it does today," rather than from stylized twins nobody had built yet. That comment was accurate the day it was written and had quietly become a landmine, because castIdentityRefFor pins a cast member to its photograph for the entire run unless that member's first appearance happens to be solo — and in an ensemble, nobody's first appearance is solo. A run on three men at one table proved it by accident: one member's identity reference happened to already be a stylized image, so he rendered "unmistakably Pixar" — round eyes, smooth bald head, caricatured features — while the other two, pinned to ordinary stock photographs, rendered fully photoreal. Same frame, same prompt, three different identity references producing three different mediums.

Fixing the frames alone would have shipped a new, subtler version of the same bug. PR #1068 found the other half: the two code paths that resolve a cast member's reference photographs for a clip — as opposed to a still frame — had never asked the question at all, reading straight from the catalog. A single-presenter module had already worked this out explicitly, with a comment reasoning that handing back real catalog photographs "would bind the voice at the price of the look, and the look is what the operator asked for." That reasoning simply hadn't been copied to the cast paths, so an animated two-hander would have opened on a stylized start frame and then handed the clip model several photoreal stills of the real people as the thing to match motion against — the exact failure mode a decision-log entry already called "the standing failure," now proven on a cast rather than a solo. The fix made one identity-twin derivation answer the question once, with three consumers reading the same answer instead of three places re-deriving it differently.

Independently stylized is not consistently stylized

Even after every member in frame was correctly drawn rather than photographed, a reviewer flagged that "the friends looks differently animated" — the protagonist fully stylized, a companion barely so. PR #1122's own first diagnosis was wrong, and it says so directly: the author expected a member had fallen back to a raw photograph, and the cost ledger disproved it before any code needed to change — every member had in fact billed a stylized-twin render. The actual defect was structural: each twin came from an independent, unseeded image-edit call over the same fixed prompt and a different source photograph, with nothing comparing the three outputs to each other. Three independent calls against one text description land at three different points on a photoreal-to-stylized spectrum, the same way three artists working from one written brief and no reference sketch will draw three visibly different styles. The fix didn't touch the prompt at all — it passed the first finished twin along as a second reference image to every subsequent member's stylization call, so the second and third renders have something concrete to match rather than another independent roll of the same dice. The number of paid vendor calls didn't change; what each call was given to work from did.

A plan that widened the cast and forgot to tell everyone

A separate pass, holdCastAcrossNarration, exists to keep the same person's face on screen across a narrated sequence — real identity continuity, and the reason the pass exists at all. PR #1086 found it conflating that fact with a second, unrelated one: "all of the cast are present." On one run, the planner wrote a beat as a two-shot of two specific friends; the hold shipped it as three people, discarding which pair the planner had actually staged while quietly widening the headcount. The root cause was a regex trying to infer shot size from the prose description of motion, returning nothing usable on ten of twelve beats — so the pass fell back to taking the full cast union almost every time, which is precisely the "uniform three" its own code comment says it exists to prevent.

Widening the cast without updating everything downstream that described the old, narrower staging produced two more bugs in the same shape. PR #1070 caught frames being graded against acceptance criteria the plan itself had made impossible — eight beats in a row scored against "one person," rendered with two or three people in frame, because the hold rewrote the cast list but nothing touched the grading criteria that referenced the cast size directly. PR #1072 found the shot's own motion description — the field that tells the image model how the shot should actually look — suffering the identical staleness, and called out why it's a harder fix than the acceptance criteria: criteria can simply be dropped when they go stale, but motion is the entire shot description and can't be discarded, so the fix moves the cast to match the shot's declared size instead, always trimming to the same anchor member in a consistent order rather than an arbitrary one.

Teaching the planner to say who a shot is about

Once a shot could legitimately hold two or three people, a new question had no owner at all: of the people present, who is the shot actually about? PR #1087 found every downstream consumer of that question — which member gets photographs, which gets the element label, which gets the voice — silently defaulting to index zero in the cast array, agreeing with each other by coincidence rather than by any actual contract. A reviewer's real complaint, "who is John? we're zooming in on the guy in the middle, but apparently John is the one in the blue collar," was this exact bug: the shot's composition followed array order, and array order read left-to-right by whoever was authoring the frame.

The fix had to clear a real constraint first. The team had already ruled out solving this with more prompt injection — the project's own decision log states the principle plainly: "a prompt guard is a guess; a measurement is not" — and a vendor capability sweep across five different providers confirmed there was no structured subject-of-camera binding available anywhere to lean on instead. So the planner itself was given a new field to declare which specific member a shot is about, rather than asking a human to intuit it from prose or asking a model to infer it from context. PR #1088 found two fresh bugs in that same change within a day of shipping it: the new directive told beats to say "the subject" while every existing substitution pass still only recognized the literal phrase "the presenter," so the new field fired on zero live beats until the vocabulary was widened; and the directive itself had quietly asked for two different things — "whoever the camera favors" and, separately, "whoever the line is about" — which genuinely diverge on a reaction shot. PR #1079 is the honest coda to the same arc: an earlier subject-clause fix had been reported as working, verified off the rendered board frames, and the report was wrong — the clause had never actually reached the vendor request at all, because it had been attached to the wrong one of two parallel prompt-assembly paths, and the real improvement that run showed came from an unrelated change in setting and cast size.

A redeploy that silently re-bought the most expensive image in the run

PR #1093 is the sharpest bug in this cluster because nothing about it is visible in the output — only in the bill. The stylized "twin" is the single most expensive image a run generates, and the code cached it by checking whether the file already existed on disk. That check was written against Railway's ephemeral container filesystem, which a mid-run redeploy wipes clean. A real run showed the signature directly in its cost ledger: the same stylized-character render, for the same member, billed twice at the same price, tagged with two different process identifiers — the unmistakable fingerprint of a container restart. The fix had, in fact, already been half-written down as a TODO in an unrelated module months earlier, explicitly anticipating this exact failure for text artifacts and noting that binary artifacts like images would need the same treatment "later." Later had arrived in the form of real paid, wasted spend, and the fix extended the existing download-and-restore-from-Drive path to cover binary files the same way it already covered text ones.

Letting a mascot share the frame instead of banning it

The last gap in this arc wasn't about people at all. PR #1095 found that a mascot character sharing a frame with human cast members fell into the pipeline's "people-free" rendering branch by construction, regardless of what the actual plan called for — meaning an ad whose reference genuinely depends on its mascot appearing alongside its humans in most shots could not be reproduced at all under the existing binary. A second, related bug fell out of fixing the first: a mascot-only beat with no human cast was being given the generic product negative prompt, which explicitly bans "arm, silhouette, figure" — banning the exact anatomy a mascot character is made of, because the two files responsible for choosing the right branch disagreed on what information they even had access to decide with.

The pattern: a feature isn't done at one, it's done at N

Every bug in this arc has the same shape: a path built and verified against exactly one cast member — one presenter, one identity reference, one subject, one cached twin — that silently stopped being correct the moment a second or third person entered the frame. None of these were algorithmically hard problems. They were places where "the" quietly meant "the one," and nobody had gone back to ask what happens when there's more than one. The through-line that actually closed each gap wasn't a smarter model or a cleverer prompt — it was measuring the real artifact a run produced, over and over, instead of trusting that a fix verified on a solo presenter would generalize for free.

Styles That Fight Their Own Prompts, Previews That Lie, and Artifacts a Redeploy Erases

Humza Tareen — Mon, 12 Oct 2026 10:31:24 +0000

Three unrelated-sounding questions turned out to share one root cause this round on the main video-generation service: why does every archived run use the same visual style, why does a $0.05 review-gate preview sometimes show something the paid render never produces, and why does restoring a run after a redeploy occasionally come back missing things nobody noticed were missing. Different symptoms, same answer each time — somewhere in the pipeline, two components had quietly stopped agreeing about what was true, and nothing was checking.

A preview that looks healthy and a render that looks healthy are not the same claim unless something has verified they're describing the same picture.

Styles: an operator's choice silently overridden by a default

The question that started this cluster was simple: do styles other than the pipeline's original Pixar-like default actually work? The measurement was blunt. Across all 53 archived runs on disk, pixar was the only visualStyle value ever recorded — and PR #1138 found that wasn't for lack of trying. resolveRunMediumWithStyle read visualStyleMedium(vs) ?? imageMediumSetting, and visualStyleMedium("custom") returned "live-action" — a non-undefined answer, so the ?? fallback never fired and a hand-set imageMedium: animated was silently discarded. An operator could pick "Others," type "claymation stop-motion," explicitly set the medium to animated, and still get live-action back. An entire category of output was unreachable through any combination of settings the form offered, and the deep machinery gated on medium === "animated" — the stylized twin, animated-person rules, realism-critic stand-downs — had simply never fired in production.

Fixing the gate didn't finish the job. PR #1139 ran two $0.20 probes at two operator-named styles and opened the resulting character twins at full size: a claymation request came back as genuine stop-motion, but a "2D hand-drawn anime, flat cel shading, no 3D rendering" request came back as a smooth 3D render — essentially the Pixar look again, just reachable now instead of blocked. No metric would have caught this; both runs reported medium: animated and carried their own style clause correctly. The cause was in the prompt itself: STYLISE_PROMPT opened by asserting "a STYLISED 3D-ANIMATED CHARACTER" and then appended the operator's 2D request afterward, rather than replacing the 3D assertion with it. PR #1140 traced this to a real finding in the generative-image literature the team already cited elsewhere — appending a contradicting instruction after an existing assertion is the weakest repair pattern tested, working in roughly 13% of cases, versus directly replacing the unwanted assertion with the desired one. The fix wasn't a better sentence; it was restructuring which clause got to make the claim at all.

The same contradiction showed up one layer down. PR #1154 found a character portrait — the single photograph that anchors every human frame's identity in a run — deciding its own visual register from a hardcoded string rather than the run's actual style clause, so a custom 2D request would get a Pixar-style portrait as its identity anchor regardless of what the operator typed. And PR #1160 found that a style's own described set and materials only ever reached cutaway shots, never person beats, because the compiler chose between a "has person" and "personless" style clause and routed the set description only through the personless branch — leaving every custom style with a person in frame and no description of the room around them at all.

A fallback that quietly won by being prettier, not by being asked

The sharpest bug in this cluster involved two different image models disagreeing about who should render a scene, with nobody deciding the winner on purpose. PR #1152 found one ad shipping with two image models' faces mixed across its clips — a scene's candidate list can hold outputs from more than one model when a fallback fires, and the scoring step that picks the best candidate has no opinion about which model drew it. The best-looking fallback output simply won, silently.

The initial diagnosis blamed the fallback mechanism itself. PR #1157 went back to the cost ledger, grouped by actual in-process task, and found that explanation didn't survive contact with the data: one task in the run held 34 frames from one model and 7 from another, mixed inside a single task — something the fallback logic can't produce, since it refuses to revise a pin once a frame already exists. The real cause, in PR #1159, was that the image-model "pin" — an in-process lock recording which model a run committed to — lived in a plain in-memory map that only got populated at the start of a full task. Editor-triggered redos and regenerations dispatched through a different code path that bound the run's scope but never acquired a pin, so a redo on a Gemini-pinned run would silently fall through to an unset ambient default that happened to resolve to a different model entirely. The team's first explanation — a vendor failing over — was wrong, and correcting it mattered, because the real fix (recover or re-acquire the pin on every dispatch path, not just the first one) is nothing like what you'd build to guard against vendor flakiness.

Previews: the cheapest gate lying about the thing it exists to show

A review-gate preview is supposed to be the inexpensive look at an ad before real money commits to it. This round found that promise broken in several distinct ways, each one undermining the entire reason the gate exists.

PR #1141 found every 4:5-aspect preview returning a 400 from the vendor — the requested resolution sat under a minimum pixel floor that 9:16 and 16:9 happened to clear. Because the failure surfaced as previewStatus: null rather than an explicit error, it read as "not generated yet" instead of "broken," and 4:5 happens to be the aspect every run using the pipeline's default style renders at. PR #1144 found the same null-reads-as-nothing-happened failure mode on reference-less runs: the batch preview pass skipped any scene missing a reference keyframe, and a reference-less run has no reference keyframes by definition — so eighteen of nineteen scenes on one measured run silently produced no preview at all, with only the single b-roll beat (which takes a different code branch) rendering anything.

PR #1150 found a preview that rendered successfully and was still wrong: a three-person cast beat's preview passed a single identity photo, while the real frame-generation step resolved every bound cast member and handed the rest over as additional reference images. The preview and the real frame answered "who is in this shot" from two different code paths, and only one of them was correct — so an operator approving nineteen beats of what looked like a solo presenter was approving a three-person ad they hadn't actually seen. PR #1153 found the plan-gate preview — the cheapest possible look at a swipe-mode ad, priced at five cents specifically so a bad plan could be caught before any paid frame exists — anchoring every post-scene-one person beat on "the first human scene's rendered frame," which is correct once frames exist and impossible at the plan gate, whose entire premise is that nothing has rendered yet. Those beats were handed a file path that didn't exist and couldn't until the operator had already approved and paid. And PR #1165 found the fallback preview template for every single scene using an identical, unconditional selfie composition — eyes at the lens, UGC framing — regardless of whether the beat was actually a narrated voiceover scene where the subject shouldn't be addressing the camera at all; the bug survived because every real, authored frame overrides the template entirely, so it was invisible on any run doing real work and only visible on the exact fallback path nobody was checking.

A gate is also supposed to stop a run, not just flag it. PR #1145 found a storyboard-error severity that the code's own documentation said should block approval without an explicit human acknowledgment — and nothing implemented that block. The only consumer was an autopilot "hold," which merely pauses automatic advancement; it doesn't make the approve route refuse. The team's own fleet-testing harness then cleared every such hold automatically after a 240-second stall, meaning a genuine plan-infeasibility or cast-contradiction error parked an unattended run for four minutes and was then waved through by the automation built to run tests, at precisely the one checkpoint where stopping was supposed to be mandatory and free.

Checks that exist, fire correctly, and change nothing

The most expensive bug shape in this round wasn't a missing check — it was a check that worked exactly as designed and had no path to acting on its own output. PR #1155 found that a lip-sync desync detector, built specifically to catch narrated beats whose rendered mouth movement can't possibly match the voice muxed in afterward (the two are generated independently, by construction), fired correctly on twelve of nineteen beats on one run and eleven of thirty on another — and nothing downstream ever read the result. The redo pipeline filtered only on a different, unrelated flag; the approve gate only blocked on error-severity findings, and this detector reported at warning severity. The detection logic had been deliberately split out of a broader check for accuracy, and in the split, silently lost the only code path that would have turned "detected" into "fixed."

A second judge is similarly only useful if its verdict is actually consulted correctly. PR #1148 found a redo mechanism whose own code comments insisted, twice, that only a criterion both judges failed should justify spending a paid retry — the intersection, not the union — while the actual implementation took the second judge's failure list wholesale. A clip got bought whenever the second judge failed any criterion at all, including ones the first judge had explicitly passed, defeating the entire reason a second judge exists.

Restore: artifacts that a redeploy quietly fails to bring back

The restore bugs in this round form their own small escalation. PR #1161 found that a bound cast's per-member reference photographs didn't survive a container redeploy — the restore routine's pattern-matching for "the character photo" matched only the single-presenter filename exactly, so per-member files written for a bound cast were never pulled back from storage. The failure mode was specifically deceptive: the legacy single-photo endpoint still answered with a 200, because a different, older photo genuinely did exist under that name, so the run looked healthy right up until something asked for a specific cast member by ID and hit a bare file-not-found five times in a row.

PR #1162 treated that as a symptom of a broader, unaudited mapping rather than a one-off miss, and checked every file-mirroring call site against every restore pattern in the codebase — finding four more mismatches, including the inverse of the cast-photo bug on a different code path, and a saved clip take whose filename convention (a dot the mirror step turns into a dash) simply had no corresponding restore rule at all. PR #1164 went one level further and asked a sharper question: what does the restore audit's own completeness check actually verify? The answer was that it could only catch artifact classes that are mirrored but map to the wrong name — an artifact nobody mirrors at all has no entry to fail on, so a true blind spot read as a clean bill of health. The fix wasn't more pattern-matching; it was an explicit, must-maintain list of every deliberately-unmirrored artifact and the reason each one is excluded, so the next person meets a documented decision instead of a silence that looks identical to correctness.

The pattern: verify the second claim, not just the first

Every bug in this post is the same shape wearing different clothes. A setting existed and a different code path silently overrode it. A prompt made an assertion and a second prompt contradicted it without replacing it. A preview rendered successfully and rendered the wrong thing. A check fired correctly and nothing consumed its output. A restore routine ran without error and restored the wrong subset of files. In every case, the first-order fact — "the setting is stored," "the check ran," "the restore completed" — was true and completely insufficient. The actual defect only became visible by checking the second, harder claim: does this setting reach the place that renders pixels, does this check's output reach a path that can act on it, does this restore step's coverage match everything a run actually writes. A system with enough moving parts will always have components that are each individually correct and collectively wrong — the only defense is measuring the handoff between them, not just the components themselves.

Stop Paying Twice: Hardening a Reframing Tool's Generation-Collection Sweep

Humza Tareen — Mon, 12 Oct 2026 10:30:51 +0000

A vendor error called VendorQueuedTooLong carried a comment that read, in effect: "nothing was rendered, so resubmitting is free." The reframing tool's render pipeline believed that comment and acted on it every time a generation sat in queue too long. Both halves of the claim were false, and they cost real money. The vendor pre-charges at submission time and only refunds a generation that reaches a documented failed state — and the "abandoned" generations weren't failing. Twelve for twelve of the ones this exact error killed in production had actually reached completed at the vendor, sitting there, paid for, with nobody ever coming back to collect them.

That discovery is where a 27-PR arc started: build a background sweep that finds generations the vendor has already billed for and reclaims them instead of quietly resubmitting and paying twice. What makes this cluster worth a post of its own isn't the sweep's existence — it's that the sweep itself turned out to need the same scrutiny as the pipeline it was auditing, right down to a defect that made it structurally blind to its own primary use case.

A cost-reconciliation process is still software. It can have the exact bug class it exists to catch.

Measuring the vendor instead of trusting the error message

Across the account's whole history — 1,020 generations between May and September — 1,005 completed normally and only six ever reached a true failed state. The remaining nine had sat in a queued state for 122 to 146 days with no assets attached: the vendor, it turned out, holds a generation indefinitely rather than ever failing it outright, which meant any sweep built on "wait for failed" needed a give-up condition of its own. PR #16 recovered the twelve stranded result URLs directly and, just as importantly, stopped creating new orphans: the queue-timeout error now parks a run in an awaiting_vendor state instead of marking it terminally failed, so a boot scan or sweep still has something alive to reconsider later.

PR #17 is a small, disciplined piece of measurement worth calling out on its own. The render-budget logic depended on observing a specific vendor transition state — dreaming — and nothing had ever confirmed that state was actually observable at the poll interval being used. A history tally couldn't settle it either, since every past generation has already resolved one way or another; a snapshot reads zero dreaming states whether or not the state is really samplable. The only way to know was to spend one real, paid 540p probe and watch the transitions land in real time. It confirmed the state is entered within about five seconds and held for roughly nineteen — comfortably inside the poll window — which is the only reason the rest of the budget logic downstream could be trusted at all.

The sweep that couldn't see the case it was built for

PR #21 is the center of this whole story. A high-effort review of the accumulated sweep code turned up fourteen findings, and the first one undid the premise of everything before it: the sweep could never actually observe a parked run. The in-memory job registry's "live" check returned every non-terminal job, including parked ones — so the very container that had parked a run kept heartbeating it forever, which meant its lease never went stale, which meant both of the sweep's own filters excluded it: the process-local id check, and the stale-heartbeat condition in its query. The one container that could have collected the parked generation was, by construction, the one container that never would. That is precisely the failure mode parking was built to solve, and the sweep had been quietly unable to touch it the whole time.

The same review surfaced a cluster of smaller but equally real defects, each pinned down with a test that fails the moment the fix is reverted:

Finding	Consequence
A hand-back path skipped the resume-attempt cap	the sweep itself could push a run past its maximum resume attempts
A give-up wrote nothing to the row	every later cycle re-probed the vendor for a case already settled
A write's return value was discarded	a silently dropped write reported as a successful collection
The give-up clock read the row's own update time	candidates read as minutes old when they were really 11–19 days old
A 404 was treated as a vendor failure	indistinguishable from what a rotated API key also returns
Status vocabulary duplicated in three places	a second and third copy could silently drift from the first

None of these are exotic bugs. They're the ordinary failure modes of any stateful background job — and they existed in the code written specifically to catch ordinary failure modes in the pipeline next door.

A fallback ladder that was buying the same seconds twice

PR #22 found a second, more direct instance of the same root cause. The fallback ladder that escalates across vendor brokers caught every subclass of a generic vendor-error type and advanced to the next broker — including the two states, queue-timeout and stall, that the system's own error documentation explicitly flagged as already billed. The ladder's entire justification had been written as "one queue starving is the whole problem this exists for" — but a starved generation is, by the vendor's own pricing model, one that's already been accepted and paid for. Advancing past it didn't recover from a starved queue; it bought the same seconds a second time on a different broker. The fix narrows which error types are allowed to advance the ladder to ones that provably bought nothing — a bad request, an auth failure, an exhausted retry with no generation id ever issued — and makes everything else degrade to the output already on hand instead of spending again.

A database constraint that was actively hostile to the feature it sat next to

PR #23 found a uniqueness constraint that made the sweep's entire purpose impossible at the schema level. A vendor-generation id was marked unique across the whole table, on the reasoning that one generation belongs to exactly one segment. But a rerun is specifically designed to reuse a prior run's generation ids — that's the entire point of recording them — so writing a reused id against a table-wide unique index threw a constraint violation that got silently swallowed as a warning. Measured against the three reruns that existed in production at the time: zero of them had successfully recorded a single shared id. Every one of them had re-attached to a prior generation for free in memory, written nothing durable about it, and was one interruption away from re-buying every segment it had just silently reused, at full price. The fix narrows the constraint to per-run uniqueness — the actual invariant worth enforcing — rather than per-table.

A receipt written too late to survive the exact failure it needed to survive

PR #24 found that a fallback generation's payment receipt was held in an in-memory Python list and only persisted to durable storage after the entire fallback block completed — after the poll, the download, and the length check. Every way a segment could die in between (a container redeploy mid-poll, an editor cancellation, an ffmpeg fault) lost the receipt for a generation that had already been bought, at up to roughly eighty cents a run. The primary generation path had followed the opposite, correct rule since it was first written — record the id the instant the vendor returns it, before the first poll — and the fallback path's own documentation claimed it did the same thing. It didn't. The fix records the id at the moment of purchase, and deliberately does not yet let the sweep act on fallback ids until a separate vendor-routing fix (below) makes probing them safe.

Guessing which vendor to ask is itself a bug

PR #34 is maybe the cleanest illustration in the whole arc of a sweep auditing a three-vendor pipeline while only being able to speak to one vendor. The query selected a vendor column and then discarded it, defaulting every probe to a single hardcoded API. Tested against one real, completed generation routed three different ways: asked correctly, it returned complete with a usable URL and the verdict "collect." Asked as either of the two wrong vendors, it came back not_found — read by the decision logic as "this generation is gone" — and the verdict was "give up." Two routings out of three threw away a paid, finished asset. The fix removes the default entirely: an unrecognized vendor or a missing credential now returns "could not ask," which is a different answer from "the generation doesn't exist," and the sweep is required to tell the two apart.

Honest messaging to the editor, not just a correct ledger

A smaller cluster (PR #29, #33, #35, #36) fixed what the editor actually sees once the sweep does its job correctly. A give-up verdict was being written to a field the run-detail panel never read, so an editor was told to keep waiting for — and explicitly not resubmit — a generation the system had just internally concluded was never coming. A successfully collected and finished run could still display a stale VendorQueuedTooLong error sitting right next to the word "Done," because nothing cleared the field once the generation was recovered. And a rerun's own cost-savings estimate was wrong in the specific population the whole sweep exists to serve: a segment only counted as "reusable" if every part of it was free, so a recovered generation with only its upscale step still pending read as zero savings, telling an editor "this is a full re-render" on a job that was actually 78% already paid for.

Operability and the parts that touch real money directly

The remaining PRs round out the picture: a confirmation dialog for stopping a parked run used to fire only once a segment had completed, which meant a run parked on its very first segment — with a generation already bought and in flight at the vendor — could be cancelled with one unconfirmed click, forfeiting money that cancellation (unlike collection) never recovers. A cost ledger was charging $0.00 for a completed, billed generation that had simply been rejected on quality grounds after the fact — real line items from production showed every other segment billing at the same consistent per-second rate while the rejected one logged as free. A quality audit that only checked whether an upscale URL was present missed an entire letterboxed, defective master shipping to an editor, because the two fallback outcomes it could distinguish both looked identical through that one narrow signal. And a logging bug meant five of seven real warning conditions in one production run never reached the log at all — a plain Python print statement reaches a platform's log aggregator classified only by which stream it came from, so warnings printed to stdout were filed as routine "info" lines sitting beside ordinary request chatter.

Woven through nearly every PR in this batch is a documentation discipline worth naming directly: a README was rewritten to describe files that no longer existed and omit the directory holding essentially all the maintained code; a gunicorn dependency got pinned to the exact version its configuration had been silently relying on, because a looser version range had already let one redeploy quietly resolve a new major and undo a prior log-level fix without anything alerting on it; and a one-write-per-poll bug inflated a 37-minute render into a database row that looked, from its timestamps alone, like it had taken 68 seconds — hundreds of identical writes whose only lasting effect was whichever one happened to land last.

The pattern: the auditor needs an auditor

Every bug in this arc shares the same shape. A background process was built specifically to catch money the main pipeline was losing — and it lost money the exact same way, for the exact same reasons: state that doesn't survive a restart, a constraint that doesn't model the real invariant, a default that stands in for a verdict nobody actually computed, a receipt written at the wrong moment to survive the failure it needed to survive. Building a reconciliation layer on top of a system doesn't automatically inherit correctness from good intentions. It's new code, with its own state machine and its own edge cases, and — as PR #21 proved outright — it can be built with such confidence that nobody notices it's structurally incapable of seeing the one case it exists to handle, until someone runs a high-effort review against real production data and checks.

The Narrated-Beat Repair That Kept Causing Its Own Bug

Humza Tareen — Mon, 12 Oct 2026 10:30:47 +0000

A "narrated" beat in the main video-generation service renders silent — the person on screen doesn't speak, the voiceover is added at the stitch — which means the frame-authoring prompt has to actively un-say things a normal talking beat would say: don't look at the lens, don't open the mouth, don't gesture like you're mid-sentence. That's a narrow, specific rewrite, and over five straight days it turned out to be one of the hardest things in the whole pipeline to get right, because every fix for one failure mode of "don't do that" created a new way for the beat to come out wrong.

This post is about that repair step, and about a parallel cluster of identity and relevance critics that had the opposite problem — not saying the wrong thing, but being structurally unable to say anything useful at all.

A repair that fixes what it looks at and never checks what it leaves behind is not a repair. It's a relocation of the same bug.

The repair that produced the defect it prevented

PR #1126 is the fix that should have closed the book on narrated-beat framing, and instead opened a new chapter. Measured on a real reference-less run, the strip step that removes camera-facing language ran on 15 of 30 beats — and the plan critic flagged six of them anyway, not for anything the strip left in, but for what it left dangling. "Tipping it side to side" survived with no object ever established to tip. "The other hand gesturing outward" survived with no first hand established. The strip was removing individual clauses correctly and leaving the clauses around them referring back to something that no longer existed.

The fix PR #1126 shipped was a cascade: once a clause gets removed, drop anything downstream that refers back to it. PR #1128 reverted that same-day, measured against a different run, because the cascade over-corrected — three scenes came back with every action clause stripped, leaving nothing but a bare shot size and the standard not-looking-at-the-lens note. The plan critic's own words on the result: "no staging at all beyond a shot size." The fix for dangling references had traded one defect for a worse one, found and reverted before a single clip was rendered.

PR #1127 caught the regression's quieter half: #1126 correctly made the narrated-direction repair reachable on every narrated beat instead of only the ones whose role had just changed — and that exposed that the repair had only ever touched a beat's motion field, never its accept field, the list of criteria the critic grades the rendered clip against. Measured: 18 of 30 beats had their motion stripped of camera-facing language, and all eighteen still carried an accept list requiring "facing camera." The repair and the check it fed were now disagreeing about what a narrated beat even is, on every single beat the repair touched — and the check, correctly, was the one catching it.

PR #1129 found a third failure shape in the same system: the repair wasn't just removing language, it was appending a denial — "not looking at the lens" — directly next to direction it had deliberately chosen to keep, like "leans in hard toward the camera." A lean toward the lens while narrating is legal, established direction. Writing a denial of eye contact one sentence after affirming a lean toward the camera is a contradiction the module wrote itself, and both the plan critic and the image model reacted to it as exactly that.

The instruction that existed and had no way to arrive

PR #1130 is the root-cause finding underneath the whole cluster, and it's almost funny in how simple it is once stated. The prompt template had, all along, contained the correct instruction: a per-beat block telling the planner a scene's role is "narrated," already decided, don't change it, write motion and accept for someone who is not talking to the camera. That block was gated on a field stamped only by the code path that reads a reference video's own shots. A reference-less run has no reference — so the field was never set, the block was never emitted, and the planner was simply never told. The correct instruction existed in the codebase the entire time; it had no path to reach the one case where it mattered most.

The twin that forbade and requested in one breath

PR #1124 caught the same disease from a different angle: a narrated beat's per-scene message introduced its own voiceover line as "they are mid-delivery of this" — explicit direction to draw an open, speaking mouth — and two lines later, the same message appended "mouth relaxed and closed." One author was told to do a thing and forbidden from doing it in the same breath, which isn't ambiguity, it's a coin flip the model has to resolve on its own. The reviewer's note on the delivered ad was that the cast "appear to be talking over a narration they are not speaking" — exactly the failure mode the contradiction predicted.

And a narrower version of the same bug, caught by PR #1106: a strip rule matched the literal phrase "to the lens" but not the paraphrase "toward the lens," so a clause aiming a face at the camera survived the strip intact and sat right next to the note the module had just appended saying the opposite. Measured on a real run: the critic flagged visual mismatches on 6 of 33 beats, each one a variant of the same near-miss string match.

A check that only detects, and a redo that isn't allowed to help

A parallel thread this same week was about critics that could see a problem and had no way to act on it, or acted in a way that made the problem worse. PR #1114 found that an identity-anchor check added specifically to catch cast drift between frames contradicted a repair mechanism meant to fix exactly that drift: the anchor check fires when a frame shows the wrong person, and a candidate that successfully repairs that has, by definition, changed who's in the frame relative to the one it's replacing — so a second check, comparing the fix against the broken original, correctly says "this is a different person" and reverts the fix. The detector and the repair couldn't coexist on the one defect the detector existed to catch.

PR #1115 found the same check failing in the opposite direction: comparing an anchor frame against later frames by matching people on where they physically stand in the shot, rather than by who they are, produced consistent false positives on a cast that never actually changed — three correct frames failed, every time, because the people in them had simply moved around the room between shots. PR #1110 closed a related gap at the source: a frame-redo mechanism had no rule against changing who the redo produces, so a redo meant to fix a lighting or composition flaw could — and on one real animated run, did — swap in a visibly different person, aging one character down twenty years and completely changing another's hair and glasses between two adjacent scenes of the same ad.

Grading on criteria nobody showed, verdicts about frames already gone

PR #1113 found a b-roll cutaway that shipped into a finished ad carrying its own critic's warning — "no person, face or hand in frame" on a span that was supposed to show someone — with a status field reading "ready" right next to it. The root cause: the generator that paints the cutaway is never shown the acceptance criteria it's later graded against. The criteria exist, the critic reads them, and the thing being critiqued never saw them — which makes a failing grade not a safety net but a formality nobody could have passed.

PR #1111 found a stranger version of the same blind spot at the whole-board level: a storyboard verdict reporting "matches all 2 of the plan's checks, zero failures" was describing frames that, by the time anyone read that verdict, had already been regenerated three times by the step that runs immediately after grading. The board graded real pictures and told the truth about them — the pictures just didn't exist anymore by the time the truth reached anyone.

The audit that found eleven more

PR #1131 is the capstone of the week, and it's refreshingly blunt about what it is: an adversarial audit — 88 agents, 7 lenses, 3 independent skeptics per finding — run specifically against this session's own changes, the same day they shipped. Seventeen raised findings, none refuted, deduplicating to eleven distinct defects, every one introduced earlier that same session and fixed in the same PR that confessed to them. One finding alone — a product-reveal rule that shipped and was structurally inert because the one code arm every real run actually takes never called the function that read it — echoes the exact shape of bug this whole cluster kept rediscovering: correct logic, unreachable from where it needed to run.

The pattern: narration is a double negative, and double negatives don't compose

Every bug in this cluster is a version of the same thing. "Narrated" isn't a positive instruction — it's a list of things a normal talking beat does that this beat must not do, layered on top of direction that was written assuming the beat could talk. Stripping language correctly requires knowing what else in the same message depends on it; appending a denial correctly requires knowing what else in the same message it might contradict; checking a repair correctly requires grading the thing the repair touched, not just the field everyone remembers to update. None of these are hard algorithmically. They're the specific, grinding difficulty of maintaining two true statements — "do this" and "but never like that" — in the same generated paragraph, across every beat, every run, every time the underlying direction changes. The fix was never a smarter model. It was finding, by measurement rather than assumption, every place the pipeline had quietly let the two statements drift out of sync.

Locking Down Sign-In: Workspace-Only Auth and a Session Lifecycle That Matches Its Provider

Humza Tareen — Mon, 12 Oct 2026 10:30:13 +0000

An internal ops platform that started as a project tracker had, by this point, grown into the shared front door for eight internal tools — which means its authentication and session handling weren't just protecting a tracker anymore. A single evening security audit found four real gaps behind that front door, and the two weeks after it were spent building something auth code rarely gets: an explicit, logged, correctly-attributed session lifecycle instead of a vague sense that "sessions expire eventually."

A valid token proves who someone is. It doesn't prove they belong in this specific shared table. Those are two different checks, and conflating them is how open data looks like access control.

Restricting sign-in to a verified company domain

PR #94 replaced password sign-in entirely with Google OAuth, scoped to the company's own Google Workspace domain — not "sign in with any Google account," but specifically the organization's verified domain, enforced at four independent layers: the Google consent screen itself is configured Internal, direct Supabase signups are disabled, the OAuth callback explicitly refuses any email outside the domain, and the session gate plus the tool proxy both re-check the domain on every single request rather than trusting the callback's one-time check. Four layers sounds like overkill until you consider what each one is actually independent protection against — a misconfigured consent screen, a stale signup path, a forged callback, and a long-lived session created before the policy existed.

The migration detail worth noting: existing accounts kept their role, tool grants, and editor links intact because Supabase links a new Google identity to the existing user record by matching verified email addresses — the underlying user id never changes, so nothing downstream had to be re-provisioned. PR #97 followed up with the unglamorous polish that makes an auth migration usable rather than merely correct: a clear message instead of a raw invalid_request when someone takes longer than Google's 5-minute OAuth state window to pick an account, an email input for admin-created users that can't be tricked into doubling the company domain, and a sign-in button built to Google's exact branding specification — a new primitive built specifically because the platform's own design-system lint forbids raw unstyled buttons everywhere else, and Google's own brand guidelines forbid restyling their button as anything else.

The audit: four gaps, found and fixed as one coordinated plan

PRs #89 through #93 are explicitly labeled as parts of a single security audit conducted in one evening, and reading them as a set is more illuminating than reading any one in isolation — each finding is a different flavor of the same root mistake: trusting a signal that looked like proof of identity or membership but wasn't.

PR #89 found that a "fast path" serving tool API calls read the caller's role from an x-hub-user-role cookie — HttpOnly stops a browser script from reading that cookie, but says nothing about stopping a person from setting it directly in a request. Any authenticated user with one legitimate tool grant could send a crafted role cookie and reach every tool API call as a superadmin. A second header, the caller's forwarded email, was read out of the unsigned JSON payload sitting next to the one actually-verified field in the auth cookie — so an attacker holding a real token could still rewrite which email every downstream tool believed it was talking to. The fix reads signature-checked claims directly and derives the role from the same database lookup the full authentication path already computed, instead of trusting a second, separately-settable copy of it.

PR #92 is the audit's sharpest finding: because the identity provider allowed open signup with auto-confirmation, anyone holding the public API key could obtain an authenticated session — and nearly every shared table trusted "authenticated" as sufficient proof of belonging. The project tracker could be read, rewritten, and deleted wholesale by that session. A specific account-creation trigger, matched by email string, silently granted one particular address full superadmin access with every tool grant attached — a backdoor that existed by construction, not by exploit. The fix introduces a real membership check — an active profile row, not merely a valid session — and gates every shared table on it. New profiles now start inactive by default, so "I have a session" stops being equivalent to "I am a member of this team."

PR #91 closed the same gap at the admin layer: a form field accepting an arbitrary profile id to link to an editor record, with no check that the id existed, was active, or wasn't already linked elsewhere — meaning an editor's evaluations and work history could be silently redirected onto any account. The same PR fixed granted_by recording a self-promoted user as their own granter, which is exactly the kind of audit-trail bug that only matters the day someone needs to answer "who approved this."

PR #93 is the audit's closing phase, and it's a good model for knowing when a cleanup pass is actually done: it confirms row-level security is live everywhere, every view runs with invoker security rather than inherited privilege, and the few genuinely sensitive database functions aren't callable by low-trust roles — and then revokes TRUNCATE, TRIGGER, and REFERENCES privileges that the database platform grants to anonymous and authenticated roles by default. None of those three are reachable through the application's actual API surface, which is precisely why the PR frames the fix as hygiene rather than a live exploit — closing a door nothing was using yet is still worth doing before something learns to.

A session cookie that doesn't comma-fold, and other quiet correctness bugs

Several of the fixes in this cluster aren't access-control gaps so much as the kind of subtle correctness bug that security hardening tends to surface as a side effect of reading code more carefully than usual.

PR #88 found that the function responsible for ending a disabled user's sessions had been calling the wrong overload of the identity provider's sign-out method for its entire existence — the first argument that API expects is the user's own JWT, not their id, so every single invocation failed with an invalid-token error that was logged and silently swallowed. Disabling, demoting, or deleting an account through the admin panel had never actually ended anyone's session. The bug was caught the ordinary way real security bugs get caught in a working system: while closing a specific person's access after an email change, and it stayed caught because the fix shipped with a new test that fails against the old code and passes against the new — proof the original code path never worked, not just an assertion that the new one does.

PR #90 is the single largest fix in the cluster, closing five separate session-correctness gaps the audit's other PRs had exposed: the one-hour idle timeout had never actually fired, because the activity-tracking cookie expired at exactly the idle window and an idle session simply arrived with no stamp at all — which the code read as "first contact" rather than "this session has been idle." Disabled accounts kept access to every page except the ones that explicitly checked an active-status flag. Sign-out and access revocation never propagated to the tool layer at all, so a revoked token kept working on tool API calls until its underlying JWT naturally expired up to twelve hours later. An open-redirect vector let a crafted path slip past validation by exploiting how browsers collapse a tab character into what the URL parser then reads as a protocol-relative double-slash. And the session cookie itself was readable by any script on the page, rather than marked HttpOnly — which matters precisely because this platform's tools share its origin, making that cookie reachable from any tool-side cross-site-scripting bug, not just a bug in the hub itself.

Choosing a session lifetime nobody guessed at

The idle and absolute session timeouts went through three revisions in two weeks, and the sequence is a better story than any single number would be. The first policy — one hour idle, twelve hours absolute — sounded reasonable until it meant every editor was being signed out once a day regardless of when they'd started working, since twelve hours from a 9am sign-in lapses mid-afternoon rather than at a natural break point. PR #87 widened the absolute limit to thirty-six hours specifically so a single morning sign-in survives through the end of the following working day — a deviation from a published security guideline's recommended twenty-four-hour ceiling, documented explicitly as a deliberate, justified exception rather than an unexamined default.

That still wasn't the end of it. Production logs showed editors being signed out mid-session for reasons the hub's own policy couldn't explain — PR #100 traced it to the identity provider's own separate inactivity timeout, which was returning a specific "session expired (inactivity)" error hundreds of times over three days while its own dashboard reported the setting as disabled. PR #102 found the actual root cause: an earlier attempt to configure that timeout had been applied as a hot-reload the provider's own config-loading code can't actually unset once a value is set, and the dashboard reporting "0" meant "unchanged from whatever's still running," not "off." The real fix was setting the provider's own inactivity timeout to a value functionally equivalent to "never" and documenting, in detail, why that field should never be set to zero expecting it to mean disabled. PR #96 then landed the actual final numbers — twelve hours idle, forty-eight hours absolute — chosen not arbitrarily but specifically to match the identity provider's own session limits, so the platform's policy and the provider's policy tell the same story instead of two different, silently conflicting ones.

Revision	Idle	Absolute	Why it changed
Original	1h	12h	Signed everyone out daily, at an arbitrary time
PR #87	1h	36h	One sign-in should survive to the end of the next working day
PR #96 (final)	12h	48h	Matched to the identity provider's own actual limits

Four reasons a session ends, and none of them were being logged

PR #103 is a small PR carrying a large point: a session on this platform can end for four genuinely distinct reasons — idle timeout, absolute timeout, a domain check failing, or an account being disabled — and until this PR, none of the four were logged. One morning's cluster of idle sign-outs had to be reconstructed after the fact from the identity provider's own auth logs, because the application itself had no record of what it had done or why. The fix is purely additive logging with no behavior change, which is exactly the right scope for this kind of gap — you don't need a new mechanism to end sessions correctly if the four mechanisms already exist and just aren't visible. PR #101 fixed a sharp edge in that same logging: the new log line was reading the session owner's id from a cookie jar after the very refresh failure that deletes that cookie, so every production log line it had produced so far recorded no user at all. Reading the id first, before the operation that invalidates it, fixed a bug in the fix.

The small bugs a real deploy cycle finds

PR #98 fixed sign-out breaking for anyone who had a tab open across a deploy — the sign-out action was implemented as a framework server action, and a server action's identifier changes with every build, so a stale tab's sign-out request simply 404'd. The fix routes sign-out through a stable URL instead, hardened to accept only same-origin POST requests so a stray link or prefetch can't trigger it. PR #99 preserved the query string through the sign-in redirect, so a deep link into a specific filtered view survived a forced re-authentication instead of dumping the user back at a bare top-level page. PR #104 closed out two silent failure paths found in a late-night log review: a proxied request that failed at the connection level with no logging at all, fixed with one same-method retry for safe, idempotent requests only, and a cold-start race where concurrent first requests after a deploy each tried to fetch the same cryptographic key set independently, producing a false key-mismatch error that a shared, de-duplicated fetch eliminates entirely.

The pattern: a session isn't one event, it's a lifecycle with four exits

Every fix in this cluster traces back to the same realization stated different ways: "the user is signed out" is not one event. It's at least four distinct events — ran out of activity, ran out of absolute time, left the company's identity boundary, or had access explicitly revoked — and each one needs its own correct detection, its own correct cookie cleanup, its own propagation to every layer that enforces access, and its own log line explaining which of the four just happened. Treating session termination as a single undifferentiated "logged out" state is exactly how a disabled account keeps working for twelve more hours, how an idle timeout silently never fires, and how a genuine security incident becomes a mystery nobody can reconstruct from the one bit of information the system bothered to record.

Forty Hand-Rolled Variants Retired: A Design System Consolidation for an Internal Ops Platform

Humza Tareen — Mon, 12 Oct 2026 10:30:09 +0000

An internal ops platform I work on had never had a design system, and it showed in specific, countable ways: thirteen primary-button call sites across seven distinct class strings, eleven tables with six different header conventions, a type scale with sixteen arbitrary pixel values, and — the one that actually mattered — roughly half the form controls in the app with no visible keyboard focus indicator at all. Over about twenty-four PRs, the fix was deliberately boring: lay down tokens, build a handful of primitives on top of them, migrate every screen one at a time, and write a lint rule that makes the drift mechanically impossible to reintroduce.

Six PRs of unification are worth nothing if the next screen hand-rolls an eighth primary button. The guard is not the cleanup; the guard is the point.

Tokens first, components second

The first PR restyled nothing. It laid down surface, text, line, brand, and status color tokens as CSS custom properties, each one annotated in the source with the exact contrast ratio that justified it — and fixed every accessibility failure that was measurable without touching layout. The sharpest of those: globals.css ended in a single rule, button:focus { outline: none; }, and only about half the form controls had ever put a ring back. Every button and checkbox in the tool had no visible keyboard focus indicator at all, a WCAG 2.4.7 failure sitting in the global stylesheet the entire time. It was replaced with one :focus-visible rule in the base layer, so components that already carried their own ring kept it and everything else inherited this one for free.

Forty variants, four primitives

With tokens in place, Button, Badge, EmptyState, and Field landed together and immediately retired the worst of the duplication: thirteen primary-button sites, five icon-button variants, the app's only purple button, and disabled-state opacity that had drifted across four different values. Button applies tone — default, success, critical — orthogonally to hierarchy rather than as an extra rank, because a destructive action can just as easily be a dialog's main action as a quiet row action, and a rank can't express that. Field replaced a shared class string that four different copies had each drifted slightly from, and switched its border from a decorative gray that measured 1.47:1 to one that clears the 3:1 minimum WCAG 1.4.11 sets for a border that's how you identify a control — fields are visibly darker now, and that's the fix, not a side effect.

The component tests caught the author's own mistakes twice in this one PR: Button's critical tone hardcoded a raw red hover shade — exactly the raw-palette-for-meaning the token layer exists to prevent — and a test asserting no raw palette class in the component failed on its own first draft. The same slip had already shipped one PR earlier, on the tokens PR's own success-tone buttons, and got caught the same way.

A bug the screenshot caught and the source code didn't

The next PR unified four different tile treatments — different radii, different label sizes, two different label-to-value orderings — into one StatTile, and in doing so hit a bug that no amount of reading the diff would have surfaced: a tint prop was meant to layer on top of a tone, but both supply a background color, and CSS resolves the conflict by stylesheet order, not by the order the two class names were concatenated in the source. The code read correctly. On screen, four of five category tiles rendered with no tint at all. It was only visible by actually rendering the component against the real compiled CSS and comparing it to a screenshot — which is exactly how it was found, and exactly the discipline that runs through this entire project: build it, then look at it, because looking at the diff proves nothing about what a browser does with cascading styles.

Tabs that were never tabs

A table-spec PR found that tabular-nums existed in exactly one of eleven tables, so numeric columns visibly shifted on every refetch everywhere else — the precise problem one table's own code comment claimed to have already solved. A segmented-control PR then applied a sharper distinction: by the standard rule that tabs navigate between distinct content areas while a switcher toggles alternate views of the same content, five of six "tab" strips in the app weren't tabs at all — they were filters dressed as navigation, with zero ARIA semantics and zero keyboard support between them. SegmentedControl is a radiogroup, not a tablist, and the new tests pin that distinction specifically: that it renders no tabpanel, exposes exactly one aria-checked, and that a group of one option renders nothing at all, because a group of one isn't a choice.

A guard that proved it could fail, in both directions

Six PRs of unification are only as durable as the thing stopping the seventh hand-rolled button. The lint rule that landed next fails the build on any raw <button> or <table> outside four explicitly-named, explicitly-justified exceptions — and the PR's own verification is the part worth noting: a raw button and a raw table were each deliberately reintroduced to confirm the rule actually fires, then reverted to confirm it passes again. The first draft of the guard had exempted an entire directory, which also accidentally exempted the three shared components that most needed the rule to apply to them. Run for the first time, it immediately found four raw tables that six prior PRs of "done" migration work had simply missed.

Verified, not assumed

A small /api/version endpoint shipped next, deliberately public and deliberately boring, for one reason stated plainly in the PR: every "deployed ✅" claim up to that point had been an inference from the service still answering, which it would also do before a bad deploy. Within a minute of the first production deploy, the endpoint itself disproved a claim its own author had just written in the same PR — that the host exposed no git commit variable, so the field would always read null. Production returned the exact merge commit. The author had checked the wrong surface: a CLI command that lists user-defined environment variables, not the actual runtime environment a deployed process sees. The fix was a one-line doc correction, and the honest write-up of being wrong about your own infrastructure in the same week you shipped it is a better tell for engineering maturity than getting it right the first time would have been.

Two outages the migration itself caused

The riskiest part of a consolidation like this isn't the components — it's moving them across a server/client boundary that doesn't forgive a wrong guess. A route split that made every manager module independently linkable also, in the same PR, took down every single one of those routes in production: a helper function was exported from a file carrying "use client", and in the App Router, every export of a client module becomes a client reference on the server, including a destination that's a plain exported function. A server component that calls it isn't invoking a function anymore — it's holding a reference, and the render throws. Four hundred and some passing tests said nothing, for a specific, structural reason the author wrote up rather than shrugged off: the test suite renders everything as a client component, so there's no server/client seam in the test environment for the bug to cross, and the build step never executes a dynamic route at build time either. The local gate was architecturally blind to this exact class of bug.

The same mistake recurred one PR later in a different shape — a const string exported from a client module, not a function, breaking /login and all five admin panels — and the scanner written specifically to catch the first occurrence missed the second, because it only looked for called functions, and a string is read, never called. The fix wasn't the component; it was the detector, rewritten to flag any lowercase import from a client module regardless of how it's used, and to walk the actual server-reachable file graph instead of trusting the presence or absence of a directive. Worth saying plainly: the fix for /login was reported as resolved once already, on the strength of a working deploy and clean route sweep, before the root cause — a React render prop crossing a boundary it can't cross — was actually found by loading the page and reading the error.

Eighteen silent bombs

Admin was the single riskiest surface in the whole migration, with zero render-test coverage across sixteen files, because the whole area is server components with server actions that fetch on render — nothing in the test harness can mount them. Eighteen of its twenty-one buttons were bare elements inside a form relying on the browser's implicit submit-on-click behavior. The shared Button primitive deliberately defaults to a non-submitting type, specifically so a button can't accidentally submit a form — which meant migrating any of those eighteen without also adding an explicit submit type would silently stop the form from doing anything, with no error, on a diff that reads correctly line by line. The guard for this was written and proven before a single button moved. It's also where the project's most repeated lesson shows up explicitly three separate times across three different PRs: a verification script that prints a success message instead of asserting a precondition can report a clean pass while having checked nothing at all, and the fix each time was the same — stop trusting a quiet result as evidence, and confirm the check can actually fail before trusting that it can pass.

What rendering caught that the Tailwind sweep couldn't

A tracker-migration PR found a progress-bar color applied through an inline style rather than a class — invisible to every one of the prior color-token sweeps, because none of them touch inline styles, and it had survived as roughly 2.4:1 contrast the whole time. The same PR fixed a delete-confirmation dialog with no dialog role, no focus trap, and no focus return — Tab walked a keyboard user straight through it into the page underneath, and Escape did nothing. And a follow-up PR closed the gap in the lint guard itself: color had never had a checker at all, which is exactly how over a hundred raw palette classes survived an earlier "completed" pass. Once written, that checker immediately found a single UI state — "AI usage: Major" — rendered in three different colors across three different files, one of which was dead code reachable from nowhere, from a decision nobody could any longer explain.

Safari rings every clicked button, and nobody can test that from a laptop

Fixing the original missing-focus-ring bug correctly for Chrome accidentally broke it for Safari: Safari's native :focus-visible heuristic matches after a mouse click, where Chrome's doesn't, so identical CSS produced two different behaviors depending on which engine rendered it. The fix stopped trusting the browser's heuristic and tracked the last input modality explicitly, defaulting to "keyboard" so a tracking failure fails toward more accessibility rather than less. The honest limit, stated directly in the PR: there's no browser automation installed in this environment and a page script can't produce a trusted click event, so the one thing that actually needed verifying — does a real mouse click in real Safari produce a ring — could not be verified by the person who wrote the fix. It shipped flagged as needing a human to physically click a button after deploy, rather than quietly marked done.

The loading skeleton that forgot where it lived

The very next PR was a regression from the one before it: clicking into Admin or the Tracker rendered a full-bleed wall of loading bars with no sidebar and no page margins, while the one section whose layout had already been restructured during the route split looked correct. A route's loading.tsx replaces the page but leaves the surrounding layout alone — and in the two sections that hadn't been touched yet, the shared shell component lived inside the page itself, so the skeleton took the chrome down with it. The fix was caught, by the author's own account, specifically because it was verified against a running production build rather than an isolated component harness — the exact check that had been skipped the first time, and the one that mattered.

The pattern: a consolidation is a migration, not a cleanup

Every one of these production incidents happened because the consolidation was working as intended — pulling shared logic into fewer places is precisely what makes a server/client boundary violation in one of those places take down every screen that now depends on it, instead of just the one screen that used to hand-roll its own copy. The lesson that recurs across this whole arc isn't "write more tests." It's that a check which can pass vacuously is worse than no check, because it gets quoted back with confidence — and the fix, every time it happened, was the same one: before trusting a green result, go break the thing on purpose and watch the check turn red.

Config-Driven Gate Thresholds and an Analytics Layer for an Enterprise Review Pipeline

Humza Tareen — Mon, 12 Oct 2026 10:29:35 +0000

On the enterprise workflow platform, two separate quality gates — External QC and Final Review — both decided pass or fail off a single hardcoded bar. Every category of feedback a verifier could raise (task design, source data, prompt, verifiers, golden data) was judged against the same binary cutoff, and the two gates had no way to disagree with each other even when the work they were reviewing called for it. AGT-1774 replaced that with per-category, admin-editable thresholds for both gates independently. The feature itself is a month of work. What makes it worth writing about is that it got built twice — once as four separate PRs, and then again, from scratch, as a five-PR stack, because the first version hit a wall that had nothing to do with the code.

A quality bar is a policy decision, not a code constant. The month this took wasn't about computing a threshold — it was about building the plumbing so that decision could live in an admin UI instead of a pull request.

Build one: four PRs, shipped in order

The first pass landed in a straightforward sequence. PR #1028 extended the existing External QC config infrastructure with an optional gateThresholds field — per-category passing percentages stored as JSON on the existing config row, deliberately reusing the existing table rather than adding a migration. PR #1029 replaced a binary isVerifierFeedbackItem classification with a five-category classifier and a pure evaluateThresholdDecision function that groups feedback by category, computes a per-category pass rate, and compares it against the configured number — mirroring a scoring pattern (computePostqaScore) that already existed elsewhere in the codebase rather than inventing a new shape. PR #1030 did the same for Final Review specifically, adding a loadGateThresholdConfig helper with per-field fallback to sane defaults. PR #1031 put an admin UI on top: five threshold input fields, reusing the existing config fetch/update pattern with no new endpoint.

That's a complete, working feature in four PRs. It also had a commit with an unsigned signature somewhere in its history, which blocked the repository's verified-signatures branch-protection rule from letting it merge.

Build two: the same feature, five PRs, properly signed — and better architected

Rather than hunt down and re-sign one commit in an otherwise-merged history, the whole stack got rebuilt. PR #1154, explicitly labeled "replacement for #1027," is stack position one of five: the core terminal/rework lifecycle, durable completion recovery, and shared gate contracts — threshold configuration and its UI deliberately deferred to later PRs in the stack. #1105 (replacing #1028) added independently persisted thresholds for both gates with atomic config updates. #1106 (replacing #1029) applied the shared evaluator to External QC specifically, with decision consumption for Final Review intentionally left out. #1107 (replacing #1030) applied it to Final Review, with admin editing intentionally deferred again. #1108 (replacing #1031) closed the stack with the admin UI and a canonical cache-invalidation fix.

The rebuild isn't just the same four PRs re-signed. Splitting "add thresholds to External QC" and "add thresholds to Final Review" into two sequential PRs that each explicitly say what they're deferring is a more disciplined decomposition than the first attempt's four roughly-feature-sized chunks — each PR in the second version has a narrower, independently-testable claim, and the stack notes spell out exactly which PR depends on which. A signing problem is a bad reason to redo a month of work. A redo that uses the occasion to ship a cleaner decomposition than the original is the only good version of that story.

The redirect bug that keeps needing the same fix

Threaded through both builds is a recurring bug shape worth calling out on its own. An earlier fix (AGT-1760, covered in a previous post) made a golden-data rework redirect fire only for its actual source version, not every older version. PR #1024 found that guard still fired one hop too eagerly — clicking the immediate predecessor version in a dropdown redirected forward again, because nothing distinguished "this session is the one that just triggered the fork" from "a user navigated here on purpose to inspect it." PR #1180 found the identical bug shape on External QC's own version dropdown, independently, and fixed it the same way: a prevManifestVersionRef plus a pure shouldFireExternalQcRedirect helper, explicitly modeled on the golden-data fix rather than reinvented. The same underlying mistake — conflating "a transition just happened" with "the user wants to leave" — showed up on two different gates built by different people at different times, which is a reasonable argument for making the transition-gate pattern a shared primitive rather than something each gate re-derives.

A false "Submit failed" from a schema that wasn't lying, exactly

PR #1018 traced a confusing failure: the golden-data finalize route could return status: "qc_in_progress" without two fields — effectiveVersion and reworkVersionCreated — that the client's response schema required on every branch of the union. The server had genuinely succeeded; the client's own validation rejected the honestly-formed response before the UI ever got to react to it, so the panel never transitioned to the QC view and the user saw "Submit failed" for a submission that had, in fact, succeeded. Schema validation did exactly what it's supposed to do here — reject a response that didn't match the contract — the actual bug was that the contract was stricter than what one legitimate server branch produced.

Building a real analytics layer, not a bigger spreadsheet

Alongside the threshold work, PR #1167 added a Task-Level Metrics tab to the task-detail page — and per direct stakeholder feedback during review, the original plan to show raw-data tables was replaced with interactive charts before it shipped, because tables of per-step durations are much harder to spot a trend in than the equivalent line chart. PR #1097 built the top-level admin Analytics dashboard separately, deliberately sourced only from the existing rl-world API and GCS manifests — "zero v0 code remains" — rather than adding a new direct-to-database query path that would have had to be maintained as a second source of truth.

PR #1170 is the bug that would have quietly undermined both: the step-timing charts derived their list of steps and versions from GCS step_started/step_completed events, and when those events were missing or incomplete, the charts rendered empty — "no step timing data available" — even on tasks where the more authoritative rl-world API had the real timing data the whole time. A first attempted fix unioned both sources together, which meant a step that only ever existed in sparse GCS events still forced its way into the list and diluted the result. The real fix makes rl-world the primary source outright, falling back to GCS events only as a last resort — a priority order, not a union, because one of the two sources is simply more trustworthy than the other and averaging them together doesn't produce a better answer.

Smaller fixes that are really about who's being shown what

PR #1178 found the Final Review gate's confirmation modal collapsing a verifier.json artifact holding nineteen individual per-verifier results — eight approved, eleven rejected — into a single row reading "1 Rejected · 0 Approved · 0 Incomplete — 1 artifact." The modal's job is to tell a reviewer what they're about to approve past; showing one artifact instead of nineteen decisions is technically accurate about file count and completely misleading about what's actually being waved through. The fix expands aggregation to count individual verifier results rather than the artifact file that happens to contain them.

PR #1169 closed three separate permission bugs — a visible Clone Task button, an accessible Events tab, and a reachable Force QC Pass action — all under platform-admin impersonation of a lower-privileged role, and all three traced to the exact same root cause: the client's auth context tracked that impersonation was active but never tracked which role was being impersonated, so every permission check kept reading the real admin's role underneath. One shared fix (threading the impersonated role through fetchUserProfile via authFetch) closed all three, which is the useful signal that it was one bug with three symptoms rather than three unrelated ones — bugs findable only by actually testing as an impersonated user, not as yourself.

A smaller cluster rounds out the access-and-trust side: PR #1187 blocked reassigning a reviewer-step task to a role explicitly ineligible to hold it — a gap on the older "classic" reassign route that a newer route had already closed, found only because the two routes had drifted apart. PR #1188 widened the new analytics dashboard's access from platform admins only to standard admins too, specifically scoped to analytics and nothing else, because the dashboard's own access model and the platform's broader admin-tier model hadn't been reconciled when the dashboard first shipped. PR #1182 made a post-claim run lookup fail loudly instead of swallowing a transient GCS error as "not found" — the quiet version of that bug could have wiped an already-approved review off the back of a storage blip that had nothing to do with the review's actual status. And PR #1181 fixed a dropdown option plainly mislabeled "Block approval for rework" in a context — Final Review — where rework was never actually a possible outcome; a small fix, but exactly the kind of thing that erodes an admin's trust in every other label on the same screen if left uncorrected.

The pattern: a quality bar only works as policy if it's actually editable

The headline feature here — admin-editable thresholds instead of a hardcoded cutoff — is the kind of change that sounds almost too simple to take a month. It took a month because making a number configurable is the easy 10%; the other 90% is every place downstream that assumed the number would never change, every version-navigation path that needed to agree with every other one, and an analytics layer built at the same time that needed its own data sources prioritized correctly rather than averaged. None of the individual fixes in this cluster are large. What they have in common is that each one is a place where "the system does roughly the right thing" and "the system does exactly the right thing for the person relying on it" turned out to be different claims — and closing that gap, repeatedly, is what actually shipping a policy-configurable gate costs.

Seven Flips: A CI Gate That Could Never Go Green, and What a Dark Month Cost

Humza Tareen — Mon, 12 Oct 2026 10:29:30 +0000

A CI gate that is sometimes red and sometimes green looks, at a glance, like a flaky test. On the main video-generation service, the gate spent roughly a month doing something different: it kept getting refused by GitHub itself, for reasons that had nothing to do with the code under review, in a pattern specific enough that the team started numbering the incidents. By the time it was over, they'd counted seven flips — and the honest accounting of what that month actually cost turned out to be as valuable as the fix. Alongside it, a separate but related cluster of bugs was quietly letting single paid renders run far longer, and far more expensively, than anyone intended.

A flaky gate you can ignore case by case. A gate that is unsatisfiable by construction will train everyone who touches it to stop trusting green, which is worse than training them to ignore red.

The shape that isn't a test failure

The first thing worth understanding is that most of these "flips" weren't CI failing — they were CI never starting. PR #1171, the fifth recorded instance, captured the exact signature: a Merge Gate job reporting conclusion=failure with steps=0 in three seconds flat, Build and Unit Tests both silently skipping, and an annotation reading "the job was not started because recent account payments have failed or your spending limit needs to be increased." That's a billing failure on the CI provider's account, surfacing as a red check indistinguishable at a glance from a real test failure — unless you know to look at the step count. A job with zero steps didn't run long enough to fail anything; it was refused before it started.

The team's own fix for telling the two apart wasn't a script — it was a paragraph in CLAUDE.md instructing every future reader to distinguish the shapes: a real failure has steps and a duration; an account block has neither. That paragraph needed updating five separate times as the state kept recurring, and each update recorded the "cleanest specimen yet" so the next person wouldn't have to re-derive the diagnosis from scratch.

A gate that was unsatisfiable, not merely unlucky

The sixth flip, PR #1175, is the one that mattered most, because it found CI finally executing for the first time since September 1st — and discovered two real conditions sitting underneath all the noise, neither one a regression in the code being tested. The first was a build step running under three different names across two workflows, all hitting the exact same out-of-memory ceiling; patching one name by itself had left the other two silently broken. The second was sharper: a test suite that drove real data from a directory where exactly one file was actually tracked in git. On a CI runner, that directory is empty by construction, so the test's own guard against a vacuous pass had been failing — correctly — since an earlier PR, number 1011, with nobody connecting the dots that the test had never once passed in CI.

The fix doesn't relax the test. It separates "the corpus directory is absent" (skip loudly, and say explicitly that a skip is not evidence of anything) from "the corpus directory is present but too short" (fail, exactly as before). That distinction — a permanently red guard teaches people to skip it, but a guard that goes vacuously green teaches them to trust a lie — is the same discipline a dead-man's-switch needs anywhere: silence and success have to be distinguishable states, or the gate stops being a gate.

Flip	What actually happened
Fifth	Account-block signature finally pinned down precisely, by shape
Sixth	First green run since Sept 1 — found the real OOM and vacuous-corpus bugs underneath
Seventh	A partial block — only the final summary job refused, four real jobs upstream had already passed

Getting the diagnosis wrong, in writing, on purpose

The seventh flip, PR #1221, is the most interesting entry in this whole saga because the PR description documents its author's own mistaken first read. The block landed mid-run, so four real jobs — lint, build, unit tests, and the merge gate itself — had already started and completed successfully before a fifth job, the PR summary, was refused with the same billing annotation. The author's first instinct was: the other jobs ran for fifteen minutes, so the account can't actually be blocked. That reasoning is wrong whenever a block arrives partway through a run rather than at the start, and the PR corrects it in the same document that contains the mistake, specifically so the next person doesn't re-derive the same wrong conclusion. A partial block, it turns out, is also the cheapest version of this failure to live with: the substantive gate genuinely ran and its green result is real evidence — only the final summary comment is missing, which calls for naming the refused job rather than writing off a mostly-green run as entirely red.

PR #1176 and #1224 close the loop with the part of this story that's easiest to skip: a direct accounting of what a month of unreliable CI actually cost. Three separate breakages — the memory ceiling, the vacuous test corpus, and a duplicate YAML key that silently invalidated an entire workflow file in a way that never showed up in the normal gh pr checks output — had all been accumulating underneath the account-block noise, invisible precisely because the account block was the more obvious, more dramatic failure mode sitting on top of them. #1224 goes further and tallies real numbers: 616 merged PRs since the last clean baseline, cross-checked against the actual dev/main branch divergence (zero, despite an earlier assumption that dev was lagging) and a re-priced cost line in the roadmap docs that had been wrong by roughly 20% because nobody had gone back to update a constant after a vendor's real billed rate was measured.

The other half: runaway cost from processes that never die

A parallel cluster of fixes this cycle tackled a different flavor of the same root problem — a safety mechanism that can't fire because the failure it's guarding against doesn't look like a failure. PR #1109 found that an Anthropic streaming call had no bound on total output: max_tokens is where the model stops, but the code was draining the entire stream via finalMessage() before ever checking the stop reason. Measured directly on two real jobs: a healthy response used 6,254 output tokens; a runaway one, given the identical prompt, generated 64,000 tokens over eight minutes before being caught as truncated and thrown away — all of it already billed. The fix bounds total output tokens directly and aborts the stream early, stopping the runaway "at under a third of the price" instead of waiting for it to finish failing.

PR #1117 found a related but distinct bug: a vision call that stalled completely — zero output, zero error — for 75 minutes, holding a render open that had already cost $37.87 in rendered clips. It never threw, so every .catch written around it was inert — a hang isn't an exception, so exception-handling code can't see it. The fix bounds idle time rather than total time, specifically so a legitimately long generation that's still producing text is never cut short, while true silence trips a watchdog. PR #1118 found the same failure mode one layer down, in an ffmpeg child process: a stitch that could hang forever is, in the PR's own framing, "an ad that never ships" — not a metaphor, a literal description of the failure. And PR #1119 is the fix that made the other two trustworthy: the kill mechanism for a stuck process was firing silently, leaving no trace of what it had killed or why — which meant diagnosing a slow render meant re-discovering the same root cause from scratch every time. Three separate bugs were investigated and ruled out by actual measurement before the real one, a 30-second-per-panel contact-sheet render wedging ffmpeg past roughly 30 panels, was found — because the first local repro used a cheap, downscaled test image that made an expensive filter graph look fast.

The pattern: a safeguard only works if its trigger condition can actually occur

Every failure in this cycle — the CI account-block, the vacuous test corpus, the inert .catch blocks, the silent kill — shares the same shape. Each one had a safety mechanism already in place, written by someone who had reasoned correctly about the failure they expected. What none of them accounted for was a failure mode where the triggering condition itself never arrives: a job that never starts isn't a failed test, a hang isn't a thrown error, a silent kill isn't a kill anyone can learn from. A safeguard is only as good as its ability to detect that it should have fired at all — and the fix, every time in this cycle, wasn't a smarter retry or a bigger timeout. It was making the absence of a signal into a signal of its own.

Billing What the Vendor Billed: A Cost Audit That Found a Tracker Recording $0 for Its Entire Life

Humza Tareen — Mon, 12 Oct 2026 10:28:53 +0000

Langfuse had ingested 31,321 observations across 1,315 traces on the main video-generation service. Every model name, every token count, every prompt label, every parent span — all correct. Every cost field: null. The dashboard that was supposed to answer "where does the money go" had never once answered it, for the entire time it had been running, and nobody had noticed because a dashboard reading $0 doesn't look broken. It looks like a quiet day.

A cost instrument that silently reads zero is not a quiet day. It's an outage in the one system whose entire job is to make outages like this visible.

The tracker that never worked

PR #1177 found two defects, and they had to land together because each one hid the other. Langfuse reserves the key total inside its costDetails object as the aggregate it actually sums and displays — any other key name gets stored as a separate custom bucket that the dashboard never reads. The pipeline's cost-reporting code had been writing to a differently-named key the entire time. Separately, the cost was being attached to the wrong span in the trace tree, so even a correctly-keyed value wouldn't have rolled up to the totals the dashboard shows. Two independent bugs, same symptom, and fixing only one would still have shown $0.

The scale of what that blindness cost is the real headline. Two separate runs made 1,555 and 844 high-resolution vision calls each, against a per-run median of seven. Nothing alerted, because the one system built to notice spend anomalies had never been capable of noticing anything. Meanwhile a separate, simpler cost-notes mechanism had quietly recorded the true number for the same 22-day window: $1,191.85. The money was never actually untracked — it just wasn't visible anywhere a human would look.

36% of the bill, reporting $0 on every call

PR #1177 fixed the bug class for Anthropic and ElevenLabs calls. PR #1185 found the same bug class had never been checked at all on fal — the single transport behind every Kling clip, lipsync pass, multi-shot bundle, split, b-roll animation, preview, and voice-create call in the product. A grep for the string "usd" anywhere in the fal integration code returned nothing. Measured against the honest cost-notes ledger for a single day — 15,170 notes, 181 runs, $1,227.51 total — fal clips alone were $443.58, or 36.1% of everything spent that day. Reported in Langfuse: zero, on every single observation, since the day #1177 shipped and the team believed the dashboard had been fixed.

That's the sharpest lesson in the whole cluster. Fixing a bug class in two of three places and believing it's fixed everywhere is itself a kind of bug — the fix needs to be verified against every vendor integration independently, not assumed to generalize just because the root cause was the same shape.

A retry that was already right, and a ladder that billed for one

PR #1179 found a vision call that retries three times whenever it comes back with an empty result — reasonable, except that on a scene whose plan never placed a product in frame, an empty box set is the correct answer, not a failure. Two of the three retries were buying the same correct nothing again, at roughly 2,981 high-resolution input tokens each, across 2,796 calls measured over 26 runs. The retry logic had no way to distinguish "the call failed" from "the call succeeded and the honest answer is nothing."

PR #1180 found the inverse failure shape on a different feature: a provider fallback ladder designed to represent multiple distinct paid attempts, where the billing call sat below every quality check on the result. A generation that failed its checks after being purchased fell back to per-scene generation — paid for once, billed for zero, then billed again as new per-scene clips. Measured impact today: $0, because the feature defaults off and the failure path hadn't fired yet in production. The PR ships the fix anyway, on the reasoning that an unmeasured bug costing nothing today is exactly the cheapest moment to close it — before the feature is on and the bug has a real dollar figure attached.

A name that meant something different from what it said

PR #1181 is a small bug with an expensive incident attached. A setting called CLIP_REDOS_PER_RUN reads as a once-per-run budget. The function that resets it actually ran at the end of every stitch pass — and re-stitching is a normal, frequent operation, triggered by a lip-sync pass, an editor's scene redo, or an editor's clip redo. Every one of those re-armed the budget from scratch. On one measured incident, a 21st clip finished rendering at $32.20; the run was cancelled 70 minutes later at $42.97, having silently re-run its entire stitch sequence a second time — three clips the budget existed specifically to refuse, purchased anyway because its name promised a guarantee its implementation didn't keep.

The $290 line item hiding inside a label nobody read

PR #1186 is the best example in this cluster of a cost leak that was visible the whole time, in a number nobody had looked at closely. anthropic:ugc-continuity was the second-largest Anthropic cost line in the product at $290.87 across 49 runs — and it should, by design, cost roughly one call per scene. The call's own label happened to carry an exchange number, which meant the ledger had been recording exactly how many times each run re-authored its continuity context, in plain sight, the entire time. Of 49 runs, 25 reached exchange 25 or higher on plans with only 10 to 17 scenes — and those 25 runs accounted for 81% of the entire line item. Worse, every single one of 27 measured runs showed calls getting about 1.85x more expensive as the exchange count climbed, because the context being resent grows with every exchange. The fix wasn't exotic — it was reading a number the system had already been recording and asking why it kept climbing instead of resetting.

That finding fed directly into one of the more interesting reversals in the cluster. An earlier decision had marked an append-only version of the same continuity mechanism DO-NOT-RETRY, on the belief that resending the full growing context at a 1.25x premium was obviously worse than periodically trimming it — a roughly 40% win, on paper. PR #1184 measured it for real on a specific fork and found the opposite: $2.54 for five calls under the trimmed approach against $0.36 under append-only, because the trim sends roughly 14,375 tokens per call against append-only's ~100,000, but the frames being authored five to eleven minutes apart meant the expensive long-context prefix was being paid for and then expiring, unread, before it could be reused. A follow-up measurement a week later (PR #1208) made it official: append-only continuity is 27.5% cheaper overall, and the prior DO-NOT-RETRY was overturned by the same kind of direct measurement that had originally justified it.

One timestamp was fine; the call next to it was not

PR #1188 started from a one-line operator question — "I turned off bbox stamping completely, why is it still here?" — and the honest answer was that the feature the operator turned off was in fact off, correctly, and contributing zero calls. The spend was coming from a different, default-on feature that happened to share a code path: a product-consistency check that locates the product in every identity panel to compare color across scenes, changing no pixels but running the same expensive vision call. Two separate features, one expensive call between them, and only one had ever been switched off. PR #1187, investigated in parallel, found a related coordinate bug: a Claude vision tier enforces two separate limits — a long-edge cap and a total visual-token cap — and the bounding-box math had only ever respected the one the team happened to have coded for, silently downscaling images wrong whenever the untested limit bound first.

Finishing the instrument, and correcting your own last round

PR #1189 closed three smaller holes specifically so the dashboard could be trusted, not just read: speech-to-text calls reporting $0 on every span despite having carried a real charge since #1177 (fixed carefully, as a cost-only note rather than a normal charge, since double-billing the ledger would have been a worse failure than the original gap); two routes that had never been traced at all; and a usage counter that structurally could never fire. PR #1190 found the single largest uncached Anthropic line in the product — a clip critic re-sending the same two product reference photographs, unchanged for the whole run, on every one of 4,724 calls, at roughly 2,350 of each call's 5,264 input tokens.

And PR #1191 is the honesty check the whole cluster runs on itself: a documentation pass that had to correct four claims, three of which the same author had written the round before — including a six-row vendor pricing table where the last three rows turned out to just be the first three doubled, cited as its own evidence for being credible. Measuring your own prior claims against reality, on a fixed cadence, catches the kind of error that confidence alone never will.

A sibling tool, the same discipline

The measurement habits built here traveled. On a sibling AI-presenter tool, a later audit (presenter-generation PR #29) found that every default clip was being priced 2.33x too high — a Kling v3 voice bind riding on an "element" was being charged the element-rate multiplier on top of the voice rate, based on a constant explicitly commented "NOT YET CONFIRMED AGAINST AN INVOICE." The correction came directly from this cluster's work: PR #1194 on the main service had already found that fal returns an x-fal-billable-units header on every response, and that fal's own documentation says that header is authoritative. Once one tool in the family learned to read the vendor's own receipt instead of a hand-transcribed rate table, the fix for a sibling tool's pricing bug was a lookup, not a new investigation.

The pattern: the instrument can fail exactly like the thing it measures

Every bug in this cluster is a variant of the same failure: a cost signal that was computed, or should have been computed, and didn't reach the place a human would look at it — a wrong key name, a call site that forgot to pass a charge, a label nobody read closely, a feature flag that looked like it governed the spend it didn't. None of these are exotic bugs. They're the same class of wiring failure that shows up anywhere a value is produced in one place and consumed in another, except here the value is money and the consequence of it silently vanishing is that the team spends weeks believing a system has been priced and verified when it's never once actually been checked. The fix isn't a smarter dashboard. It's treating the dashboard itself as a thing that can be wrong, and auditing it against the vendor's own invoice on a schedule — not once, and not assuming a fix for one vendor integration generalizes to the next one before you've actually measured it.

One Brief Instead of Four: Unifying Create Flows Into a VSL Generator and an AI Presenter

Humza Tareen — Mon, 21 Sep 2026 18:13:30 +0000

The main video-generation service had accumulated four separate ways to start an ad: clone an ad, original from script, video sales letter, and — briefly — a fourth. Each had its own page, its own header, its own place in the sidebar. They looked like four products. Underneath, they were four thin page files sitting over one shared wizard component, and the words that told them apart never made it past the page layer into the code that actually planned and rendered anything. Collapsing that into one brief-driven flow, and then building two genuinely new product surfaces on top of the result, took about three weeks and touched navigation, script splitting, sample pricing, and — closing the loop — the first real finished output the new flow ever produced.

A chooser that asks a question whose answers converge is not a choice. It is a detour.

Four routes, two runs

PR #993 found the tell before it fixed anything: videoTypeForCreateMode mapped two of the four entry points — "AI Presenter" and "video sales letter" — to the exact same short_vsl job tag. Different headers, different marketing copy, byte-identical jobs underneath. The fix collapsed the chooser to one card, with the four original types demoted to a picker inside a single brief: storyboard, UGC video, VSL, or reference recreation. /create/vsl became the only page that actually mounts the form; the other three routes now just redirect into it. The pipeline itself — /api/wizard/* — wasn't touched at all, which is the property that made the change checkable: nothing about how a job renders was supposed to change, only how many doors led to it.

A second card, justified by a different job

PR #994 is the interesting counter-move to #993, and it's worth sitting with the order these two shipped in: the very next day. A stakeholder asked for a dedicated AI Presenter wizard — one person to camera, no reference video, none of the fancier modes — in the same conversation that asked to delete the "no reference" button from reference recreation. The PR's own comment names the standard #993 had just set: a chooser that converges to the same job is a detour, not a choice. So the new card only earns its place by producing something the merged flow didn't already produce — a distinct videoType: "ai_presenter" rather than a second front door to short_vsl. Two products that looked identical a day earlier now diverge by design, not by accident.

Eight back buttons, none of them the same

The AI Presenter card shipped with an unglamorous bug: no way back to the chooser screen. PR #995 could have patched that one path. Instead, asked to audit the whole tool's navigation rather than add a ninth arrow, it found eight hand-rolled back treatments across five different glyphs — ←, ‹, ↺, a chevron, a bare dropdown caret — and router.back() appearing exactly zero times anywhere in the codebase. Two of the eight were the same bordered ← chip routed to two different destinations depending on which screen rendered it. The fix is one shared back component with one behavior, replacing all eight, closing not just the reported bug but the entire class it came from.

Splitting a script where a viewer would actually cut

PR #996 fixed a scene-splitting bug that had been hiding behind a workaround written into the product's own documentation. A line like "When she told me about this, I smiled and said I'd look into it" is two beats of performance carried by one beat of picture — a natural cut point sits at the comma. The splitter had no way to see it: both the sentence-partition logic and the scene-count cap treated a sentence, not a clause, as the smallest unit a cut could land on. The operator-facing guidance for this exact problem was to punctuate a script with extra full stops to manufacture more cut points — grammatically wrong on purpose, as a workaround for a limit that shouldn't have existed. The fix moves the atom from sentence to clause boundary, following the same rule every subtitling authority independently converges on for exactly this reason.

A sample that finally shows what the money buys

The richest thread in this whole arc starts with a plain product complaint: the pre-review stage — the one screen positioned specifically to catch a bad plan before the pipeline renders and bills for the full fleet — showed nothing but still frames. Pacing and line delivery are both fundamentally time-domain; a still frame can't show either. PR #997 changes pre-review to actually render two real scenes before the operator commits to the rest, with a feedback box underneath so a correction can be applied to the sample and re-checked before the expensive part starts.

Three fast-follow PRs hardened it, and the middle one is worth reading as a genuine self-correction rather than a straight line of fixes:

#998 closed a race: applying a feedback note re-renders and re-stitches the sample, which is safe only at the exact gate it was written for — mid-sample it collides with a render already in flight, and after approval it silently re-bills work the operator already paid for. The route now refuses anything outside that one window by name, because a safety property that depends on the screen never sending the request late is one stale tab away from being violated anyway.
#999 found the quote was reading half its own bill: the cost estimator priced a voice-bound clip at the bare per-second rate, while the actual billing code multiplied that same rate whenever the request carried image-set elements — which, on the UGC path, is every clip, single presenter included. A mirror test had been passing the whole time because it only checked that the base rate matched; nothing checked the multiplier.
#1000 then made a call that turned out to be wrong, and the PR that follows it says so directly rather than quietly overwriting it. #1000 reasoned that the sample quote and the full-run quote should share one number — the plan's average beat length — since they're "an inch apart" and answer what looked like the same question. #1001, written against a real paid run, showed they don't: the sample quote and the full quote answer two different questions ("what does this click spend" vs. "what does the whole ad cost"), and forcing them onto the same average produced a 12% under-quote on the exact number an operator is about to commit money against. The fix reverted to pricing each figure off the beats it actually renders — the sample from the sampled beats, the full estimate from the whole plan — and the PR is explicit that this reverses a call made one PR earlier, with the measured run that proved it wrong attached as evidence.

That reversal is the part worth remembering longer than any individual bug: the fix that shipped first wasn't obviously wrong when it shipped. It took a real paid run, not more code review, to show the two numbers needed to diverge on purpose.

A vendor's zero exit code is a claim, not proof

Three defects landed on the same day, and the RCA that followed them is a small case study in not trusting your own fix. PR #1002 traced a transcription failure back to its real cause: ffmpeg had already refused to demux the uploaded file and logged exactly why, and the code downgraded that free, authoritative, local signal to a warning and handed the same unreadable file to a paid remote transcription vendor instead — which then failed with a much less specific error. The fix makes a failed local probe stop the pipeline immediately rather than pass the problem downstream.

PR #1005 is the sequel, and its own opening line states the discipline plainly: it exists because the author tested the #1002 fix instead of trusting it, by uploading a deliberately truncated file to the live deployment. The new guard didn't catch it. Every individual check the previous PR had added — video probe, ffmpeg exit code, non-empty output — passed cleanly on a corrupted result: ffmpeg reported success while writing a 799-byte MP3 with no decodable audio frames inside it, because a zero exit code only claims the tool didn't crash, not that its output means anything. The real fix probes the extracted audio file itself, not the process that produced it.

In between, PR #1003 traced a "scene cuts not applied" report back to the clause-splitter from #996: one sentence in a real script carried five commas and a dash, every individual clause candidate fell under the minimum word floor, and the merge-short-atoms step folded all of them into one indivisible 33-word atom — 30% of the whole script riding on one shot. The fix caps how large a single clause atom is allowed to grow before it's forced to split anyway, so a floor meant to prevent scenes that are too short can no longer produce one scene that eats a third of the ad. And PR #1004 traced a "the character doesn't look like Pixar, it looks like our normal avatar" report to a filename: the animated character had been generated and paid for correctly, uploaded to Drive under a name the serving route's lookup table didn't recognize, so every request for it silently 400'd and the pipeline fell back to the photographic original nobody had asked for.

A progress bar that says how long "generating" actually takes

PR #1006 is a smaller, purely UX fix worth including because of the number behind it: the render stage — the one that takes roughly five minutes per scene, the better part of an hour on a nine-scene ad — showed a bar frozen at 0% with the word "starting…" for the entire duration. It was one of roughly forty hand-rolled waiting states across the app, and only two of them used the shared progress component that already existed. The fix consolidates onto one component that names the current scene, elapsed time, and a realistic estimate range, mounted everywhere a screen can run for minutes rather than seconds.

A seventh stage, because "done" was a dead end

Before PR #1007, a delivered ad was terminal. Every edit allowlist in the repository named only the review steps that come before delivery, so the instant an ad shipped, the only way to change anything about it was to pay for an entirely new render — a standing do-not-retry. #1007 adds an eighth-hour-of-the-day feature that sounds simple and touches everything: a seventh pipeline stage, Edit, with per-scene regenerate priced on the button itself, saved takes, master version history, and a genuinely free editorial cut. It's registered once, in the shared STAGES array every flow already reads from — there's no separate per-flow wiring to forget.

The three PRs that followed it are the now-familiar shape of shipping something and then actually using it in production before calling it finished. #1008 found two of the new stage's refusal messages contradicting the condition they were checking — a delivered ad refused with "can only be edited once the ad is finished," describing a state the ad was already in. #1009 found the new edit route was the only mutating wizard route doing a bare in-memory job lookup instead of falling back to the Drive-backed restore every other route already used, which meant reopening an edit on an ad that had survived a container restart — the exact case the feature exists for — 404'd. #1010 found a fourth refusal site with the same contradictory wording that #1008's by-hand fix had missed, and closed it by centralizing the message next to the predicate it describes rather than leaving four hand-written copies of one rule for a future bug to hide in.

The first finished VSL, and what it actually surfaced

PR #1019 is where this whole arc reports back. Taking a real product script through the new VSL Generator, in the Pixar visual style, with an AI-generated presenter, produced the first ad this exact configuration had ever finished rendering — and finishing it surfaced nine real defects, four fixed same-day and five more detailed in this PR. The most interesting one is a retraction: an earlier report had blamed a density control for doing nothing. It wasn't inert. A second, unrelated cap on how many generated lifestyle spans could ship was silently deleting the density control's own output after placement — the density setting worked exactly as configured, and something downstream threw its answer away. Measured against two real finished runs, the arithmetic lines up exactly: four dropped spans totaling 10.42 seconds, and 20.56 minus 10.42 is 10.14 — the shipped total to the decimal. The bug wasn't where the first report pointed; the first report was simply reading the wrong layer.

The pattern: a flow isn't unified until someone finishes an ad through it

Every stage of this arc reads clean in isolation — collapse four routes to one, add a card that's actually different, fix the navigation it broke, split scripts at the right boundary, make the sample honest, make the quote honest, give delivered ads a way back in. What ties them together is where almost every fix came from: not code review, but a real run, watched end to end, with the actual numbers pulled from logs and job documents rather than assumed from reading the code that produced them. The reversal in the sample-pricing arc and the retraction in the closing PR are the clearest examples, but the shape repeats throughout — a fix that looks complete when it merges and turns out to be answering the wrong layer of the question, caught only once someone pushed a real script all the way through to a finished ad.