On the enterprise workflow platform, two separate quality gates — External QC and Final Review — both decided pass or fail off a single hardcoded bar. Every category of feedback a verifier could raise (task design, source data, prompt, verifiers, golden data) was judged against the same binary cutoff, and the two gates had no way to disagree with each other even when the work they were reviewing called for it. AGT-1774 replaced that with per-category, admin-editable thresholds for both gates independently. The feature itself is a month of work. What makes it worth writing about is that it got built twice — once as four separate PRs, and then again, from scratch, as a five-PR stack, because the first version hit a wall that had nothing to do with the code.
A quality bar is a policy decision, not a code constant. The month this took wasn't about computing a threshold — it was about building the plumbing so that decision could live in an admin UI instead of a pull request.
Build one: four PRs, shipped in order
The first pass landed in a straightforward sequence. PR #1028 extended the existing External QC config infrastructure with an optional gateThresholds field — per-category passing percentages stored as JSON on the existing config row, deliberately reusing the existing table rather than adding a migration. PR #1029 replaced a binary isVerifierFeedbackItem classification with a five-category classifier and a pure evaluateThresholdDecision function that groups feedback by category, computes a per-category pass rate, and compares it against the configured number — mirroring a scoring pattern (computePostqaScore) that already existed elsewhere in the codebase rather than inventing a new shape. PR #1030 did the same for Final Review specifically, adding a loadGateThresholdConfig helper with per-field fallback to sane defaults. PR #1031 put an admin UI on top: five threshold input fields, reusing the existing config fetch/update pattern with no new endpoint.
That's a complete, working feature in four PRs. It also had a commit with an unsigned signature somewhere in its history, which blocked the repository's verified-signatures branch-protection rule from letting it merge.
Build two: the same feature, five PRs, properly signed — and better architected
Rather than hunt down and re-sign one commit in an otherwise-merged history, the whole stack got rebuilt. PR #1154, explicitly labeled "replacement for #1027," is stack position one of five: the core terminal/rework lifecycle, durable completion recovery, and shared gate contracts — threshold configuration and its UI deliberately deferred to later PRs in the stack. #1105 (replacing #1028) added independently persisted thresholds for both gates with atomic config updates. #1106 (replacing #1029) applied the shared evaluator to External QC specifically, with decision consumption for Final Review intentionally left out. #1107 (replacing #1030) applied it to Final Review, with admin editing intentionally deferred again. #1108 (replacing #1031) closed the stack with the admin UI and a canonical cache-invalidation fix.
The rebuild isn't just the same four PRs re-signed. Splitting "add thresholds to External QC" and "add thresholds to Final Review" into two sequential PRs that each explicitly say what they're deferring is a more disciplined decomposition than the first attempt's four roughly-feature-sized chunks — each PR in the second version has a narrower, independently-testable claim, and the stack notes spell out exactly which PR depends on which. A signing problem is a bad reason to redo a month of work. A redo that uses the occasion to ship a cleaner decomposition than the original is the only good version of that story.
The redirect bug that keeps needing the same fix
Threaded through both builds is a recurring bug shape worth calling out on its own. An earlier fix (AGT-1760, covered in a previous post) made a golden-data rework redirect fire only for its actual source version, not every older version. PR #1024 found that guard still fired one hop too eagerly — clicking the immediate predecessor version in a dropdown redirected forward again, because nothing distinguished "this session is the one that just triggered the fork" from "a user navigated here on purpose to inspect it." PR #1180 found the identical bug shape on External QC's own version dropdown, independently, and fixed it the same way: a prevManifestVersionRef plus a pure shouldFireExternalQcRedirect helper, explicitly modeled on the golden-data fix rather than reinvented. The same underlying mistake — conflating "a transition just happened" with "the user wants to leave" — showed up on two different gates built by different people at different times, which is a reasonable argument for making the transition-gate pattern a shared primitive rather than something each gate re-derives.
A false "Submit failed" from a schema that wasn't lying, exactly
PR #1018 traced a confusing failure: the golden-data finalize route could return status: "qc_in_progress" without two fields — effectiveVersion and reworkVersionCreated — that the client's response schema required on every branch of the union. The server had genuinely succeeded; the client's own validation rejected the honestly-formed response before the UI ever got to react to it, so the panel never transitioned to the QC view and the user saw "Submit failed" for a submission that had, in fact, succeeded. Schema validation did exactly what it's supposed to do here — reject a response that didn't match the contract — the actual bug was that the contract was stricter than what one legitimate server branch produced.
Building a real analytics layer, not a bigger spreadsheet
Alongside the threshold work, PR #1167 added a Task-Level Metrics tab to the task-detail page — and per direct stakeholder feedback during review, the original plan to show raw-data tables was replaced with interactive charts before it shipped, because tables of per-step durations are much harder to spot a trend in than the equivalent line chart. PR #1097 built the top-level admin Analytics dashboard separately, deliberately sourced only from the existing rl-world API and GCS manifests — "zero v0 code remains" — rather than adding a new direct-to-database query path that would have had to be maintained as a second source of truth.
PR #1170 is the bug that would have quietly undermined both: the step-timing charts derived their list of steps and versions from GCS step_started/step_completed events, and when those events were missing or incomplete, the charts rendered empty — "no step timing data available" — even on tasks where the more authoritative rl-world API had the real timing data the whole time. A first attempted fix unioned both sources together, which meant a step that only ever existed in sparse GCS events still forced its way into the list and diluted the result. The real fix makes rl-world the primary source outright, falling back to GCS events only as a last resort — a priority order, not a union, because one of the two sources is simply more trustworthy than the other and averaging them together doesn't produce a better answer.
Smaller fixes that are really about who's being shown what
PR #1178 found the Final Review gate's confirmation modal collapsing a verifier.json artifact holding nineteen individual per-verifier results — eight approved, eleven rejected — into a single row reading "1 Rejected · 0 Approved · 0 Incomplete — 1 artifact." The modal's job is to tell a reviewer what they're about to approve past; showing one artifact instead of nineteen decisions is technically accurate about file count and completely misleading about what's actually being waved through. The fix expands aggregation to count individual verifier results rather than the artifact file that happens to contain them.
PR #1169 closed three separate permission bugs — a visible Clone Task button, an accessible Events tab, and a reachable Force QC Pass action — all under platform-admin impersonation of a lower-privileged role, and all three traced to the exact same root cause: the client's auth context tracked that impersonation was active but never tracked which role was being impersonated, so every permission check kept reading the real admin's role underneath. One shared fix (threading the impersonated role through fetchUserProfile via authFetch) closed all three, which is the useful signal that it was one bug with three symptoms rather than three unrelated ones — bugs findable only by actually testing as an impersonated user, not as yourself.
A smaller cluster rounds out the access-and-trust side: PR #1187 blocked reassigning a reviewer-step task to a role explicitly ineligible to hold it — a gap on the older "classic" reassign route that a newer route had already closed, found only because the two routes had drifted apart. PR #1188 widened the new analytics dashboard's access from platform admins only to standard admins too, specifically scoped to analytics and nothing else, because the dashboard's own access model and the platform's broader admin-tier model hadn't been reconciled when the dashboard first shipped. PR #1182 made a post-claim run lookup fail loudly instead of swallowing a transient GCS error as "not found" — the quiet version of that bug could have wiped an already-approved review off the back of a storage blip that had nothing to do with the review's actual status. And PR #1181 fixed a dropdown option plainly mislabeled "Block approval for rework" in a context — Final Review — where rework was never actually a possible outcome; a small fix, but exactly the kind of thing that erodes an admin's trust in every other label on the same screen if left uncorrected.
The pattern: a quality bar only works as policy if it's actually editable
The headline feature here — admin-editable thresholds instead of a hardcoded cutoff — is the kind of change that sounds almost too simple to take a month. It took a month because making a number configurable is the easy 10%; the other 90% is every place downstream that assumed the number would never change, every version-navigation path that needed to agree with every other one, and an analytics layer built at the same time that needed its own data sources prioritized correctly rather than averaged. None of the individual fixes in this cluster are large. What they have in common is that each one is a place where "the system does roughly the right thing" and "the system does exactly the right thing for the person relying on it" turned out to be different claims — and closing that gap, repeatedly, is what actually shipping a policy-configurable gate costs.
Top comments (0)