You will learn why first-pass rate and cohort delivery need separate measures. They cover different populations and answer different questions. A change can pass its first measured gate without reaching delivery.
Code volume sits near the start of delivery. It does not show whether a change survives review, passes later checks, meets its business intent or reaches a usable state.
A high first-pass rate measures gate performance, not end-to-end delivery.
The denominator mismatch
The supplied measurement draft reports:
- First-pass rate: 91.7%
- Delivered share of started work: 52.2%
The measures use different denominators:
First-pass rate
= changes passing their first recorded gate attempt
/ changes with a recorded first gate attempt
Delivered share
= changes reaching delivery
/ changes in the defined started cohort
Started work that has not reached the gate is excluded from the first calculation but included in the second. Work can also pass the gate and then remain pending, enter repair, be superseded or be abandoned.
The gate boundary also matters. Moving it from implementation to review, or from review to production readiness, changes the meaning of the result. The reported first-pass rate applies only to its defined gate. It is not a proxy for delivery.
The measurement boundaries should remain explicit:
- Use gate attempts to measure first-pass performance.
- Use a defined started cohort to measure delivery.
- Separate pending work from terminal non-delivery.
- Record transitions between the two boundaries.
The reported results remain unauditable
The source measurement draft and raw counts are unavailable. No citation or link has been supplied. The metric definitions, cohort rules, observation window, collection methods and source systems are also missing.
The figures therefore remain reported observations. They cannot yet support comparisons across teams, periods or work classes.
Other reported measures have the same limitation:
| Metric | Reported result | Intended scope |
|---|---|---|
| Share requiring a second pass | 0% | Recorded repeat passes |
| Repair cycles per started change | 0 | Recorded correction cycles |
| Blocking review findings | 63.3% | Severity of recorded findings |
| Approved-plan coverage | 34.8% | Share of changes with an approved plan |
Publication of the source should include a raw-count appendix covering:
- Numerators and denominators for every percentage and zero value
- The exact gate definition
- Definitions of started work, gate attempts, repair cycles and delivery acceptance
- Cohort inclusion rules and dates
- The observation window
- The age distribution of unfinished work
- Finding-level dispositions and linked revisions
- Collection methods and source systems
Until these materials are available, the results should not be treated as a substantive case.
Delivery needs a validated lifecycle model
End-to-end measurement needs consistent states and transition events. The following table is a proposed baseline, not a confirmed representation of the measured workflow.
| Current state | Transition event | Next state |
|---|---|---|
| Started | Implementation submitted | Implemented |
| Started | Work replaced | Superseded |
| Started | Work stopped | Abandoned |
| Implemented | Gate attempt passes | Gate passed |
| Implemented | Gate attempt fails and correction starts | In repair |
| Implemented | Work replaced | Superseded |
| Implemented | Work stopped | Abandoned |
| Gate passed | Required reviews and checks pass | Approved |
| Gate passed | Accepted finding requires correction | In repair |
| Gate passed | Work replaced | Superseded |
| Gate passed | Work stopped | Abandoned |
| In repair | Revised implementation submitted | Implemented |
| In repair | Work replaced | Superseded |
| In repair | Work stopped | Abandoned |
| Approved | Delivery acceptance occurs | Delivered |
In this baseline, delivered, superseded and abandoned are terminal states. Pending is not a separate terminal state. It describes work with no later transition at the observation cut-off.
The model must be checked against the actual workflow. Validation must determine how it represents:
- Rollback after delivery
- Rejection after approval
- Partial deployment
- Reopened work
- Work that leaves the tracked workflow
If these events occur, the model needs corresponding transitions. Otherwise, it will hide outcomes or assign them to the wrong state.
Each transition should record a persistent change identifier, timestamp, responsible stage and applicable work or risk class. Each review finding should link to its disposition and any revision that addresses it.
Pending work changes the result
A recent cohort may contain substantial unfinished work. Counting every unfinished change as non-delivery lowers the delivered share without showing whether those changes will later reach delivery.
Reporting should distinguish:
- Delivered work
- Superseded work
- Abandoned work
- Work still pending at the cut-off
The following example is hypothetical. It does not represent the reported data.
A cohort contains 100 started changes. Its status changes between two observation dates:
| State | Earlier cut-off | Later cut-off |
|---|---|---|
| Delivered | 40 | 58 |
| Pending | 45 | 17 |
| Abandoned | 8 | 12 |
| Superseded | 7 | 13 |
| Started cohort | 100 | 100 |
At the earlier cut-off:
Delivered share = 40 / 100 = 40%
Pending share = 45 / 100 = 45%
At the later cut-off:
Delivered share = 58 / 100 = 58%
Pending share = 17 / 100 = 17%
The cohort size does not change, but its observed outcome does. Some pending work reaches delivery. Other work becomes abandoned or superseded. A delivered-share figure therefore needs both a cohort definition and an observation date.
The method must also define cohort maturity. It may use a documented maturity threshold or a documented time-to-delivery treatment. The available material provides neither, so no method can yet be selected or assessed.
Blocking findings expose a recording gap
The supplied draft reports that automated reviewers raised all recorded findings and that every recorded finding was categorised as a correctness issue. It also reports that 63.3% were blocking, while the share requiring a second pass and repair cycles per started change were both zero.
These observations do not prove that repair occurred. They raise a narrower question: what happened after each blocking finding?
A finding record should show whether the affected change:
- Was corrected before the measured pass
- Continued without a recorded repair cycle
- Remained pending
- Was superseded or abandoned
- Left the tracked workflow
- Reached delivery
Without these links, the data cannot show whether findings create repair work or how much engineering capacity that work consumes.
Keep the measures separate
Complete transition records support several distinct measures:
| Measure | Numerator | Denominator |
|---|---|---|
| First-pass rate | Changes passing their first recorded gate attempt | Changes with a first recorded gate attempt |
| Delivered share | Changes reaching delivery | Started changes in the defined cohort |
| Repair cycles per started change | Transitions into repair | Started changes |
| Terminal non-delivery share | Superseded or abandoned changes | Started changes |
| Pending share | Changes without a terminal state at the cut-off | Started changes |
These measures describe gate quality, correction activity, workflow progress and final outcomes. Combining them produces an ambiguous result.
Measurement priorities
The immediate task is to make the existing claims auditable:
- Publish or cite the source measurement draft.
- Add a raw-count appendix for every reported result.
- Define the measured gate and each lifecycle state.
- Publish the cohort rules, dates and observation window.
- Define the treatment of immature and unfinished work.
- Validate the proposed state model against the actual workflow.
- Record transitions against persistent change identifiers.
- Link findings to dispositions, revisions and final states.
- Compare teams or periods only after definitions and collection methods remain stable.
What this means
First-pass rate describes one gate. End-to-end delivery requires a defined cohort, explicit outcomes and transition records from start to observation cut-off.
Originally published at AFT Group Engineering Insights.
Top comments (0)