We learned that an AI-written change can look complete long before it’s safe to merge. The implementation may be coherent, the explanation persuasive and every review comment resolved, but none of that shows how the candidate behaved in the delivery pipeline.
That gap changed how we treated the gate. It now fails closed: a positive result has to be observed independently. If the required continuous integration evidence is missing, incomplete, stale or unverifiable, the change stays blocked.
Not verified means not ready.
The author can’t authorise the work
The authoring process asks whether the request seems to have been understood and implemented plausibly. CI asks a different question: did this candidate pass the repository’s configured checks in a controlled environment?
An authoring agent can inspect its own reasoning, edits and local tool output. That context helps while implementing the change, but it isn’t independent of the artefact being assessed. Creation and authorisation can’t sit inside the same trust boundary.
For CI evidence to support advancement, it has to identify:
- the candidate under consideration
- the relevant pipeline run
- the result of every required check
- a verifiable status observed independently of the author’s claim
A result for another candidate doesn’t verify the current one. A partial result doesn’t show that every required check passed. A successful local command, an expectation that tests will pass or a prediction from the authoring agent can’t fill a hole in the evidence chain.
Across the recorded delivery activity, 93.2% of evidence records were verified. All recorded verification was classified as independent. We use claude-opus-5 to verify work at any risk, but model judgement can’t turn missing pipeline evidence into a pass.
There are limits to what that number tells us. The source material doesn’t define an evidence record, explain how evidence is bound to a commit or candidate, or state the test used to classify verification as independent. So 93.2% is a reported operational rate rather than an independently reproducible measurement or a complete account of the verification mechanism.
The gate itself is less ambiguous. If implementation and review are complete but the required CI result isn’t available, we don’t infer success from the work already done.
The candidate waits.
That can pause flow. It also stops infrastructure faults, stale results and missing signals from being reported as engineering success. Leaders can distinguish between changes that passed the configured path and changes still waiting for evidence, rather than allowing the level of assurance to vary silently from case to case.
Repository readiness complicates the picture
The readiness assessment reported that observation of CI checks passed throughout. It also showed uneven control presence:
| Readiness control | Reported result |
|---|---|
| Presence of continuous integration | 85.7% |
| Presence of an agent trigger | 85.7% |
| Checks for committed secrets | 28.6% |
| Default-branch protection | 14.3% |
There’s an awkward detail here. Passing the observation assessment doesn’t mean CI was present in every repository. The source doesn’t explain how that observation relates to the reported 85.7% presence rate, so we can’t infer anything from the relationship between those measures.
The percentages describe repository controls. They don’t tell us whether individual changes passed those controls. Nor does the source say whether each percentage represents repositories, checks or some other assessed population.
That uncertainty doesn’t justify weakening the candidate-level gate. Where required evidence is unavailable, can’t be attributed to the candidate or can’t be checked independently, the candidate doesn’t advance. Uneven repository readiness simply makes the evidence harder to produce or trust.
Repair attempts need a boundary too. Most changes required no repair cycle. A further pass was needed in 3.8% of cases, while the same proportion exhausted the configured repair budget.
Again, the source leaves some important terms undefined. It doesn’t say what counts as a repair pass or cycle, how large the repair budget is, what population was measured, what conditions cause exhaustion, or what terminal state follows once the budget has been used.
Even with those gaps, the policy is clear. Once the process can’t produce acceptable evidence within the configured limit, more agent activity doesn’t count as more confidence. The limit prevents an unresolved candidate from sitting in an indefinite loop of edits and reassessment.
Planning, review and CI answer different questions
Independent review can find questionable logic, unsafe assumptions and mismatches with the requested behaviour. Planning can make the intended implementation clearer. Neither shows that the resulting candidate builds, tests and integrates successfully.
A green pipeline has the opposite limitation. It records the outcome of the configured checks, but it can’t show that the requested business behaviour is correct beyond those checks.
Automated reviewers raised every recorded finding. Of those findings:
- 57.3% were classified as blocking
- 29.8% were classified as major
- 12.9% fell into categories the source doesn’t specify
We don’t have the number of findings, the reporting period, the classification criteria or a statement of which classifications prevent advancement. Resolving a finding removes a known review objection. It also changes the candidate, which still needs fresh CI evidence.
Planning showed the same separation between useful preparation and evidence of execution. An approved plan covered 48.1% of started changes, and no plan approval was reported as later revoked.
First-pass acceptance was:
| Planning status | First-pass acceptance |
|---|---|
| Approved plan present | 77.8% |
| No approved plan | 72.7% |
Those figures need care. The source doesn’t define a started change or first-pass acceptance, and it gives no sample sizes or measurement window. They show an association, not proof that planning caused the difference. A plan describes the intended work; it doesn’t prove the implementation passed its required checks.
Verified evidence rose from 90.5% in the early build phase to 93.7% in the current phase. Over the same period, the share requiring a further pass fell from 5.1% to 2.5%.
The measures moved together. That surprised us enough to pay attention, but the source doesn’t define the phases, their dates or their populations. It gives us no basis for saying one change caused the other.
What the figures can’t tell us yet
No underlying source, dataset or citation was supplied with these figures. Readers can’t audit their provenance from this article alone.
The material also leaves out:
- the assessment and reporting periods
- populations, denominators and sample sizes
- the assessed repository population and repository count behind the readiness figures
- definitions of
evidence record,started change,first-pass acceptance,early build phaseandcurrent phase - definitions of
blocking,major,repair cycleandexhausted repair budget - the candidate identifier and the method used to associate a candidate with a pipeline run
- the criteria used to mark evidence as verified or independent
- the operational representation of missing, stale or mismatched evidence
- the required checks and who configures them
- the rounding method
That means the percentages are reported operational observations. They aren’t general benchmarks or independently reproducible measurements.
Our readiness work now focuses on default-branch protection, committed-secret checks and agent tooling. The first two had the lowest reported control presence. We’re also tightening the way we classify evidence gaps and review findings, so unknown states are easier to diagnose without quietly turning into approvals.
There’s still more to define, especially around how evidence is tied to a candidate and how independence is assessed. In the meantime, the gate remains deliberately blunt: if the required pipeline result isn’t present, attributable and independently verified, the change waits.
Originally published at AFT Group Engineering Insights.
Top comments (0)