DEV Community: AFT Group

Engineering Vehicle Fraud Checks as Point-of-Sale Transaction Gates

James Smith — Fri, 28 Aug 2026 19:31:12 +0000

A point-of-sale fraud check sits in the transaction path. This article shows how to define its deadline, state model, latency budget, verdict policy, failure behaviour and audit evidence. The check must return before the finance decision or follow an explicit deadline policy.

A check that misses the transaction deadline cannot control the transaction without a defined fallback.

The framework also applies these criteria to the reported MotorVerify design. The product assessment remains limited because the supplied material contains no attributed documentation, engineering testimony, observed traces or performance measurements.

Product assessment

The draft reports the following MotorVerify capabilities. The supplied material does not independently confirm them.

Reported capability	Evidence needed
Guided video capture at the point of sale	Product documentation, engineering testimony or an observed capture trace
Vision models that assess whether a vehicle is real and physically present	Model description and observed processing trace
Detection of screen replays and printed photographs	Anti-spoofing documentation and validation evidence
Cross-references to DVLA and MOT records	Dependency documentation and observed request traces
Assessment of more than 60 data points	Representative categories and their roles in the decision
Parallel execution of checks	Architecture description and component timing trace
Streaming of partial results	Interface documentation and observed event trace
Timestamped storage of each data point	Data model, retention policy and audit controls
Pass, Refer or Fail verdicts	Approved definitions, aggregation rules and transaction consequences

The reported design is plausible as a transaction-gate pattern. The available material does not establish that MotorVerify implements this pattern or meets point-of-sale timing and decision-quality requirements.

The transaction-gate model

A gate begins when the inspection starts and ends when a transaction outcome becomes available. Its design must cover capture, request validation, queueing, checks, evidence aggregation and verdict publication.

Capture
  |
  v
Validate request -----> Reject invalid request
  |
  v
Queue work
  |
  +----------+-------------+
  |          |             |
Vision   Anti-spoofing   Record lookup
  |          |             |
  +----------+-------------+
             |
       Aggregate evidence
             |
      Pass / Refer / Fail
             |
   Proceed / Review / Stop

The nominal flow is not enough. The design must also define:

Which checks can start during capture.
Which checks require complete capture.
Dependencies between vision, anti-spoofing and record checks.
Queue boundaries and maximum queueing time.
Evidence required for each verdict.
Whether partial results affect a verdict or only report progress.
Timeout and retry rules for external requests.
Treatment of cancellation, duplication, late completion and stale evidence.

Request states

A compact state model makes terminal outcomes explicit.

                 +----------------+
                 |     Queued     |
                 +----------------+
                    |          |
               start|          |cancel
                    v          v
              +-----------+  +-----------+
              |  Running  |  | Cancelled |
              +-----------+  +-----------+
                |    |   |
       complete |    |   | deadline expires
                |    |   v
                |    | +-----------+
                |    | | Timed out |
                |    | +-----------+
                |    |
                |    | cancel
                |    v
                | +-----------+
                | | Cancelled |
                | +-----------+
                v
          +-------------+
          |  Completed  |
          +-------------+

Cancelled, Timed out and Completed are terminal request states unless a documented policy permits another transition. A late component result must not silently move a cancelled or timed-out request into Completed.

Duplicate submissions need stable request identities and idempotency rules. The policy must state whether a duplicate returns the existing state or creates separate work.

Latency budget

The point-of-sale deadline must be defined before feasibility can be assessed.

Let:

D be the maximum permitted time from inspection start to transaction decision.
C be guided capture time.
Q be queueing time.
V be vision-model processing time.
S be anti-spoofing processing time.
R be vehicle-record lookup time, including permitted retries.
A be aggregation and verdict-publication time.

If processing starts after capture and the three checks run concurrently:

Total latency = C + Q + max(V, S, R) + A
Requirement:   Total latency <= D

If checks start during capture, the overlap must be measured rather than assumed:

Total latency = elapsed time from capture start
                to publication of the transaction verdict

Requirement: Total latency <= D

Parallel execution can reduce elapsed time. It does not guarantee an earlier verdict. The slowest mandatory check, queueing delay, retries and aggregation policy still determine completion time.

Streaming partial results improves progress visibility. It reduces decision latency only when the aggregation policy permits a valid verdict before all checks finish.

Measurement boundaries

Boundary	Starts	Ends	Required evidence
Capture	Inspection begins	Required media is accepted	Timestamped capture trace and completion criteria
Queueing	Work is submitted	Processing starts	Queue-delay percentiles
Model processing	Model work starts	Required outputs are available	Per-check latency percentiles
External lookup	Request is issued	Accepted response or terminal timeout	Lookup, timeout and retry measurements
Aggregation	Inputs are available or expire	Verdict is published	Aggregation latency and decision trace
End to end	Inspection begins	Transaction outcome is available	Latency percentiles against `D`

A hypothetical deadline path

Consider a request where vision and anti-spoofing finish but the vehicle-record lookup does not return. This is a design example, not observed MotorVerify behaviour.

The outcome depends on policies defined before deployment:

The retry policy decides whether another lookup attempt is permitted.
The deadline policy decides whether that retry can fit within D.
The aggregation policy decides whether the completed checks support any verdict.
The verdict policy maps the available evidence to Pass, Refer, Fail or another permitted outcome.
The transaction policy decides whether the finance process proceeds, pauses or stops.

If the record lookup is mandatory for Pass, the system cannot return Pass because the dependency timed out. It must follow the approved timeout path. A timeout must not become an implicit pass.

Latency therefore cannot be assessed separately from verdict policy.

Request trace

A sourced end-to-end trace is required to test the reported architecture. The supplied material does not include one.

At minimum, the trace should record:

request_id
inspection_started_at

capture_accepted_at
work_queued_at
processing_started_at

external_request_issued_at
external_retry_issued_at
external_response_or_timeout_at

evidence_aggregated_at
verdict_published_at

transaction_deadline_at
request_state
final_verdict
verdict_reason
transaction_action

Policy decisions attach to specific points in the trace:

inspection_started_at
  |
  |-- Deadline D begins
  v
capture_accepted_at
  |
  v
work_queued_at
  |
  |-- Queueing limit applies
  v
processing_started_at
  |
  +-- Internal checks
  |
  +-- External request
        |
        |-- Timeout policy applies
        |-- Retry policy applies
        |-- Retry must remain within D
        v
external_response_or_timeout_at
  |
  |-- Aggregation policy selects usable evidence
  |-- Staleness and conflict rules apply
  v
evidence_aggregated_at
  |
  |-- Pass / Refer / Fail definition applies
  v
verdict_published_at
  |
  |-- Transaction policy maps verdict to action
  v
transaction_action

The trace must identify mandatory checks, retry attempts, state transitions and the evidence used for the verdict. Success traces alone are insufficient. Timeout, cancellation, duplicate and late-completion paths also require observation.

Verdict policy

Pass, Refer and Fail need approved operational definitions. These definitions can specify transaction behaviour without exposing detection thresholds or other sensitive controls.

Path	Policy to define	Evidence needed
Pass	Whether the transaction proceeds and which checks must complete	Aggregation rule and complete decision trace
Refer	Whether the transaction pauses, who reviews it and how the review ends	Review procedure, routing record and reviewer outcome
Fail	Whether the transaction stops and what evidence is retained	Stop policy and retained decision record
Deadline missed	Whether the request refers, stops or takes another defined path	Deadline policy and timeout trace
External timeout	Retry limit, permitted verdicts and degraded behaviour	Dependency trace and aggregation record
Duplicate request	Deduplication and idempotency behaviour	Request identifiers and state-transition record
Cancellation	Which work stops and what record remains	Cancellation trace across each component
Stale result	Validity period and recheck policy	Result age and policy decision
Conflicting evidence	Precedence, referral or stop rule	Input evidence and aggregation explanation
Customer departure	Whether processing continues and the transaction remains blocked	Session state and recorded outcome

Rates derived from these paths require a defined evaluation population and period. False-positive and false-negative rates also require an agreed reference outcome.

Failure behaviour

Retries can extend the critical path and create duplicate external requests. The design must define retryable failures, maximum attempts, delays between attempts and the outcome after retry capacity is exhausted.

Cancellation can occur while work is queued, running or waiting on an external dependency. The cancellation policy must state:

Whether queued work is removed.
Whether active internal work stops.
Whether external calls can be cancelled.
Whether late results are discarded or retained.
Whether a late result can alter the terminal request state.

Captured media and record responses must represent the same decision point. Validity periods and recheck rules prevent old evidence from being reused without an explicit decision.

Conflicting evidence requires a deterministic path. The aggregation policy must state whether a conflict produces Refer, Fail or another permitted outcome.

Audit evidence

Timestamps support later investigation. They do not establish audit integrity on their own.

The design needs documented controls for:

Retention periods.
Access authorisation.
Protection against alteration.
Links between requests, evidence, retries and verdicts.
Records of cancellation, late completion and state changes.

Evidence needed for a product conclusion

A product-specific assessment requires:

Attributed documentation or testimony for each reported capability.
A point-of-sale deadline supported by dealership workflow evidence.
End-to-end and component latency percentiles under representative load.
Observed traces for success, timeout, retry, cancellation, duplication and late completion.
Approved definitions and transaction consequences for Pass, Refer and Fail.
Decision-quality measurements using defined populations and reference outcomes.
Representative categories and decision roles for the reported 60-plus data points.
Documented retention, access-control and alteration-protection mechanisms.

What this means

A point-of-sale check can gate a transaction only when its deadline, state transitions, mandatory evidence, verdict consequences and failure paths are explicit. This framework defines how to assess that gate, but the supplied material does not support a conclusion about MotorVerify’s implementation or performance.

Originally published at AFT Group Engineering Insights.

Planning Gates: Recording Approval, Revision and Final Acceptance

James Smith — Fri, 28 Aug 2026 19:30:35 +0000

This article shows developers how to record plan approval, material revisions and final acceptance. The gate defines what is approved, who may approve it and which changes require renewed approval.

A recorded plan gives implementation and review a common reference point without claiming that planning alone improves delivery.

Record the plan

A plan captures decisions expected to remain stable during implementation:

Intended behaviour
Relevant business rules
System boundaries
Failure handling
Testing strategy
Evidence required for acceptance

Product and engineering participants agree the outcome and constraints. The implementer decides how to meet them within the repository.

Without this agreement, reviewers reconstruct more of the requirement from implementation choices. Those choices can anchor the discussion. Replacing an unsuitable approach appears more expensive after implementation than selecting one before work begins.

The plan should not prescribe incidental code structure. Excess detail takes time to review, becomes stale and restricts implementation decisions that do not affect the agreed outcome.

Use an explicit approval gate

The gate requires three decisions:

Approve or reject the plan before implementation.
Continue, pause or seek renewed approval when a material change arises.
Accept or reject the completed change against the current approved plan.

Rules for each risk band must state:

How the risk band is assigned
Who is authorised to approve the initial plan
Who is authorised to approve revisions
Who is authorised to give final acceptance
Whether those authorities may be the same role
Whether independent challenge is required
Whether implementation pauses during revision
How unresolved disagreement is escalated
What evidence final acceptance requires

Plan approval, revision approval and final acceptance are separate decisions. The control model must identify the authorised role for each rather than assume that one approval covers the full lifecycle.

A risk label without these rules cannot determine which controls apply.

Keep one lifecycle record

The following synthetic table defines the minimum lifecycle record. It is a proposed template, not evidence that the control has operated.

Stage	Field	Required entry
Identification	Work item	Identifier linking the plan to implementation
Planning	Intended behaviour	Observable outcome of the change
Planning	Business rules	Rules the implementation must preserve
Planning	Boundary	Included and excluded systems or components
Planning	Failure handling	Required behaviour when the operation fails
Planning	Acceptance conditions	Testable conditions for acceptance
Planning	Validation method	Tests, checks or inspection used as evidence
Classification	Risk band	Assigned classification
Initial approval	Plan approver	Role or person authorised for the assigned classification
Initial approval	Approval state	Draft, approved, rejected or superseded
Initial approval	Approval record	Decision, date and recorded location
Revision	Revision trigger	Material change requiring reassessment
Revision	Revision record	Changed field, reason, proposer and decision
Revision	Revision approver	Role or person authorised to decide the revision
Revision	Work status	Continue or pause while approval is pending
Implementation	Code reference	Link between the approved plan and implementing code
Validation	Acceptance evidence	Links to tests, checks and review findings
Final acceptance	Acceptance authority	Role or person authorised to accept the completed change
Final acceptance	Acceptance state	Pending, accepted or rejected
Final acceptance	Accepted plan version	Exact approved version used for final review

Each approved version should link to the version it supersedes. Missing records should remain visible rather than be inferred from surrounding entries.

Define material change

A material change affects the approved intent or the basis on which it was approved. It requires reassessment through the revision fields in the lifecycle record.

Area	Material change	Normally not material
Behaviour	An operation returns a different result for an accepted input	Internal refactoring preserves the same result
Boundary	The change writes to a component excluded by the plan	Code moves between files inside the approved component
Risk	New information meets the defined criteria for another risk band	An implementation detail changes without affecting the assigned criteria
Acceptance	A required test is removed, weakened or replaced with different evidence	The same acceptance condition is tested at a different internal layer
Failure handling	A specified rejection becomes a retry or partial success	Error construction changes while externally observable handling remains unchanged

These examples define the decision test, not the risk-band rules. Each organisation still needs operational criteria for assigning risk and identifying authorised approvers.

When a material change arises, update the lifecycle record with the affected field, reason, risk classification, work status, decision and superseded plan version.

Review against the plan

An approved plan reduces initial requirement discovery and business-rule interpretation during normal code review. It does not remove them. Review can still uncover mistaken assumptions, unrecognised dependencies or missing requirements.

Reviewers can ask:

Does the implementation preserve the agreed behaviour and invariants?
Are error paths handled as specified?
Do tests exercise each acceptance condition?
Has the change stayed within its declared boundary?
Is every material departure recorded and approved?
Does the acceptance evidence refer to the current plan version?

If a finding changes behaviour, boundaries, risk or acceptance conditions, the plan requires reassessment. Otherwise, the reviewer can treat it as an implementation issue within the approved scope.

Apply the workflow

This synthetic example demonstrates the mechanics rather than a delivery result:

A plan states that a component must reject an invalid input without writing outside its declared boundary.
Acceptance requires a test showing the rejection and confirming that no external write occurs.
An authorised plan approver approves the plan under the assigned risk band.
During implementation, the engineer finds that the proposed path requires a write to a component excluded by the plan.
The engineer records the boundary change and requests reassessment.
Work continues or pauses according to the rule for that risk band.
The authorised revision approver accepts or rejects the revised boundary.
Code review checks the implementation and tests against the approved version.
The authorised acceptance role records the final decision and links it to the supporting evidence.

The external write is material because it changes the declared boundary. A local refactoring that preserves the rejection behaviour, boundary and acceptance test does not require renewed approval.

The example does not establish whether the three approval decisions should be made by the same person or by separate roles. That choice belongs in the organisation’s authority rules.

Link repository controls to evidence

Planning cannot compensate for a repository that cannot reproduce or enforce its validation process. Repository controls should support named acceptance evidence rather than form a general checklist.

Repository control	Acceptance evidence it supports	Review question
Runtime pinning	Test results produced under the declared runtime	Was validation run with the recorded runtime version?
Continuous integration with observed checks	Recorded execution of required tests and checks	Did the required checks run against the accepted change?
One-command validation	A repeatable command covering the stated validation method	Can a reviewer repeat the required validation from the repository?
Default-branch protection	Enforcement of required checks before integration	Could the change enter the default branch without the recorded checks?

A plan should not imply that evidence is repeatable unless the repository provides the required mechanism. The lifecycle record should link the approved plan, implementing code, validation results, review findings and final acceptance decision.

Define delivery measures

Planned and unplanned work can only be compared when each measure has an operational definition.

Measure	Proposed definition
First-pass result	Whether the change meets all recorded acceptance conditions at its first formal acceptance review
Review finding	A recorded issue raised during review that requires a response or change
Finding classification	The defined category and severity assigned under the review process
Final acceptance result	Accepted or rejected against the current approved plan
Change size	The organisation’s declared sizing method, applied consistently to both cohorts
Change complexity	The organisation’s declared complexity method, applied consistently to both cohorts
Revision frequency	Number of approved-plan revisions per change
Approval latency	Time between submission for approval and the recorded decision
Missing data	A required field or record that is absent, unavailable or cannot be linked to the change

Teams must define sizing, complexity and severity schemes before collecting results. Retrospective classification risks inconsistent comparisons.

Do not force ambiguous records into a category. Record them as unknown and state why. A missing approval or acceptance record is not evidence of rejection, acceptance or an absence of findings.

Reports should include:

Definitions and reporting period
Planned and unplanned cohort sizes
Risk band, repository and scope
Change size and complexity
First-pass and final acceptance results
Finding counts and classifications
Revision frequency
Approval latency
Missing and ambiguous records

Compare work with similar risk, repository and scope characteristics. Where direct matching is impractical, group results by those factors and report size and complexity within each group. This can reduce unsuitable comparisons, but it cannot establish causation.

Evidence limitations

This article proposes a control model and synthetic record. It does not include a real change record, repository artefacts, operational risk-band rules, defined outcome measures or a source-backed separation of approval roles. No governance, change-management or software-assurance standard has been cited because no supporting source was provided.

No reported percentage is justified without its source, population, numerator, denominator and definition. The model therefore does not demonstrate that planning improves delivery or makes intent clearer in practice.

What this means

An approval gate gives implementation and review a recorded reference for intent, revision and acceptance. Claims about delivery impact require complete lifecycle records, repeatable repository checks and consistently defined comparisons between planned and unplanned work.

Originally published at AFT Group Engineering Insights.

Cross-Family Code Review: An Operational Control Without Comparative Evidence

James Smith — Fri, 28 Aug 2026 19:30:34 +0000

Cross-family code review separates implementation from review. This article sets out what that separation controls, what evidence reviewers need and how to test whether it detects more defects than same-family review. The available evidence does not establish that advantage.

A model reviewing its own output may preserve assumptions made during implementation. A misunderstood business rule, unsuitable abstraction or missed edge case can survive a second pass. Another model family introduces a different perspective, but difference does not prove independence or correctness.

Cross-family review creates useful disagreement, but its advantage over same-family review remains unproven.

Undated routing snapshot

The documented workflow assigns implementation and code review to different model families.

Risk level	Implementation	Code review
LOW	`gpt-5.6-sol`	`claude-sonnet-5`
STANDARD	`claude-sonnet-5`	`gpt-5.3-codex`
HIGH	`claude-sonnet-5`	`gpt-5.6-sol`
CRITICAL_FINANCIAL	`claude-opus-5`	`gpt-5.6-sol`

Other responsibilities are assigned as follows:

gpt-5.3-codex challenges plans for LOW, STANDARD and HIGH-risk work.
gpt-5.6-sol challenges plans for CRITICAL_FINANCIAL work.
gemini-3.1-pro provides visual review for UI work.
claude-opus-5 performs verification at every risk level and handles escalation where required.

Plan challenge, implementation, code review and verification remain separate controls. Each examines an artefact it did not produce.

This table has no effective date or model-version context. It should therefore be treated as an undated operational snapshot, not a durable routing specification.

Evidence available to reviewers

A different model name is not enough. Reviewers need explicit business rules, repository context and reproducible checks to distinguish defects from intentional constraints.

The described repository readiness checks cover:

pinned runtimes;
lockfiles;
architecture maps;
testing guidance;
common failures;
one-command validation.

These controls provide observable conditions against which to assess a change. A failing test, inconsistent business rule or missing validation path gives a firmer basis for a finding than a preference about naming, formatting or architecture.

The supplied material does not include reviewer prompts, context packages, tool permissions or deterministic validation commands. It also omits the risk taxonomy, classification procedure, fallback behaviour, escalation criteria and severity calibration.

Limits of the reported evidence

The internal measures cover first-pass performance, readiness checks, approved-plan coverage, agent-tool support, default-branch protection and finding severity. Their reported rates are omitted here because the material does not provide the underlying counts, denominators, definitions or collection dates.

These measures describe parts of the current process. They cannot test whether cross-family review performs better than same-family review.

The material also lacks:

a defined repository and change population;
inclusion and exclusion criteria;
finding-level audit records;
false-positive, disagreement and override rates;
escaped-defect rates;
a controlled same-family versus cross-family comparison;
an end-to-end case linking a change, finding, correction and verification result.

Agreement between reviewers does not establish correctness. Model families may also share training patterns. Neither model naming nor reviewer agreement is a substitute for adjudicated outcomes.

Comparative evaluation

The experimental unit should be one eligible code change reviewed under both defined conditions, or assigned to one condition through a recorded allocation procedure. Where the same repository or model contributes multiple changes or review runs, those observations must not be treated as independent.

Equivalent review conditions require the same:

code change or matched change criteria;
business requirements;
repository state and context;
reviewer instructions;
tool permissions;
deterministic checks;
time and execution limits;
finding format;
adjudication criteria.

Model identity should be removed from finding records before adjudication where practical. Reviewers should not see findings from the other condition before submitting their own.

Outcome records

Outcome	Required evidence	Adjudication requirement
Accepted defect	Finding linked to the change and supporting rule, test or check	Confirm defect and severity
Rejected finding	Finding linked to the change and contrary evidence	Record rejection reason
Overridden finding	Original finding and authorised override evidence	Record decision owner and rationale
Duplicate finding	Links to findings describing the same defect	Identify the retained record
Deterministic-check finding	Reproducible command and observed result	Confirm that the check establishes the defect
Model-only finding	Finding without a decisive deterministic check	Assess against requirements and repository evidence
Escaped defect	Later defect record linked to the reviewed change	Reassess the earlier review outcome

Severity levels and consequential-defect criteria need to be defined before collection starts. Style preferences and other non-blocking observations should remain separate from correctness defects.

Statistical analysis

Sample size should be determined from the comparison being tested, the outcome rate of interest and the required precision. The source material does not provide the inputs needed to calculate it.

Analysis should account for clustered observations when repositories, changes or models contribute repeated results. Confidence intervals should accompany comparative rates. If multiple model runs are used, the protocol should define how run-to-run variation is recorded and whether results are combined or reported separately.

The analysis should also distinguish a planned primary comparison from secondary measures. False positives, escaped defects, disagreements and overrides should be reported separately rather than combined into one success rate.

What this means

Cross-family review provides operational separation, but the current material does not establish an advantage over same-family review. A valid comparison requires defined review conditions, auditable adjudication and analysis that accounts for repeated observations.

Originally published at AFT Group Engineering Insights.

Commit-Bound Evidence and Human Acceptance in an Internal Software Factory

James Smith — Fri, 28 Aug 2026 19:29:56 +0000

You will learn how commit-bound evidence, independent review and human acceptance combine to control code changes in an internal software factory. The factory generates code changes, but it does not yet provide an auditable record of why a specific change should be accepted.

Early coding agents received work without an agreed plan, acceptance gate or independent review. The implementing agent also reported completion. Acceptance depended on that account rather than evidence tied to the resulting commit.

An agent’s account of its own work is not evidence that the resulting software meets the requirement.

The proposed control path combines an approved plan, continuous integration evidence bound to a commit, independent review and a human acceptance decision. The design is clear. The available records do not show the complete path operating.

Acceptance workflow

APPROVE PLAN AND ACCEPTANCE CONDITIONS
    |
    v
IMPLEMENT CANDIDATE
    |
    v
RUN RELEVANT CONTINUOUS INTEGRATION CHECKS
    |
    v
VERIFY EVIDENCE AGAINST REVIEWED COMMIT
    |
    |-- evidence absent, malformed or mismatched --> REJECT
    |
    v
INDEPENDENT REVIEW
    |
    |-- accepted --> HUMAN ACCEPTANCE DECISION
    |
    `-- rejected --> REPAIR
                        |
                        |-- budget remains --> IMPLEMENT CANDIDATE
                        |
                        `-- budget exhausted
                                |
                                v
                    HUMAN JUDGEMENT | REPLAN | CHANGE SCOPE

Automated review does not make the final acceptance decision. An accepted review passes the candidate to a person.

Assessment limits

The available material does not define:

The assessment period
The codebase estate
Eligibility rules
The complete record population
Representative control invocation records
A complete anonymised delivery trace

This assessment uses the following classifications:

Classification	Threshold
Specified	Expected behaviour and failure handling are documented
Reported in use	Records state that the control is used, but do not include representative invocation records
Deployed	Invocation records show the control running within a defined scope
Operating effectively	Records show the control running across that scope, handling failures as specified and producing assessable evidence
Untested hypothesis	A proposed design choice lacks comparative evidence

A statement that a control is used does not meet the threshold for deployment. The available material does not support classifying any control as deployed or operating effectively.

No concrete commit-binding protocol or anonymised control trace is available. Neither can be reconstructed from partial statements without inventing missing events.

Mandatory controls and design hypotheses

The acceptance path depends on mandatory controls. Model-family diversity is separate. It is an untested review design choice, not an acceptance control.

Type	Control	Current assessment
Mandatory	Approved plan and acceptance conditions	Reported in use, with non-universal coverage
Mandatory	Continuous integration evidence bound to the reviewed commit	Specified
Mandatory	Fail-closed evidence verification	Specified
Mandatory	Review separate from implementation	Reported in use
Mandatory	Human acceptance decision	Specified in the workflow
Mandatory	Bounded repair and terminal outcomes	Specified
Supporting	Classified review findings	Reported in use
Untested hypothesis	Different-model-family review	No comparative defect-detection evidence

The records state that implementation and review use an agreed plan. They also state that the reviewing agent uses a different model family from the implementing agent, and that findings are classified by severity and category.

Invocation records, finding counts and a defined assessment scope are absent.

The plan is the contract

Implementation starts from an agreed plan and specification. The plan defines:

Intended behaviour
Affected boundaries
Validation approach
Acceptance conditions

Implementation and review use the same plan as their reference.

A precise plan can still contain an unresolved product decision. If ambiguity reaches implementation, an agent must stop or infer intent. An inferred answer may produce coherent code while addressing the wrong requirement.

Product clarification therefore happens before implementation. Business rules, exceptional cases and acceptable trade-offs belong in the specification. A person decides before code is written.

The factory must implement an approved interpretation of the requirement, not merely produce a plausible change. Acceptance conditions must exist before implementation begins. Current records are insufficient to measure how consistently this happens.

Evidence must belong to the reviewed commit

A plan defines the target. It does not prove that a candidate meets it.

The specified gate requires evidence from the relevant continuous integration checks. Evidence counts only when independently verified and matched to the exact commit under review.

A test summary copied into an agent response is insufficient. It may:

Refer to another revision
Omit a failed check
Describe an environment that cannot be reproduced

The specified response is fail-closed. Missing, malformed or mismatched evidence causes rejection.

A concrete commit-binding protocol is not documented. Missing design details include:

How the gate obtains the reviewed commit identity
How evidence carries the tested commit identity
How the identities are compared
How the evidence producer is authenticated
Which systems may produce or store trusted evidence
How evidence reuse across revisions is prevented
How verification failures are recorded

Until these interfaces and trust boundaries are documented, commit binding remains a specified control.

Minimum auditable evidence record

An auditable record needs enough data to link the approved requirement, candidate, checks, review and disposition.

Record area	Minimum fields
Plan	Plan identifier, approved version, approval state and acceptance conditions
Candidate	Repository reference and candidate commit identity
Continuous integration	Checks invoked, result for each check, tested commit identity and evidence provenance
Verification	Candidate identity, tested identity, match result, verification result and failure reason where applicable
Review	Reviewer identity or type, reviewed commit identity, decision and classified findings
Repair	Attempt number, repair-budget state and resulting candidate identity
Disposition	Human decision and final state: accepted, replanned or scope changed

These fields define an assessable record. They do not imply that such a schema is implemented.

The minimum state transitions are:

PLAN_APPROVED
    -> CANDIDATE_CREATED
    -> CHECKS_COMPLETED
    -> EVIDENCE_VERIFIED
    -> REVIEW_COMPLETED
    -> HUMAN_DECISION

EVIDENCE_REJECTED
    -> REPAIR_STARTED
    -> CANDIDATE_CREATED

REVIEW_REJECTED
    -> REPAIR_STARTED
    -> CANDIDATE_CREATED

REJECTED_WITH_BUDGET_EXHAUSTED
    -> HUMAN_JUDGEMENT
    -> REPLANNED | SCOPE_CHANGED

Each transition needs a recorded source state, destination state, candidate identity, reason and time. The available material does not show these transition records.

Independent review

The reviewing agent uses a different model family from the implementing agent. Rejection requires a stated engineering reason rather than a general expression of confidence.

Review findings are classified by severity and category. Counts by classification, reviewer type, period and scope are unavailable.

All recorded review findings in the current phase came from automated reviewers rather than people. However, the records do not define the phase or provide the underlying counts.

People still:

Define intent
Approve plans
Resolve ambiguity
Decide whether a reviewed candidate is acceptable

Different-model-family review may reduce shared assumptions, blind spots or interpretations of the specification. No comparison with same-family review shows that it detects more faults. It therefore remains an untested hypothesis.

Repair needs an enforceable state machine

A rejected candidate can return to implementation, but it cannot circulate indefinitely. A repair budget provides the intended boundary.

REJECTED
    |
    |-- budget remains --> IMPLEMENTATION
    |                         |
    |                         v
    |                    EVIDENCE CHECK
    |                         |
    |                         v
    |                       REVIEW
    |
    `-- budget exhausted --> HUMAN JUDGEMENT
                                  |
                                  |-- REPLAN
                                  `-- CHANGE SCOPE

The budget is a safety control, not a measure of engineering effort. Assessing it requires:

The configured repair limit
A record of each repair attempt
The budget state at each rejection
The terminal state reached after exhaustion

Those records are unavailable. Bounded repair remains a specified control.

The codebase is part of the control system

A controlled workflow cannot compensate for an environment that agents cannot reproduce.

Readiness requirements cover:

Written engineering conventions
A pinned runtime
Dependency locking
An architecture map
Testing guidance
Common failure modes
Consistently invocable checks

Agent tooling and default-branch protection are reported as the least consistently satisfied areas. The assessed estate, scoring method, dates and underlying counts are not available.

Where readiness differs between codebases, the same workflow can provide different levels of assurance. Each undocumented convention or environment-specific step pushes a decision back into inference and makes acceptance harder to audit.

Measurement remains incomplete

Previously reported percentages and the delivery-activity comparison are omitted because they cannot be reproduced from the available records.

Measure	Reporting status	Evidence required
Readiness controls passing	Not independently reportable	Checklist, passing count, assessed count, scoring method, measurement date and estate coverage
Approved-plan coverage	Not independently reportable	Eligible work-item definition, approved-plan count, eligible count, sample size and period
First-pass acceptance	Not independently reportable	Accepted candidate count, assessed count, period, eligibility rules and treatment of withdrawn or replanned work
Verified-evidence coverage	Not reportable	Eligible delivery count, verified-evidence count and documented verification mechanism
Delivery activity change	Not independently reportable	Activity unit, phase dates, phase durations and rules establishing comparability
Review findings	Not reportable by classification	Counts by severity, category and reviewer type, with period and scope

First-pass acceptance means acceptance through the configured evidence and review path. It does not mean code perfection.

Implementation priorities

The work should proceed in dependency order.

Define assessment scope
- Set the assessment period.
- Define the codebase estate and eligible work.
- Identify the complete record population.
Standardise plans
- Make approved plans and acceptance conditions universal.
- Record product clarifications as durable decisions.
Specify commit-bound evidence
- Define the evidence fields.
- Document producer authentication.
- Document identity matching, storage and trust boundaries.
- Define rejection behaviour for absent, malformed, stale or mismatched evidence.
Record control execution
- Store verification results and failures.
- Store review decisions and classified findings.
- Retain the candidate identity at every stage.
Enforce bounded repair
- Record the configured limit.
- Record each repair transition.
- Ensure budget exhaustion produces a recorded terminal outcome.
Produce an end-to-end trace
- Use an existing delivery record.
- Do not reconstruct missing events.
- Include plan approval, commit identity, continuous integration provenance, verification, review, repair attempts and final disposition.
Establish reporting
- Define metric numerators, denominators, periods, eligibility rules and exclusions.
- Report verified-evidence coverage and review findings only when the required records exist.
Address codebase readiness
- Strengthen agent tooling.
- Strengthen default-branch protection.
- Record the assessment method and codebase coverage.

What this means

The acceptance design has the required control points, but the records do not show the complete path operating against a specific commit. The immediate task is to define the evidence protocol, record state transitions and produce an end-to-end trace without reconstructing missing events.

Originally published at AFT Group Engineering Insights.

Why First-Pass Rate Does Not Measure Engineering Delivery

James Smith — Fri, 28 Aug 2026 19:29:55 +0000

This article explains why a 91.7% first-pass rate does not measure engineering delivery, which definitions and records are missing, and how lifecycle events can make gate, repair and delivery states queryable.

The reported 91.7% first-pass rate describes performance at one gate. The gate definition, denominator and source are unavailable.

The same material reports zero second passes, zero repair cycles and delivery of 52.2% of started work. These figures may use different boundaries or definitions. Without the underlying records, they cannot be reconciled or treated as substantive evidence.

A first-pass result measures one gate. It does not measure delivery.

A change can pass a gate on its first recorded attempt and still await later checks, be superseded or fail to meet the definition of delivered.

Evidence limitations

The original report, dashboard or dataset has not been supplied. The figures cannot be cited or independently verified.

The available material also omits:

Raw numerators and denominators
Cohort dates and observation windows
Exclusion and ageing rules
Definitions of started work, attempt, pass, repair cycle and delivered
The position of review findings relative to the measured gate
Breakdowns by risk class, repository, change type or work age
Repository counts, readiness results and aggregation rules
A policy linking risk classes to planning, review and approval controls
A recorded end-to-end change trace
Cost-attribution rules and comparable cost periods

This matters when classifying unfinished work. A change still open at the end of an observation window is not necessarily a failed delivery. That judgement requires an explicit ageing rule.

The reported figures below identify the claims under examination. They are not verified results.

Measure	Reported result
First-pass rate	91.7%
Delivered share of started work	52.2%
Share requiring a second pass	0%
Repair cycles per started change	0
High-risk share of work	21.7%
Critical financial share of work	13%
Approved-plan coverage	34.8%
Overall repository-readiness pass rate	92.6%
Agent-tools pass rate	33.3%
Default-branch-protection pass rate	33.3%
Blocking share of recorded review findings	63.3%

The material states that automated reviewers raised all recorded findings and that every finding was categorised as a correctness issue. It also lists model-routing assignments. Those assignments describe configuration, not measured performance. They provide no evidence about assurance outcomes.

Define the gate and delivery state

A first-pass measure needs a fixed boundary. Its definition must state:

Which artefact is assessed
What starts an attempt
Which checks apply
What records a pass or failure
How corrections before the result are treated
How reopened or resubmitted work is treated

Moving the gate from implementation to review, or from review to production readiness, changes the result without necessarily changing the engineering work.

Delivery needs the same precision. It should be represented by an observable event, not reconstructed later from incomplete workflow states.

Denominators change the result

Consider four synthetic changes. These records are illustrative and do not come from the reported data.

Change	Recorded events	Final state
A	First attempt passed	Delivered
B	First attempt passed	Still open
C	First attempt failed, repair completed, second attempt passed	Delivered
D	First attempt passed	Superseded

These records produce different measures:

first-pass rate = 3 ÷ 4 = 75%
delivered share = 2 ÷ 4 = 50%
recorded second-attempt share = 1 ÷ 4 = 25%
repair cycles per started change = 1 ÷ 4 = 0.25

The first-pass rate is valid for the specified gate. It says nothing about whether changes B and D were delivered.

This distinction is missing from the reported 91.7% and 52.2% figures. Both may be correct within their respective boundaries, but the underlying records are needed to establish that.

Make repair activity visible

The recorded review findings require explanation:

Automated reviewers raised all recorded findings
Every recorded finding was categorised as a correctness issue
63.3% of recorded findings were blocking

The material does not show when these findings occurred or what happened afterwards. Corrections may have taken place before the recorded gate, appeared as continuations rather than repair cycles, or moved outside the tracked workflow.

Zero recorded repair cycles does not establish that no correction occurred. Until repair states are explicit, the first-pass rate should be reported only as gate performance.

Unrecorded correction also affects capacity and forecasting. Two changes can share the same recorded outcome while requiring different amounts of engineering work.

Record lifecycle events

A common event record would make attempts, repairs and delivery states queryable. This is a proposed measurement design, not a description of the current data.

Field	Purpose
`change_id`	Identifies the change across its lifecycle
`event_id`	Identifies an individual event
`event_time`	Orders events and applies observation boundaries
`event_type`	Records start, attempt, result, repair or delivery
`gate_name`	Identifies the measured gate
`attempt_number`	Distinguishes first and later attempts
`result`	Records pass, fail or blocking disposition
`risk_class`	Supports comparison between work classes
`repository_id`	Supports repository-level analysis
`superseded_by`	Records replacement by another change
`recorded_by`	Identifies the event source

A minimum event set could include:

change_started
gate_attempted
gate_passed
gate_failed
repair_started
repair_completed
change_superseded
change_abandoned
change_delivered

Corrections before gate_attempted would not count as post-attempt repair. Corrections after gate_failed would appear through repair events and a later attempt number.

The resulting measures would be:

first_pass_rate =
  changes passing the specified gate on attempt 1
  ÷ changes with recorded activity at that gate

delivered_share =
  started changes with a delivered event
  ÷ all started changes in the cohort

recorded_second_attempt_share =
  changes with attempt_number = 2 at the specified gate
  ÷ changes with recorded activity at that gate

repair_cycles_per_started_change =
  completed repair cycles
  ÷ all started changes in the cohort

A recorded pass counts as evidence that the change reached the gate. This avoids requiring a separate gate_attempted event when a gate_passed event already exists.

A full reference query belongs in an implementation appendix or companion repository. The essential gate logic is:

MAX(CASE
    WHEN gate_name = :gate_name
     AND event_type IN ('gate_attempted', 'gate_passed', 'gate_failed')
    THEN 1 ELSE 0
END) AS reached_gate,

MAX(CASE
    WHEN gate_name = :gate_name
     AND event_type = 'gate_passed'
     AND attempt_number = 1
    THEN 1 ELSE 0
END) AS passed_first_attempt,

MAX(CASE
    WHEN gate_name = :gate_name
     AND attempt_number = 2
     AND event_type IN ('gate_attempted', 'gate_passed', 'gate_failed')
    THEN 1 ELSE 0
END) AS recorded_second_attempt

This detects a recorded second attempt. It does not show that the attempt was required unless the query also identifies a preceding failure.

Separate results by risk

The material reports that high-risk work accounts for 21.7% of the mix and critical financial work accounts for 13%. These figures remain unverified without the underlying source.

Such work warrants stronger planning, challenge and review than low-risk implementation. A throughput comparison that combines risk bands may favour easier work rather than safe delivery of valuable work.

Approved plans reportedly cover 34.8% of changes. That result cannot be judged without a policy stating which risk classes require approval. The relevant comparison is coverage against the applicable policy, not coverage in isolation.

Report readiness by control

The material reports an overall repository-readiness pass rate of 92.6%. It also reports complete pass rates for:

Runtime pinning
Lockfiles
Observed pipeline checks
One-command checks
Testing guidance
Absence of committed secrets

Agent tools and default-branch protection each reportedly pass at 33.3%.

The aggregate cannot be interpreted without repository counts, repository-level results and the weighting method. Reporting should show each control by repository and state whether the total weights repositories, checks or observations equally.

Tie cost to delivery

Cost per delivered change keeps the denominator tied to an accepted outcome.

Attributable cost should include:

Agent use
Human planning and review
Delivery-pipeline activity
Verification
Repair effort

cost per delivered change =
  attributable cohort cost
  ÷ delivered changes in the cohort

Started work that does not reach delivery should remain visible unless an explicit attribution rule excludes it.

No directional cost claim can be made from the supplied material. Comparison requires consistent cost attribution, delivery definitions, observation windows and risk bands.

Publication criteria

A later report should not present first-pass performance as delivery efficiency until:

The underlying source can be cited and verified
Numerators, denominators, cohort dates and exclusions are published
Gate, delivery, observation and ageing rules are explicit
Lifecycle events expose attempts, repairs and final disposition
Risk policy defines required planning and review
Readiness aggregation and repository-level results are available
Cost periods and attribution rules are comparable
At least one anonymised change has a complete recorded trace

No owners or delivery dates are supplied, so none can be assigned.

What this means

The current figures describe reported gate performance, delivery share and control indicators. They do not form a reconciled delivery measure. The full change lifecycle must be recorded and verified before delivery, cost and risk can be compared on the same basis.

Originally published at AFT Group Engineering Insights.

Why Agent-Led Code Repair Needs a Hard Cap

James Smith — Fri, 28 Aug 2026 19:29:18 +0000

This article explains why agent-led code repair needs a hard cap, how the policy applies that boundary and where its published evidence remains incomplete. Repeated attempts can compound the assumptions behind the first failure. A hard cap limits that exposure and returns unresolved work to a person, although the policy’s cap values remain unpublished.

A repair cap is a safety boundary, not a throughput target.

Why repair needs a hard cap

An agent that misunderstands a requirement must interpret two things during repair:

the original requirement;
the reviewer’s objection to its implementation.

The second attempt therefore depends on the same reasoning that produced the first. A correction can extend beyond its intended scope, alter behaviour, disturb unaffected code and introduce further assumptions.

Consider a hypothetical sequence:

An agent interprets an unsettled business rule and implements that interpretation.
Review rejects the resulting behaviour.
The agent treats the objection as a local code defect rather than evidence that the rule remains unsettled.
Its repair changes the rejected path and adjacent behaviour that depended on the original interpretation.
Review must now assess both the initial misunderstanding and the wider correction.

This sequence is illustrative. The available material does not include a representative repair trace. It does, however, show why unlimited retries are unsafe: each attempt can carry the original misunderstanding into a broader change.

The hard cap stops that process. It prevents repeated automated attempts under unresolved assumptions and forces re-specification when the budget is exhausted.

The policy boundary

The current policy assigns a repair budget to each risk band. Only blocking or major findings can trigger repair, and budget must remain.

Within the available material, the severity labels define eligibility rather than detailed operational criteria:

Blocking: eligible to trigger repair when budget remains.
Major: eligible to trigger repair when budget remains.
Minor: does not automatically reopen implementation.
Correctness: reported as a finding category, but its relationship to blocking and major classifications is not defined.

The classification criteria are not available. The actual budgets and cap values for each risk band are also unpublished.

Risk affects the permitted repair budget because the consequence of an incorrect repair varies. A low-risk change and a critical financial change do not receive identical permission to iterate. The evidence does not state how their budgets differ.

When a change exhausts its budget, it returns to a person for re-specification. It does not receive another automated attempt under the same assumptions. The person can:

narrow the requirement;
resolve a disputed rule;
split the change;
reject the implementation approach.

The repair workflow

The policy supports the following state model:

State	Condition	Next action
Review and evidence	Implementation enters its configured checks	Accept or reject
Accepted first pass	Checks complete without another implementation pass	Complete
Rejected	A review or evidence check fails	Assess repair eligibility
Repair eligible	The finding is blocking or major, and budget remains	Attempt a tightly scoped repair
Repair ineligible	The finding is minor, or another eligibility condition fails	No automatic repair
Cap exhausted	No repair budget remains	Return to a person for re-specification
Human escalation	Assumptions or approach require reconsideration	Narrow, clarify, split or reject the approach

The hard cap matters at two points. It bounds the number of eligible attempts, then changes ownership when automated repair can no longer proceed safely.

First-pass acceptance records whether a change completes its configured review and evidence path without another implementation pass. It measures an outcome, not effort, elapsed time, code volume or responses to comments. It also does not establish code quality in the abstract.

Using first-pass acceptance as a proxy for effort would encourage teams to reach review quickly rather than remove uncertainty before coding.

Reduce repair pressure before implementation

The working hypothesis is that much second-pass pressure begins with ambiguous intent. Possible sources include:

incomplete acceptance conditions;
unresolved business rules;
unclear ownership of side effects;
plans that defer too much design to implementation.

This remains a hypothesis rather than a measured causal result.

Current policy requires approved plans and settled clarification questions before implementation. A useful plan establishes:

behavioural boundaries;
affected components;
validation expectations;
explicit exclusions.

The aim is to turn clarification answers into implementation constraints rather than review discoveries. A cap then acts as the final boundary when those controls fail to remove enough uncertainty.

Evidence and policy limitations

Two measurements are reported:

Measure	Reported result
First-pass acceptance	91.7%
Repair-cycle distribution	Entirely in the no-repair category

These measures describe different events. First-pass acceptance records passage through review and evidence without another implementation pass. Repair-cycle distribution records whether repair was executed. They cannot be treated as one funnel without a shared cohort and linked state data.

The source also reports blocking, major and correctness findings. It does not explain how those findings relate to a distribution containing no executed repairs. They may cover different periods, populations or workflow stages, but the evidence does not establish that.

The source states that roughly a third of changes have an approved plan. Planning is selected partly by risk and complexity, so a direct comparison between planned and unplanned work could confuse selection effects with plan quality.

The evidence does not provide:

repair-cap values or budget assignments by risk band;
definitions and stages for the configured review and evidence path;
cohort sizes, dates, inclusion criteria or denominators;
linked states from rejection through final outcome;
classification criteria and context for the reported findings;
a representative repair sequence;
evidence of a causal relationship between planning and first-pass acceptance.

These gaps limit independent assessment of both the policy and the reported measurements. They do not remove the causal case for bounding repeated repair under unresolved assumptions.

What needs to be tightened

The stated instrumentation and process changes are to:

publish the cap and budget assigned to each risk band;
define the configured review and evidence path;
align first-pass and repair denominators;
define cohort dates, sizes and inclusion criteria;
link rejection, repair eligibility, repair execution and final outcome;
record why rejected changes are not eligible for repair;
trace correctness findings to unresolved requirements only where evidence supports the link;
extend consistent use of the plan gate;
settle clarification questions before implementation begins.

The cap remains hard under the current policy. Publishing its values is necessary to make the control operationally clear.

What this means

A hard cap limits the propagation of unresolved assumptions across automated repair attempts and returns exhausted work to a person. The policy establishes that boundary, but its cap values, severity criteria and supporting measurement context still need to be published.

Originally published at AFT Group Engineering Insights.

Bounded Agent Repair: Measuring Review Demand Requires Aligned Evidence

James Smith — Fri, 28 Aug 2026 19:29:17 +0000

This article defines bounded agent repair, first-pass acceptance and the evidence required to measure repeated review demand. The policy is clear, but current event records and metric definitions are not sufficient to verify its effects.

Agent-led repair can avoid a visible hand-off. It also asks the agent that misunderstood a requirement to reinterpret its work and the reviewer’s objection.

Correction may then extend beyond the original change, alter unaffected code or introduce further assumptions. Each cycle requires reviewers to understand and verify another implementation pass. No sourced change trace is available to quantify these effects.

The objective is not to make agents better at looping. It is to stop ambiguity reaching review.

Bounded repair is a policy position. Current evidence does not establish whether it reduces review demand, defect escape rates or failed acceptance.

First-pass acceptance

First-pass acceptance asks whether an implementation completes its configured review and evidence path without another implementation pass.

It measures an outcome, not activity. It does not reward generated code, responses to comments or repeated attempts. Nor does it measure effort, elapsed time or code quality in the abstract. Review can assess only the requirements, checks and evidence available.

A carefully specified change may require substantial reasoning and still pass on its first attempt. A quick change may fail because a requirement remains unsettled. Using acceptance as a proxy for effort would encourage teams to rush towards review rather than remove uncertainty before coding.

Bounded repair policy

Repair is a bounded exception:

Only blocking or major findings trigger repair.
Minor observations do not automatically reopen implementation.
Each risk band has a repair budget with a hard cap.
Exhausting the budget returns the change to a person for re-specification.

An agent cannot continue iterating merely because it can propose another patch.

After budget exhaustion, the person can:

narrow the requirement;
resolve a disputed rule;
split the change;
decide that the implementation approach is wrong.

The cap is a safety boundary, not a throughput target. It is intended to prevent repeated implementation and review cycles from becoming hidden delivery churn.

Intended process

The policy separates rejection, repair eligibility, repair execution and human disposition:

Implementation
    ↓
Configured review and evidence path
    ├── Accepted → Complete
    └── Rejected
          ├── Not eligible for repair → Human disposition
          └── Eligible for repair
                ├── Budget available → Bounded repair → Review
                └── Budget exhausted → Human re-specification

This is an intended policy model, not a verified event schema.

Human disposition may narrow the requirement, resolve a rule, split the change or reject the implementation approach. No sourced case currently traces these steps from rejection to final disposition.

Measurement and evidence limits

A complete first-pass acceptance metric requires:

a numerator covering changes accepted without another implementation pass;
a denominator covering all eligible changes entering the configured path;
a defined cohort and measurement period;
explicit treatment of human-routed and abandoned changes;
documented inclusion and exclusion rules.

These details are unavailable. The previously reported 91.7% rate is therefore omitted until its basis is documented.

The available measurements describe different events:

First-pass acceptance records acceptance through the configured review and evidence path.
Repair-cycle distribution records whether repair was executed.
The recorded repair-cycle distribution contains only the no-repair category.

These measurements cannot yet form a single funnel because their denominators are not aligned. The data do not fully account for rejected changes that were ineligible for repair, abandoned or routed to a person. The zero-repair distribution therefore cannot be reconciled with first-pass failures.

Further missing evidence includes:

a verified event schema linking rejection, eligibility, repair execution, abandonment, human routing and final disposition;
counts for intermediate and terminal dispositions;
published definitions for finding severity, risk bands, repair budgets and configured review paths;
documented review controls and reviewer-independence criteria;
a sourced change trace covering rejection through final disposition;
aligned cohort counts for testing review demand, acceptance outcomes or defect escape effects.

Until these definitions and records exist, the operational effect of bounded repair cannot be assessed.

Untested hypothesis: ambiguity creates repair pressure

The working hypothesis is that second-pass pressure often begins with ambiguous intent. Possible sources include:

incomplete acceptance conditions;
unresolved business rules;
unclear ownership of side effects;
plans that leave too much design to implementation.

Current evidence does not quantify this relationship or establish that approved plans improve first-pass acceptance.

Planning is selected partly by risk and complexity. A direct comparison between planned and unplanned work could therefore confuse selection effects with plan quality.

The policy places approved plans and settled clarification questions before implementation. A useful plan establishes:

behavioural boundaries;
affected components;
validation expectations;
explicit exclusions.

Clarification answers then become implementation constraints rather than review discoveries. Whether this reduces repair pressure remains untested.

What we are tightening next

We are:

defining the first-pass numerator, denominator, cohort and measurement period;
aligning first-pass acceptance with repair-event records;
recording accepted, rejected, repair-eligible, repaired, abandoned and human-routed dispositions separately;
testing whether those states are mutually exclusive and collectively exhaustive;
recording why rejected changes are or are not eligible for repair;
tracing correctness findings to unresolved requirements only where evidence supports the link;
applying the plan gate consistently;
requiring clarification questions to be settled before implementation;
documenting review controls, risk bands and repair caps.

The repair cap remains hard.

What this means

Bounded repair is intended to expose repeated implementation and review demand rather than conceal it inside an open-ended loop. Complete metric definitions, aligned records and a sourced change trace are required before that intended effect can be treated as an observed outcome.

Originally published at AFT Group Engineering Insights.

Designing Commit-Bound Acceptance Gates for an Internal Software Factory

James Smith — Fri, 28 Aug 2026 19:28:47 +0000

This report explains how to design acceptance gates around approved plans, commit-specific evidence and independent review. It also identifies which controls are intended, partial or reported as operating, and why the available records do not support an end-to-end audit.

Generating changes is not the difficult part of building an internal software factory. Establishing a trustworthy basis for accepting them is.

An agent’s account of its own work is not evidence that the work is correct.

Why self-reporting is insufficient

During the early build phase, coding agents received work without a plan contract, acceptance gate or independent review. The implementing agent also reported what it had done.

Acceptance rested on that report rather than on evidence tied to the resulting software. This supported exploration of agent-led delivery. It was not sufficient for dependable engineering.

The acceptance model now separates:

Approved plans and specifications
Product clarification
Implementation
Commit-specific CI evidence
Independent review
Repair, rejection or acceptance

These controls are at different implementation stages. The records do not show them operating together on a single candidate change.

Planning is the contract

An agreed plan and specification provide the reference point for implementation and review. The plan defines:

Intended behaviour
Affected boundaries
Validation approach
Acceptance conditions

A technically precise plan can still contain an unresolved product decision. If ambiguity reaches implementation, an agent must stop or infer intent. That inference may produce coherent code that solves the wrong problem.

Product clarification therefore belongs before implementation. The specification records business rules, exceptional cases and acceptable trade-offs. A person decides the intended behaviour before code is written.

The target is not merely a plausible change. It is an approved interpretation of the requirement, implemented against conditions defined in advance.

Approved plans are not yet universal. Product decision records are also too limited to measure coverage.

Evidence must be independent and commit-specific

A plan defines the target. It does not prove that a candidate change reached it.

Continuous integration is intended to provide evidence from the required checks. That evidence counts only when independently verified and attached to the exact commit under consideration.

A test summary copied into an agent response is insufficient. It may:

Refer to another revision
Omit a failed check
Describe an environment that cannot be reproduced
Lack evidence that the required checks ran

A commit-bound evidence interface needs four defined elements:

Element	Required definition
Identity	The immutable commit identity under consideration
Inputs	CI results, required attestations and provenance records
Validation	Checks that the evidence belongs to the commit and meets the required conditions
Failure states	Missing, malformed, incomplete, mismatched or non-compliant evidence

The intended gate fails closed for each listed failure state. However, the records do not describe how commit identity is verified, how CI results are bound to that identity or how attestations are validated. They also provide no workflow evidence that acceptance is blocked.

Commit-bound evidence and fail-closed enforcement therefore remain design requirements rather than auditable controls.

Review is separate, but rejection policy is incomplete

The records state that the reviewing agent comes from a different model family from the implementing agent. They also state that findings are classified by severity and category.

Classification and rejection are separate controls. A workflow can classify findings without a complete policy for deciding which classifications block acceptance. The current records do not define:

The severity taxonomy
Rejection thresholds
Findings that require escalation
Findings that permit acceptance
How multiple findings are combined

Severity classification is reported as operating, while rejection policy remains unspecified. Without workflow records or review artefacts, neither claim can be independently audited.

Cross-family review may reduce repeated assumptions and blind spots. This remains a hypothesis. No comparative evidence shows whether it finds defects that same-family review would miss.

Rejected work needs a defined exit

A capped repair budget is intended to prevent a candidate circulating indefinitely between implementation and review. When the budget is exhausted, the work should return for:

Human judgement
Replanning
A change of scope

The records do not define how the cap is selected, configured or enforced. They also do not define the workflow state entered when the budget is exhausted.

The repair limit is therefore a design principle. It becomes an auditable control only when workflow definitions record the budget, transitions and terminal states.

The codebase is part of the control system

A strong workflow cannot compensate for an environment that agents cannot reproduce. Readiness requirements cover:

Written engineering conventions
A pinned runtime
Dependency locking
An architecture map
Testing guidance
Common failure modes
Consistently invocable checks

The records identify weaknesses in agent tooling and default-branch protection. Repository policies or configurations supporting those findings are not available here.

The requirements also lack recorded enforcement methods. The available material does not show which checks are automated, which are reviewed manually or which block acceptance. Codebase readiness must therefore be treated as partial and not yet reproducibly assessed.

Every undocumented convention or environment-specific step pushes a decision back into inference. It also makes acceptance harder to audit.

How control status is assessed

This report uses four status terms:

Status	Meaning
Intended	Required behaviour is described, but implementation evidence is unavailable
Partial	Some constituent behaviour is reported or required, but coverage, enforcement or supporting records are incomplete
Reported operational	The records state that the control operates, but linked configurations or artefacts are unavailable for independent verification
Auditable	Implementation records show the control acting on a specific candidate and commit

No control can be marked auditable from the available material. The source records and repository policies needed to verify operational claims have not been supplied.

Current control status

Control	Status	Basis
Agreed plan and specification	Partial	Plans provide a shared reference but are not universal
Product clarification before implementation	Partial	Human clarification is required, but durable decision records remain limited
Separate automated review	Reported operational	The records state that implementing and reviewing agents are separate; supporting workflow records are unavailable
Severity and category classification	Reported operational	Classification is reported, but the taxonomy and review artefacts are unavailable
Review rejection policy	Intended	Thresholds, escalation rules and rejection criteria are unspecified
Cross-family review	Reported operational	Different model families are reported as in use; effectiveness is unproven
Commit-bound CI evidence	Intended	Binding and verification mechanisms are not recorded
Attestation and provenance validation	Intended	Required attestations and validation checks are unspecified
Fail-closed acceptance gate	Intended	Failure conditions are stated, but enforcement evidence is unavailable
Capped repair budget	Intended	Configuration, enforcement and exhaustion transitions are unspecified
Codebase readiness controls	Partial	Requirements exist, but testing and enforcement methods are not recorded

The missing delivery trace

A source-backed trace is needed to show one candidate passing through:

Specification and plan approval
Product clarification
Implementation
Identification of the candidate commit
Commit-bound CI checks and attestations
Independent review and classified findings
Repair, rejection or escalation
Final acceptance

Each stage must link the control to its corresponding artefact. Required artefacts include:

The approved plan
The product decision record
The commit identifier
CI evidence
Attestations
Review findings
Repair history
The acceptance decision

No such trace is available. Adding an example without those records would repeat the self-reporting problem that the acceptance model is intended to remove.

Measurement limits

The records do not support publication of figures for:

Readiness
Approved-plan coverage
First-pass acceptance
Verified-evidence coverage
Delivery activity

Denominators, dates, reporting windows, work-item counts and eligibility rules are missing.

The records also do not establish a time saving, productivity multiplier or increase in individual working speed. Comparative data for cross-family and same-family review is unavailable.

What changes next

The remaining work is to turn stated requirements into recorded controls:

Make approved plans universal
Record product clarifications as durable decisions
Define metric denominators, reporting windows and eligibility rules
Publish the readiness checklist, enforcement method and scoring method
Specify how CI evidence is bound to an immutable commit identity
Define required attestations, provenance checks and validation failures
Record an anonymised end-to-end delivery trace
Define the severity taxonomy, rejection thresholds and escalation rules
Establish a comparison method for cross-family review
Define repair-budget configuration, enforcement and exhaustion states
Provide workflow records and repository policies for reported controls
Strengthen agent tooling
Strengthen default-branch protection

What this means

The acceptance model separates implementation from planning, evidence and judgement. The available records establish design intent more clearly than implementation. Dependable acceptance requires reproducible evidence that every required control operated on the exact commit accepted.

Originally published at AFT Group Engineering Insights.

Why First-Pass Rate Does Not Measure End-to-End Delivery

James Smith — Fri, 28 Aug 2026 19:28:26 +0000

You will learn why first-pass rate and cohort delivery need separate measures. They cover different populations and answer different questions. A change can pass its first measured gate without reaching delivery.

Code volume sits near the start of delivery. It does not show whether a change survives review, passes later checks, meets its business intent or reaches a usable state.

A high first-pass rate measures gate performance, not end-to-end delivery.

The denominator mismatch

The supplied measurement draft reports:

First-pass rate: 91.7%
Delivered share of started work: 52.2%

The measures use different denominators:

First-pass rate
= changes passing their first recorded gate attempt
  / changes with a recorded first gate attempt

Delivered share
= changes reaching delivery
  / changes in the defined started cohort

Started work that has not reached the gate is excluded from the first calculation but included in the second. Work can also pass the gate and then remain pending, enter repair, be superseded or be abandoned.

The gate boundary also matters. Moving it from implementation to review, or from review to production readiness, changes the meaning of the result. The reported first-pass rate applies only to its defined gate. It is not a proxy for delivery.

The measurement boundaries should remain explicit:

Use gate attempts to measure first-pass performance.
Use a defined started cohort to measure delivery.
Separate pending work from terminal non-delivery.
Record transitions between the two boundaries.

The reported results remain unauditable

The source measurement draft and raw counts are unavailable. No citation or link has been supplied. The metric definitions, cohort rules, observation window, collection methods and source systems are also missing.

The figures therefore remain reported observations. They cannot yet support comparisons across teams, periods or work classes.

Other reported measures have the same limitation:

Metric	Reported result	Intended scope
Share requiring a second pass	0%	Recorded repeat passes
Repair cycles per started change	0	Recorded correction cycles
Blocking review findings	63.3%	Severity of recorded findings
Approved-plan coverage	34.8%	Share of changes with an approved plan

Publication of the source should include a raw-count appendix covering:

Numerators and denominators for every percentage and zero value
The exact gate definition
Definitions of started work, gate attempts, repair cycles and delivery acceptance
Cohort inclusion rules and dates
The observation window
The age distribution of unfinished work
Finding-level dispositions and linked revisions
Collection methods and source systems

Until these materials are available, the results should not be treated as a substantive case.

Delivery needs a validated lifecycle model

End-to-end measurement needs consistent states and transition events. The following table is a proposed baseline, not a confirmed representation of the measured workflow.

Current state	Transition event	Next state
Started	Implementation submitted	Implemented
Started	Work replaced	Superseded
Started	Work stopped	Abandoned
Implemented	Gate attempt passes	Gate passed
Implemented	Gate attempt fails and correction starts	In repair
Implemented	Work replaced	Superseded
Implemented	Work stopped	Abandoned
Gate passed	Required reviews and checks pass	Approved
Gate passed	Accepted finding requires correction	In repair
Gate passed	Work replaced	Superseded
Gate passed	Work stopped	Abandoned
In repair	Revised implementation submitted	Implemented
In repair	Work replaced	Superseded
In repair	Work stopped	Abandoned
Approved	Delivery acceptance occurs	Delivered

In this baseline, delivered, superseded and abandoned are terminal states. Pending is not a separate terminal state. It describes work with no later transition at the observation cut-off.

The model must be checked against the actual workflow. Validation must determine how it represents:

Rollback after delivery
Rejection after approval
Partial deployment
Reopened work
Work that leaves the tracked workflow

If these events occur, the model needs corresponding transitions. Otherwise, it will hide outcomes or assign them to the wrong state.

Each transition should record a persistent change identifier, timestamp, responsible stage and applicable work or risk class. Each review finding should link to its disposition and any revision that addresses it.

Pending work changes the result

A recent cohort may contain substantial unfinished work. Counting every unfinished change as non-delivery lowers the delivered share without showing whether those changes will later reach delivery.

Reporting should distinguish:

Delivered work
Superseded work
Abandoned work
Work still pending at the cut-off

The following example is hypothetical. It does not represent the reported data.

A cohort contains 100 started changes. Its status changes between two observation dates:

State	Earlier cut-off	Later cut-off
Delivered	40	58
Pending	45	17
Abandoned	8	12
Superseded	7	13
Started cohort	100	100

At the earlier cut-off:

Delivered share = 40 / 100 = 40%
Pending share   = 45 / 100 = 45%

At the later cut-off:

Delivered share = 58 / 100 = 58%
Pending share   = 17 / 100 = 17%

The cohort size does not change, but its observed outcome does. Some pending work reaches delivery. Other work becomes abandoned or superseded. A delivered-share figure therefore needs both a cohort definition and an observation date.

The method must also define cohort maturity. It may use a documented maturity threshold or a documented time-to-delivery treatment. The available material provides neither, so no method can yet be selected or assessed.

Blocking findings expose a recording gap

The supplied draft reports that automated reviewers raised all recorded findings and that every recorded finding was categorised as a correctness issue. It also reports that 63.3% were blocking, while the share requiring a second pass and repair cycles per started change were both zero.

These observations do not prove that repair occurred. They raise a narrower question: what happened after each blocking finding?

A finding record should show whether the affected change:

Was corrected before the measured pass
Continued without a recorded repair cycle
Remained pending
Was superseded or abandoned
Left the tracked workflow
Reached delivery

Without these links, the data cannot show whether findings create repair work or how much engineering capacity that work consumes.

Keep the measures separate

Complete transition records support several distinct measures:

Measure	Numerator	Denominator
First-pass rate	Changes passing their first recorded gate attempt	Changes with a first recorded gate attempt
Delivered share	Changes reaching delivery	Started changes in the defined cohort
Repair cycles per started change	Transitions into repair	Started changes
Terminal non-delivery share	Superseded or abandoned changes	Started changes
Pending share	Changes without a terminal state at the cut-off	Started changes

These measures describe gate quality, correction activity, workflow progress and final outcomes. Combining them produces an ambiguous result.

Measurement priorities

The immediate task is to make the existing claims auditable:

Publish or cite the source measurement draft.
Add a raw-count appendix for every reported result.
Define the measured gate and each lifecycle state.
Publish the cohort rules, dates and observation window.
Define the treatment of immature and unfinished work.
Validate the proposed state model against the actual workflow.
Record transitions against persistent change identifiers.
Link findings to dispositions, revisions and final states.
Compare teams or periods only after definitions and collection methods remain stable.

What this means

First-pass rate describes one gate. End-to-end delivery requires a defined cohort, explicit outcomes and transition records from start to observation cut-off.

Originally published at AFT Group Engineering Insights.