Measure Copilot Cost per Retained Change, Not Accepted Suggestion

An accepted AI suggestion is an event, not a durable outcome. If the code is rewritten tomorrow, acceptance rate still calls it a success.

For an adoption review, I would connect three timestamps:

suggestion:
  accepted_at: 2026-07-19T09:00:00Z
  repository: api
  task_type: test
change:
  retained_lines_24h: 31
  rewritten_lines_24h: 9
  reverted_at: null
review:
  human_minutes: 12
  incident_link: null

Then report a funnel rather than one flattering percentage:

shown → accepted → merged → retained_24h → retained_14d

A useful unit is cost per retained task:

(tool cost + review labor + rework labor) / retained tasks

“Retained” needs a written contract. For example: the change remains merged after 14 days, passes required checks, and has not caused a linked rollback. Line survival alone is weak because formatting and refactoring can change lines without rejecting the solution.

Segment the result by task type and repository. Boilerplate tests and unfamiliar security changes should not be blended into one portfolio average. Also publish counter-metrics: review time, escaped defects, rollback rate, and developer-reported interruption.

GitHub documents available fields and limitations in its Copilot metrics API. Those product metrics can be inputs, but the retained-task join belongs to the adopting organization and should be versioned like any other analytics contract.

My pilot gate would be simple: expand only if retained-task cost beats the existing workflow without worsening rollback rate. Otherwise, change the workflow before buying more seats. Record the baseline before enabling the tool, and keep one comparable task cohort outside the rollout; without that counterfactual, a rising retention rate may only reflect easier work entering the sample.

What retention window would make an accepted change meaningful for your team?