Design AI Usage Metrics So Teams Can Challenge the Story They Tell

Primary source: https://github.blog/changelog/2026-07-17-repository-level-github-copilot-usage-metrics-generally-available/.

Repository-level Copilot metrics are now generally available, according to GitHub's July 17 changelog. A dashboard can show where activity occurs, but activity is not automatically useful work.

Design a review card with two layers:

observed:
  repository: stable identifier
  period: explicit dates
  usage: API metric with definition
not_observed:
  accepted_work: requires workflow evidence
  quality: requires review/incident data
  causality: not established

Ask researchers to evaluate three scenarios: a high-usage repository with low merge volume, a low-usage critical library, and a renamed repository with split history.

Question	Stop condition
Can users state what metric means?	definition misunderstood
Can they find missing data?	absence looks like zero
Can they challenge recommendation?	dashboard presents inference as fact
Is identity protected?	unnecessary user-level exposure

Include date range, denominator, freshness, exclusions, and a link to the metric definition beside any recommendation.

This is a design hypothesis, not evidence that usage causes productivity. What caveat must be visible before a manager compares two repositories?