GitHub Copilot MCP Usage Metrics Without False Signals

GitHub's latest Copilot usage metrics update finally names things. Instead of lumping every extension into one undifferentiated bucket, the usage metrics API now reports the names of your customer configured skills, custom agents and MCP servers, and each record can carry 25 entries instead of five. That is great news for adoption dashboards, and a trap for anyone who reads "used a lot" as "works well".

TL;DR

• The usage metrics API now names your own skills, agents and MCP servers, with 25 entries per record where it used to be five. You can finally see which internal tools people actually touch.
• A usage count is an adoption signal, not a quality signal. Join it to evaluation outcomes before you make any rollout call.
• Telemetry has gaps (clients that never report, telemetry that gets blocked), and instrumentation does not enforce permissions. Treat the numbers as a floor and keep your guardrails somewhere else.

What changed in the Copilot usage metrics API

Short version: named entries, and more of them.

Before this change, if your org ran three internal MCP servers and a handful of custom agents, the usage data mostly told you that "something custom" got used. Now the record carries the names you configured, which means you can answer questions like "is anyone actually using the read only billing server we shipped last sprint?" without sending a survey nobody fills in.

The jump from five to 25 entries per record matters more than it sounds. With five slots, a busy developer who touched a dozen tools in a day got truncated, and the long tail (usually the internal tools you care about most) fell off the end. At 25, most of that tail survives.

What it does not tell you: whether the call helped. A connection is not a success, and an invocation is not a merged pull request. That gap is the whole point of this post, and it is the same gap I keep running into when evaluating LLM agents in production: activity metrics look fantastic right up to the moment you check outcomes.

Collect named MCP and skills usage

Pull the data on a schedule and land it in your own table. Don't build dashboards straight off the API response. You'll want history, and you'll want to join it with your own data later.

I'm not going to reproduce GitHub's response schema here; read it from the API docs for your plan and map the field names once. What matters is the normalized shape you store. Here is the normalizer I'd write, with the raw record type kept deliberately loose:

// normalize-usage.ts
// Map whatever the usage metrics API returns into one row per named tool, per user, per day.

type RawUsageRecord = {
  day: string; // map from the API's date field
  user: string; // map from the API's user identifier
  entries: Array<{
    kind: "skill" | "agent" | "mcp_server";
    name: string; // the customer configured name, now reported
    interactions: number;
  }>;
};

export type ToolUsageRow = {
  day: string;
  user: string;
  toolKind: "skill" | "agent" | "mcp_server";
  toolName: string;
  interactions: number;
  truncated: boolean; // the record hit the entry limit, so its tail may be missing
};

const ENTRY_LIMIT = 25;

export function normalize(records: RawUsageRecord[]): ToolUsageRow[] {
  return records.flatMap((record) =>
    record.entries.map((entry) => ({
      day: record.day,
      user: record.user,
      toolKind: entry.kind,
      toolName: entry.name.trim().toLowerCase(),
      interactions: entry.interactions,
      truncated: record.entries.length >= ENTRY_LIMIT,
    }))
  );
}

Two details in there are easy to skip and expensive later. Normalize tool names, because people rename servers and casing drifts. And keep the truncated flag: if a record is full at 25, the long tail might still be cut, and your dashboard should say so instead of pretending the count is complete.

Join event counts to evaluation outcomes

This is where usage metrics turn into something you can actually decide with. Counts tell you who touched a tool. Outcomes tell you whether touching it was worth it. You need both on the same row.

The join key is the hard part. GitHub's numbers are aggregated, your evaluation results are per run, so pick a grain both sides can honestly share. Tool name plus day is the one I'd reach for. Per user is tempting, but it quietly turns an adoption report into a surveillance report, and people will route around it.

On your side, emit your own spans for agent operations and their outcomes. OpenTelemetry's semantic conventions (version 1.44.0) include GenAI attributes for operations and for evaluations, which gives you a vendor neutral way to separate "the agent did something" from "what it did was any good":

// record-outcome.ts
import { trace } from "@opentelemetry/api";

const tracer = trace.getTracer("agent-evals");

export async function recordToolRun(
  toolName: string,
  run: () => Promise<{ passed: boolean; score: number }>
) {
  return tracer.startActiveSpan(`execute_tool ${toolName}`, async (span) => {
    span.setAttribute("gen_ai.operation.name", "execute_tool");
    span.setAttribute("gen_ai.tool.name", toolName);
    try {
      const result = await run();
      // Check the GenAI attribute registry for the exact evaluation names in your semconv version.
      span.setAttribute("gen_ai.evaluation.name", "task_passed");
      span.setAttribute("gen_ai.evaluation.score.value", result.score);
      return result;
    } finally {
      span.end();
    }
  });
}

Then the join itself is boring SQL, which is exactly what you want. Aggregate each side first, otherwise the join multiplies your counts:

WITH usage AS (
  SELECT day, tool_name,
         SUM(interactions) AS interactions,
         BOOL_OR(truncated) AS any_truncated
  FROM tool_usage
  GROUP BY day, tool_name
),
evals AS (
  SELECT day, tool_name,
         COUNT(*) AS evaluated_runs,
         AVG(CASE WHEN passed THEN 1.0 ELSE 0.0 END) AS pass_rate
  FROM tool_evals
  GROUP BY day, tool_name
)
SELECT u.day, u.tool_name, u.interactions, u.any_truncated,
       e.evaluated_runs, e.pass_rate
FROM usage u
LEFT JOIN evals e USING (day, tool_name)
ORDER BY u.day DESC, u.interactions DESC;

A NULL in evaluated_runs is not a bug in the query. It is a finding: a tool people use that nobody is measuring.

Avoid sampling and telemetry bias

Copilot counts are GitHub telemetry, and telemetry has holes. Three of them will bend your numbers if you ignore them.

1. Missing clients. If a tool is mostly used from a surface that doesn't report, it looks unpopular when it isn't.
2. Blocked telemetry. Teams with strict network rules, often the same teams running your most sensitive MCP servers, undercount without anyone noticing.
3. Counts are not success. Interactions and connections measure activity. Only your evaluations measure whether the activity helped.

So treat every count as a floor, compare a tool against its own trend rather than against other tools, and never turn the data into a team leaderboard. My opinion, from building dashboards like this: the first chart someone screenshots is "most used tools", and it is the chart most likely to mislead.

There is a security angle too. OWASP LLM06 describes excessive agency as coming from three drivers: excessive functionality, excessive permissions and excessive autonomy. Instrumentation alone does not enforce any of those boundaries. A popular MCP server with write access it doesn't need is still a footgun; the usage chart just tells you how many people are holding it. If you're reviewing those boundaries, my production defense guide for AI agent security covers the permission side in detail.

Build a rollout decision table

Once usage and outcomes sit on the same row, the rollout call stops being a vibe. This is the table I'd pin in the platform team's channel:

Usage	Outcomes	Decision
High	Good	Expand. Make it a default and document it.
High	Poor	Fix or restrict. People want it and it is hurting them.
Low	Good	Promote. This is a discoverability problem.
Low	Poor	Retire. Remove the server and its permissions.
Any	No evaluations	Wait. Add evaluation before you decide anything.

The last row is the one teams skip. A tool with heavy usage and zero evaluations feels like a win, and that feeling is exactly the false signal this whole setup exists to catch.

FAQ

How do I implement github copilot mcp usage metrics safely?

Pull the usage metrics API on a schedule into your own table, normalize tool names, and keep a flag for records that hit the 25 entry limit. Emit your own OpenTelemetry spans for agent operations and evaluation results, join both sides on tool name and day, and make rollout decisions from usage and outcomes together. Keep permission boundaries enforced separately, because instrumentation does not enforce them.

What changed with github copilot mcp usage metrics?

GitHub's usage metrics API now reports the names of customer configured skills, custom agents and MCP servers instead of grouping them together, and each record can hold 25 entries instead of five. That makes internal tool adoption visible, but the counts still measure activity, not success.

---

Related reading: The LLM evaluation metrics that actually predict production quality and MCP server security in production: what actually breaks.

If you're still deciding which agent stack to measure in the first place, the AI agent framework chooser narrows it down in a few questions.

---

Drop a comment if your setup looks different. Curious how people are joining usage data to outcomes in production.