I Built a 5-Metric Dashboard for AI's Impact on My Team - Here's the One I'd Refuse to Skip

A US state shipped an observability tool for AI yesterday and most engineers didn't notice.

California turned on a monthly tracker (June 25) that links AI-exposure data to unemployment claims, so the state can watch AI's effect on the workforce in near real time. It's a macro dashboard for a thing nobody was instrumenting before.

You already know where I'm going with this. You instrument your services. Latency, error rate, saturation, the whole golden-signals stack. You'd never run a system in prod with no dashboard. So why is the AI workforce inside your own org running completely uninstrumented?

The metrics we actually have are the wrong ones

Pull up how most teams "measure AI" today. Two dashboards, tops.

One is spend. Tokens, seats, API bills. Useful for finance, tells you nothing about impact.

The other is model evals. Benchmark scores, pass rates on a test set. Useful for the model, tells you nothing about your team.

Neither one answers the question an exec is about to start asking: what is this actually doing to how the work gets done? That's not a cost question and it's not a benchmark question. It's an observability question, and it's ours to own before it gets handed to someone who'll measure it badly.

So I sketched the dashboard I'd build. Five signals, one screen. Here's the shape:

# ai-workforce-dashboard.yaml
metrics:
  agent_task_coverage:
    desc: "% of a real workflow's steps running on agents"
    query: agent_steps / total_steps
    note: "measure the Tuesday reality, not the demo"

  role_reclassification_rate:
    desc: "roles that changed shape vs roles eliminated"
    query: reclassified / (reclassified + eliminated)
    note: "the augmentation signal — counts shift, not loss"

  upskilling_completion:
    desc: "% of changed-role people who finished training"
    query: completed / affected

  error_escalation_rate:
    desc: "agent outputs caught + kicked back to a human"
    query: escalated / agent_outputs
    note: "your reliability trend — rising is trouble"

  output_vs_headcount_delta:
    desc: "is output growing faster than the team?"
    query: d(output) - d(headcount)
    refuse_to_skip: true

The two metrics a dev will actually trust

error_escalation_rate is the one that should feel familiar. It's just your error budget, pointed at agents instead of services. An agent ships output, a human catches a bad one and escalates, you log it. The ratio over time is a trust curve. Falling means you can safely hand the agent more. Rising means something regressed and you're about to find out in production.

The honest part: this one is hard to instrument cleanly, because "a human caught it" isn't always a logged event. The first version is manual and noisy. I'd still rather have a rough trend line than pretend reliability is fine because no one filed a ticket.

output_vs_headcount_delta is the one I'd refuse to skip. It's the only line on the screen that ties everything to economics in one number. Output growing faster than the team is the entire argument for the budget you're spending, and it's the number a non-technical board will actually read.

Why reclassification is the metric that keeps it honest

The peg here is a job-displacement tracker, and the lazy version of this article is "here's how to watch the layoffs arrive." I think that frame is wrong, and role_reclassification_rate is why.

When you measure roles that changed shape instead of only roles that vanished, you catch what the doom story misses. The reviewer who now supervises an agent and owns the judgment calls didn't get deleted. Their job moved up the stack. If your dashboard only counts headcount, every shift looks like loss. If it counts reclassification, you can see the work redistributing and actually manage it.

That's the difference between an instrument and an alarm. I want the instrument.

You already own observability. Extend it.

None of this needs a new platform. It needs the discipline you already apply to your services, pointed one layer up at how agents and people are splitting the work. Five queries, one board, checked weekly.

If you had to ship this dashboard with three metrics instead of five, which two would you cut, and which one would you defend in the room?

Comments

[TuanPK Builds]

One metric I'd add is human intervention. If AI-generated work still requires extensive review or rework, the productivity gains may be smaller than they appear. Measuring both speed and quality gives a much clearer picture.