One day my dashboard showed that bugs in one module had gone up by 65%. The explanation was right on the surface: the developers had gotten sloppy, time to tighten up code review. It was convenient, easy to understand, and wrong. The real cause was somewhere else, and I would never have found it if I had looked at that number apart from the others.
For about two years I have been using a metrics dashboard that I built myself in my spare time, and over that time I learned a simple thing: almost any metric on its own can lie. Not because the data is bad, but because a single number knows nothing about its context. Below I will show which metrics I watch, which of them I read together in pairs, and how to check in one minute whether your own dashboard is fooling you. There is a short checklist at the end.
TL;DR
- Read metrics in pairs: review speed with MR size and defects, Cycle Time with WIP, a "green" rate with the number of events behind it.
- "Lead Time" and "Cycle Time" mean different things in different sources. Until the boundaries are written down, the numbers are not comparable.
- Little's Law checks your dashboard for honesty in one minute.
- A metric for an individual person is, for me, a reason for a one-on-one talk, not a line in a ranking.
What my dashboard is
It is not a product and not a SaaS, it is my working tool. At first it pulled data from Yandex Tracker, Notion and GitLab; now Jira has been added. Changes land in a separate database through CDC (change data capture) and a few workers, and everything else is computed from there. On top of it run analysis scripts, the dashboard itself, and an AI agent that I ask pointed questions like "why did this release slip", and that uses the database for RAG.
The sources changed; the database and the metrics stayed.
The metrics fall into four groups: planning and estimates, flow speed, quality, and review and releases. The set itself is ordinary; similar lists have been published by Alfa-Bank and SimpleOne (both in Russian). The interesting part starts when the metrics begin to argue with each other.
A disclaimer up front. All numbers in this article, except the ones linked to research, are illustrative. I have used this approach in several companies, some of them under NDA, and not all of them are on my résumé. So I do not show real team data, and the examples are chosen to show only the general logic.
How the dashboard lied to me after a move
The most useful lesson came when I moved the dashboard to another company. Same code, same formulas, fresh data. And the metrics started showing nonsense.
The reason was mundane. Processes in the new company were set up differently, and the same formulas were now measuring different stretches of work. The status names matched, the meaning did not. I had to adapt the dashboard to the process. Until then Cycle Time honestly showed a number, it was just a number about something else.
After that I stopped trusting metrics whose boundaries are not written down. And it turned out I was not the only one confused. In the Kanban Guide, Cycle Time is the time from when work on an item starts until it is finished. DORA has Change Lead Time, which runs from commit to production. Same word "lead", different intervals. This is how I define them now:
Wait Time is the difference between Lead Time and Cycle Time.
I measure all flow metrics in calendar time and kept working hours only for estimate accuracy. It sounds like a detail, but the next section shows what it costs.
A one-minute lie detector
There is Little's Law: average Cycle Time equals average WIP (work in progress, the number of items being worked on) divided by Throughput (the number of items finished per day). It holds on historical data and for a stable flow, where work does not pile up endlessly (Vacanti, Scrum.org). For a team lead it is a free lie detector.
For example: on average 40 items are in progress, and 280 were closed in four weeks.
wip, closed, days = 40, 280, 28
throughput = closed / days # 10 items per day
cycle_days = wip / throughput # 4.0 days
print(cycle_days * 24) # 96 hours
Little says: 96 hours per item on average. The dashboard shows a Cycle Time of 32 hours. One of them is lying, and it is usually not the math.
I check three things. First, units: 32 working hours and 32 calendar hours are different durations. Then windows: WIP and Throughput must be computed over the same period. Finally, what counts as "in progress": if the queue got into WIP, it is closer to Lead Time than to Cycle Time. The mismatch almost always turns up at one of these steps.
Little's Law is also useful for forecasting. If Throughput stays flat and WIP grows, every item will take longer. That is not "the team got slower", that is arithmetic.
The team works at the same speed; items just wait in the queue longer.
So I never read a rise in Cycle Time on its own. First I look at WIP. If it went up by half, what needs fixing is the number of parallel items, not people's speed.
The pairs I never open the dashboard without
Time to First Review, MR Size and Escaped Defects
Fast review is pleasant to look at. But picture a chart where the time to first review dropped over a quarter from six hours to 36 minutes. You want to praise the team. Now put next to it the size of merge requests (MRs) in lines and the defects found after release.
The top chart without the two below looks like a success.
If MRs get bigger while review gets faster, review has almost certainly turned into a formality. Research backs this up. Kemerer and Paulk, on data from 371 and 246 student programs, showed that at review rates up to 200 lines per hour more than half of the defects in the code are found; going slower does not improve it further, and going faster makes it drop (IEEE TSE, 2009). A 500-line MR approved in ten minutes is not speed, it is a miss. And given the AI slop all around us these days, some very interesting things can make it into the repository.
That said, fast review is not evil in itself. At Google the median wait for a first response is under an hour for small changes, and the median change is only 24 lines (Sadowski et al., 2018), and that was before the AI boom. It is fast there because the changes are small. DORA's 2023 report links fast code review to higher software delivery performance, but warns that if review is already fast, speeding it up further is pointless (DORA 2023).
A green metric and the number of events
Twelve deployments in a month, none rolled back, Change Failure Rate 0%. Great result? In fact 0 out of 12 proves almost nothing. One failed deployment next month gives 8%, and the team will not have gotten any worse.
One deployment separates "excellent" from "alarming".
MTTR is even funnier: no incidents, no metric. So next to every rate I show n, the number of events, and metrics with a small sample are labeled "not enough data". Boring, but it saves you from celebrating nothing.
Reopened tasks and defects
The share of reopened tasks is low, and that looks like quality. But if a bug that is found gets filed as a new task instead of reopening the old one, the reopen counter simply does not see it. A low reopen rate with a high number of defects means one thing: bugs are going around that counter.
Estimate accuracy and delivery time
A team can accurately estimate a task at eight hours or 3 story points and deliver it a week later. Because it waited three days in the queue and then sat in review for two more. Good estimates with bad delivery times point to a flow problem, not an estimation problem. To calibrate expectations: according to a review of surveys, 60–80% of projects overrun their effort or schedule, by 30–40% on average (Moløkken, Jørgensen, 2003).
You can see an overloaded release in advance
The most practical pair for a team lead: team capacity and the release plan. Capacity comes from the people and their availability, the plan from the estimated tasks. When more is planned than there is capacity for, or one person is loaded twice over, you see it before the release starts, not at the retro afterwards. I have had such cases, and the overload was visible on the dashboard in advance.
There is a catch: tasks without an estimate do not count toward the load. So the real load is higher than what is on the screen. That is why next to the load I always show the number of unestimated tasks.
A person's metric is a reason to talk
The dashboard has a breakdown by developer. It is its most controversial part, and I understand why. DORA warns directly: these metrics work at the team or service level, and once they are turned into targets, people start gaming them (DORA). People quickly learn to split tasks, take the easy ones and put off the hard ones. The metric goes up, the work does not.
So I do not compare people by these numbers. For me it is a signal. If someone's output dropped or their bug count went up, it is time for a one-on-one to ask what is bothering them. In my experience, a drop in output is often not about work but about what is going on in the person's life outside it. That is an opinion, not research. But a plain conversation between two people always shows much more than any chart.
How the module story ended
Back to the 65% more bugs. I think every team has run into a situation like this. I did not tighten up review. It turned out that tasks for the module were arriving half-baked. The PM was rushing the team toward a deadline that had been badly planned from the start, and requirements were worked out in a hurry. The code was reviewed, but review does not cure a bad task definition.
We fixed the input. We started working out requirements together with the analysts and the PM before a task goes into development. And we deliberately slowed development down so that features could be thought through before any code was written. Roughly, it looked like this: two months later bugs in the module were down by about 40%, and the average time per task in the module was up by about 15%. I was happy with that trade.
The conclusion for me is simple. A module's bug rate is not only a development metric. Often it shows planning problems, and it needs to be discussed not only with the developers.
What changed
In short, this dashboard helped reduce the number of bugs and people's overload, speed up fixing production bugs, shorten time-to-market and raise Throughput. The real data is under NDA, so below is an illustrative order of magnitude of the changes over a year in one of the teams:
| Metric | Before | After |
|---|---|---|
| Defect Rate (share of tasks with defects) | 43% | 18% |
| People loaded above capacity in a release | 8 of 15 | 2 of 15 |
| Median time to fix a production bug | 3+ days | <1 day |
| Lead Time (from task creation to production) | 14 days | 7 days |
| Throughput (tasks per week) | 32 | 115 |
The numbers in the table are illustrative and show the direction of the changes, not the result of a specific team.
Where I might be wrong
The research I rely on is not final. McIntosh and colleagues found a link between review coverage and post-release defects, but a re-analysis could not confirm it. Kemerer and Paulk worked with student PSP data, not with team code review in GitLab. The DORA reports are based on surveys and show correlation, not causation.
The popular figure "flow efficiency in most teams is 5–15%" comes from consultants' experience, not from research. There is a good critique based on data from 63 teams, so I do not turn such benchmarks into targets. Most importantly, in our teams we always want to build great things, and this story shows how one tool can make life a little better for everyone.
And the main point: my conclusions come from my teams, my experience and my definitions, which give repeatable results.
Checklist: check your dashboard in ten minutes
- For Lead Time, Cycle Time and Change Lead Time, it is written down from which status to which status the clock runs.
- All flow metrics use the same units, with no mixing of working and calendar hours.
- Little's Law holds: WIP divided by Throughput is roughly equal to the average Cycle Time.
- Every rate shows the number of events next to it; small samples are flagged.
- Time to First Review is shown together with MR Size and Escaped Defects.
- The reopen rate is checked against the total number of defects.
- Release load is shown together with the number of unestimated tasks.
- Individual metrics are used for conversations, not for rankings.
If three of the eight are not met, your dashboard is more likely to reassure you than to help. Metrics are best read as a system, where one number explains another. A board full of green lights is pleasant, but it holds less truth than it seems.
Which pair of metrics has fooled you the most when you looked at it alone?
Top comments (0)