“When a measure becomes a target, it ceases to be a good measure” Goodhart's law
I have read the McKinsey article “Yes, you can measure software developer productivity” and this is my response. As a software engineer who was also a management consultant at BCG at some point in his career, I have a unique perspective on this matter.
My main argument is that McKinsey's proposed framework measures the organizational health of the engineering team, not the productivity of developers. The difference between the two is mainly: while health is measured in a one-time snapshot, productivity is measured as the improvement (or lack of) of health snapshots.
To provide some context, I have worked as a software engineer since the early 2000s and have experienced various shifts throughout my career. After completing my MBA in 2012-2013, I transitioned to management consulting at BCG before returning to software engineering. This experience broadened my perspective and equipped me with the skills to effectively manage engineers and communicate between the technical and business aspects of a project.
The McKinsey suggested framework
The article outlines a framework of three metric types and three focus areas and then organizes various metrics on a 3x3 matrix.
Metric types
- the system level
- the team level
- the individual level ### Focus areas
- Outcomes focus: Are you delivering products satisfactorily?
- Optimization focus: Are you optimally delivering products?
- Opportunities focus: Are there specific opportunities to improve how you deliver products, and what are they worth?
Using metrics from DORA, SPACE, and other metrics the article gives a sample for each cell in the matrix:
| Outcomes | Optimization | Opportunities | |
| System | Deployment frequency Customer satisfaction Reliability/Uptime |
Code-review timing Velocity/flow through the system |
Satisfaction with the engineering system Inner/Outer loop time spent |
| Team | Lead time for changes Change failure rate Time to restore service Code-review velocity |
Story points completed Handoffs |
Quality of documentation Developer Velocity Index benchmark Contribution analysis |
| Individual | Developer satisfaction Retention |
Interruptions | Contribution analysis Talent capability score |
As a former management consultant, I find this framework to be a work in progress. Most of the metrics mentioned are interdependent, and one is even listed in two different cells. Where is the MECE (mutually exclusive, collectively exhaustive)?
Using this framework, can you compare the productivity of a startup with 10 developers to the one of a financial institution with 500 developers? Ok, a startup should probably focus more on growth and less on measuring… So, can you compare two companies with similar size and maturity? I seriously doubt it. Can you compare the productivity of different teams in a single organization?
Generally speaking, any framework that does not enable you to compare against a benchmark is faulty.
DORA metrics, i.e., deployment frequency, lead time for change, mean time for recovery and change failure rate, are good indicators that your software is developed and delivered well, but not of developers’ productivity.
SPACE metrics, i.e., satisfaction & well-being, performance, activity, communication & collaboration, efficiency & flow are not even metrics. They are areas to consider when assessing a developer or a group of developers.
Ok, so what do I offer?
As an engineering manager, I developed my approach to enabling engineers. I set goals for my team which are aligned with the company goals, and after a few iterations with different managers (both upper management and peers), I committed to achieving them.
Next, I collaborate with my engineers to determine the goals that are in line with those of the team and the ways in which I can support them in attaining those goals. My primary responsibility is to enable the engineers to reach their objectives, which ultimately leads to the achievement of the team's goals.
In case I identify any shortcomings in the personal goals, I evaluate the level of engagement of the developer using quantitative metrics (some from DORA & SPACE). I also discuss with relevant stakeholders to confirm my opinion and conduct a performance evaluation with the developer.
If I detect any gaps in the team's goals, I discuss them with key team members to confirm my observations and formulate a preliminary plan. I then convene the entire team to establish and commit to a plan of action to address the issue.
I found this approach works in sales teams & recruitment teams. Make goals that are easy to measure, and when there are gaps in the progress, address them at the right level.
Notice that the approach is set goals and then measure progress, not set metrics and then measure, as Goodhart's law says:
“When a measure becomes a target, it ceases to be a good measure”
Engineers, including software developers, are naturally curious about the inner workings of things. Their job is to break things down into their basic components and find ways to improve them. When setting goals and determining metrics for evaluation, engineers will often spend time breaking down the metrics and improving their results. However, if the metrics are not aligned with the overall goals of the business, then it begs the question of whether it was worthwhile to measure them in the first place.
Top comments (0)