How Measurement Elevation and Aggregation Change Behaviors

The costs of measuring the wrong thing or the right things in the wrong way vastly outweigh the costs of collecting and processing the data.

In software delivery, conversations about metrics and measurement wax and wane as people discover useful techniques or get hurt by unintended consequences. Understanding how measurement elevation and aggregation levels change people’s behaviors and business outcomes will help you create a healthier metrics system that better informs your decisions.

The Economics of Measurement

A measurement system is subject to economics. There is a cost of collecting data and a value in using it. The economics, particularly the cost, are commonly overlooked. The widespread nature of computer systems means many leaders believe the cost of collecting metrics is negligible, which causes them to misunderstand these trade-offs.

For example, the cost of getting a time series of data onto a graph is very cheap, but the consequences of concentrating on this data can be surprisingly high. When you publish metrics, you signal that they are important. It’s natural for people to believe you want to see some form of improvement on those metrics, and they will make decisions based on this signal (whether or not that’s your intent).

If you shift people’s attention to the wrong measurements, the cost to your business could be huge.

The Trouble with Measurement Elevation

One classic problem with metrics occurs when they are elevated inappropriately. Measurements useful to one team can cause problems when shared higher up the organization. Software delivery metrics can be a helpful way for teams to inform their continuous improvement (CI) activities. However, when the same metrics are elevated throughout the organization, they may cause undesirable effects on behavior, especially if they become key performance indicators (KPIs).

Metrics are a way to make sense of abstract concepts. In business, they are used to connect actions across the organization back to the company strategy. Problems occur if you put more emphasis on the metrics themselves than the concept they represent or if they don’t appropriately represent the concept. When you fixate on metrics, you can force people to do clearly unwise things in their service.

For example, I worked on a team that missed a crucial release date for a finance system. One of the lead developers stopped working on the project to write a training guide. This behavior damaged the team and organization, but the developer was given a goal to write five training guides and their annual review was coming up. If they had done the right thing and helped with the release, they would have missed their goal-related bonus.

Was this the team member’s fault, or were they doing exactly what the system demanded?

I recently collaborated with Google and DORA on an article about how business leaders can empower software delivery teams that discusses how to avoid the elevation problem.

Problems with Measurement Aggregation

Aggregation levels are another crucial factor to consider in designing measurement systems. Measurement aggregation refers to the data you can measure at different organizational layers. On a basic level, you could measure at any of these levels:

• Individual
• Team
• Business unit
• Organization

I’ve simplified the levels to make it easier to apply this concept across multiple organizations. You’ll need to determine how this concept fits your organization’s context and structure.

Measurement aggregation could involve combining data or displaying multiple sets. For example, aggregating data at the business-unit level might mean calculating averages across all teams in the unit, or it might involve plotting each team as a separate line on a chart.

Aggregation changes the economics because the effect on behavior will change based on which organizational level you apply the measurement. For example, measuring performance at the team level encourages team members to help each other succeed, whereas measuring this at an individual level will reduce attention on crucial tasks if they don’t directly impact individual performance. A rough sketch of this change is shown below, with the measurement’s value dramatically changing with each movement through the layers.

This chart is easiest to explain by starting at the team level. When you measure a team, you encourage the members to help each other improve their performance over time. Junior team members are coached by more experienced people, and the process adapts over time. Lots of great things happen as the team finds better ways of working. This metric has high value at the team level.

But if you apply the same metric at the individual level, many of those beneficial behaviors stop. Time spent coaching other team members directly lowers an individual’s score. Time spent on automation and process improvement is more relevant to hitting your own numbers than increasing someone else’s.

You can’t blame someone for doing things that result in achieving the numbers you decided to measure — even if those things may seem wrong. When you make a metric too important, it changes how people perceive their work in ways you might not predict.

But what if you move up a level from the team to the business unit? You now have a problem with the measurement competing with contextual noise. One team may be working on a modern application with a loosely coupled architecture, another might be wrestling with heritage software that’s not factored well. One team’s struggles might be hidden by the another team’s improvements. The measurement’s value is reduced because the signal isn’t detectable.

A common solution to this problem is to measure at the team level and elevate the metrics for comparison at the business-unit level. However, great care must be taken here to avoid causing unintended competition or creating teams nobody wants to work on.

If you shift up to the organizational level, you once again obtain highly valuable information. This is the level where you are closest to measuring outcomes. Not only can these metrics help you tell whether you are profitable and hitting goals, but you can also assess whether improvements at the team level are translating into tangible results. (Although there is usually an unfortunate delay in confirming that leading indicators are converting into bankable results.)

Balancing Competition and Collaboration

The key to understanding the complete reversals at each level of aggregation is that they naturally tend toward either competition or collaboration. In knowledge work, collaboration is usually the best path to positive outcomes for the people, the organization and its customers.

The team and organization levels are units of collaboration. Individuals and business units are often units of competition. There are healthy places for competition — usually between your organization and others offering a similar product or service. That’s why they’re called competitors.

In business, collaboration gets you further than competition.

Healthy Measures Make a Healthy Organization

Understanding measurement economics, aggregation and elevation will help you design a system that encourages positive behaviors, those that leverage human ingenuity and collaboration to create value.

For more insight on designing healthy measurement systems, consult my white paper Measuring Continuous Delivery and DevOps.

When it comes to the economics of measurement, the costs of measuring the wrong thing or the right things in the wrong way vastly outweigh the costs of collecting and processing the data.