What counts as a productive day for a software developer? What about engineering managers and leaders - how do they define and measure their developers’ productivity?
Developer productivity has always been hard to measure since there is no paradigm on how to comprehensively value their work. Nevertheless, as we embark on building a new era of hybrid work, there is an urgent need and an important opportunity to develop a nuanced and expansive understanding of developer productivity.
The measurement challenge in engineering management
Engineering teams and their leaders have always faced the challenge of comprehensively and meaningfully measuring team and individual developer performance. Most business verticals, such as sales or marketing, use a commonly agreed-upon framework to understand, measure, and monitor performance. However, engineering teams lack such a unified system due to engineering practice’s complex and refined nature.
The reason that measuring developer productivity has eluded the industry for so long is because unlike other business functions where quantifiable metrics like output and worker activity can serve as an indication of success, engineering practice cannot be measured against the same yardstick. Developer and engineering team productivity cannot be appraised by counting the number of lines or the number of PRs merged or any one other metric. It is also harder since engineering is a team activity and it is hard to gauge individual contribution in a team effort. In software development, it is critical to identify the outcome of developer activity and the value created by this activity; and measuring value of an outcome is immensely more nuanced than tracking a few metrics.
Measuring developer productivity
The foundation to succeeding at engineering team performance measurement is to appreciate the many nuances of a software development team’s activities. This leads to acknowledging the need to measure and stay cognizant of several performance metrics that inform a practical framework of metrics acting under tension, providing a comprehensive view of engineering practice performance.
Every organization values and measures performance through differing lenses. Creating a framework of metrics allows every team the opportunity to evaluate the goals of the team and develop a system that helps teams progress towards the right priorities. The SPACE productivity framework provides a holistic and multidimensional view to approaching productivity from multiple dimensions that, in relationship to each other, can help decode the actual factors impacting productivity within a business context.
SPACE Productivity Framework
Developed by researchers at GitHub, University of Victoria, and Microsoft, SPACE is a practical and multidimensional viewpoint into developer productivity that proposes a new approach to defining, measuring and predicting it. The SPACE framework presents five categories to measuring productivity:
S – Satisfaction & Well Being
P – Performance
A – Activity
C – Collaboration & Communication
E – Efficiency & Flow
This framework captures different dimensions of developer productivity, going beyond measuring just activity or individual performance, and proposes the use of several measures to draw a holistic interpretation about engineering productivity.
Satisfaction & well-being
Satisfaction is a measure of developer fulfillment – the feeling of engagement that developers feel with their tasks, the tools, and the processes that their teams use. It captures the influence and interrelationship that work and happiness have on each other. Well-being captures the health index of developers and how their work affects their health. Satisfaction and well-being are much more than a happiness index, this metric captures how teams work together, cohesively, to create value.
As a leading indicator of productivity, there is a correlation between satisfaction and burnout and therefore, engineer satisfaction has a direct impact on productivity.
What to measure?
Satisfaction and well-being are best measured using qualitative feedback, surveys or polls, and discussions with employees. Leaders are also able to measure degree of satisfaction using polls such as
Employee satisfaction – Will your developers recommend your team to others?
Employee efficacy – Are your developers equipped with the right tools and processes to succeed?
Burnout – Do your employees have enough work-life balance to combat workplace stress or exhaustion?
Why is it important to measure satisfaction?
Workplace satisfaction means that developers are engaged in their work and feel a sense of achievement and intrinsic motivation to produce quality work. This results in improved innovation, since developers are motivated by their own success and potential, also resulting in higher employee retention. Similarly, measuring and managing developer well-being can help build resilient teams that can integrate sustainable productivity for the long term.
Performance
Performance captures the outcome of a team’s systems and processes and helps gauge how effectively a team or a workflow functions. When necessary, performance metrics help leaders to correlate processes to the resulting outcome achieved, thereby finetuning them to produce optimal results.
What to measure?
It is hard to quantify the performance of individuals given the high degree of teamwork that contributes to a successful engineering outcome, that in turn leads to a successful business outcome. The quality of each developer’s work, the impact of the product on the business, and the importance of each developer’s role in the team are all variables that make measuring individual performance much harder. However, a simple starting point could be measuring the health and effectiveness of an engineer’s code – did the code do what it was supposed to do? Some metrics that can help measure performance,
1) Quality – The reliability of a developer’s code, the absence of errors and bugs, and ongoing service health can be measured and tracked using dashboards for metrics like velocity and acceptance of code reviews, code churn, code complexity, static code analysis, etc., to help quantify the quality of a team’s performance.
2) The impact that the product has on customers and the business can be measured as customer satisfaction, retention, and adoption of products, cost optimization, feature adoption rates, etc.
Activity
Activity is the most popular and often misused measure of productivity. It captures developer activity as a count of things or quantifiable output. However, it is a futile effort to correlate developer activity count as a measure of developer productivity; just because of the complex, intractable, and multidimensional nature of a developer’s work. Nevertheless, a count of activity, even if it doesn’t capture the complete impact of a developer’s contribution, can still serve as a useful tool to gain visibility into developer efforts and contributions. These metrics, when used in tandem within the framework of other metrics, can help leaders understand the phases of software development life cycle.
What to measure?
Some developer activity metrics that are tractable and easily quantifiable:
Count of actions such as volume of work items, pull requests, commits, or code reviews, or lines of code written as a measure of developer activity.
CI/CD metrics such as a count or frequency of build, test, deployment/release, and infrastructure utilization.
Volume of operational activity like volume of incidents and their corresponding mitigation, on-call participation, etc.
Communication & Collaboration
How do team members connect and work together as a cohesive unit? Are developers communicating and collaborating effectively? Is there a healthy and meaningful inter and intra team interaction? The communication and collaboration metrics capture the insights to such questions. Software development is a team effort and the team that succeeds at seamless and effective communication, succeeds at innovation, alignment, and delivery.
In order to maintain high levels of integral cohesiveness, teams have to be built on a culture of high transparency, commitment to tasks, and accountability. Particularly in distributed or hybrid teams, inclusive diversity sparks innovation which is sustained by effective knowledge sharing and documentation, and effective async communication.
Capturing collaboration metrics allows leaders to build processes that can find a balance between async work and collaboration. Collaboration analytics and insights from network metrics can lead to devising strategies that preserve focus time by pruning meetings, and in many teams, it can facilitate team cohesion and collaborative processes that lead to improved performance and lesser stress for individual developers.
What to measure?
Most often, these metrics help indicate the health of a team’s communication and collaboration processes:
Meeting metrics – quality, frequency, and effectiveness.
Ease, quality and efficiency of documentation and knowledge sharing – can the right people find the right documents in a timely and organized manner?
Thoughtfulness of collaboration measured as code review score and PR merge times that indicates the quality and timeliness of the code review.
Efficiency & Flow
The efficiency and flow metrics keep all stakeholders in the loop on the progress of tasks in a team. It helps managers and leaders to stay in sync with delivery timelines and helps developers to gauge how successfully their team brings a work to completion. These metrics often include measures of time or speed through a system, number of handoffs, interruptions, and a developer’s ability to stay in a state of flow.
Measuring these metrics are helpful in spotting and removing inefficiencies in the software delivery process.
Measuring and improving efficiency and flow often call for a balanced approach between individual and team priorities. For example, when managers try to optimize an individual developer’s maker time by reducing meetings or communication requests, it has a negative impact on that developer’s ability to participate and contribute to a teammate’s requests or team activities. Similarly, when leaders optimize team efficiency, it is easy to prioritize completion metrics while paying only little regard to testing or defects. Hence, leaders optimize efficiency and flow metrics by moulding processes that provide customer satisfaction and user value.
What to measure?
Teams capture the efficiency and flow using:
Efficiency and cohesion in a team and system performance can be measured using code review quality and timing.
Availability and usage of maker time to capture the ability of an individual or team to stay in flow and practice deep work.
Volume, quantity, timing, and spacing of interruptions and their corresponding impact on developer focus.
Number of handoffs in a process and across different teams measured to minimize delays.
How to use the SPACE framework?
As leaders approach the task of measuring their engineers’ productivity, it is imperative that they develop a holistic perspective of developer productivity, keeping in mind the many intractable but immensely important activities a developer does to get a project to completion. In order to fully appreciate this complex engineering process, a measurement system has to capture several metrics across multiple dimensions of the SPACE framework.
More importantly, leaders have to always add perceptual metrics using feedback and survey data to get more accurate and complete information about a developer’s work.
Such a system of metrics creates “ a constellation of metrics under tension” that leads to a balanced and holistic view that helps to reinforce smarter decisions and tradeoffs among team members.
While embarking on the process of measuring employee productivity, organizations and leaders have to renew their commitment to maintaining employee privacy and creating a metric system free from biases by ensuring anonymized data processing.
Understanding engineering productivity in the era of hybrid work
The baseline to approaching and understanding engineering team productivity has seen a tremendous shift with the adoption of the hybrid work model. What was once a rather obscure, complicated, and archaically output-based system of counting git activity is now evolving to be more expansive, more nuanced, wholesome yet complex, and increasingly multidimensional.
This shift can be attributed to the demands of the location-agnostic workplace, with team members being distributed across geographies and work spaces. This distributed nature obscures managers’ and leaders’ visibility into their team’s activities and contributions and introduces challenges to gauging and preempting blockers and bottlenecks. This led to a need to develop a data-driven understanding of engineering productivity as a tool to get visibility into developer activity, task progress, and workplace processes so that managers and leaders could equip teams for better alignment and focus, better-informed decision making, and providing better leadership.
Along with the lack of visibility into team effort and task progress, hybrid leaders also battle the obstacle of not being able to gauge and improve employee well-being, satisfaction, and engagement. Thus, it became imperative that any measurement of developer productivity should include measuring and managing developer health, satisfaction, collaboration, and innovation. This expanded approach to productivity to ensure sustainable gains can enable organizations and managers to provide better leadership.
Additionally, in the hybrid workplace, more developers’ focus time and makers’ time is impacted due to distractions such as meetings, communication notifications, and a prevalent culture of staying always-on. Although the hybrid workplace needs robust communication, engineering leaders have to start relying on collaboration analytics and insights in order to balance communication with optimum makers’ time.
Engineering teams have been pioneers of tech-focused work models that use highly sophisticated and tried and tested work methods like async work, agile methods, and management that were suited for remote or distributed work. Now, as the world of work reimagines a new future of work, engineering teams have the unique opportunity to leverage their expertise to reimagine a new normal. As all businesses grow to become powered by digital technology, engineering teams have taken the spotlight of rebuilding the notion of work on the foundations of productivity, innovation, and continuous optimization that consistently creates immense value. A good place to begin this paradigm shift will be by changing how engineer productivity is measured.
💡 Hatica is an engineering analytics platform that helps development teams build sustainable productivity and ensure optimum employee experience by tracking several SPACE metrics like cycle time, DORA metrics, focus and meeting time and more.
Top comments (0)