This post was originally published on the Shipyard Blog.
Measuring DevEx has been a major discussion point over the last few years. How exactly can you measure something that relies so heavily on individual experiences? The DevEx framework introduces a new way to do this by surveying developer perceptions and system data around three key dimensions.
What is developer experience (DevEx)?
Developer experience (DevEx) is the quality of the processes and culture surrounding a development team. Developers write better code (and deploy it faster) when they’re given the tools they need for success. At its core, DevEx aims to help developers focus on staying in the inner dev loop, instead of getting sidetracked with maintenance, getting blocked from testing, or waiting around for infrastructure.
DevEx has become a major initiative in many engineering orgs because development teams build better software when they’re productive and happy. But how exactly do you measure productivity and happiness? Both of those skew more qualitative, which makes it tricky to calculate a direct ROI.
What is the DevEx framework?
The DevEx framework exists to solve the biggest challenge of developer experience initiatives: measuring them. It was introduced in 2023 by Abi Noda, Dr. Margaret-Anne Storey, Dr. Nicole Forsgren, and Dr. Michaela Greiler as a system to measure the productivity gains from DevEx that DORA and SPACE metrics can’t quite capture alone.
Three facets of DevEx
Noda et al. found three dimensions from which DevEx could potentially be measured. These three categories focus on measuring developers’ perceptions to best assess where blockers and friction come into the scene, and can be supplemented with quantitative data (e.g. lead time, frequency rate of improvements). Here are the core dimensions:
- Flow state: how enabled/supported a dev feels during focus time, amount of disruption caused by non-critical tasks
- Feedback loops: how satisfied a dev feels with automated test time, lead time, and deploy time
- Cognitive load: how difficult documentation is to use/understand, how complex overall codebase feels to a dev
The DevEx framework pairs suggested workflow measurements along with the above perceptions, which orgs can use to drive positive changes in developer attitudes.
Why measure DevEx?
Measuring system efficiency only goes so far. For example, an organization might score really well on their mean time to restore service after an outage, but if a developer perceives these efforts as a flow-disruptor, there is obviously room for improvement. It’s important to have this reflected in your org’s measurements, since developer satisfaction can set a bar for your team’s delivery potential, which can likely exceed its current output.
Traditional frameworks don’t take these human factors into account. Understanding the real-world context around your quantitative metrics can help you better assess where your team stands, and what their trajectory might look like.
High-performing teams recognize that software quality and delivery improvements come from people, process, and tooling (in that order), but very few frameworks exist to measure the relationship between people and tooling. Measuring DevEx’s impact on your organization can fill in these gaps and justify with facts and figures on why keeping your developers happy eventually converts to improved software delivery.
DevEx framework vs DORA Metrics
The DevEx framework and DORA Metrics both give teams a baseline for continuous improvement. Since they serve distinct yet important functions, they can be used strategically to complement each other.
DORA Metrics are a set of measurements that benchmark your team’s output frequency and quality. Scoring well on DORA means that you’re able to push code quickly, deploy often, and experience limited prod failures (but remediate them quickly when that inevitably happens).
DORA is comprehensive when it comes to true delivery performance. Through DORA’s research program and studying top-performing orgs, the team has identified what industry-leading deployment standards look like.
The DevEx framework also suggests some concrete, easily-measurable system benchmarks. Naturally, the DevEx framework will hone in on the inner dev loop, so these include a few things that fall outside DORA’s scope, e.g. pipeline runtime and number of blocks.
Using these frameworks together can give a more complete assessment of your engineering org. The DevEx framework can answer some of your DORA weaknesses (e.g. we have a long lead time because developers feel they are spending too much time in meetings to be productive). And DORA can pick up where the DevEx framework leaves off, particularly in the outer loop, and show valuable data that you can tie back to your bolstered developer experience initiatives.
Conclusion
The DevEx framework has been a helpful resource for teams who have been looking for a standardized way to measure the impact of developer experience initiatives. This framework works as an excellent supplement to your org’s DORA Metrics and can help identify a few important pain points of your developers’ workflows so you can best enable them.
Top comments (2)
Very cool article. I'll read it many times tomorrow to see how to put/add KPIs to the journey i'm working on for my team.
Thanks for putting all these resources at the same place.
Thanks @adriens! The full DevEx framework paper has some helpful info on designing KPIs queue.acm.org/detail.cfm?id=3595878