Developer Experience is Dead: Long Live Developer Experience! 🫠

Right now, here in early 2024, we seem to be experiencing YAPP (Yet Another Productivity Philosophy), and that philosophy is converging on developer experience or ‘DevEx.’ The “revelation” that better developer experience leads to better productivity outcomes is not a revelation at all, but a truth that’s been, perhaps bafflingly, deprioritized in favor of post-CI software delivery improvement initiatives like DORA that have all but ignored the role of the human in the loop.

It seems that with every new method we invent for the delivery of products, whether physical or virtual, we reinvent productivity philosophies to go alongside them. From lean principles and “just-in-time” manufacturing in the 1970s to DevOps and platform engineering today, process stakeholders update their respective buzzwords and slang to match what they observe in their surrounding culture, but the cross-functional behaviors don’t shift much. Conclusively, physics is physics, and the way we work is subject to those universal constraints.

GitHub’s recent highlight of the Microsoft/DX study of the productivity outcomes of improved Developer Experience is being lauded as “finally” presenting the much-needed hard data behind the productivity outcomes of improved DevEx. Their article begins with: “The wait is over: we finally have data to back up the benefits of developer experience (DevEx).”

A highlight of my own work and study has been some of the frequent and fantastic exchanges I've been lucky to have with Peggy Storey and Abi Noda, so they know this is coming, and there is excellent data in this study, everyone absolutely should read it, but…

Finally? … really? ;)

Veterans of the developer productivity space may immediately be reminded, for instance, of “The Coding War Games,” an ongoing study from the mid-1980s to the mid-aughts which provided the same conclusions and led to the publication of the book ‘Peopleware.’

Without a lot of centralization of knowledge available, almost everyone else will agree with GitHub, unperturbed, that the data is the first of its kind. But we already have:

• The Coding War Games - Taking place over several years across almost a hundred different software organizations, this study provides ample data over years that developer experience metrics are a great predictor of organizational productivity
• The Code Red Study (CodeScene) - A study completed in March of 2022 demonstrating conclusively that time wasted by low-quality code led to 9x delays in cycle time
• The Only Way to Measure Developer Productivity Without Causing a Revolt - A comprehensive response to Dan North’s “The Worst Programmer I Know” blog, illustrating the inefficacy of many supposed productivity metrics
• Cat Hicks on Developer Thriving - A perspective that draws conclusions that subjective developer experience impacts overall productivity

Why do these cycles repeat? Why do we spend our energy discovering and rediscovering that the experience provided for the worker will almost universally predict better productivity outcomes when instead we could invest that energy in the cultural and environmental shifts necessary to enshrine those outcomes universally? More on what those investments would look like later.

What failed before will probably fail again

Chomsky said “If you are teaching today what you were teaching five years ago, either the field is dead or you are,” and this wisdom has reached anthem-scale for cultures of continuous improvement. This guidance shouldn’t direct us to forget the lessons of the past, but remind us that we stand on the shoulders of giants. I think the majority will forget, though, and I think that’s why we see clear indications that, up to this point, DevEx efforts have largely failed:

1. Leaders invest in software delivery, not software creativity - Everyone loves to talk about Developer Experience improvement, but forty years after Coding War Games, decision makers are still wary of ROI. Perhaps we just haven’t seen comprehensive enough solutions, it’s possible that the ROI doesn’t exist because the right solutions aren’t being invested in.
2. DevOps has descended into Dev ← Ops, where Developers have taken on additional responsibility for Ops, while the opposite has not been true. Developers have been expected to learn Docker and Kubernetes, SREs have not seen similar expectations. Developer self-service solutions have not kept pace with this change.
3. Release Engineering and SRE has become Production Support - When software quality is negatively impacted because of increased pressure and a lack of self-service options for developers, that leads to a culture where supposedly proactive resources become almost fully reactive. SREs spend time answering tickets instead of improving systems. This can be considered a cascading failure, where poor developer experience leads to failed adoption of proactive SRE/DevOps measures.
4. Everyone is waiting for Generative AI to save the day - While LLMs show great promise, they are unproven as of yet, nevertheless we are already abandoning the current state of the art in favor of the next shiny thing, and we’re not even sure the cavalry is coming.
5. DevEx Improvement Adoption Rates Are Abysmally Low - The critical cultural shift following adoption of better DevEx practices such as implementing Backstage often fail with a lack of organizational acceptance, with most teams never realizing more than a 10% adoption rate. Typically, the most elite developers in the organization are responsible for curating the portal, and it may not align to what will actually be consumed by the more mainstream engineers in the organization. These failures may even further drive divides between these classes of engineers.

At some point, the tension will snap, and the data suggests that we are inching towards this milestone. A 2021 study by JL Partners observed that a significant percentage of developer burnout can be blamed squarely on flawed productivity metrics, and that these flawed metrics are responsible for perpetuating a vicious cycle. When goals such as release dates are calculated using incomplete metrics, the accuracy of those calculations is negatively affected. This leads to missed deadlines, which turn into efforts by the business to “fix productivity” by trying to find new ways for developers to ship faster. Unsurprisingly the end result is generally longer hours for developers, complemented with eroded trust because of continually perceived “failure.”

To complicate matters further, the data shows us that we still haven’t aligned on the proper way to communicate about productivity and developer experience, despite the work that has been done over the decades. A criminally overlooked April 2022 study, seriously please go read it, by many of the same researchers who worked on SPACE concluded that developers and their leadership tend not to prioritize the same metrics when thinking about productivity, often creating tension between developer experience and company velocity.

This is dizzyingly ironic in a world where developer salaries increasingly exceed $500k annually and digital transformation is a universal driver for nearly every business. In almost every other industry, the environment is curated for workers and bound by open and enforceable standards such as the ones provided by OSHA.

A perfect storm may be brewing where:

• As predicted by IDC, 65% of the Global GDP is software driven, a huge responsibility for a single workforce
• Developer salary continues to drive higher business cost while developer outputs are not improving
• Workforce fatigue leads to unprecedented economic waste and lack of innovation

None of these indicators show any significant signs of improvement, despite billions of dollars in DevEx investment in 2023. And can we even define “improvement,” when we still have not done a thorough job of aligning on what exactly those indicators even are? Digital transformation is only accelerating with an industry most recently invigorated with the promise of breakthroughs in AI and augmented or extended reality. Developer salaries will continue to surge with demand. Without real breakthroughs in developer productivity, cognitive fatigue and burnout will only become more ingrained.

We need productivity engineering, not productivity management

There is good news emerging out of the platform engineering and developer productivity engineering (DPE) circles, in that many who have been spending time thinking about better ways to measure and improve productivity have discovered long-hidden sources of bottlenecks and friction in the developer experience, and are starting to take action to deal with them. When we move past speculation and use solutions that let us gather quantitative data about developer workflow, the parts of the workflow deleterious to developer experience show up in high resolution. With this a new set of metrics are emerging, ones that more closely resemble the efficiency of a developer’s workflow. With more accurate metrics, we can drive real improvements.

These metrics deal with pains felt acutely by developers, and more chronically by organizations. They come from dozens of sources and represent the galaxy of tools relied upon by developers in their daily workflows. Build tools like Maven and NPM, testing frameworks like Cucumber, security and quality scanners like Snyk and Checkmarx, deployment substrates like Kubernetes, requirements management tools like JIRA and even on-call management frameworks like PagerDuty, all capable of creating new frustrations and sources of context switches for developers.

These revelations are proving what many of us have known for decades, and what countless studies have proven time and time again. There is no one metric, or even one set of metrics, that can even hope to paint an accurate picture of developer productivity, and so no one metric or set of metrics can fix it. As the researchers on the SPACE framework put it, we need to capture a “constellation of metrics in tension.”

The right solutions prioritize integration… and they already exist

We need to invest in tools that can act as predictive engineering systems of record and analyze the full spectrum of metrics, not just ones that are evident in CI/CD. A fair criticism of DORA is that it only captures metrics post-CI, and Conway’s Law would then lead to tools which only look at data generated in CI. Those data are important to understand parts of delivery, but to see the full picture, we need solutions that allow us to look at data generated by every part of the developer workflow, from the laptop all the way to the deployment substrate, and they need to adapt quickly to changes in the infrastructure.

The right solution for this problem will prioritize gathering data easily and up-to-the-microsecond from the full bloom of developer tools used by engineers in the organization. The solution must also use these data to drive new effects and behaviors in the organization, through gamification and other proven techniques. These tools should not overburden teams and should require little administration past the point of integration and data processing.

Lucky for us, “The future already exists, it’s just not evenly distributed yet.” We have no need to reinvent another productivity approach when effective solutions such as internal developer portals already exist, and already prioritize the correct mechanisms for facilitating successful productivity engineering initiatives. As business leaders, we just need to invest in and create urgency behind these solutions today, so that we don’t find ourselves having the same discussion again in a decade.

Comments

[Benoit COUETIL 💫]

Such a nice and deep post my friend, thank you.

I will have to read it several time to grasp everything, especially the external links. For now, i do not agree with everything, but that is the beauty of diverse views.

Developer Experience around me have been my profound will for the past 18 years, and the main problem is poor management afaik.

A few things come to mind, that I'm not sure you addressed... And you seem so knowledgeable in the field that I might look ridiculous, but I will try anyway.

1) DORA is indeed technical delivery metrics at the core, which can lead to burn-outs if misused. But it can serve different purposes. I has helped me advocate to managers that "quality ultimately means more revenue on top of nice quality of life".

2) Shipping fast with good tools is a first step for more interesting talks, and is wanting by lots of (good) developers in my experience.

3) is there really a need of metrics for DevEx ? It is a matter of people, could we just ask them what they think ?

4) Shipping fast will never surpass shipping the right things, no matter the DevEx. Ingeneers tend to be happy to solve a wrong problem.

5) Burn-outs is also a matter of technical leadership. It is the role of the trusted ingeneering leaders to give boundaries to managers. Ultimately human problems have to be solved too, not just tech ones.

What do you think ?

[Justin Reock]

Thanks Benoit, I really appreciate your thoughts on this and I'm glad you are finding these posts to be insightful -- I think disagreement is really important right now too, as I think the entire industry is seeking thought leadership and the "right" way forward for platform, devex, etc, so we need to start answering these questions accurately. I'll do my best to respond, I think these are very thoughtful observations!

1) DORA is indeed technical delivery metrics at the core, which can lead to burn-outs if misused. But it can serve different purposes. I has helped me advocate to managers that "quality ultimately means more revenue on top of nice quality of life".

• Agreed, I didn't mean to imply that DORA is bad, even though the tone of the article is playfully critical. DORA is a really effective distillation of dozens of important metrics, and it can help us tell an important part of the DevEx and productivity story. But even DORA was created within the SPACE framework, and so it also by definition only fills in one part of the overall image. I think the challenge for management is that human and socio-technical metrics are really hard, and there are a lot of invisible tensions and dependencies between them. That's why I think portals are such a smart solution, is that they allow us to create a picture of DevEx that is inclusive of DORA and other important data like on-call data, developer productivity engineering metrics, unexpected escalations and other sources of toil, etc, and allows us to explore the way those metrics are transitively related.

2) Shipping fast with good tools is a first step for more interesting talks, and is wanting by lots of (good) developers in my experience.

• I think that's right, in that Deployment Frequency tells us a lot about the health of the underlying delivery pipeline, and Throughput, in the context of the Theory of Constraints, is the most important component of a complex system and it should always be optimized for with the highest priority. But, Throughput is distilled from hundreds of other measures, including the more human and DevEx or Developer Productivity Engineering metrics, so I think they are actually included in that picture, we just haven't been painting a high enough resolution picture to see where those data fit.

3) is there really a need of metrics for DevEx ? It is a matter of people, could we just ask them what they think ?

• Qualitative metrics are important, and we can get very creative about how we capture those metrics, through microsurveys, etc, to try and keep things accurate and avoid survey fatigue. I think that's all important, and when we use a framework like SPACE to normalize those metrics, we get another very important part of the story. It's still not the full picture though, and so we still run the risk of making poor business decisions off of incomplete data. For instance, we already know that cognitive fatigue can skew survey data, rather confoundingly. Burned out teams tend to correlate with toxic management, and this can chill qualitative results. On the other hand, highly productive teams tend to communicate more, and so survey bias favors them. We should survey our developers and collect qualitative metrics, but we should look at those metrics and find the tensions that exist with quantitative ones, which can help us discover problems that were not visible because we weren't capturing all of the right data and correlating it.

4) Shipping fast will never surpass shipping the right things, no matter the DevEx. Engineers tend to be happy to solve a wrong problem.

• Agreed, again, that Throughput is the most important metric. I would say though, that here in 2024, the biggest bottlenecks to throughput seem to be in DevEx, as automation and DevOps tooling has removed a lot of the bottlenecks that were present a decade ago, especially in the most elite engineering organizations. This can include bottlenecks to delivery caused by cognitive fatigue and developer burnout, which we know is rife, and absolutely is part of the critical path of the value stream. With scale it may no longer be the most critical part, but, it is still tied to TTM and revenue, and hasn't been solved for in the ways that CI/CD, cloud app delivery, etc, have been.

5) Burn-outs is also a matter of technical leadership. It is the role of the trusted engineering leaders to give boundaries to managers. Ultimately human problems have to be solved too, not just tech ones.

• Absolutely, technical leadership is accountable for burnout, especially when this is a problem that's been mostly just ignored and not fully solved for, so there's high potential for improvement. That's the chip on my shoulder for this article, actually, that we have been speculating long enough, and its time for technical leadership to start making real investments, and making room in budgets, for the right types of human / socio-technical solutions.

Thanks again Benoit for your thoughts and your kind words, I really appreciate this dialogue, we have to solve these problems, so much is riding on our workforce!

[Ben Halpern]

Great read