DEV Community: Ulrich Kautz

I failed my own Turing Test

Ulrich Kautz — Mon, 19 Oct 2026 23:00:00 +0000

I have a lifelong affliction that has often massively frustrated me and certainly has held me back in countless ways over the years. Here it is: I suck at short-form writing. Or so I thought, until I ran into a conundrum last week. Note: this doesn’t mean I don’t suck, but if I do, then for other reasons than I thought. And that is freeing!

What I thought, or rather observed, was that I was entirely unable to write any short text about any topic I wanted to write about. I came to each text with an idea, usually an observation I made or a question that plagued me for which I found an answer I liked to share. Something that surely could be expressed in a few hundred words over the course of an hour or three. Or so I thought. What then inevitably happened, each time, was that the text blew up; both in space - thousands of words - and in time - weeks, sometimes months. Each time! It vexed me, but I simply couldn’t help myself and I didn’t know why! Until now, that is.

My newly gained perspective emerged from a somewhat distant place, yet closely related. As an engineer, I love to build. I love to find technical solutions for all the problems that I face - just because. In that spirit I was recently building a custom AI integration designed to ingest a person’s whole writing corpus - both content and style (if you must know: RAG + stylometrics behind MCP server). Armed with this corpus the AI is then enabled to emulate the person’s voice while drawing on their opinions, beliefs and knowledge to really imitate them as closely as possible. Sounds somewhat scary, but is not nefarious at all. Quite the opposite: it’s a prosthetic, a tool to help the likes of me to express themselves in their authentic voice when they are otherwise not able to. Be that for a strange ineptitude to write short-form texts, or social anxiety to share their thoughts, or overwhelm when using social media, or any reason you can think of.

In that process I came to a point where I had to fine tune my tool. My approach was a simple, blind experiment: a series of triplets of writing examples. One taken from my actual texts, one generated by the AI based on my integration and one generated by AI without any of my influence. Only the topics matched - roughly.

And here it got hairy: I could not tell them apart. Not at all! That absolutely stumped me. I should be able to recognize at least a text I wrote myself in the past, no? I mean, there are only so many options. Let’s say my integration indeed worked as advertised. Then it might “fool” me - that was the point - and I would not be able to tell it from my original writing. If, on the other hand, my integration was bad, then I might not see a difference between the texts generated with and without my voice. Or I might see three entire different styles. Still, there should always be at least one text that peeks out: either my authentic one, or the generic one. Neither was the case for me! Mind you: my writing problem forced me to spend countless hours of detailed research and long thinking on how to condense that into individual sentences that really express what I mean. How could I not recognize these products of weeks, if not months of sweat and tears?

That was surprising, to put it mildly. It also got me thinking. Well, investigating and then thinking. What I found is that there is a large body of research in cognitive psychology about the topic of text comprehension (Kintsch, van Dijk). It segregates the comprehension into three distinct levels:

Surface Code: The How. The exact words; remembering a poem or a text passage verbatim
Textbase: The What. The distilled information; knowing the exact facts of what was written.
Situation Model: The Why. The underlying model, abstract concept; making sense of the writing.

Not everyone, it turns out, excels at all the levels equally. I am terrible at (1), ok’ish at (2) and excel at (3). That means if I read a text, I get what it’s saying, I mostly remember the points that were made, but I have not the foggiest as to the precise words. Or even shorter: I discard the envelope, but retain the message. This is why I cannot even tell my own writing apart from a machine generation - as long as the structure of the points being made is similar. Yes, that was the case. That was my only instruction to the text generation: “make this point”.

Learning this about myself was freeing. It matches flawlessly with my whole life experience. There are countless examples, starting in school never being able to recite any poem or text. I remember being baffled how friends - who were otherwise much better in school than I - told me they had to memorize math formulas to use them in the tests. Math was my only saving grace, but not for my ability to remember, but because the models (as I would call them now) made sense, and once I got them it was effortless to apply them when the patterns matched.

So how did that connect to writing short stories? Well, it didn’t. Or not immediately. All I knew at this point was that I understood something about my ability to read, that also frustrated me throughout my life, that wasn’t just me “sucking at reading”. I am actually pretty good at reading, just not in the way I measured it, which was: exactly remembering passages.

Understanding this felt empowering. I mulled it over and over in my head in the following days. I shared my insight with my wife. It took a while for the coin to drop: The Why. My brain’s happy place is “The Why”. I may want to share an anecdote, a short “aha” I experienced while tinkering. Then I start writing it down. Then something happens: My brain kicks into gear, and steers me into the direction of its feel-good place: let’s research what the related field of sciences knows about that. Let’s put that experience into a bigger perspective of the whole world. Let’s build a definite ontology around it. And don’t forget a practical framework, that can be used for all future cases for everyone everywhere and forever!

Well, it’s not that bad - but close. And yes, that shoe fits me perfectly. I want to wear it. I feel safe and giddy wearing it. Yet, it kept me from writing short stories, because any short-form I ever attempted simply devolved into a research paper or essay - most of which never saw the light of day, as that got me into a gazillion tangents, deep angst of “not getting it fully right” and just provided weeks or months of surface go astray or otherwise find a reason not to finish.

So here we are. I am writing a short-form text, yet I am drawing (almost) only on memory. I am using even a story arc, of sorts. I wrote it in hours, not a day, not a week, least of all a month. Knowing about myself how I comprehend text, and subsequently how it steers me in writing helped me to now see clearly when I veer off into essay territory. Not that I never want to write long-form again, quite the opposite, but I rather have the choice.

Of course, it wasn’t that easy and straight forward. There was much more reflection, discussions with my patient wife and, yes, research in between my insight and this article. But the cat is out of the bag, to quote a German proverb. You may be the judge of the outcome. Sure, this article is not as short as I wanted it to be and, yes, I could not help but investigate at least a little bit while writing. However I am sure it’s as with everything: it’s a journey. I just found my starting point.

Either way: how do you understand text and what keeps you from publishing all you want?

Why care about DevEx?

Ulrich Kautz — Mon, 14 Apr 2025 23:00:00 +0000

Software runs the world. From small start-ups to big tech, software businesses are ubiquitous and affect almost everyone daily. Moreover, traditional companies that once thrived without much technology now find themselves embracing software as a core business requirement. Where there is software, there is software development. Where there is software development, there are often hidden costs, bottlenecks and waste - and humans who burn out from frustrating processes and high-stakes pressure.

There is one discipline in the software development world that addresses these problems and it is called Developer Experience - or DevEx for short. In recent months, I’ve often found myself explaining what DevEx is to people outside of the tech industry. That turned out to be really helpful for me, even though it wasn’t easy to describe the impact of DevEx without using technical concepts like CI/CD, platform, IDE, IaC, etc. This led me down a few rabbit holes, but eventually I gained a new perspective that I’d like to share.

DevEx in a Nutshell

If you run a business, then you care about the bottom line. To have any bottom line worth thinking about, you need a product or service that provides value for a large enough number of customers. Then you need to make them aware and want to buy whatever it is that you are selling. In short you need at least a viable market, product-market fit and the right marketing strategy.

If the business has software development at its core then this bottom line is also directly linked to the software engineering work that goes into it: How fast are features shipped? How quickly can the course be corrected when market conditions change or new information arises? How much time does maintenance take away from moving forward? Is decision making supported by high quality feedback loops? Are customers complaining about reliability? And so forth.

Multiple disciplines have emerged over the years that take on these challenges: The DevOps movement revolutionized how we think about building, delivering and shipping software. Platform engineering aims to solve common scaling problems. Part of this ecosystem is the field of DevEx that this article is about.

So what is DevEx then?

From my own experience, reading articles, blogs, job descriptions and social media posts over the years I found there is a broad spectrum of how people think of DevEx. On one side of this spectrum is the view that DevEx is in charge of building tools that help developers more easily and faster test and release their software. More specifically, that often translates to the team that owns the software delivery pipeline (i.e. the processes and tools that get the software from the development to the user). I think of this as Functional DevEx.

On the other side of the spectrum DevEx looks at the whole developer journey (i.e. from hire to leave) as if it were a critical system. That makes all frictions and blockers that affect the impact that the developer can have (on the outcome) a concern. This implies a potential agenda everywhere that can affect the developers doing their work: processes, culture, and of course also the tools that influence day-to-day tasks. I call this Comprehensive DevEx.

I am a big proponent of the latter, holistic approach. This is what I will be referring to when writing about DevEx in this article. I believe this is the only sensible approach that leads to significant positive impact for the business that applies it. What this impact is and how DevEx manifests it is what I want to convey here. In my previously mentioned discussions outside of the field I noted that doing so is not easy, because it often requires deep knowledge about software development work. There is a knowledge gap that hinders discussion. I want to bridge that gap with a thought-model that explains the impact of DevEx from first-principles.

Unpack Impact

The impact that DevEx has on said bottom line is indirect: It acts upon the factors that lead to the outcome.

So what are these factors then?

Established DevEx frameworks, like SPACE, focus on measuring both human performance aspects (satisfaction, well-being, cognitive load, etc) and system performance aspects (error rate, throughput, stability, activities, etc). I won’t dive into any of that in this article nor will I expect that you, the reader, has any familiarity with the topic. Instead I will introduce four new factors. They are not intended to be used as directly-measurable metrics for day-to-day DevEx work. Their value is in providing an intuitively understandable framework that facilitates the discussion that I want to make happen - within and outside of engineering war rooms. I strongly believe that this will open up significant opportunities and alleviate real pain. Both on the engineering side and for the business.

Consider the following:

Education , encompassing all learning and knowledge gained that is relevant for impact.
Competence , measuring how well education can be leveraged for impact.
Motivation , describing the willingness to be impactful towards business goals.
Environment , evaluating the structure and frictions of the system within which work is being done.

The first three of these factors are traits of the engineers, the last is a trait of the organization.

To establish that these factors are sensible I will share a short thought experiment for all of them. In each I will provide a scenario where two potential hires only differ in one of the factors. If this difference is reasonably sufficient to predict the impact that the hires will have on the business outcome, then it is reasonable to assume that they are indeed good factors.

Education

Education - as a container word for any learning or knowledge gain, whether formal, informal or otherwise - should be almost intuitively clear. An example in the world of software engineering is easy to construct:

Say you run a business that provides a service or product through a custom iPhone App for your customers. Now you need to hire a developer to add more features to this App. You have two candidates: Alice, who has an impressive computer science degree and literally wrote the book on iPhone App development practices. Then there is Bob, who you know just started out looking into coding in his free time. You would expect Alice to be able to be productive and ship features that translate to increased adoption and revenue much quicker than Bob, because she has the knowledge to do so and Bob hasn’t. You would expect Bob to take a long while to catch up to where Alice is already. You would very likely be right. Knowing only this difference, you can make a good prediction.

Competence

Competence - as all the experience and developed understanding to apply your education for effect - is equally intuitively obvious. Considering again that hiring scenario, this time Alice and Bob both have a comparable computer science degree, but Alice has already worked for ten years in the field at various well-reputed companies. Her resume and self-presentation in the interview show that she has seen a lot of interesting tech problems in her career - and that she was able to solve them creatively and learn from them. The specifics of her past work align well with what she is expected to do in this new role. Bob however is fresh from the university. He has great aspirations and a puppy-eyed enthusiasm. He has not yet any idea of what kind of challenges await in field-work, but he feels his university degree prepares him for every eventuality. Knowing this it is reasonable to predict that Alice is much more likely to be impactful than Bob - just on the basis of their perceived work competence.

Motivation

Motivation - as the internal drive, interest and willingness to exert effort towards a goal - clearly predicts outcome too. This time Alice and Bob have comparable education and work experience. The difference this time is their drive. Assuming you can look in their head and see that Alice is genuinely interested in the work and really wants to help reduce the pain that your product is mitigating for your customers. On the other hand Bob is bored, burned out and considers this opportunity just as another gig to make some easy cash. Based solely on this, you can correctly predict that Alice is much more likely to deliver a higher impact towards your goals than Bob is.

Environment

Environment - as the sum of all the tools, organizational structures and processes that are part of the value generation and affect the developer - is certainly factoring into the outcome. It is not under the control of the candidates, but it is the context in which they are being hired into. To see the effect Alice and Bob are moved after the hiring stage. Both have equal education, competence and are highly motivated. Alice works in an organization where she can focus on developing the App. She gets effortless highly accurate feedback from the users and can roll out changes within minutes. Bob’s workplace requires him to spend a lot of time in coordination meetings, he rarely has an hour of uninterrupted time to work on the App. New releases require a lot of approvals from different departments and many highly stressful, manual steps. They are done only monthly, bundled together with many other developers changes. Given this, it should be obvious that Alice is much better positioned and therefore much more likely to contribute a lot more outcome.

Hiring Problem?

While the environment is not in the control of the candidates, the other factors are independently good predictors for the outcome of their potential work impact. Surely, a good hiring pipeline can account for them? Well, no. Or rather: only somewhat, and it depends on the nature of the business that is hiring.

Ramp-Up Costs

Say your business is a restaurant and you have poached a known top chef from a competing eatery. It would be reasonable to expect that your new hire is able to serve their signature dish equally well in your place as they did at their previous employment. Assuming access to all the right equipment, ingredients, etc, of course.

However, if your business is in tech and you hire a senior engineer then things look quite different. Unless that is your first hire in a new venture, there is likely already a large code base, or many, and often a rather complex technical infrastructure that goes along with that. To understand the current state the engineer often needs to understand the history that led to it as well. More so, previous design intentions and choices often still make up a good part of the active code, even if they are superseded by novel approaches. It will take time for this new engineer to be able to safely roll out changes. It will take even longer until that engineer is anywhere near their peak performance. In large infrastructures a year and more are perfectly expected ramp-up times. This ramp-up time is longer for higher level engineers, which usually goes along with more experience. The reason is of course that higher levels come with greater responsibilities and therefore larger scopes of complexity that must be integrated.

Domain Costs

During this ramp-up time the new engineer will improve their Education by integrating the domain knowledge: Understanding of the code base, infrastructure, organization, culture, and so forth. They add to their Competence by applying their knowledge and navigating the domain (the workplace). Their Motivation is also affected by the experience of the engineer while going through this process and being confronted with the Environment. Negative experiences (like frustration, confusion, anger) usually drain motivation quickly, whereas positive experiences (like success, feeling supported, aha-moments, leveling, etc) build it up.

In other words, in addition to the education, competence and motivation that the engineer comes with there is:

Domain Education, that can only be acquired after joining the company and learning about the local setup, code, structures, etc.
Domain Competence, that can be only gained when already working within the structure of the company.
Domain Motivation, that is shaped by the experience of the individual engineer in the Environment.

I am going to call this the **Domain Cost. **The price that has to be paid to approach peak or even high performance.

Somewhat a Hiring Problem

Back to the question whether this is a hiring problem: If there is a very low domain cost, then sure. If there is high domain cost, then only insofar that hiring can filter for potential peak performance and potential timeframe to get there. With tech companies, unless they just start out, there is almost always a high domain cost, because it simply reflects the size and complexity of the structures that make it up and that individual contributors must deal with.

To be clear, my use of peak performance may imply that this is only about getting the last few percentage points of performance: It is not. This is about even low to moderate performance. The best software engineer in the world won’t be able to be productive if they don’t know how to ship code, have no clue where to make changes, or are unable to get the right approvals if that is needed.

Process, not Destination

Complexity usually tends to breed more complexity. Your tech business is not static. The code-base is not static: it may be refactored and restructured massively and many times over its lifetime. The infrastructure is not static: it may evolve through designs, or arising opportunities, it will shrink, grow and generally scale many, many times. Processes resulting from organizational structures are not static, because organizations tend to re-organize themselves periodically, they move through expansions and contraction, they are influenced by many changing factors like changes in leadership or market conditions.

That means: the domain specific knowledge continuously evolves, corrects itself, extends, deprecates and so forth. This is why I chose Domain Costs over Ramp-Up Costs. The expenditure is continuous. It starts with a steeper curve (i.e. onboarding and ramp-up). It still will have many small and large curves ahead.

Do not fall into the trap of looking at this only as a “cost of doing business”, because this would leave a large opportunity on the table: leveraging this understanding well to vastly increase the outcome. This brings me to the topic of this post:

DevEx for Impact

A general recipe for DevEx work is hard to come by, although I will try to provide commonly applicable examples shortly. In real life the work is highly situational. DevEx is fundamentally interested in every step of the developer journey: starting with hiring, on-boarding, ramp-up, day-to-day and ending in off-boarding. Every aspect of the systems and organization that affect developer work along this journey is a concern of DevEx. While DevEx will not “run” all these places, it will certainly have an agenda for them if they cause friction or provide opportunities. The specific measures then depend on the situation of the organization: where is the largest friction to be removed? Where are the easiest gains to get?

Having this said, let’s have a look at concrete actions that DevEx can undertake to improve the four factors. I will need to use some software development jargon to illuminate their effects, but I will try to limit that as much as possible.

Mind that what I propose are large scale initiatives, which bundle many individually useful efforts that could be undertaken separately, or at least iteratively. I aim for the bigger picture. Furthermore, none of these efforts need to be done by DevEx, although DevEx will be interested in ensuring that they come to be and is well positioned to initiate and drive them at the start.

Pave the Delivery Pathway: CI/CD

Create standardized, automated pipelines that reliably build, test and deploy software changes with minimal or no manual intervention. Yes, this is “classic CI/CD”. The goal is to increase the efficiency of the software assembly line.

What does this impact?

Environment : Drastically remove friction in code delivery by categorically removing many manual error scenarios and eliminating manual efforts.
Competence : Provide fast feedback loops for developers enabling rapid iteration and fast validation.
Motivation : Eliminate frustration and anxiety that come with manual, high-risk software releases. Increase job satisfaction by allowing engineers to focus on development work.

Unlock Engineering Know-How: Knowledge Management

Create a centralized knowledge hub (sure, a wiki) that enables knowledge ownership and provides quality feedback pipelines to these owners. Enable domain actors to contribute with ease. Advocate organizational buy-in to prioritize and remunerate knowledge management work of engineers. Based on this, create and maintain a streamlined learning pathway for onboarding.

What does this impact?

Education : Flatten learning curves, especially during initial onboarding. Reduce distribution and access costs for crucial knowledge, lowering learning barriers.
Competence : Reduce cognitive load of “hunting for information”. Increase confidence to act through reliable, authoritative information.
Motivation : Signal interest in long-term growth of individual engineers, improving trust and goodwill.
Environment : Reduce friction of navigating within systems. Disperse tribal knowledge boundaries.

I think a note on using AI (or specifically LLMs) is needed here: I consider this an implementation detail, as it is “only” an interface for the information. A powerful one, but it builds on reliable and recent domain-specific information as any other interface would. The above proposed measures for knowledge ownership and ease of contribution are unchanged by that and I maintain that guided learning pathways are also not obsoleted, but amended by this technology.

Fast-Track Talent: Systematic Leveling

Establish a buddy system for onboarding and a career-long mentor system that incentivizes participation and is supported by best-practice knowledge and complemented by self-guided learning paths.

What does this impact?

Education : Extremely short feedback loops enable quick learning. Largest possible bandwidth for tacit knowledge and culture transfer. Enables continuous learning.
Competence : Quickly overcome beginner problems in onboarding and ramp-up. Early exposure to production work, with safe-guards, builds confidence fast, reducing ramp-up strongly. Enables continuous skill development.
Motivation : Strongly reduce anxiety and confusion during onboarding and ramp-up. Demonstrates deep investment in person, building loyalty and engagement.
Environment : Supports collaborative and knowledge-sharing culture. Builds fast support pathways and deep connections that reduce communication friction.

Empower Builders: Developer Platform

Create a unifying platform where developers can make use of standardized templates, tools and services (like requesting test environments, managing monitoring, accessing application logs, configuring scaling for applications, etc) that are entirely self-service and reduce the amount of complexity developers have to deal with.

What does this impact?

Environment : Increased consistency and reduced complexity free up mental resources. Reduce cognitive load reserved for secondary activities that are not development.
Competence : Empowers engineers to manage their applications with low effort, giving them control and enabling them to focus on their core competencies.
Motivation : Less frustration when engaging with infrastructure, through hardened and performance-optimized workflows.
(Education): Persist organizational knowledge in code, countering knowledge degradation and enabling knowledge validation.

Bridge Team Silos: Cross-Team Exposure

Create structures to give individual engineers the chance to rotate temporarily in other teams (especially adjacent ones) to transfer both explicit and tacit knowledge peer-to-peer.

What does this impact?

Education : Creates a channel for tacit knowledge transfer. Breaks down knowledge silos. Enables T-shaped knowledge building (deep learning in specialization, broad understanding of context).
Competence : Improved decision making abilities through better understanding of related systems and organizations.
Motivation : Forms bonds, contributes to motivating we-feeling.
Environment : Establishes shorter communication pathways. Improves collaboration.

How Much Impact?

You may have noticed a lack of any hard numbers that predict the business outcome in the previous examples. That could not be helped. DevEx is similar to adjacent disciplines like security or reliability. They all affect the very “fabric” and shape of the systems that they are part of, which makes predictions rather difficult. It is very challenging to quantify the damage to customer-trust - and the bottom line in the years to come - that was prevented by thorough security measures. It is almost impossible to count the outages that did not happen due to top-notch reliability efforts. Similarly, it is futile to attempt to measure the impact of inventions that were made because of continuous learning structures that have been put in place. Or the time saved by automating annoying processes that then freed engineering resources to improve documentation, which then prevented countless person-years of information hunting, and so forth.

The field of medicine provides an excellent comparison. A medical doctor knows what “healthy” looks like, generally speaking. Similarly DevEx practitioners know what a “healthy” system looks like and what a “healthy” developer journey entails: smooth workflows, fast feedback, uninterrupted information flow, stable and efficient operations, etc. Medicine uses “pain” as an indicator for health problems. So does DevEx, although it calls this “friction”, which manifests as developer frustration, slow processes, unreliable tooling, or production incidents. Both medicine and DevEx can formulate treatments to improve the state towards “healthier” (see the previous examples). Both think of their domain (the human body and respective tech organization) as complex systems where changing one part influences many others.

Now, a medical doctor can still confidently say that healing a broken leg will enable a patient to walk and run substantially faster. However, they would not be able to predict how many seconds faster the healed patient would run in a 100 meter sprint. The notion itself is ridiculous. In that same sense can DevEx identify significant friction (i.e. the “broken leg”) and confidently state that addressing that will improve the outcomes: faster delivery, higher quality, more innovation, etc. So while the precise impact of individual measures may be forever elusive, there is no question whether “curing” unhealthy components is tremendously beneficial to the whole.

Having this said: DevEx, reliability, security and medicine have plenty of hard numbers to go on. They are all strongly leaning on empirical evidence. However, these are proxy metrics. Like DORA’s metrics, which are well established by now, they can be used to measure the health and identify friction, but they won’t give you a good prediction for the bottom line of the next quarter.

Conclusion

I hope this article illustrates how a comprehensive DevEx can impact the bottom line by improving the factors that lead to it. Needless to say, I have only I provided a snapshot of what is possible

I am deeply curious how others look at DevEx and see the opportunities it offers. Write to me, if you have opinions!

The Expert Deadlock Dilemma

Ulrich Kautz — Sun, 02 Jun 2024 23:00:00 +0000

This article is written for software engineers that work in collaborative settings. It is also interesting for anyone on the business end of tech and everybody that works in supporting tech teams with decision making (if only to read the perspective of someone on the receiving end of that support).

If the following story speaks to you, then keep on reading:

A small team of software engineers sits together in a cramped, windowless conference room. It smells like sweat and sugary drinks. The AC tries to make up for its inability to sufficiently air the room by producing more noise. Snacks, laptops, and whiteboard markers litter the table in the center. This is the fifth time this week that these people have come together to talk. It is only Wednesday. The last two weeks looked no different.

Despite everyone on the team being really experienced in their domain, which they all share, you can feel the tension. Nobody is at ease. Odd glares are exchanged. People shift in their seats and seem not to be able to find a comfortable position. One member frantically scrolls through a large document. Another cleans the whiteboard on the wall with quick and needlessly powerful strokes that speak of anger and frustration, something that is palpable in the room. You can see varying degrees of the same frustration and the same anger written on everyone’s face. And exhaustion. Everyone looks as if they haven’t slept for days. They have. Their fatigue is caused by hours upon hours of discussion over days and weeks. Discussing what? A simple question, really: what is the best approach to solve ?

_By now, everyone in the team has developed a strong opinion about that. The trouble is just: Not two of them agree on what the best approach is. Only that the other’s methodology is flawed and ridden with pitfalls. Only that much is agreed. One of the engineers makes a throat-clearing sound and starts to say, “I think I have found another perspective to illustrate why we should not consider the outcomes of yesterday’s proof-of-concept a valid sample, because…” which is as far as that goes, before a chorus of moans and snarls from the others around the table puts an end to this attempt. “Really?” asks another. “You spent the rest of your time yesterday trying to prove that we are unable to agree on even that much?” Heads around the table nod vigorously, and the first speaker starts to turn red. “Yeah, really,” the exasperated reply bursts out, “we cannot go forward if it is clear that we will fail on this path!” More irate sounds emerge, and the mood in the room noticeably sours by at least one more octave.

Let’s pause here. This illustrates what I’d call an Expert Deadlock. Nothing is moving forward. Nothing is being decided. Everyone is mad. It is bad for every individual involved because it smokes you up, burns you out. It is bad for the business because it is the most unproductive state possible: total stalemate. It may not be as bad as illustrated in every case. Sometimes you are able to still progress in some areas while being in a deadlock with only specific decisions. Sometimes a deadlock lasts only for days and not weeks (or months). Either way. It is always very taxing in the moment and it always has the potential to cause long lasting damage in the team relations.

Have you been here? Are you in that picture? How do you think a mess like this came about? How can you fix it?

Well, I’ve been there. I’ve seen it happen to others, and I’ve heard about it, too. Speaking to a friend who currently is in a similar place is what triggered me to write about it now. There are many variants of this scenario. Often, it is not as bad as my fictional story above - at least not yet. Sometimes it is worse. From what I can tell, this is not an uncommon occurrence at all. Alas, among software engineers, which is my realm of experience. If you haven’t seen it yet, count yourself lucky. Or not, because then you might not know how to handle it when you encounter it. We’ll get to that later.

Casus Belli

So, how do we end up in a dilemma like this then? From my observations there are many individual reasons and contextual circumstances that contribute. I am not interested in those. What I care about are the common, underlying patterns. If they are known you will fall prey less easily to them. Let me also be clear: this is not about bad actors that have bad intentions, are motivated by greed, or are malicious - or any such notions. I am sure all of that could easily yield a situation as described - but it is not needed. This happens to juniors and highly experienced people alike, who work together in good faith, and with good intentions.

The following reasons I’ll be discussing here are all contributing. All of them are sufficient, as in: they are enough to produce an Expert Deadlock. All of them are also probabilistic, as in: You can get away with any of them, if your circumstance allows it. I highly doubt that this list is comprehensive. It is only the list that I have experienced myself so far.

Goal Glitches

A common contributor to messy situations is the good ol’ misalignment of goals. If this divergence is known, then it’s easy: you can account for that - or choose to part ways. However, if it is not known, then it becomes awkward. You may pitch your idea with the best reasons that are fully supported by data and still you find yourself rejected. And probably hurt. The other person simply doesn’t share your priorities as they have a different goal in mind. Imagine we are both bakers and I try to convince you that blue icing is the way to go, as proven by extensive empirical research. You simply don’t care, because you cannot see how this contributes to the goal of healthy cakes that you have in mind. We are badly misaligned here. The chance of us finding an approach we both agree on is very low – especially as long as we both operate under the assumption that the other is sharing our goals.

This contributes to Expert Deadlocks in two ways: First, by people simply talking past each other. I’ve taken part in hours of heated discussion only to find out that everybody was talking about something different entirely. You might think this could only yield a brief deadlock, because such misunderstandings are certainly uncovered quickly, right? Well, if you are lucky, yes. However, it doesn’t take long for emotions to become entangled and then for stung egos to supersede rational argument. Once at that stage, it is not easy to think straight and even the obvious may elude you. Not only can it take much longer than you think to even identify the misalignment when in this state, people that are that riled up may still choose to continue arguing either way.

Deep Ends

The holy triangle of the three Ps - Perfectionism, Pragmatism, Passion - is both a curse and a blessing in any software engineering endeavor. They must be carefully balanced to arrive at a desirable outcome.

Perfectionism makes us focus on details that might be essential. It drives us to optimize and iterate until we can make true breakthroughs. Too much perfectionism leads to many yaks losing their fur and many rabbits being disturbed in their holes. Nothing is ever shipped because something can always be improved, or polished, or refactored, or …

Pragmatism makes us focus on solving the problem. It allows us to be open to any approach. It drives us toward efficiency. Too much pragmatism results in a whatever-attitude and sloppy solutions. Trashware systems, the eventual outcomes, are as maintainable as they are usable: not.

Passion is The _key motivator. It feeds us energy. It makes us stay after hours and come back early in the morning. It drives us towards impossible goals – and makes them possible. Too much passion makes us blithely impervious to facts and data: “_The critics are just afraid” or “You just fail to see the vision”. A marvelous approach to create spectacular art. A terrible mistake for the business of engineering.

Expert Deadlocks can easily arise from an excess of both perfectionism or passion. Either makes for a great foundation for last stands on the least relevant hills. I cannot see extreme pragmatism leading to a deadlock. However, it can create meetings where nobody gives a sh*t and that comes with its own pitfalls.

Tunnel Vision

This happens to the best of us: You are deeply into a topic, say a technical solution of sorts, then you gradually lose perspective of the big picture until you are so focused on the task that you forget why you started it. You simply lose sight of the reasons for your actions (and everything else) by focusing entirely on those actions. That is not a wholly wrong approach either: we humans are single-taskers after all, and we are much less prone to errors when doing one thing, than when doing multiple things. This focus increases the quality of our work. Even if this work is unnecessary or counterproductive.

I’ve seen Expert Deadlocks arise – even with rather seasoned engineers that you would think know better (and yes, I was one of them) – when people just get too invested into a solution to be able to back out easily. It hurts to do that. Nobody wants to get hurt. It’s so much easier just to keep going. A room full of engineers that are afraid to lose all the efforts they have put in over the last days / weeks / months – not to speak of all the self-worth that is bound up in that – is not the most rational place.

Being able to step back, take a breath, detach yourself from the emotions of an approach and refocus on the bigger picture is both a skill and a function of culture. It can be learned and it can be taught. And it can be prevented if acknowledging or making mistakes is punished.

Fear Is The Mind-Killer

This brings me to another common cause. And it is a sneaky one. While being able to step-back and regroup is a great skill, it can also be overdone. Being afraid to commit to a pathway. Ever seeking the comfort of the higher-level perspective. While that may not end in the clash of strong opinions, it ends in a myriad of evaluations, trials and proof-of-concepts that eventually have no value because they never go far enough to have any effect at all.

Thereby it can also lead to an Expert Deadlock by creating a room full of critics and nitpickers. Everyone then only sees problems with everyone else’s solution. Always one more alternative that needs to be reviewed. Effectively nobody is willing to take the risk of having signed off on a future failure. The only proposals are then either so outlandish that there is no chance in hell that they will be accepted (so there is no risk to propose them) or they are so vanilla that doing them or not makes no difference (and there is no risk of failure).

Dog Eat Dog Ethos

Somewhat related to the previous point is a culture of hyper-competitiveness. A place where everyone is pitted against each other, where any advancement requires continuously winning against somebody else (or cutting them down, to step over them). I know this may sound like the bad actors I said this would not be about. To this I’d say (with sadness): if it is culturally / structurally caused, then the actors are not bad, just rational.

Either way I think it’s pretty obvious how this creates a breeding ground for Expert Deadlocks: The overarching incentive is to promote yourself. If it is more important to look good than to make headway towards a shared goal, then why would you? Agreeing to a proposal from your boss, so as to increase the chances of climbing up the ladder yourself? Sure, a concession here may get me a promotion. Agreeing to a proposal of a peer, however suitable and beneficial? Nah, if they shine and get the promotion then I won’t.

Feedback Void

This is the big one. Big as in: All of the above can be mitigated or aggravated by how this is done. The previous point about hyper-competitiveness illustrates how feedback loops that stimulate bad behavior lead to a deadlock. This is about how the lack of them can be even worse. Feedback is essential to being able to formulate goals and specify the right work that needs doing to achieve those goals.

How those feedback loops are implemented depends on the individual organization: They might come as leaders that are sufficiently knowledgeable and involved to be able to make decisions and can act as tiebreakers. Or they emanate from a community culture where proposals of solutions can be discussed without fear of giving critique nor reluctance to receive criticism (think RFCs). Or both. Or something else entirely.

If such mechanisms are lacking and an individual team is entirely free-wheeling, then any of the above factors can easily take hold and grow. There is no steering mechanism that provides direction or even keeps us from falling into the abyss. The Expert Deadlocks that will eventually arise are likely one of the lesser problems that come out of it.

Make Progress Not War

Nobody wants to be a character in the story at the beginning. Still, we find ourselves in such situations. We discussed some factors that make it more likely that we arrive in such a mess. The questions now are: How can we make sure that we do not end up here? And: if we do, how can we get out again? Prevention and remediation.

Prevention Strategies

Measures that reduce the risk that deadlocks arise are almost exclusively structural. Which means they must be implemented at organizational level. They must be part of the culture.

Culture Of Candor

If you do not know where to start, then start here. The following is absolutely fundamental for highly effective teams. You can look at it as a general antidote that strengthens teams’ abilities to work together and therefore immunizes against deadlocks and other ailments. It comes out of a body of decades of research, among which a multi-year study at Google that involved hundreds of their teams to answer the question: “What makes teams effective?”. It was called Project Aristotle and you can read up on the journey that it was here.

The primary learning that you should take from that is the concept called Psychological Safety – a term coined already at the start of the millennium by Prof. Edmondson from the Harvard Business School. It measures the degree to which people in teams / organizations believe that they can be candid without being punished for it. There is of course more to it, but this is the gist. A team that rates high on Psychological Safety is much more likely to be effective. As a matter of fact: Google’s study did not find any other factor that came even close in predictive prowess. This efficiency also includes the ability to reach decisions smoothly, that is: without falling into Expert Deadlock traps.

So how can you help to improve the perception that candor does not backfire then? Well, if you are coming from the DevOps-world as I am, then you are probably immediately reminded of Blameless Postmortems. Postmortems are processes to analyze previous incidents, so as not to repeat them. The practice of Blameless Postmortems shifts the focus away from individuals as the primary contributing factors and instead concentrates on the systemic causes. It has been very effective in addressing root causes and improving systems, hence it is by now widely accepted across the industry. Another consequence is that it makes candor without punishment a valued, nay necessary virtue.

Of course, this practice is only relevant for very specific situations: reviewing past incidents. In Systems Thinking exists an idea that arrives at extremely similar conclusions as Blameless Postmortems, only it applies them to a much larger scope. It is called Just Culture and it seems to be spreading through many industries. The principles are again the same: instead of blaming individuals when something goes wrong, it asks “What are the causes?” and then focuses on solving the systemic issues that such questions uncover. Optimized systems are the outcome. Again, this only works where candor is not punished but encouraged, thus looping back to Psychological Safety.

Note: this a prevention measure because it takes time to build a culture with sufficient trust to be candid. Do not attempt this in the moment of crisis.

Escalation Path, Safety Net

Setting everything up to lessen the likelihood of arriving at an Expert Deadlocks, as the previous point aims to, is where you start. However, consider that it takes only one nasty Expert Deadlock that leaves a formerly high-achieving team in tatters. Even if it doesn’t fully destroy the team, each such highly-emotional stalemate chips away at the bonds of trust that tie the team together. High cost, even if you are able to make it low risk.

My years of work in reliability topics left me on the constant lookout for risk mitigation strategies. What comes to mind here is the need for safe-guards that trigger automatically before a critical threshold is reached. Those safe-guards should have the shape of escalation paths and a set of guidelines when they trigger. Like: If we come out of the second meeting about the same topic without agreement, then we escalate. The simpler the better. It is more important to establish this as a habit than to optimize for efficiency. Err on the side of escalating too early rather than too late. Escalating is cheap, failing to is not.

As for how the escalation plan should look: the best I have seen requires simply having an empowered lead engineer on each team. In less formal situations this role can also be filled by an inofficially agreed tech lead. Whatever the name, the point is that one person is the predesignated decider. They are handed the mighty decision-hammer that they can take out in the time of need to break a tie or to end a deadlock. A technical manager that is sufficiently involved in the day-to-day can also work. Worth mentioning here are the perils of micromanagement and running amok on a power-high (neither of which I will detail in this article). In any case: Decision making through escalation should be the exception, never the norm (if it is then this should ring alarm bells about the state of the team).

To emphasize the previous point about candor: this goes for tiebreakers as well. If they are afraid to make a decision their value is blunted at best and negative at worst (i.e. when their decisions are not based solely on best knowledge considering all input).

A single level of escalation will already mitigate the risk significantly. Another level, like a director or CTO that is indeed willing and able to be of value here gives a warm and fuzzy feeling to everyone involved – even if never really used.

I found that not only a lack of organizational planning but also an overemphasis on a culture of consensus leads to situations where such escalation paths are missing (or worse: are seen as bad, because they are “too hierarchical” or some such notion). While this approach may be admirable and can work in specific situations for a while, I think it ultimately leaves teams vulnerable. Without clear escalation paths, every disagreement has the potential to escalate into a full-fledged confrontation, putting the team at risk of falling into an Expert Deadlock.

Note that this is a prevention measure because designating anyone in a position that can serve as tiebreaker requires good faith and ideally is giving credence over time by solving smaller team issues. Attempting this in an active stalemate situation can easily backfire and burn more trust and bridges than you aim to repair. Otherwise see the following headline “Utilize Peer Perspective”.

Remediation Tactics

Having the above prevention measures in place goes a long way. However, as you are here reading this, chances are it is too late for you to think about this now.

If you are currently in a stalemate of sorts then your first priority is to get out of that. Following I will give you tactics that I found useful in the past. Of course, they won’t be all applicable to your current situation. You will have to cherry pick what makes most sense for your case. To do that you first need to know your options:

Peer-to-Peer Arbitration

Shift the question! While you are in a deadlock to decide a particular question: can you still agree on something else? Like which pizza to order? If so, then good. There is hope for an easy out. Try to agree on the following: an external arbiter. This person must be experienced enough so that you all will trust their decision and far away enough so that you do not suspect bias. This is somewhat similar to the prevention measure of a tiebreaker, but different in that it is a one-off and cannot be someone already part of the team that is part of the stalemate.

Whomever you end up deciding on I would strongly recommend not to involve them in your (potentially heated) discussions but instead to give them a short(!) written summary of each branch of the decision tree that you are unable to agree on. Allow them to pick all of your brains if they have questions. Ideally in 1:1 meetings, so as not to have them influenced by the eye-roles and moans of despair that might come up in a group-setting. Now to the hard part: Await their decision. Trust it is good enough.

Note that only full agreement in advance of the arbitration will help you to break the stalemate. Pulling somebody new in without having the team’s consent only aggravates the issue and is not helpful (doubly so if they support your position).

Less Planning, More Driving

I had a really hard time to put this into words, so I have to fallback to story time again:

Imagine a gaggle of disagreeable Tolkien’esque dwarves that visit the surface for the first time. They are quarreling over the right route to take on a map. They all concur on the destination – a place full of precious treasures, gold and gemstones of course – but they cannot settle on how to get there. The destination is a long way out.

The map they are looking at is badly drawn. Some roads were clearly added later by an amateur. Annotations litter the canvas: “road does not exist anymore” it reads here or “don’t trust this overpass, it has fallen down” is written over there. The dwarves are loudly squabbling and arguing with each other. One proclaims angrily “Your path is curving away from the destination! How could it possibly bring us there?” Another exasperates “The bridges you claim are there do not exist on the map! How would we get across those canyons?” And so it keeps going on and on.

Riddle me this: how can you find out which of the dwarves knows the right path? Well, you can’t. The thing is: they are all wrong. The reason is that the map is just too bad. Any plan based upon it is doomed to be wrong! There is really only one strategy that delivers the bearded travelers their destination: Stop looking at the map and look around instead, then head in the direction of their destination. Ideally find higher ground. Maybe scout ahead. Use the map only to navigate around large, unchangeable features like mountains or canyons. If they keep on doing that they will arrive eventually. Unless they discover an even better opportunity on their route, which might happen.

Why do I tell you this story about dwarves and maps? What does this have to do with remediation of Expert Deadlocks among engineers? Everything. If you are one of the engineers in a deadlock, then you did put your own proposal forward. Withdraw it. Now! Acknowledge that all of your solutions are based on incomplete / invalid information about the technical landscape. Worse, the landscape is ever shifting and evolving with new pathways popping up all the time, while old ones deteriorate or disappear entirely. The further ahead, the further in the future, the more wrong your assumptions will become.

If you cannot make this plain to everyone (remember: you are in a deadlock) you must break the deadlock by instead picking one of the other proposals that a) goes mostly in the right direction and b) is the most likely to get more support by others. While you should not assume that this chosen route will deliver you to your goal, you need to accept that it is enough to move ahead towards where you want to go. By that you take on the role of the driver that gets everyone moving in the right direction. Prioritize progression on the right trajectory over the specifics of the movement!

Yes, there are a multitude of asterisks to this, many of which I simply cannot put into words. I give this advice with the assumption in mind that most engineers strongly err on the side of overthinking. I am also harping on this point and wrote a whole story to bring it across because my advice often necessitates to go along with or even champion flawed plans. This seems especially hard to do for engineering-minded people. It certainly was (is?) for me. I think this is the case because among engineers subpar solutions are considered almost offensive. This is why criticism of our own solutions are often received as personal slights to our skills and abilities. Get over it. It is a deterrent from being a driver of progression. You are building an obstacle where you should build a ladder.

Use The Right Tool

If there is one thing that humans excel at it is certainly tool creation and use. Tools are the distillation of learning into a device or process that makes doing specific things easier (or even possible). There is a whole pantheon of tools that are geared towards team decision making. I am by no means deep into this topic, but I have employed some such tools in the past and I have also seen them employed by others. More importantly I am in the lucky position to know someone closely that is extremely experienced in this field. My wife. I picked her brain so that I can give you some immediately useful tools. Useful meaning: circumventing or hacking intentional or unintentional behaviors that are obstacles in the pathway towards making a decision.

Know that I have no affiliation with any of the following (nor do I use affiliate links). You can take this as “to the best of my knowledge”.

1. Workshop Tactics

I write this article with people like myself in mind: software engineers. I assume you are one. That means you are likely specialized in some deeper field in that domain – as I am. Hence you can conceive that there are other people specialized in other equally deep fields. You probably also know how rarely such a focused immersion coincides with the ability to communicate or explain it to someone on the outside. Are you with me so far? Ok. Here is the thing: there is someone that is deeply into the field that pertains to team decision making. This someone is named Charles Burdett and he happens to also be an amazing communicator. He used that to encode his knowledge in a way that is immediately actionable (drumroll) in the form of a deck of freaking playing cards! You can buy this deck for small money (search “workshop tactics deck”). Or you can use it entirely for free online. It even comes with a whole decision-tree interface that guides you exactly to a very useful set of immediate things that actual work to tackle. Try it.

2. Design Sprint

So there is this big internet company that has a lot of very expensive (because very good) people employed. They have a huge amount of great ideas that they want to bring to the rest of the world as products. As execution is generally hard, many such attempts to productize ended in a mess. Or if not: they took longer than anticipated and were more expensive than necessary. Consequently the big internet corpo was eager to find a reliable and reproducible way that allowed them to go from idea to product. This way also needed to be easy enough so that everyone could follow it without creating a mess. The company is named Google and the methodology they came up with is called Design Sprint. It is a formalized 5-day process that starts with understanding the problem space, goes to sketching solutions, then making decisions and lastly validating the approach (it can restart from there if this fails). The leading inventor at Google was Jake Knapp who later published the recipe of the sprint in a book called “Sprint”.

Both of the above proposals share a couple of elements that make them rather efficient:

Less overthinking, more momentum through time-boxing and other forced constraints
Focus through guidance of structured activities vs free-wheeling
Quick decisions through processes that whittle down options fast

There is more to it, but this is what convinces me. I didn’t like the gamification aspects (just seemed contrived / fake). Still, not a great fan of that part. However, I am swayed as I have seen reliable results with good quality coming out of this. And quickly too.

Seek Professional Help

Who would have thunk: there is a whole industry that is facilitating the above. They are called (drumroll) facilitators. Mind blown. But seriously: if you cannot do this alone, there is no shame in getting help. You are already great engineers (I assume), there is no need for you to master this additionally. I mean, do it if you are interested! It is a great skill to have in any collaborative environment. Whatever you decide: do not stay and wallow in your misery.

You may have these kinds of resources in-house. Often Product Managers / Owners are knowledgeable about this. If you happen to have teams that care about Learning and Development (L&D) they are certainly a good address. Generally Human Resources may be experienced as well. If none of that exists or is unhelpful then you will find aplenty resources externally. Ask your peers and friends if they have worked with anyone they can recommend – you might be surprised. Otherwise a google search for a local facilitator or team coach will certainly yield you a lot of results.

The equation that you should consider / pitch to business is simply this: X engineers are not productive for Y days due to stalemate equals Z money. Additionally, through team burn-out and breakage resulting from this, efficiency will be significantly down at least short- to mid-term. Likely long-term. Is this cost smaller or larger than professional help?

Note: If you are in a position to make that call: Make that call. If not then consider putting a limit on how much time you spend in your attempts to get the funds. I found myself frustrated in the past fighting against the wrong windmills (if you keep hearing “sort it out yourself, you are all professionals” then consider it a sign of resigned helplessness and move on).

Get Out

This is the hardest. And the easiest. It can feel like giving up. It is not. If you tried, really tried, to resolve your disagreements and still could not do it then acknowledge that and move on. It doesn’t mean you gave up, it means the cost of going in that direction exceeded the value it yields. This can feel almost like a break-up in a relationship. It can go equally wrong and leave everyone hating each other afterwards. Or it can go well and people agree to disagree and everybody just moves towards where they want to go next on their own. No harm done in this case.

Advice how to get this right probably fills books. The only one I can give you is to do it quickly, kindly and firmly. Above all be clear with yourself and anyone else – this is probably the greatest kindness of all. Remember that you can only control your own actions and nobody else’s. This is also where your responsibility ends.

Fix it?

From what I can tell things (at least in the software engineering industry) are moving somewhat in the right direction – but not across the board, only in some select companies that invested the time and money to gain the metrics that make the factors of productivity crystal clear to them. So I think it is on all of us to change that. I do not exclude myself from this at all. Here is how I think you can help, depending on where you are:

Tech Side (developers, engineers, product?, etc)

Do you believe in using the right tool for the right job? If you do then make sure you act the part. Saying things like “don’t bother me with this non technical nonsense” is a direct pathway to more frequent, more painful and longer meetings. Unnecessarily, because there are well researched practices and tools that are indeed getting the job (of reducing needed organizational chores) done. If you are already familiar with strategies and tactics to that effect then share them. If not then consider learning them. At the very least be open to take part in them. There are best-practices for collaborative environments. This does not come easy to me as well and I am getting it more often wrong than right. I get you. We just have to get this fixed.

Business Side (managers, executives, product?, etc)

The hidden costs of Expert Deadlocks in your engineering teams are significant. Beyond the immediate loss of productivity, these situations erode team cohesion and can lead to long-term and permanent efficiency declines. Investing in prevention and mitigation strategies isn’t just a feel-good exercise – it’s a necessary business expense of the same token as continuously investing into technical debt reduction is. The return on investment in terms of increased productivity, faster time-to-market, and improved team retention can be substantial.

People Side (L&D, HR, product?, etc)

From where I am sitting your value proposition is not being received in the way I think you intend it. Speaking for myself as a software engineer: I am convinced by fewer, shorter and less painful meetings and subsequently more engineering time. You should focus on virtues like that if you aim for an enthusiastic “hell yeah, I’ll do whatever you say” from the likes of me. To put it bluntly: Right now you are often perceived as overhead by us. Additional things we have to do that detract from the things we want to do. This is a framing problem. This is a communication problem. I think you are also not convincing business as it stands. Alas the preventive measures I spoke about are rather elusive. Given the business risks they mitigate, they should be absolutely everywhere. TL;DR: Plz change your pitch.

Outro

Wow. You made it. What a long post. Thanks for reading! I’d be curious to hear about your experience – whether you are a fellow code monger, or one of the brave souls that tries to help us heathens. Comment or DM me on LinkedIn.

The Rubber Duck That Talked Back

Ulrich Kautz — Mon, 01 Jan 2024 00:00:00 +0000

In software engineering there is a practice that is called “rubberducking”. Or more precisely: rubber duck debugging. Look it up, it is real. The expression comes from a hugely transformative book, called The Pragmatic Programmer, that was written at the end of the last millennium. Rubberducking is a very simple technique that has been profoundly useful to sufficiently many people so that it stuck around. It goes like that: When you have to reason through an intricate problem, then explaining it to a rubber duck that sits on top of your monitor, will help you to figure it out. That’s about it. In its original form, rubberducking was specifically about finding hard-to-spot bugs in a large amount of lines of source code – but you can use it for any kind of problem. It also does not need a yellow rubber duck sitting on a monitor. You can use any toy or other inanimate object that you prefer. It will work. It will work brilliantly.

Why does it work?

Well, for a couple of reasons. First, by speaking your thoughts out loud you are forced to transform them from your inner thinking into externalized structures like words, phrases, or even full sentences. You are thinking aloud. Doing so inadvertently surfaces all the silent assumptions and subconscious conclusions you made – and the gaps therein. You will easily recognize those gaps, for they are hard to put into words. Once those gaps are known you can of course close them by attaining the missing knowledge. This in turn will increase your comprehension of the topic. In short: the rubber duck helped.

Another reason is that through speaking aloud you hear your words back. Sounds trivial. And it is, but it is also powerful. It sounded better in my head is a common saying for good reason. Alas I can personally attest that most of the things that sound great in my head somehow become incoherent noise once they leave my mouth on a gust of air. I am sure you have observed this with yourself and others as well. It is a rather common thing. A human thing. Hence, the rubber duck helps again (not only the speaker, but also the audience) by being the silent recipient of those bumbling first attempts to formulate a thought.

When doesn’t it work?

The rubber duck is very helpful, but it is no magic bullet. While it assists with applying reason to find a solution, it still requires you to do the legwork – so to speak – of gathering all the arguments and acquiring all the information that underpin them. In a word: research.

Research is the process of expanding your knowledge. There are endless possible ways to approach that. My own “standard algorithm” – that I apply when writing more involved blog posts – starts with finding and reading popular articles that target a non-expert audience in the domain I am interested in. This usually teaches me the right words to find broader “meta-studies”: writings that summarize the learnings in a discipline. This step makes me dig into the methodologies and the language that the field uses to communicate – at least to some depth. Armed with this knowledge, I can find, read and understand deeper papers and articles. This is usually sufficient to get the answers to the questions that come up while writing.

If you think this sounds like a lot of effort then let me assure you that you are correct. It is.

In my day job my research approach is similar, although I am usually targeting solutions to specific problems and I am normally already deep into the topic that I am investigating. You are probably doing something comparable. At least if you are working as any kind of knowledge worker.

None of that is new. It comes with the territory of working with information. It takes a lot of time.

Enter: Conversational AI Assistant Systems

For about a year or so, I have had help with research. I hired an assistant. Well, not quite. I started using Conversational AI Assistant Systems (CAAS) like ChatGPT, Bard and Claude. A lot. Concurrently. It is impossible for me to convey how much easier my research and subsequently writing has become. It made the process at least two magnitudes less painful – maybe more. It, quite literally, got me thinking. In this case, it got me thinking: why does this work so well? I mean, I had the same information accessible before. Via normal web search. Googling. Why is this new interface so much better supporting my research?

From what I can tell there are multiple answers. Let’s start with the obvious one: CAAS are large language models (LLM). They are trained on mind bogglingly huge datasets and can aggregate the interconnected information into coherent answers. This is a significant improvement over googling one thing, which just gives you two more things to look for, which each give you two more things and so forth. Keeping track of all these different threads of thought is not easy. Human error is likely. The risk of finding yourself in a deep rabbit hole is continuously mounting. Not so with CAAS, that gives you a combined result in one understandable chunk. This compares best to the meta-studies I was mentioning earlier – only they are made on-the-fly for arbitrary questions that I have.

Now I can get to why I started with the story about rubber ducks above: CAAS helps you massively to formulate your thoughts. It turns out that it matters not whether you speak your thoughts out loud or write them down. As long as you do so with the intention that they can be understood by someone else the same corrective effect occurs. With one slight difference: This rubber duck talks back! And this is not a trivial difference. At the very least your questions will be reflected back to you through the answers the LLM spits back to you: If your formulation was ambiguous you will find that so is the reply. If the answer is seemingly wrong or unfitting: chances are your question was not what you wanted to ask. In short: CAAS gives you a continuous feedback loop like a conversation with a (human) expert would (and yes: you can actually have a spoken conversation instead of a typed chat, if you prefer). More so, this feedback loop is guiding you to the answer in a way that smells very familiar to another known good practice: the root cause analysis (or “20 questions”).

Lastly the questions you ask a CAAS are rarely simple singular words or short sentences. You can formulate immensely more complex and nuanced queries that you can express in multi-paragraph length. And you will. The amount of information and conditions such questions can contain is vastly greater than what you can type into a web crawler search input like Google or Bing (until they become a direct interface to a CAAS as well, of course). You can also iterate repeatedly, which allows you to keep spinning your thoughts almost frictionless. CAAS allows your thinking to flow unhindered.

All good, then?

As with any good thing: there are caveats. For CAAS, or rather the LLMs that underpin them, you will find serious concerns enumerated everywhere so I will make this short:

Hallucinations, where produced results (answers) are entirely wrong, nonsensical, surreal, etc. Trusting results blindly can easily produce harm in the real world.
Bias, where generated results are skewed due to unknown (or when malicious: known) trends within the training data. Prejudices propagate unhindered and can even threaten lives.
Privacy concerns and security risks , where inputs leak into the training data and become available to other users or the providers that own the systems or malicious third parties.
Creativity deterrent , which results from outsourcing thinking. I, for one, have a hard time navigating without Google Maps these days. How about you?

This is certainly not an exhaustive list, but it should paint a picture of the scope of the problem. My stance is simple: Use with care. You can harm yourself (or others) with any tool. CAAS is just that: a very complex and very sophisticated tool. Personally, my biggest concern is the last item on the list: impairing our ability to think creatively. The other items are rather technical problems that will find a solution, given sufficient time and resources. But this last item has indeed the (slight) potential to cause significant harm in the long term.

However, where there are serious risks, there are also equally substantial chances and I am very excited to see what comes next.

What’s next?

I think that CAAS will become ubiquitous, because it is the next generation of interface and it will enable a whole new generation of massively more powerful (and complex) tools that will make life and work easier. Here comes my wishlist:

“In 1986, if you added up all the information being blasted at the average human being - TV, radio, reading - it amounted to 40 newspapers-worth of information every day. By 2007, they found it had risen to the equivalent of 174 newspapers per day.” – Johann Hari, Stolen Focus, 2022

Living in the Information Age is wrought with the problem of every individual being exposed to an ever increasing bandwidth of information. CAAS is the first tool that gives me hope that we will be able to mitigate or even reverse this destructive trend. Not by decreasing the amount of information that hits you, but by increasing the amount of information you can handle. Imagine a fully integrated CAAS that has access to your emails, slack messages and any other communication channels you use. And your documents. And your pictures. And everything else. Now imagine your CAAS automatically summarizes and prioritizes all communication streams for you – on the same level and with the same quality a highly trained, full-time human assistant can do today. Basically: CAAS has the potential to give you the signal while keeping away the noise. It could do even more: replying in your stead and interest to (some of the easier) communications. Or at least gathering all requested information and preparing replies or reply variants for your review. You get the gist. All of this would be possible today, however (afaik) it is not (yet) offered by any provider. I would call this a fully integrated, personal CAAS.

This immediately brings up another item: Trust. Currently, all CAAS are provided and maintained by private companies. You do not know what will happen to your data that you store with them. Maybe they are all great people with good morals - but who knows who buys them tomorrow? Maybe they have great intentions – but their security posture is far less comprehensive than they think and they lose all your highly private communication to some internet pirate from overseas. This is just a small excerpt of all the possible horror scenarios. Independent of this, there is also the question of data accessibility: say only your documents are with Google, but your company messaging is with Slack and your mails are hosted by a small local outfit. Google’s CAAS will likely not be able access all of that. So to bring about this shiny future where CAAS can double as a full personal assistant there needs to be a different approach that assuages privacy concerns and is not tightly coupled with a single data-sphere. This could be solved in multiple ways. One would be to “offline CAAS”. This is entirely possible. A lot of very big LLMs are already available for download. Big players like Apple are actively working to bring LLM execution to mobile devices. This can be done! We need trusted, private, vendor-independent CAAS.

Lastly, if I can have my way, all of the above will be carried on the wings of an agreed upon CAAS standard. Right now, as a user, you need to come up with creative workarounds like the word document that my wife uses to store an “initial conversation” that she uses to “seed” the CAAS with context. This is a hack. It is a hack that works, but it is a hack nonetheless. What I want is a transportable behavior profile that contains all the information of how you want the CAAS to behave, what it should know and what of your data it should have access to. Maybe there are even transportable discussion profiles, that capture the state of a investigation, so that it can be easily continued at a later time with a different CAAS. All of that I want of course to be vendor-independent, so that I can utilize different CAAS providers - online or offline. In short: I want the execution (LLM) to be separate from the data (profile) and the data to be transportable.

Ok, that is it for now. The future stays exciting. Thanks for reading.

Disclaimer: I am not professionally involved in developing AI, ML, LLM - or anything related to that. I am writing as a user of such technologies.

Models, a story of human success

Ulrich Kautz — Tue, 24 Oct 2023 23:00:00 +0000

It is not what you think. I think. I am referring to the abstract concept, not the profession in the fashion industry. Models that simplify, that abstract and approximate. These are the models I am speaking of and they are awesome – truly! Why? They help us to think way beyond our ken and through that they drive all human intellectual endeavors, all technological innovation and all scientific progress ever made, since we left the trees.

Why, you ask, can models do that? Well, because they enable us to have meaningful discussions about complex topics. With ourselves, in our own mind, or with others. That said, those discussions may still not be easy, but we can at least have them. I know, that does not answer the question why models can do that. I’ll get to that by separating the question into two:

What makes discussions about complex topics hard?
How do models make these discussions easier?

To get to the first part we need to agree that complexity means that there are many parts with many connections in between. That means a thing or topic is simpler than another thing or topic, if it has comparatively less parts and connections than the other (I don’t know if parts and connections count equally, I will assume it’s about the total sum). With that definition, and our current understanding of cognitive sciences, we already know that the human capacity to hold many things in our brain is limited (see working memory capacity). That alone means: we need to find a way to whittle down the quantity of things that we cram in our head. But it gets worse from there: when discussing any topic of real complexity, there are multiple layers of perspectives, of resolution to it. We need to switch in between them as we think and argue. Again cognitive sciences quantify how poor our abilities are in that regard (see memory decay). While I am sure there are even more reasons, these two alone suffice to make any discussion about complex topics in detail hard or even impossible.

Now that we have established we simply cannot speak about the whole complexity at once: how then can models help us to still do it? Well, the answer to that follows from how we arrive at a model of a thing. We do so, in very basic terms, by removing both parts and connections, until we are left with something that still has the general shape, the general character of the original, but has a sufficiently small amount of parts and connections to fit into our finite faculties. This is the model. A simplified picture of the original, where only the broad strokes, the outlines, survive.

Sure, the process I described may be simple, but simple in the sense that running a marathon is simple: just put one foot in front of the other until you arrive. Simple: yes. Easy: no. To make this easy, or at least easier, we humans came up with a great strategy that begins by finding another thing that is somewhat similar to the thing we actually want to talk about. For example, explaining that good ideas are like seeds that, once planted into a person’s mind, can take root, grow and bear fruits - and become seeds again, that this person will then carry forth and plant into another person’s mind and so on. Most of us, even ardent city dwellers like myself, understand deep down what seeds are, how they become plants and produce more seeds, which in turn easily explains how ideas propagate and conveys what properties an idea must have to spread better.

These analogies are not models, but they give us a starting point, a first glimpse of what will become a model of an idea. More so: Finding a useful analogy forces us to zoom in on the quintessential properties, these aformentioned broad strokes of the thing that are the real subject which we want to discuss. In that sense analogies and models are also alike: they both serve a specific perspective on the matter, they both leave out any details that are not relevant to that and they both aim not for completeness, but for understanding.

The amazing Yuval Noah Hararari – the author of Sapiens, Homo Deus and many more world shaking books (I am a big fan, if you can’t tell) – famously said and keeps repeating that the human superpower is to invent and believe fictional stories. That this gives us the ability to collaborate on a huge scale and to build societies that are magnitudes of size beyond the scope of our innate ability to create stable social relationships. Something no other creature on this planet can claim. I wholeheartedly agree with that sentiment and I think that stories are the infrastructure on which another thing travels, that feeds this superpower of ours: the model, the abstract concept, that permits us to tame seemingly endless complexity, that enables us to share meaning and to teach understanding. I like to imagine that the model, the abstract thought, has pre-dated spoken language, has given rise to it. I think that is what makes us different, what triggered our long success story and that may enable us one day to go beyond our lone planet and seize the stars.

The humble model, the key to human success.

Limited Warranty : As a side-effect of reducing the amount of parts and connections your models will be incomplete and will not show every aspect of the thing that they are encoding. Some situations may require multiple overlapping and potentially partially conflicting models. Use wisely and with care. Beware of mathematicians: they often leave behind fully coherent models that are tangential to reality. Ask your local apothecary for alternative treatments.

Scaling End-to-End Responsibility with Platforms

Ulrich Kautz — Sat, 27 May 2023 23:00:00 +0000

In my previous article I wrote about the benefits of platforms and how platform engineering and DevOps fit together. In that context I thought about end-to-end responsibility. The thing is: it doesn’t work at scale. It just gets way too complicated! Well, that is not the whole truth. You can make it work, even in the largest of structures. Here is how:

What is End-to-End Responsibility?

Ok, let’s pause for a moment. Get a coffee or a tea (or whatever beverage you prefer) and settle in for I am going to tell a story.

The story is about two amazing teams. One team is great at developing software. They take the most intricate and complicated requirement and solve them by creating easy to understand and beautiful to behold applications. What they produce is free of annoying bugs and unexpected behaviors. They pride themselves in testing and benchmarking all functionality rigorously.

The other team is brilliant in operating all kinds of software. They know the underlying operating systems and infrastructure services by heart. They have seen hundreds if not thousands of custom applications and have years and years of experience running about everything you can imagine in production.

Both teams should work splendidly together, no? One writes a marvelous application, the other operates it flawlessly. Everyone is happy ever after. Right? Wrong!

What actually happens, and this is a true story, is this: the synthetic load that was used to test and benchmark the application fails to match actual production characteristics, hence the life performance of the application is abysmal. It also has a host of external network dependencies that make it behave erratic as the external service uptime and responsiveness are highly volatile. The application further suffers from memory overflows that result in out-of-memory kills and the garbage collection leads to random freezes. None of which was visible during development or testing, as they only occur in high load situations after sufficient time.

Worse: The operations team fails to provide actionable details about the issues to the developers. Their extensive metrics that measure their infrastructure add only confusion and noise, but do not help to narrow down on the root causes of the application issue. It takes operations hours to roll out new versions and it’s rarely clear from which release the logs that are provided to the developers are generated from. The staging and production environments differ from each other, so that code changes that work in one fail in another.

In short: it is a huge mess. This is not a happy place.

To put a name to the problem that arises: silofication. Knowledge is constrained to the borders of specific departments. Processes end at those very boundaries and do not interconnect. Basically: communication pathways are broken down. As a result traversing those boundaries (like handing over code from development to operations) suffers from a huge bottleneck at best and raging storm of excrement at worst.

It is 2006. Enter Werner Vogel, freshly minted CTO of an ambitious internet bookseller named Amazon. He utters one phrase that shakes up a whole industry and that still echoes nearly twenty years later: You build it, you run it (YBIYRI). As insignificant as that may sound, YBIYRI shaped modern tech organizations as deeply as the advent of cloud computing. The meaning of YBIYRI is pretty straightforward: as the team that builds a piece of software you are also responsible for operating and maintaining it. That’s it. However, as simple as that idea is, as large are the consequences. The responsibility of running your own application causes a significant shift in perspective: reliability, security, scalability – all of these things have immediately different priorities assigned, because they directly affect you as the developer that created the code. Not only that: as the author of the application, you have deep insight into any possible arising issues as anyone else could have. Communication pathways are not only re-established but also shortened to zero length, because the people that dig into the code and dig into the operating system log files are the very same!

There is just one little problem: you have a team of excellent software developers that have little experience with operating anything in production. What do you do? Easily: merge the operations team and the software developer team. Have one team that has all the knowledge required to both develop and run the software. Have them both share knowledge and grow together into a new team that operates under a new paradigm. And with a last Puff the earlier mentioned silos are gone. YBYRI. You may also have an inkling what those teams that were staffed by developers and operational engineers were called later on – but this is another story. For now let’s just call such teams owners, as in: teams that own products and that are responsible for them. These owners are very interested in making the deployment of their software very easy. They want to have sensible insight into their application behavior at runtime. They want to be able to roll back changes fast and with ease. The good news is, they can do all of that by being staffed with engineers that focus on operations. Sure, in reality it took a bit more than just merging teams. There is a whole organizational mindset that had to be shifted and I believe the model truly gained significant adoption only after programmable infrastructures became more widespread.

So what is end-to-end responsibility then? I think you could call it YBYRI+ or something like that. While YBYRI proposes development and running of the software is done by the same people under end-to-end responsibility teams go farther in “both” directions: what happens before development and what happens after the software runs in production. On the development side in addition to building the software end-to-end responsibility also requires the team to gather all requirements and then design / architect both the software and the infrastructure. On the other “end” owners also take care of support and gather user feedback. So YBIYRI is a subset of end-to-end responsibility. However, your mileage may vary and I have seen YBIYRI setups that were indistinguishable from what I described here. I have no doubt there are end-to-end responsibility implementations that are more limited (and implementations that are more extensive) than what I describe.

In a now very old podcast with Martin Fowler and Jez Humble I heard a sentence that stuck with me: the job of developers is to take business ideas and turn them into measurable business outcomes. To me, a team that starts with such a business idea and makes it happen truly implements end-to-end responsibility.

Why does End-to-End Responsibility fail at Scale?

When you apply the above strategy of creating owner teams that are skilled in developing, running and maintaining the software, then everything will work splendidly – at the start. However, if you are growing, you will eventually run into a wall. That wall manifests itself by massively slowed down development, reliability problems in production, very short retention of employees – and a whole host of other problems that are all bad. Some of these problems may even look very familiar to you, like those you had before you switched to the end-to-end responsibility model.

Why is that?

First I need to clarify one thing: what does at scale mean? I think it is intuitively clear that at scale refers to large systems as opposed to small systems. However, what makes one system large compared to another? Amount of people working on it? Kilowatt hours consumed? Compute resources available? Sure, all of that, to some extent. However, what it comes down to is ultimately one thing: a large system is more complex than a small system. At scale it is complexity that becomes a problem for end-to-end responsibility.

Think of complexity in two dimensions:

Depth, as in: a single application becomes more complicated through adding functionality. More functionality means more components and more interconnections between components. More knots and bolts = more complex.
Breadth, as in: a large quantity of simple, but different applications form a complicated tech stack. Adding more applications that have different architectures, that use different components or use components differently all adds more complexity to whole. Relationships in between applications add further complexity proportional to the amount.

You can also think of Depth as the Team-Level perspective: the complexity each member of the team has to engage with when working on their application / service. Likewise Breadth can be thought of as the Organization-Level perspective: the complexity that results from the amount of different tech stack of the different applications / services that the whole organization owns.

With that out of the way, what does end-to-end responsibility at scale mean then? It refers to the challenge your owner teams are facing when they have to handle either a large amount of applications or very complicated applications – or usually both. This challenge relates to the complexity they have to deal with. The total complexity that owners are grappling with results from the earlier mentioned Depth and the Breadths. Think of it like an “area of complexity”, that is the product of multiplying Depths and Breadth. As long as that area is big, it doesn’t matter whether it comes from a wide Breadths or a deep Depths (or both). With scale complexity grows. With rising complexity the challenge grows.

So what exactly is the problem with high complexity for end-to-end responsibility at scale? What it comes down to is cognitive load. Complexity is a measure that describes how many parts and how many connections between those parts exist in a system. More parts with more connections means more complex. When you think of a system that you see in the real world (or the virtual world) then your mind creates a mental model of that system in your head. This mental model allows you then to reason about the system: “here we can add this feature we talked about” or “this is likely the cause of the issue we are seeing”.

Let’s make this more practical. Imagine you are the current engineer on-call of a team that owns an application. Alarms go off, some metrics turn red, you have to find out what went wrong and then fix it. Let’s say your application uses five different infrastructure components. Each of them could have failed and caused the issue. With such a small amount and you easily understand how they interact and relate to each other. It takes you only a few minutes and a couple of log lines to narrow down the issue, identify the faulty component and perform a fix. Nicely done! Now, how do you reckon the story would have played out if there would have been fifty different infrastructure components involved? What about five hundred? What about magnitudes more? I think you can see the problem. The more moving parts are involved, the harder it gets to make sense of them all. We humans are simply limited in that regard. An end-to-end responsible team must be able to model their whole product, including all related processes and components that they own. In summary that means: end-to-end responsible teams are staffed with humans that can get overwhelmed by exceeding complexity, at which point any thought of responsibility goes right out of the window.

Look at it also from the organizational perspective: Imagine you have two teams that all are end-to-end responsible for their applications. They all take this very seriously and have designed ideally fitting tech stacks for their individual needs. As both team’s applications are different and both teams are solely responsible for the tech stack architecture, both tech stacks are different as well. Over time your organization grows. You now have fifty teams, with fifty different tech stacks. Oh wait no! That was last year. Now it’s a thousand teams and a thousand tech stacks. How easy is it for moving applications in between teams, given that they were designed and thought of individually? How easy is it to move people in between teams, given that they don’t know much about each other’s setup? And especially: Who can even begin to make sense of the combined tech stack of all your applications? How can you track security issues across that mess? How can you possibly start to think about efficient resource usage? There are no happy answers to these questions. If the complexity of the whole organization is not fathomable anymore, even at a high level, then it becomes impossible to direct it. Without direction your organization is little more than a headless chicken that runs in circles until it dies. That means: too many end-to-end responsible teams that are not aligned add more trouble than they add value.

All of the above are the problems that result from end-to-end responsibility adhering teams at scale. Well, the problems arise if your approach is naive and if you apply the model almost blindly on a team level. Of course: you should not.

How does End-to-End Responsibility work at Scale?

If exploding complexity is at the heart of the problem of scaling end-to-end responsibility then surely the solution is to reduce it? Right! Easily said. How do you do that exactly?

Consider again the team perspective. Above I argued that complexity results from the amount of moving parts that individual products (applications) are made of. More parts make equals more complex. With that the solution seems trivial: less parts make it less complex? Alright, I buy that for the moment. Let’s say your goal is simplification.

We also need to look again at the organizational standpoint. Complexity here emerges from the sheer amount of different architectures. Following again the obvious pathway: reduce the amount of architectures. Your goal is standardization.

So what you are looking for is something that helps you to standardize your tech stack throughout the organization while simplifying the components that owners engage with. A tall order, for sure, but don’t fret there is an answer. You may have guessed it by now: platform engineering.

Right, that may need some more elaboration. There is two parts to that:

How does platform engineering standardize and simplify your tech stack?
How can end-to-end responsible teams work upon platforms?

The first part is easy to explain. Let’s start with the obvious: Platform engineers build platforms. Now, what are platforms? Platforms are systems that reduce the complexity of underlying systems. I wrote a whole article about that, please find it here. In summary: Using many (infrastructure) components directly means that you need to engage with the interface of each individual component. Each component adds some amount of complexity. The sum of the complexity of all the components is then the complexity that is surfaced to you, that you have to understand and handle in order to manage the components. A platform hides away the individual components and their complexity. It instead exposes a much smaller interface to you. Meaning you need to understand way less in order to manage the same resources. This is only possible at by giving up fine-grained control over those resources. This is a good thing, because it frees your resources for doing things that generate value instead. Another consequence is of course that with less choice comes less variety. Standardization. In essence, platforms refine the vast amount of complicated components into a small amount of simple components. By that, platforms are the foundation that both simplifies and standardizes your tech stack.

Now to the second part: how end-to-end responsible teams can work with platforms. Let’s have a look at a practical example in which your owner teams work directly with cloud infrastructure resources. Let’s say they are using AWS and some CDK flavor to manage their infrastructure as code. Providers like AWS have a huge range of products by now. Many of those build upon other products they have. For example, you could buy compute resources in the form of virtual servers and then install and maintain a container orchestrator like Kubernetes on it. Or you could directly buy a managed Kubernetes from AWS. Or you could buy a managed orchestration that hides away how it is implemented. That means: if you work directly with a cloud provider like AWS, then you are already building upon a platform. Actually upon multiple platforms that just pretend to be a single one (don’t get me started). The only difference between an AWS provided platform (building cloud infrastructure) and an in-house platform (building on cloud infrastructure) is that providers like AWS have to generalize their solutions to fit a large market, whereas in-house has the opportunity to optimize only for local use-cases. Hence the question of whether end-to-end responsibility adhering teams can work with platforms is entirely moot: they already do! Their responsibility is with the infrastructure components they own. The granularity of those components can differ (EC2 server vs high level orchestration abstraction), but that does not change anything about the end-to-end responsibility model. The more interesting question is whether an optimized and limited platform is better than a generalized and vast one. The answer is a resounding YES! Why? Using your own platform means that the components are optimized for your use-case. That means they are simpler. That means they much better decrease complexity than generalized components from a public provider.

However, building and running a platform is not for free. Hence one last question needs to be answered:

When does End-to-End responsibility require a Platform to work?

There are a couple of clear signals that tell you that you really, really need to reduce the complexity people are dealing with in order for them to take on the responsibility for their product. I outlined them with the wall that I described earlier (slowed down development, reliability problems, low employee retention, etc). However, if you are at that point, then you are already failing. If you don’t act, things will get worse – until they stop entirely of course. That makes it almost an easy decision. The more interesting question is how you can predict when you will need a platform way before that happens. If you are able to do that then you can formulate a sound strategy and follow it so you will never even start hemorrhaging employees, face customer backlash due to unreliable services or have to explain to investors why development stalled to a halt.

As I argued throughout this article: the decision of when a platform is required relates strongly to the complexity that individuals within end-to-end responsible teams are facing. The hard part is to get a clear measure of this complexity, so that you can gauge the current state or even begin to make predictions about the future development. To my knowledge there is no one magic metric that will give you a definite, actionable answer. However, from my observations of the years, I can give you at least a couple of proxy metrics that serve as early indicators. Which of them are best applicable depends strongly on the context of your particular situation. In general I would never rely on just one of them.

As end-to-end responsible teams own their infrastructure components, an increased complexity translates into more time that needs to be invested to manage them. Setting up monitoring for more components with more intricate relations can quickly become exponentially harder. Rolling-out changes into systems with more moving parts can add magnitudes of stress and failure potential. As a result, Deployment Frequency will decrease with increased infrastructure complexity , because each individual deployment is harder and more time consuming. And also because individuals will increasingly eschew to go through the hassle and the perils of roll-out.

Not only that, but Lead Time for features to go from ideation to production will increase with increased infrastructure complexity , for the very same reasons of more time consuming deployments and more time invested into managing the infrastructure.

When it takes longer for any change to hit production, then it will take you also longer to roll-out fixes in incident scenarios. That means the Mean Time to Recovery (MTTR) will increase with increased infrastructure complexity. This one is a doozy, because, unless you are really seeing significant increases in a short time, it will sneak up on you. One or two required recoveries within a few months’ time rarely ring the alarm bells loud enough.

Recovery is, of course, following incidents that have already happened. Before it gets to that point you should see alarms that do not result in incidents either because they are caught early enough to counter them, or because they subside on their own. Hence the metrics that you should watch like a hawk is the Amount of Oncall Alarms that will increase with increased infrastructure complexity , because more complicated systems are easier to break, easier to misunderstand and harder to fix even if understood.

Underlying this alarm increase is, more often than not, human error, caused by cognitive overload that stems from the inability to model the whole complex system in your mind. Without that big picture it is rather easy to make decisions that do not account for the interplay of all components. A proxy for that is the relation between incidents and roll-out of changes, which is captured in the Change Failure Rate that will increase with increased infrastructure complexity.

If you are able to track with some precision where the work of your teams is spent, then you will of course see that Time Spend On Infrastructure Management increases with increased infrastructure complexity. While some seasonal, temporary cost increase is fine and even continuous expenditure to keep technical debt at bay is expected, at a certain point it becomes counter to the idea of end-to-end responsible teams. After all, they were founded to speed up the time from ideation to production and increase productivity, not keep more people busy with infrastructure concerns.

Last, but certainly not least, is the obvious: ask the very people that are working in the teams. You need, of course, a comparable metric that you track over a long enough time. Adding questions about time spent in infrastructure in your regular pulse surveys is a good start. Generally Employee Satisfaction and Engagement will decrease with increased infrastructure complexity , assuming that the majority of end-to-end responsible teams are focused on delivering business value and get frustrated if they are instead bogged down in battling with infrastructure.

Again, you need more than a single metric measured in a single team to get a reliable signal. However, in conjunction with each other – of at least a subset of the metrics – you will certainly see a clear trend emerging way before you are suffering serious consequences. Keep your eyes open and be ready to simplify and standardize the complexity that development is facing. Platforms are an incredibly helpful tool here, if they are widely and wisely deployed.

Conclusion

The end-to-end responsibility model works at any scale – given that you provide the right environment for it to flourish. That means you have to perpetually rein in the complexity that members of end-to-end responsible teams have to engage with. You also need to make sure that no chaos ensues from a too large variety of solutions that result from too many teams creating optimal solutions for only their own domain. Platforms are a great way to address both concerns.

Platforms are also not the end of the story. Well, at least a single platform is not. As you keep growing, so does the complexity of your infrastructure. Over time you will see higher level patterns emerge: platforms upon platforms upon platforms. It’s a greater journey ahead!

As for my next article: I am pleasantly surprised to find myself deeply immersed and thoroughly enjoying an adjacent topic to platform engineering: developer experience. My interest, coming from the platform engineering side, has narrowed down over the years and I became more and more involved in the very intersection of development and platforming. When the rubber hits the road, so to speak. I am now actively exploring topics such as knowledge sharing and general developer tooling, which are other equally interesting problems that you face when engaging in development at scale. Stay tuned, more to come!

Platform Engineering: Development at Scale

Ulrich Kautz — Sat, 13 May 2023 23:00:00 +0000

The topic of Platform Engineering has become all the rage in the last year. Good! It is about time. Software development is at the heart of an ever growing number of organizations and platforms are a necessary response to anchor best practices and standardize resource use – which are increasingly hard problems proportional to the complexity that comes with organization and software growth.

Most of the discussions around the topic over the last year and a half have passed by me. I do have a good excuse, for I had and have the privilege to be part of one of the largest infrastructures of the planet that received all my attention instead. Needless to say that I learned a tremendous amount about platforming at scale. This of course changed my perspective since I last wrote about this topic in March ’21.

I recently picked up the ball again and started reading a lot of newly published articles about what people think that platforms are and what platform engineering does. I found that some are arguing that platform engineering is an evolution of DevOps. Even that platform engineering is the practice while DevOps is the philosophy. That made me a bit twitchy, because building platforms has been at the heart of what I was doing for the last two decades or so – the term DevOps has certainly not been around for that long.

On the other hand, I certainly support pitching the platform model to organizations that already bought into the DevOps paradigm. I can also see where those arguments are coming from and how DevOps and platform engineering intersect. However, I think conflating both terms in such a way limits your thinking unnecessarily. To me platform engineering is foremost a practice to enable software development at scale. By that, platforms are a tool. How and why you wield that tool is a question that can have many answers. One answer could be that it fits into your DevOps organization, but that is not the only one – you can gain value out of platform engineering in any case.

What are Platforms?

I want you to think about the platform concept without any preconceived notions about DevOps or anything else for a moment. Consider the following entirely non-technical example from even before the interwebs was around: a travel agency of the olden days. Back when phones had cables and dials – and no displays. You know, shortly after the dinosaurs roamed on earth. It was pretty hard to go on vacation anywhere outside of your own backyard. The same as today you needed to book a flight, find and book a place to stay at the destination, ensure access to local transport and plan activities outside of loafing by the pool. Now imagine doing that pre-internet in a country whose language you do not speak. Imagine even just calling all the airlines directly and asking them for flight times and prices. Yes, a nightmare. Travel agencies are an excellent, if mostly obsolete, example for a user-facing platform. The complexity of booking all the components and thinking of all the things, and then orchestrating everything so that it fits perfectly together, was reduced to a simple conversation with an agent and maybe looking through a couple of catalogs. This is an example of a platform. Not a platform created by platform engineers, but a platform nonetheless. The example shows the first fundamental property of a platform: a platform reduces the complexity of underlying systems. Basically: It makes complicated things simple to use.

You probably noted that I threw in the term systems. I will also soon use the related term service. Let me quickly explain what these two terms mean in this context: A service is “a single, self-contained unit of functionality”. A service implements the functionality itself. In the above example flying is a service that airlines provide. A system is a collection of components that could be platforms, services or systems themselves. The travel agency above would be a system and a platform. Each airline would be a system as well.

With that out of the way consider another fundamental platform aspect that the above example describes: the travel agency cherry picks functionality (services / systems) from the airlines, the hotels, the local transport and so forth, and offers this choice-set to their own client. More so, the agency re-packages that functionality in a way that their client may not even be aware who provides it while combining it (ideally) seamlessly with other functionality from other providers. This is done with the goal to make booking a vacation an easily accessible experience. In a sentence, a platform is a higher level system that refines one or more lower level systems. You can think of a hierarchical structure that has higher and lower levels. The platform is at the top and the other systems are below it. When interacting with the platform you only see the top layer (i.e. the travel agency) and the lower layers are hidden.

By cherry picking functionality another fundamental aspect of platforms reveals itself: a platform serves a limited set of specialized use-cases. It has to, because if you could do the same things with a platform as with all the systems beneath it, then the platform would be exactly as complicated as the combined underlying systems and thereby entirely redundant.

In the context of the above example: A travel agency would not be the place to, say, book cargo flights to transport goods from A to B – even though the airlines that the travel agency uses also offer cargo flights.

Lastly – and this is not always fully possible, but always highly recommended: A platform abstracts the implementation behind an isolated, well-defined interface. That means that changes of the underlying systems have (ideally) no impact on the exposed interface of the platform itself. Again, using the example of the travel agency, this would mean that if from tomorrow on you would not anymore travel by airplane, but by zeppelin or dragon or something, then the way you book your travel would still be the same: you talk to an agent, tell them where you want to go at what dates and they take it from there. There are some asterisks to that, because it is not always possible to fully isolate the platform interface from underlying systems (i.e. zeppelins and dragons don’t offer 1st class anymore, so you cannot book that anymore). More on that in the benefits topic below.

As I wrote earlier, platforms can exist entirely outside of the DevOs paradigm. At least the above properties do not require it.

What is Platform Engineering?

While the platform model transcends industries, the term platform engineering is – as far as I know – exclusively used in the wild world of software engineering. So the above example of a pre-internet platform is valid, but it is not descriptive of what platform engineers do. To understand what that is, you have to further narrow down the realm of software platforms: any user-facing platform – from social media to online travel agencies – is usually out of scope. From what I have observed while working on platforms in various sized companies – including an early Platform-as-a-Service (PaaS) – is that platform engineers almost exclusively create developer-facing platforms. To put it in a single sentence: platform engineers design, build and maintain platforms that facilitate development and delivery of software.

This by no means implies that there is a single type of platform that every platform engineer builds. You’ll find mobile app development platforms, web application development platforms, machine learning data platforms, and much more. Platforms come in all shapes and sizes and each is designed to make a different use-case for the specific context of an organization easier to tackle. In larger structures you will see tiers of platforms forming: higher level platforms that build on lower level platforms. In this context developer-facing can equally refer to the developer of the end-product as well as any other (refinement) step in the value chain – like developers of a higher level platform. Platforms are also almost never static: they change and evolve along with the changing and evolving needs of the business.

So what does that mean in practical terms? As a platform engineer you need to take on two perspectives. First, of course, the perspective of the developer. To do that you need to understand how the developers in your organization work. There may be even multiple contexts, like the day-to-day of the mobile app developer, who faces rather different problems than the web application developer. It is up to you to find out whether there are common denominators. In general you need to have a lot of conversations and ask a lot of questions – and listen! What tools do they use? What languages do they write in? Do they have a local development environment? How do they currently deploy? How long does that take? How are automated tests triggered? What business goals shape all of that? And so forth. Gathering answers to all these questions gives you the internal context of what is being done and why. A good platform engineer understands the needs, wants and pains of the developers. This is the first half of the equation.

The other half is being informed about what is going on in the field. Be assured that you are not the first person facing, well, almost any problem. Make use of the wisdom of the domain. Learn how others have tackled the same issue – and have proven their approach right by thousands and thousands of hours of hardening in production. Read about strategies and patterns that aid you in framing the problems and delivering working solutions. This is the external context. A good platform engineer must have knowledge of current industry best practices. Combining both contexts together is what makes the job interesting.

A warning here: Platforms are a great tool to introduce and propagate standards. However, if this is misused to steamroll and force-feed even the greatest practices without involving the very developers that are affected, then chances are adoption will wither and the rollout will fail. Been there, done that. Big mistake. Tact, common sense and pragmatism are more useful values for platform engineers than adherence to the book, idealism or zeal. The most successful platform rollouts I’ve seen were not the technically most sophisticated or the most optimized solutions, but those that had the best feedback loops to check in with the users (the developers) and make their transition the easiest. Platform engineers must advocate for the interests of developers. Without their support and buy-in, all effort is for nil. Besides, it is so much easier to iterate the platform later towards better practices, than attempting to start out in the perfect setup while fighting with the developer’s expectations.

The other perspective that platform engineers need to take on is the infrastructure side. As with any production IT system efficiency of resource use and resilience towards outages are key concerns. Since platforms are multi-tenant systems by design, isolation of tenants – for both security and accounting reasons – needs to have strong guarantees. Scalability and monitoring go hand-in-hand and are essential for any modern platform. All of this needs to be optimized, which often leads to hard coupling of the implementation with the underlying lower (infrastructure) systems. And here is where it becomes hard: you have to balance optimization with the flexibility to exchange whole underlying systems. As a platform engineer you usually work upon cloud infrastructure – whether directly or indirectly. The cloud is constantly improving and changing as well, so you need to be able to adopt new technologies and trends quickly. After all, what does it help if you squeeze out 5% more performance at the same cost, whereas migrating to the next generation gives you 50% more performance for the same price? As a platform engineer you need to be informed of the latest developments in the underlying infrastructure , so that you can leverage opportunities when they come along. Then again, this balance shifts towards optimization when your infrastructure costs make up most of your total expenses. This is certainly the case for very large scale operations, where even shaving of fractions of a percent in resource consumption can translate to millions in annual opex.

By the way: While the role of platform engineering is a new one, the same activity and context was around way before that: I am thinking of web hosting. The value of web hosting was to provide the ability to serve a website that was built by web developers, with very little effort. At least much less effort than is required to maintain your own hosting infrastructure. The people that built those web hosting platforms were not called platform engineers, but they pioneered the field (or maybe I am just nostalgic).

What is the Value of Platforms?

Again, I ask you to suspend your any assumptions about DevOps and platform engineering and just think about what value platforms independently provide.

I’ve already tried to convey what I consider to be the primary value: Platforms simplify access to complex functionality. However, that is far from all. Platforms rarely only simplify a single system, they more often merge and simplify multiple systems into a new system: the platform. That often involves additional layers of authentication that control access to the involved resources. By that platforms combine related systems into a new system with a single interface. This combined interface is simpler than the sum of the individual interfaces and reduces the complexity on the consumer side. In specific cases – especially relating to authentication and authorization – another value arises: platforms enable compliance , which may otherwise not be possible or very hard.

Another huge value is that use of the underlying complex systems is aligned. Consider cloud infrastructures like AWS or GCP: You have a multitude of different solution pathways, a huge variety of possible architectures and designs. You can run your code in lambda functions, container runtimes, virtual machines or on bare metal – just to consider a single use-case. A platform standardizes use of resources. This strongly reduces complexity for those that maintain the lower levels and often allows you to decrease cost by buying in bulk. It also helps to keep technical debt at bay by reducing the surface (=variety) for entropy to wreak havoc.

Related to that is that platforms provide a consistent, well-defined interface. The technological landscape is continuously changing and improving. Having the ability to replace basically all the systems that underlie the platform, while keeping the surface interface the same is invaluable. Why? Because all of the things that use the platform do not need to change! So think thrice before committing the cardinal sin of not separating concerns and exposing implementation details. You will pay for it down the line – promise.

If you forget anything else and just remember one thing, then make it this: Platforms allow developers to focus on developing , by reducing the amount of things they need to know to do that.

What are the risks of platforms?

When talking about platforms, it is easy to dote on the benefits and forget about the drawbacks, because there are some and it is important to recognize that not every situation calls for a platform. For example, when a software is in the early stages and is still heavily changing at high frequency, then defining a platform that serves the fast evolving needs of developing the software is not feasible. Similarly, small organizations with only a few developers are better off using pre-existing platforms than investing into building their own. Also if the software stack is too simple or too diverse, a platform does not add any value.

If the circumstances are right in principle, there is still a fundamental risk in creating a platform: over-optimization / over-fitting. Any platform naturally imposes constraints by being fitted for a set of specific use-cases. This takes away choices which allows developers to focus on the primary task of development (instead of peripheral tasks like deploying). The risk here is that you optimized the platform in the wrong direction. As a result you may have made the wrong things easier and the wrong things harder in the process. Consider Google Glasses that came out in 2013: the device was purely optimized for the tech side of an augmented reality experience, without considering privacy concerns and social tolerance, resulting in a major failure. By the same token a development / release platform that optimizes heavily for release, while making check-in code changes harder is a certain recipe for disaster. To avoid this pitfall, it’s important to integrate feedback loops and roll out iterations gradually rather than going for a big-bang approach.

Another risk of over-optimization is locking yourself too deeply into a solution that stalls innovation. Yesterday’s great platform can quickly become today’s obstacle. Again, to avoid this the design of your platform should consider changes to the developer-facing interface from the start. Providing a versioned interface makes it possible to describe migration from one version to another. Detaching the interface design itself from the implementation makes it possible to replace the whole implementation.

Platform Engineering ❤️ DevOps?

All of the above is about platforms and platform engineering. Now let’s have a look at how all of that intersects with DevOps.

First, given that DevOps is perpetually lacking a commonly agreed upon definition, let me clarify that I mean the practices that arise from the three principles that are outlined in the DevOps Handbook by Jez Humble et al:

Optimized Flow , throughout all processes by identifying the path of flow, where the bottlenecks are and where work-in-progress accumulates.
Feedback Loops , that act automatically on the signals of pervasive telemetry that is deployed throughout the system.
Continual Learning , that seeks to perpetually improve the processes and thereby the organization.

I certainly don’t want to get into a discussion of whether this is the best definition of DevOps or not. I acknowledge that there are other often cited principals. Thus far, I haven’t encountered a sensible principle that I could not derive from the ones mentioned above. Take, for instance, end-to-end responsibility, which, in my view, exemplifies a practice that ingrates all three principles at team-level. It requires (and at the same time fosters) a DevOps mindset of the whole organization where such teams are embedded in.

Anyhow, if you follow the first principle and identify how work flows through your organization across product- and process boundaries, then it is only opportune to draw circles around common and especially effortful areas and put them behind platforms. The same goes for ubiquitous work that repeats in many or all steps (again, authentication / authorization is a great example). All of that would fit neatly behind the transparent interface of a platform. Maybe your setup is even simple enough and you can draw a single circle around it and have one super platform that reduces the efforts of the whole process.

The second principle of feedback is equally supported through platforms, because for scalability and reliability reasons, you will have pervasive telemetry already deeply integrated into your platform. Exposing those metrics to each platform user as a signal for their feedback loops (of their specific products) is likely little effort.

Continual Learning may seem a bit harder to argue at first glance, because platforms hide complexity and thereby reduce the opportunities for learning for individuals. However, DevOps is about organizations and Platforms certainly supplement organizational learning: they provide a high level signal for a whole category of processes (i.e. all products that use it in their processes) and they provide a focal point to improve that whole category as well. To an extent platforms are also an implementation of organizational knowledge.

So, yes! Platform engineering fits nicely into DevOps adhering organization – even if it developed independently.

Conclusion

I hope I could make clear that platform engineering is not tightly coupled with DevOps – even if it is a great partner to it. Again, I fully endorse the pitch – and I also want to prevent an unnecessary boxing-in of thinking. To me the platform model has a huge range of applications. I can only see this range growing further in the future – especially since increased tech complexity in all kinds of organizations has become an unstoppable trend.

I am writing another article about the end-to-end responsibility practice I mentioned above. My interest here is how platforms enable that practice even at the largest scale. More in my next article.

Interruptions: The Silent, High-Performance Killer

Ulrich Kautz — Mon, 20 Mar 2023 00:00:00 +0000

Everybody these days wants high performers. Everybody wants to be a high performer. Everybody wants to hire high performers. High-performance is commonly defined as: better, faster and more efficient than others. What this definition doesn’t tell you, of course, is how to do that. I can’t tell you that either, but I can tell you that high quality outcomes and high productivity share a common arch enemy: interruptions. As Aristotle put it so eloquently more than twenty-four centuries ago: The greatest threat to the state is not faction, but distraction.

In my last article, I shared my introduction into the field of interruption research. This time I want to share some fundamental “why” questions that have been bothering me ever since. I like to understand cause and effect. Given that our understanding of all things relating to the human brain are limited by our current knowledge and insight, there is a lot of room for interpretation. So I will interpret.

Let’s start with the most fundamental “why”-question in that context:

Why Are Interruptions Disruptive?

Imagine yourself in front of a whiteboard. You are thinking about an intricate problem. Maybe a complicated engineering task. Or your upcoming wedding. Or your next tax return. Over the last hour you painstakingly added bits and pieces of information to the board with arrows of relations drawn in between them. A bigger picture finally starts to emerge. You know the solution will present itself anytime now. Then someone in a hat steps up next to you. He smiles politely – and erases half your board in the blink of an eye. Before you can even voice your outrage he quickly fills the empty half with entirely different information that hardly makes any sense to you. While he does that, he even smears the remaining picture that you have so scrupulously drawn over the last hour. Finally, to add insult to injury, he tips his hat and vanishes.

Rude! How could he? The whole hour is lost! Why would he do that? Well, he would not. Normally. Social norms would likely prevent anyone, save for borderline sociopaths, from such shenanigans. Hat or no. By the way: the whiteboard metaphor is not from me. I heard it first from the brilliant Professor Gloria Mark and it stuck with me. Anyway, what happens when you remove the clearly visible whiteboard from the storyline?

Consider a scenario in which you are equally concentrated on solving whatever you are focused on. Only instead of offloading your contemplations externally, on a whiteboard, you are keeping everything in your head. This is not entirely unheard of. You are doing this all the time. Everybody is. This inner, figurative whiteboard is called a Mental Model. When I first read about this, as a software engineer I immediately thought of program state. Whatever the name, it is a representation that your brain creates of the outer world, specifically of the thing that you are focused on. This representation is constantly updated by all the new information and relations you think of and learn about.

Now, what happens then when you are interrupted? Altman & Trafton 2002 described the Memory for Goals Model that is widely cited and referred to when pondering this particular question. The short answer is: pretty much the same will occur in your head as did to the whiteboard when the guy in the hat came along – vandalism. Whether your mental model is just maimed or wholly destroyed depends on many of the factors described in my previous article. However, two factors peek out: similarity and complexity (Cades, Werner, Trafton, Monk 2008). The closer the new, interrupting thing is to what you were already doing, the better it will fit into the existing Mental Model in your head. Ideally, it is not different at all. That means you can keep using the same model – give or take a few pieces. The further away the new thing is from the old, the more different the Mental Model that presents itself. If the two things are entirely different, then the two models will have no overlap at all.

As an example, let me tell you a short story that you may have experienced as well: For the last twenty minutes you have investigated your fridge and your pantry and a lip-smacking meal is starting to form in your mind from what you found. While you are not hangry yet, you are certainly on the verge of really hungry and the prospect of the nice, home-cooked meal makes your mouth water. Then your partner comes in, bright eyed and bushy tailed about a new idea! In the beginning, you listen with half a mind, while still thinking about what goes into the dinner you were about to start making. As your partner’s telling goes on, and is indeed very interesting, your thoughts move ever away from culinary delights and you get drawn into the story. An undetermined time later, you find yourself sitting at the kitchen table with a growling stomach - and order in Szechuan food. While that is delicious, that was not your plan. Your own Mental Model of the dinner has evaporated without you being consciously aware of that. It was replaced by the tale your partner spun – or more precisely: it was replaced by the entirely different, non-overlapping Mental Model that this tale induced in your mind. Also, you can figure yourself what my usual role is in that recurring epic.

From that anecdote I want you to take one thing: Complexity of the interrupting thing matters! Complexity refers here to how many details and relations it contains. The more complex the thing is, the more space it will occupy in your head. The whiteboard metaphor fits yet again: more details and relations drawn require more space. At some point, the whiteboard would simply fill up and the only way to add more would be by first erasing something else. And this is pretty much what happens with the Mental Model as well: If the new, interrupting thing requires more space than there is available then it will push out the old. Your old model collapses – partially or fully. The amount of available space, which is referred to as Working Memory Capacity , varies from person to person (Forgoughi, Malihi, Boehm-Davis 2016) and there is good indication that you can train to grow it (Dingfelder 2005). However, it always will be finite. It also always will be smaller than truly complex problems, hence you will still need to simplify them, so they fit in your head, so you can think about them.

But wait, it gets worse: Even if your old Mental Model survives the interruption mostly intact, it will still suffer from what the scientific literature calls Memory Decay. It is as bad as you imagine: the things in your memory rot away over time – unless you keep working with them to keep them fresh in your mind. Here the whiteboard metaphor fails us. The best I can offer up is the image of a news or social media feed: new things are continuously added to the top, ever pushing down the old contents towards the bottom until they leave your field of vision and are gone into the digital aether – unless they are being upvoted by steady interest and stick around. The same happens to the old Mental Model, that you are temporarily not using when working on the new thing. The above mentioned Memory for Goals Model predicts exactly that and many studies found strong evidence that this holds true, Monk, Trafton 2008 makes an especially convincing case.

Once more back to the question: Why are interruptions disrupting? The answer is: because interruptions screw with any existing Mental Model by either overtaking its space or moving away from it temporarily, which leads to its decay.

Unlike in the story with the hat guy messing with your whiteboard, the damage done is not obvious to the outside – nor is it always to the inside – but it is there nonetheless.

Now the question becomes: what are the consequences of that damage? Well, there are a few…

Why Do Interruptions Hurt Productivity?

Let’s go back to the drawing board. Excuse me, the metaphorical whiteboard, I mean. Consider for a moment the effort it takes to create that big picture. When you do it the first time the process of writing and drawing on the board costs time, but compared to the big effort of gathering all that information and relations together the drawing effort is small. When the guy in the hat brazenly erased the board and left, and you later had to redraw half of it from scratch, you would likely have been faster than the first time around. You would recall some of the things that were on the board and you would remember the source of where you found others. This means retracing your steps would be quicker than initially finding the pathways. Either way, it still would take you time and effort.

The exact same goes for a mental picture. In research papers you find this most often as Resumption Lag , rarer as Reimmersion Time (Tregubov, Boehm, Rodchenko 2017) or earlier Reorientation Time (Burmistrov, Leonova 2001). It is the time measured after the interruption has finished and before the interrupted task is resumed again. Meaning: it is the time it takes to rebuild the Mental Model.

You have certainly experienced Resumption Lag many times over. Think back to a time when you were intensely focused on a complicated task and the world around you faded into a distant murmur. Then, suddenly, something comes cutting into your thought process like a knife, ripping you out of your concentration with the same abruptness as a cold bucket of ice water poured on your head would have. Being so deeply immersed into the topic means that you did build an elaborate and, in all likelihood, fragile model in your mind. That model is now shattered into pieces. That certainly hurts in the moment and it will hurt again when whatever disrupted you has been dealt with and you have to pick up all those pieces and rebuild the picture in your mind to continue. This costs time. This is error prone (I’ll get back to that in a bit). It is certainly not efficient.

To make it painfully clear, here is some math: consider a 60 minute task. It is interrupted three times. Each time it takes 5 minutes to think back into it. That means the total task time is: 60 + (3 * 5) = 75 minutes. In productivity terms: instead of one task per hour this is only 60 / 75 = 0.8 tasks per hour. A 20% reduction in outcome performance.

With that it becomes pretty obvious how interruptions hurt productivity : less is getting done in the same time.

How bad is it? Magrabi, Li, Day, Coiera 2010 showed it depends on the complexity of the interrupted task. Interruptions resulting from switching in between tasks from more than one project can eat up 20% of your time , according to Tregubov, Boehm, Rodchenko 2017. That is a full day out of a five day workweek! Iqbal, Horvitz 2007 found in their study at Microsoft that an interruption by half an hour of conversation on average leads to a subsequent resumption time of another half hour to resume the task (compared to a five minute conversation that results in an average of fifteen minutes of resumption time).

Why Do Interruptions Increase Error Rate?

I’ve already hinted towards the first reason: the Resumption Lag is caused by you having to rebuild the Mental Model from an either decayed or otherwise partially destructed state. As it turns out: nobody is perfect. That means you will make mistakes in the process of rebuilding the model. Once a mistake is made, it will have ripple effects that of course will impact the outcome: it is faulty, flawed, buggy, corrupted. Take your pick. Altman, Trafton 2008 showed an up to three times higher error rate resulting from short (five second) interruptions.

There is more.

Using again the handy whiteboard metaphor as it can explain another observed mechanism that research has described: If you have ever cleaned up a physical whiteboard after using it you certainly have found out that erasing your drawings does not work perfectly. It’s a bit of a pain, to be frank. Working with others in shared office spaces I have often come into a room where someone halfheartedly cleaned the board, leaving behind ghosts and fragments of their musings. I may have been guilty of that as well. Anyway, those remnants of previous artworks can make it hard for you or someone else to separate them from the added layers of your masterwork – and then misunderstandings become mistakes.

Those leftover bits of information are not only an annoyance when you work with physical whiteboards, but they also occur in your Mental Model. The same principle applies: residual fragments from a previous task carry over to the current task and mess with the current Mental Model. Hence the scientific term: Attention Residue , the leaking of information from one Mental Model into another.

When talking about any outcome of work, you must consider both tasks: The one that was interrupted and the interrupting task. The Attention Residue affects the latter one: the interrupting task, that barges in. Again, you certainly have experienced that as well. Remember the last time you were working on something interesting, something that moved you ahead in your goals, which was interrupted by a time-sensitive chore that had to be done now. This chore may come from the hat guy, or maybe you kept pushing off a bothersome thing until it became urgent. Either way, you will not be fully motivated to do that interfering task and you will keep thinking about the intriguing thing you were forced to suspend. Recall the last time that happened to you. That is a good example of Attention Residue messing with you. Or at least messing with the outcome of your work.

Leroy (2009, 2016), who contributed fundamental research in this area, describes two human needs that underpin the effects of Attention Residue:

Need for Completion: As long as you do not meet the goal you set out to, you are distracted. This drive also motivates you to finish.
Need for Cognitive Closure: Even after meeting your goal, you may still fuss about your solution, thinking of alternatives, which distracts you from the next thing.

As long as both of the above needs are not satisfied you will suffer from Attention Residue.

There is also a related effect that has not been named (or I don’t know the name). Think back to the last time you have heard or seen something extreme, as in: something that evokes strong emotions in you. Rather than using yet another horrifying news story, think of something positive: the birth of a niece or nephew or a wedding proposal a close friend of yours received or somesuch. This kind of news persists with you for days. Once embedded, it overlays other things you are thinking about. Within the whiteboard metaphor, I would think of someone writing with a red permanent marker on it. Whatever is then modeled on that whiteboard, there is always that glaring red spot that cannot be erased.

So to the question of why interruptions increase the error rate, the answer is twofold: First there is Attention Residue, that makes you think of not-the-thing-you-are-doing, which makes you prone to make mistakes. Then there is plain imperfection of human memory that causes mistakes when rebuilding a previously interrupted Mental Model. Together they can wreak havoc on the quality of the outcome.

The amount of error rate increase depends on the context and the task. Westbrook, Woods, Rob, Dunsmuir, Day 2010 showed the error rate increases by 12% in medication administration when interrupted - while the severity of the mistakes doubled. In the study Adamczyk, Bailey 2004 it was found that error rates would increase by 25% and according to Bailey, Konstan 2006 the error rate can double as well, depending on the task.

How Can You Stop Interruptions From Diminishing Performance?

The good news is that there are actually a lot of things that can be done. In my opinion the single most important thing is: Create awareness. First for yourself, so that your confidence does not suffer from low performance caused by high interruption amounts. It’s them, not you. Then within the groups you are part of: Make it widely known and clear to anyone that interruptions are a significant factor in productivity, error rate, quality – and anything else pertaining to performance – and finding ways to reduce them is in the best interest of all.

With that in mind, I also want to acknowledge that there are as many situations as there are people around. Please consider my answers and suggestions below with an appended if your circumstances allow.

Lastly, some of the measures I will propose to you only make sense in the context of systems / groups / organizations. I will mark them clearly. Everything else can be done by every individual or group alike.

Let’s get to it then:

Decrease Amount, Frequency And Duration Of Interruptions

I know, if it were that easy, we would not be here talking about the topic. Still, there is a vast gulf between being unhappy about “feeling disrupted all the time” and having an informed discussion about the topic, with the knowledge of the underlying mechanisms and resulting consequences (see the article above).

One way to look at this is from the direction of project management and work organization. In other words: how can you make work (time) better predictable? How can you decrease the amount of unplanned work that comes in? Many books have been written on this topic. The Mythical Man-Month and Making Things Happen come to mind as great starting points.

Everyone:

Track Interruptions: You need to know the extent of the problem. Sometimes it is just perception. More often it is not. It is always quantifiable. How frequent do interruptions occur? How long do they last? At what times are they most likely to happen? The power of raw numbers that showcase the true scale of the issue cannot be overstated. This alone can turn the tide.
Be Mindful Of Other People’s Time: “Can I interrupt you for a minute” sounds polite, but it is already an interruption, if maybe a lesser one than the actual item. Do not take anyone’s availability for granted. Model the behavior you want to see. Use asynchronous means (like messengers, mail, ..) with sufficient time ahead to arrange for time. Reduce the amount of interruptions you are causing.
Set Boundaries On Your Own Time: Do not feel bad about blocking time for concentration. Make that public by sharing your calendar, hanging a sign outside of your door or whatever helps to communicate to others when you do not want to be disturbed. Do not forget to clarify also when you are best available and make an effort to consider other people’s time requirements. Reduce the amount of interruptions and funnel their occurrence (as much as you can) in time that you control.

Organizations:

Culture Of Respect: Values do not spring into being by happenstance. They are forged intentionally. Valuing each other’s time is a deliberate decision that requires purposeful design of processes. Take the time to think that through and come up with ideas / guidelines that make people aware. Provide tooling that makes it easy to arrange shared time. This will reduce the amount and frequency of interruptions for everyone.
Culture Of Focus: Find long stretches of time and designate them for concentration. Like meeting-free days each week. Communicate in batches, not continuously. Make it public what you do and encourage everyone to join in. Provide guidance and tooling to make that easy. This reduces the probability of interruptions, by making (many of) them planned events. It also decreases the average impact of interruptions, because most of them should happen outside of dedicated concentration time.

Decrease Task Complexity

The next best thing to reducing interruptions is making your work more resilient to them. The major factor in that is Mental Model complexity, which directly derives from task complexity. In that context, think of complexity as fragility. The more complex your tasks are, the easier they are disturbed, the more costly is the damage to them and the higher the price to rebuild them after the interruption passes (Resumption Lag). You need to have a good understanding of the nature of the tasks that you are engaged in. Combining this with the insight into interruptions, as proposed in the previous topic, will allow you to make good decisions.

Everyone:

Divide And Conquer: Decompose a given task into two or more similar, but simpler, subtasks. Each simpler task will require a proportional simpler Mental Model. Simpler models require less mental capacity which makes them more resilient towards interruptions by reducing their cost to rebuild.
Separate Planning And Execution: The complexity of a task is often concentrated in understanding the task and formulating a solution. Still, the execution, the doing part, may take much more time. In such circumstances, assuming that interruptions are not distributed exactly equally (e.g. less random phone calls in early morning), deliberately separating the work into two phases can be extremely helpful. Complex things in the “off hours” and simple things during “prime time”. This reduces the cumulative effect interruptions have, as you steer your most fragile work out of their way.

Organizations:

Pre-Process, Process, Post-Process: Separate a single workstream of highly complex tasks into multiple workstreams of simple tasks. Complexity at scale has known solutions since the introduction of manufacturing. With less complex tasks across the board, the impact of interruptions is dampened.

Increase Task Similarity Or Decrease Task Switch Frequency

Regardless of whether the amount of interruptions or the task complexity can be reduced, there is another pathway to reduce the cost of interruptions: Order the tasks by the shape of their Mental Model. That means: make sure that you work on similar tasks first. This way switching between tasks has a lower price, as less changes in the Mental Model are required and more parts of the Mental Model will be continuously used, hence less decay.

This is a project management concern. As Tregubov, Boehm, Rodchenko 2017 showed: you can easily lose a whole day per week just for task switching

Everyone:

KISS: Stick with one project that you are working on. Within that aim to switch between tasks that translate into closely related Mental Models. Move on to tasks that are at least somewhat similar. This will minimize Resumption Lag - a lot.
Prepare Then Do: Make it your first task to group all of your tasks on your list by similarity. Only then pick up the first actual task from that list. Abraham Lincoln famously said, “Give me six hours to chop down a tree and I will spend the first four sharpening the axe.” This is a variant of that. This will reduce the amount of Mental Model changes in between tasks solidly.
Pre-Dedicate Your Time: This is a different way to look at it. Instead of choosing from the list of already existing tasks, make sure that your list is filled only with similar tasks in the first place. For example: Dedicate days to communication, dedicate other days to engineering (or whatever you do) and yet other days to administrative toil. By doing that you limit the range of “task varieties” that make it on your list. Again: Less Resumption Lag for more similar tasks.

Organizations:

Constrain Number Of Workstreams: This is about distribution of work. Make it a project management / planning goal to optimize for similarity of work. Only comparable work goes into the same workstream. Then constrain the amount of such workstreams that individuals or teams work on during each time frame (e.g. sprint, quarter, etc). Maximizing for similarity minimizes Resumption Lag.

Increase (Work) Experience

You create Mental Models for everything that you do: Working a specific task or driving your car or cooking spaghetti bolognese. Having done something, especially repeatedly, allows you to recall a stored Mental Model for that thing at a much lower cost than creating a new one from scratch. I like to think of this as “higher level (mental) modules”, that I can load and combine. From working with complex systems for half of my life I don’t think this is even limited to repetitive work: having various Mental Models about a certain domain will help with lateral thinking in that space because each model adds perspective.

Mind though: experience does not help to mitigate increasing error rate (Westbrook, Woods, Rob, Dunsmuir, Day 2010), it only decreases the impact on productivity by decreasing the Resumption Lag.

Everyone:

Extend Your Toolbox: Learn more! Try more! Tinker! Each success gives you a new pathway. Each failure tells you where not to go. All of that will add to the bigger picture, allowing you to draw conclusions and make educated guesses even in areas that you have not experienced directly. It also increases your speed of building Mental Models: many components or even whole subsystems are already known to you. That also means less surface for interruptions to act on, and if they do, it will cost you less time to resume.

Organizations:

Culture Of Learning: Tenure comes with deepened and broadened domain knowledge - if you foster that, by providing incentives to learn. Time to grow and self-develop should be built into the organization of work. Your goal here is to establish a self-reinforcing feedback loop of self-improvement, so that everyone will have better and more Mental Models accessible. Interruptions will lose much of their punch.
Blameless Culture: Learning requires experimenting. Experimenting means taking risks and making mistakes. Likely a lot of mistakes. If that has negative consequences for the person that made the mistake – experimenting will seize. In fact it will never start in the first place under such conditions. Experience will stall at low levels and interruptions will have a major impact on the already low productivity. Don’t go there, but instead encourage experimentation. Share learnings – of failures and successes alike. Again: better and more Mental Models for the win.

Heighten Pressure, focus on the Negative

This one I found a bit surprising. Leroy 2009 investigated the question of how time pressure relates to Attention Residue. Her research revealed that more time pressure helps you to reduce the Attention Residue. Having “all the time in the world” allows you also to overthink and keep thinking of alternative solutions way after you solved the problem. In Leroy, Schmidt 2016 the question was asked whether aiming to achieve a positive outcome or attempting to prevent a negative outcome helps with moving on from previous tasks. It turns out a negative framing (“What bad thing will happen if I don’t get this done?”) works much better. I figure this relates to the evolutionarily caused Negativity Bias.

Everyone:

Religiously Timebox (i.e. predesignate a fixed amount of time for) your tasks. Attention Residue comes from the inability to disconnect your thoughts from a past activity. Having a clear boundary will help you to cut the ties to previous thoughts and move on.
Start With Unfavorable Outcomes: When beginning a new task, remind yourself of what you will lose if you don’t do this. This will make it easier for you to detach from the previous task with reduced Attention Residue. From my other readings I must warn you to approach this with caution. Do not overdo it (e.g. foster a negative mindset). If you do, it will cause stress. Stress will quickly overshadow any positive gains. Use with care!
Crystal Clear Definition Of Done: The more ambiguity in the goal definition, the more room for you to doubt your solution. Be entirely clear about what is to be achieved and how you measure whether that is the case. Err on the side of greatest possible obviousness, if in doubt.

Organizations:

Foster Confidence: The last thing you want is everyone questioning their sanity and their solutions. People get stuck in loops from which they cannot move on. Certainly not the most productive state. Bolster confidence by providing positive feedback on executed solutions. Make it easy to move on. Help mitigate Attention Residue.
Communicate Clearly: Leave no room for misinterpretation. Don’t let anyone waste time questioning their understanding of the task, which would make it that much harder to identify success. Done must be highly obvious and it must be unambiguously clear that no alternative solutions are required or intended.

The right Tooling

As an engineer, this is of course my go-to solution for, well, anything. Pracash, Koczmara, Savage, Trip, Stewart 2013 found (in the medical space) that having supplemental tooling to execute a task can reduce the interruption-caused error rate by half. This means: Understand the task, then make sure you have the right tools to do it in place. Or build them, if they don’t exist.

Everyone:

Integrate Quality Assurance: Assume you are making mistakes – because you are making mistakes. You want to arrange it so that whatever you did is automatically verified. Like spell-checking when you write a text. In software engineering this means having automatically triggered software testing integrated in your continuous deployment / integration pipeline. I am sure there are comparative and highly specialized solutions in other fields.
Automate Everything: Automation is usually associated with productivity. However, working mainly with reliability topics professionally I can assure you that error reduction is the silent hero here. Automating a process means that you are now out of the equation. Your state of mind, whether distracted or focussed, is of no relevance anymore towards the outcome of the process. This does not help you reduce a few errors here and there as much as it helps you to eliminate whole classes of errors. I cannot overstate how much automation plays a role in reliability concerns.

Conclusion

You have now created Mental Models of Mental Models in your head. You are welcome for the recursion!

Thanks for following along. I hope you now have a much better understanding of the underlying causes and mechanisms that make even seemingly innocuous interruptions so painful for your own performance. Maybe I even surprised you with one or two insights that will help you to take control of your own time and mental resources again. The Information Age that we are living in comes with many intricate pitfalls and masked dangers that ride on the trail of the vast amount of new opportunities. If I could ask you one thing, then it is this: spread the word! Help to build awareness. While interruptions are a problem as old as humanity itself, technology magnified the size and frequency to an unprecedented level. Dealing with this in a sane way requires cultural change – or we are at peril to lose a battle that we don’t even know we are fighting, likely because we are so distracted through the many interruptions that hinder us from thinking clearly.

This article will not be the end of my research journey, but merely a segue to the closely related Wild West of attention research. Stay tuned.

A note on why I chose to present error rate and productivity as separate proxies for performance: Productivity seems to change the outcome linearly whereas error rate can have an exponential effect. By that I mean: productivity impact is predictable and within a limited range of outcome. Not so for error rate. The range of outcome is in between that of reduced productivity (added time of fixing the mistakes and doing it again) and incalculable catastrophe (consider potential crippling reputation costs or something like the failure in the Mars Climate Orbiter from 1999).

Sources

Why are attentions disruptive? Why do interruptions decrease productivity? Mental Model, Memory Decay, Resumption Lag
- Burmistrov, Leonova 2001: Interruptions add Reorientation Time => Resumption Lag
- Dingfelder 2005: What influences Working Memory Capacity and how to improve it
- Iqbal, Horvitz 2007: Length of interruption directly correlates with Resumption Time => Memory Decay
- 5 min conversation => avg 15 min after resuming task
- 15-30 min conversation => avg 30 min after resuming task
- Cades, Werner, Trafton, Monk 2008: Complexity of interrupting task increases resumption lag => Complexity requires space
- Monk, Trafton 2008: Longer and more demanding interruptions increase resumption lag => Memory Decay
- Magrabi, Li, Day, Coiera 2010: Resumption Lag strongly correlated to complexity of interrupted task => Memory Decay & Rebuild complexity (com
- Tregubov, Boehm, Rodchenko 2017: More than one project => Switching between tasks between projects => 17% of time goes into switching (Resumption Lag)
Why do interruptions increase error rate? Attention Residue, Rebuilding Errors
- Adamczyk, Bailey 2004: Interruption causes increased error rate (25%), increased completion time (27%) and doubles the annoyance
- Bailey, Konstan 2006: Error rate doubles when interrupted during primary task
- Altman, Trafton 2008: Shows short interruption strongly increase error rate (5s => 3x higher, 3s => 2x higher)
- Leroy 2009: Attention Residue is caused by focus on previous activity that interferes with current activity. Need for Completion. Need for Closure.
- 40+% better performance in ideal conditions (able to finish + time pressure)
- Westbrook, Woods, Rob, Dunsmuir, Day 2010: interruptions in medication administration => 12% increase when interrupted & severity doubled
- No mitigation through nurses experience level
- Foroughi, Werner, Nelson, Boehm-Davis 2013: Interruptions decrease outcome quality, measured in rated essays => creative work badly impacted
- 17% worse scores for interrupted vs not
- 11% less words
- Pracash, Koczmara, Savage, Trip, Stewart 2013: Appropriate tooling can cut down error rate by half
- Leroy, Schmidt 2016: Regulatory Focus Theory => when focused on avoiding negative outcome Attention Residue lower than when focused on achieving positive outcome
- Forgoughi, Malihi, Boehm-Davis 2016: The higher Working Memory Capacity, the lower error rate following interruptions

If you want to join a discussion, please visit the cross-published version on LinkedIn.

The Cost of Interruptions

Ulrich Kautz — Mon, 20 Mar 2023 00:00:00 +0000

Did you know that a 4-5 second interruption of your work increases the amount of errors you are making threefold? But I am getting ahead of myself. Let me rewind.

I became interested in the topic of focus over the last year. It started out with attempts to organize myself better (a recurring theme in my life). This journey led me down a new rabbit hole, which I did not frequent before: the domicile of a hare called focus. As far as rabbits and their holes go: this is a big one. It also contains a plentitude of adjacent cavities to sink your time into. To name a few of those topics: attention, communication, conscientiousness, interruption, flow, performance, sleep - and that is cutting it short. Frankly, I don’t know the scope yet myself.

In that context I recently came across a very interesting paper with the prosaic title: Pardon the Interruption: An Integrative Review and Future Research Agenda for Research on Work Interruptions (Puranik, Koopman, Vough 2019). As the field of attention and focus research is comparatively young, there is still a lot of fundamental work to be done. This paper pertains to the interruptions research field and is aiming to build said fundamentals: as a meta study of about 250 other papers it proposes shared definitions and processes that pull together the sometimes vastly different approaches in the sources. This helps the reader – me – also to get a birds eye view of the field – while providing a plethora of sources to further get lost in.

Disclaimer: This blog is my own reflection and part of my process of synthesizing what I learned. It contains only my own opinions. I am not speaking for anyone else. I also may be wrong. I refer you to the (linked),original paper(s) for your own studies.

What makes interruptions interesting?

I personally don’t like them. Not a fan. The old adage of know thy enemy applies here: to get them under control in my life, my work, I must first understand them properly.

More so: In my profession you rarely work fully alone. Teams of four to twenty’ish people are the common configuration, in my experience. Not having a clear understanding of how interruptions influence your productivity or well-being is a big problem. The question of how to communicate within a team strongly relates to this: finding the right balance between sharing information, helping each other and having sufficient time to get concentrated work done is a tricky task. Arming myself with the data, with the latest understanding will help finding better solutions to organize teams I work with in the future.

Also, I find this topic oddly fascinating – I don’t really need more reason than that.

What are interruptions?

A question you likely would not ask – because it is pretty damn clear what an interruption is: you know it when you see it. However, it turns out that it is not clearly defined – alas not to the extent that everybody studying the phenomenon speaks about the same thing.

When studying anything the question of how to gather relevant data points arises. Consider you are working on something finicky. You are deeply into it, keeping ten threads of thought in your mind (if barely) and you are close to finishing it. Then the phone rings. It’s your boss’s boss calling. You have to pick up and suspend what you are doing. You lose your threads. That finish line you saw earlier may be lost for the day. This is, quite obviously, an interruption. Now consider a slightly different situation: Same build up, finicky task, many threads etc. Then the phone rings again. This time it’s not yours, but from the person next to you. They pick up the phone and start talking. Loudly. Laughing. Gesturing. Annoying. Your concentration is lost. The threads are gone. Another obvious interruption!

However, obvious to whom? Obvious to you, experiencing both of those scenarios? Sure. But not obvious to an observer, a scientist, who measures the effect of interruptions sitting somewhere behind you. Nor is it obvious to an automated system that counts the amount of phone calls, emails or chat messages that you receive to gauge the impact on your productivity. Hence some researchers defined interruption only as things that changed your behavioral performance (you visibly stopped working on a thing) where others included the suspension of your attentional focus (you kept doing a thing, but your mind was pulled away).

Puranik et al propose to do the latter. Their definition is: Work interruptions are unexpected suspension of an ongoing task. This definition intentionally does not specify that the interruptions need to be observable or that it must “happen directly to you”. Both phone cases above fit into it. The task is considered equally suspended, whether only your mind (invisibly) or also your body (visibly) stopped doing it.

By the way, the word unexpected is also important. A planned thing, like a scheduled phone call, does not count as an interruption - even if it disrupts your work. While that may also have an (negative) impact, the field of study here would be about work organization/planning, not work interruptions.

How do you measure the impact of interruptions?

That is of course the big question. The paper makes clear that most studies do not even use the same dimensions or otherwise agree on the same measures. Hence they propose a new model, fetchingly named:

Integrative Process-Based Model of Work Interruptions.

This model comprehensively describes the factors known to this field of research that play a role when speaking about the impact of interruptions. Understanding these factors is the first step to both predict the likely outcome and intentionally mold the outcome. Ath the very least you want to make informed decisions with known trade-offs.

The paper is so well written, that even someone new to the field can easily follow it. However, for my own purposes a complex model is not nimble enough. I need something easier to handle. What I am looking for is more shaped like a formula:

Cost of interruption = A * B + C

Something like that tells me at a glance whether a thing is relevant to the outcome or not – assuming I know what A, B and C are.

The model consists of a large number of details and reasons that lead to the outcome interruptions have. Puranik et al then groups these factors together under shared topics. These topics already come very close to the terms of an equation: this is what I am looking for! Well, not quite, but with a bit of tinkering it will be close enough.

Following I will go through the topics. Or rather: through slightly changed grouping of factors into topics that lends themselves better for my purposes. Later then, I will make each topic a term in my equation.

Here we go:

(T) The Task that is interrupted or is interrupting

This topic is about the thing that is interrupted: The task. Or the tasks.

First, there is the interrupted task, which was going on until now. How complex was it? As in: How much mental energy was bound up in it? How many “threads of thought” were held? How long did it take to “get into” the task in the first place? Also the timing is of importance: Did the interruption occur in between two steps of a long chain of steps? Were these steps independent or building upon each other? Or did the disruption happen in the middle of step? The stage of completion is of course relevant as well. Is the whole thing almost done, or just started? While not in the paper, hence not part of the visited studies, I would also add accessibility , as in an interesting measure: The time and effort it takes to start or restart the task would be multiplied for every interruption.

An interrupting task does not always need to be part of an interruption. Or it may depend on perspective (would you call a conversation that interrupts a task also a task?). Again, complexity plays a big role: how much mental resources does it cost to “load” that new task into working memory? Also similarity to the previous task: is that just a variant of the same, or something entirely different that requires a mental reset of sorts? Lastly, centrality to the “core work” plays a role. Like: is this something that you are normally doing, or something outside of your comfort zone?

(I) The occurring Interruption itself

This topic is concerned with the actual interruption that forced itself into your attention.

What kind of interruption, with what kind of immediacy is it? It can be fully-asynchronous like email or semi-asynchronous like chat or optional synchronous like phone or mandatory synchronous like face to face. At what frequency does the interruption occur? Once in a blue moon? Ten times an hour? Is this an exceptional or a habitual event? How long was the duration of the interruption? 5 seconds? 5 minutes? 5 hours?

(E) The Context or Environment the interruption happens in

This topic is concerned with the external conditions in which the task was executed in and then the interruption occurred in. It speaks somewhat to what extent interruptions can be expected.

Consider the collocation : are we talking open space offices with everyone talking to everyone, or closed offices where people have to knock, or home office? Also the organizational culture plays a huge role: is it a reactive, chaotic, multitasking environment, that expects you often to switch from task to task – or is everything planned out years ahead, with no deviation from the path feasible/needed, everyone knowing exactly what to do? Then the degree of collaboration factors in: is the work mostly done alone, or does it require multiple people forming (temporary) teams and working at it together?

(M) The Mental Factors / Mediators

This topic is about effects that interact with your finite mental resources and your perception of positivity (or negativity).

If you are interested in systems theory, you may find this factor curious, because it seems to convert the whole process into a self-reinforcing feedback loop. This makes the whole system “stateful” and non-deterministic regarding the input.

First, there is the Cognitive Pathway, which pertains to innate abilities and how they “hold up” under the stress caused by the other factors. The memory decay effect works on the suspended task and makes it hard to resume later because your brain clears out the “mental threads” you had in your mind when working on something, when you move on to something else. Also, division of mental resources does not fare well with your hominid brain: it causes conflict, which causes stress, which causes all kinds of nastiness. Interesting tasks that are interrupted also result in attention residue , which means your thoughts keep lingering on the interrupted thing, hence making it harder to do the new thing (not great for either outcome). Last, but not least: prospective memory theory predicts you will forget to get back to the thing that you were doing. Especially when the interruptions keep coming.

Next is the Self-Regulatory Pathway, that is related to motivational or energetic effects. Starting with action regulatory theory, that describes the frustration caused by slow goal progress, which greatly dissuades further progression (example: leaders that are bogged down in mail handling, which is not perceived as making progress, measurably scale down their efforts to, well, lead). Interruptions are principally perceived as goal hampering, hence adding to that frustration. All that, ego depletion theory tells us, empties the limited self-regulatory resources you have, which makes it hard or impossible to deal with other items that require the same resources, like anxiety, that otherwise may have been easily managed. The mental exhaustion ultimately leads to low job satisfaction and is a high-probability pathway to burn-out.

Lastly, there is the Affective Pathway, that is concerned with emotional impact. Simply put: the interrupting thing can be good or bad. Being interrupted by a call that confirms a big client signed up due to previous or ongoing efforts is vastly different from an email from a project manager that adds more items to an already impossible timeline. Also, relevant is the base anxiety level , which has been correlated to the amount and frequency of incoming interruptions.

(P) The Person(s) involved in the Task or the Interruption

This topic deals with people – like you, the person that was interrupted.

We are all individuals. The way we deal with interruptions, stress and multitasking is on a spectrum. Chronemic preferences play a role. A polychronic person does not mind juggling multiple things at once with little stress, whereas a monochronic person is able to focus deeply on a single task. The extent of your ability to screen external stimuli plays a role: do even slight noises pull you out of concentration or can you work next to a jackhammer? Big 5’s conscientiousness is relevant as well. Where others would have given up long ago, someone very diligent can come back again and again to an interrupted task to eventually finish it. The size of your working buffer – the capacity to juggle short term memory – relates to when overload increases the decay speed. Also self-control is necessary to fight off frustration and stay motivated in the face of adversity. Lastly, tenure and experience allow you to be more relaxed in high-stress situations that would thwart someone more junior. What the papers seemingly were not concerned with is neurodiversity. I wonder whether that is a factor.

If the interruption is caused by, or transmitted through, another person, they also play a role in the outcome. While this is not widely researched, it seems at least clear that interruptions from superiors are more distracting than from peers.

(O) The range of measured Outcome

And now finally to the other side of the equation. The paper separates the outcome into two broad domains: well-being and performance. While it is left to the reviewed papers to detail specific results, the measures that are commonly used are introduced.

A simple performance metric is completion time , which directly correlates to productivity: faster completion equals more productive – and vice versa. The task resumption time , that describes the extra time added for picking up a suspended task again, adds on overhead, which increases the total time, lowering productivity. Then there is the error-rate , that measures the frequency of mistakes when performing a task. This either impacts the quality of the outcome or adds time to fixing those mistakes. The work quality itself is also a measure of performance that is also altered by interruptions.

For well-being there is a range of more subjective metrics, most of which are also impacting performance. For example, job satisfaction is certainly affected by interruptions. A low level of satisfaction results in a different work attitude, than a high one would. Stress is the most common outcome of interruptions that affect planned tasks, as it hinders progression of goals. This goes hand-in-hand with heightened levels of anxiety , which do not help level-headed goal execution either. In the long term stressful interruptions can also cause emotional exhaustion which is strongly correlated with burn-out. Lastly, the paper cites that actual physical distress is not off the table as an outcome.

By the way: All of the performance metrics are directly observable, or at least their effect is. Well-being is not as easy to gauge from the outside and is often measured through self-reporting or interviews or through its indirect impact on performance.

What is the impact of interruptions?

You probably noted the letters that I put in parentheses before each above topic headline. These name the terms of the equation that the model comes down to in my understanding:

Yes, I intentionally ordered the letters so that it forms the Spanish word for time (or weather). Even though I do not speak Spanish (my wife does), it helps me to remember.

I am going to call this a lazy approximation. A simple equation with complex terms. The complexity of each of these terms I layed out above. Chances are the continuous research, that is going on, will add more knowledge and nuance. It likely will change, add or replace many of the factors in the terms. However the topics, aka the terms, may be more stable. The equation itself may hold true for longer.

Either way, the question was what the impact of interruptions is. I cannot tell you, yet - or at least not in any precision. However, having the concept that underlies the model in such an explicit fashion available makes it much easier for me to think about it:

Scoping the outcome

The equation I ultimately want would deal with numbers. I am not there yet. I am far from any meaningful calculation. Before I can think about that, I first need to get a handle on the scope, which will give me the possible range the formula operates in. At least approximate – that is all I need.

Beginning with the outcome (O): Mind, the outcome was separated into well-being and performance. For simplicity’s sake I am not going to differentiate and lump sum everything together as performance. So, what could the possible result here be? That is actually pretty simple. Given an interruption, one of three things could happen: Things get worse, things stay the same or things get better.

Expressing this in numbers I would use positive, decimal numbers (R>=0)):

Values below 0.0 : nonsensical. (Undoing things that were already performed?)
Value of exactly 0.0: interruptions cause performance to go to zero. Nothing is getting done.
Value below 1.0: interruptions decrease the performance.
Value of exactly 1.0: interruptions have no effect whatsoever.
Value above 1.0: interruptions increase the performance.

Going forward when I write the outcome gets worse then I mean that the performance value is influenced to decrease towards 0.0 – and vice versa for _better _means that the performance value is nudged to increase.

Scoping the terms

To better understand the following, imagine you are looking at a specific role that is offered from a specific organization. I propose that by asking pointed questions you can guesstimate how “interruptive” this role will be and thereby how well you will be able to perform in it – and how your well-being will be impacted. I am not promising that I already have the right questions, but this is fine for now. From what I read so far – they may be close anyway.

Mind also that the terms “worsen” and “improve” will be used in regards to performance – this is not a value judgment. Some functions / roles mandate certain things that cannot be changed. These things have consequences. This is about understanding and predicting these consequences.

Now let’s get to it.

Consider (T) – the Task – and ask: “In this role, how complex are the tasks that need doing and of how many steps do they consist of?” Think of a spectrum. On the left side is written “multiple, very complicated, dependent steps” and on the right side is written “a single trivial step”. Now order the answer you got for that specific role in that specific organization on that spectrum. The closer the answer is to the left, the closer it is to 0.0. The closer the answer is to the right, the closer it is to 1.0. Resulting the value for (T) either worsens (closer to 0.0) or improves (closer to 1.0) the performance.

Next up (I) – the interruption – with the questions: “In this role, how urgent and how frequently do you need to communicate and in what way?”. The spectrum here ranges from the left end (=0.0) “urgent, constant, face-to-face communication” to the right end (=1.0) “pro forma, rare, email communication” . Again, you will find whether (I) for that role worsens or improves the performance in general.

Looking at (E) – the Environment – the question is threefold: “In this role, how do you collaborate, where do you work and on how many concurrent things?”. The spectrum is ranging from the left side “in changing groups in large, open work spaces on many things” to the right side “alone, in isolated space, on a single thing”. Find where the role fits on this spectrum.

Now for **(M) **– the Mediators – is not that easy. Actually, I don’t think that this term has to do with the role or the organization, but only with the individual – and maybe with happenstance. I also think that (M) is not static, but due to change. For example through coaching (and maybe through training) towards improvement – or through workload and stress towards worsening. If that is the case then an organization would be advised to treat this the same way as technical debt: continuously invest into improvement to keep the status quo. However, that is far from a well thought out opinion. Let me get back to you on that (in a follow up article). As an individual, it may help consider the factors that are mentioned (above) and evaluate yourself. If you are scoring low (as in: lower performance) you may want to look for roles that do not additionally worsen performance – especially given that performance also includes well-being!

Lastly comes (P). It has a component that is dependent on the organization: the interrupting person. The question, which likely would not get an honest answer, would be: “Is this role micromanaged?”. Well, if that gets an answer then a “yes” worsens performance whereas an “no” improves it. Other than that, it is again more up to the individual to score themselves – and for organizations to filter for the traits that help them.

Gauging the impact

Now that I walked through all the terms in the equation on both sides, I think there is some immediate practical application: I can now guesstimate what outcome I should expect in terms of well-being and impact on my performance for any role role I come across. At least for three out of five terms in the equation, with the rest mainly pertaining to myself.

I also think any organization should reflect on these terms and truly consider whether their current configuration matches with their goals. I would not be surprised if that is not often the case – especially in the knowledge worker realm, where interruptions seem to be one of the main culprits for productivity loss (getting ahead of myself again).

In any case, I have a much deeper appreciation of the factors that play a role and the questions that I need to ask. While I don’t have any ability to provide specific predictions (like: do X and performance increases by 12%), the landscape of possibilities is much clearer to me.

What comes next?

The error rate of factor three, that I cited in the beginning, comes from a study (Altmann, Trafton, Hambrick 2013) that investigated specifically short interruptions – of three or five seconds in length – that disrupted short tasks (also in the seconds range). There are many more studies, 247 of which are linked in the paper from 2019 that caused this blog article. Many, many more studies came out since then. I just started digging into them and can already foresee that I will not squeeze all my learnings into a single post.

For me, the journey to learn about this topic has just begun. I am already reading a lot of papers and hope to post a new article soon. Most of them are like the one cited above, that is they are concerned with a highly specific experiment. The hard part, well the interesting part, is to accumulate multiple of such studies and find generalizations. In case you are wondering why some engineer is interested in this topic: I think we all should. I think this is a very interesting topic of our time. Also everybody needs a hobby.

One thought regarding the outcome: The paper only classifies the result into performance and well-being. I think a classification of short-term and long-term would be more useful, which mostly would map with the provided categories but would frame it in a perspective that would lend itself better for use (in business etc). Even better would be both (long term / short term performance / well-being).

Stay tuned, next up some more numbers. I am most interested in everything that has a practical consequence. I am glad there is so much material and that others do the leg-work that I can now comfortably consume.

If you want to join a discussion, please visit the cross-published version on LinkedIn.