Developer? Send This To Your Product Manager (if you dare)

There is a common anti pattern in modern software organisations. I've seen it leading to catastrophic outcomes multiple times. It arises from the inherent structure of these organisations and the differing skill-sets, habits and incentives of key players, particularly those in product and engineering roles. I call it "The implementation details fallacy".

What is an implementation detail?

In software, an implementation detail is codename for "something you shouldn't worry or care about" or "unimportant stuff".
For example, if you want to increment the value of X by 1, there are a few ways to implement it in single line of code:

1. x = x + 1
2. x = 1 + x
3. x += 1
4. x++
5. ++x

Whichever of these five options the developer chooses is an implementation detail. It doesn't affect performance or user experience. It doesn't impose any limitations on the system. We can always swap any of these options for another at any point in the future at zero cost. It just doesn't matter (I'm sure some developers would disagree with me, even in this simple case 🤣).

What might surprise you though, is that it's quite hard to come up with clear-cut examples of choices that truly don't matter. The "implementation details fallacy" is treating critical choices (a.k.a "design choices" or "architectural choices") as if they were implementation details.

Quiz time

What do you think about the image below? For context, it tries to illustrate the importance of "viable" in "minimum viable product". It appears in several articles on the internet; I first encountered it here. Let your brain process it for a minute before you proceed:

I used to think it's elegant and brilliant. I even printed a copy and had it displayed at my desk (when going to the office was still a thing). My manager and product managers I worked with at the time liked it too. That was a long time ago.

While I still wholeheartedly agree that every iteration should provide value to users, I completely disagree with all the other ideas that are expressed in this graphic. What's really interesting about it, is what it reveals about the mental models of the product managers who created it:

1. They think that it's a bad idea to ship stuff that are critical parts of an actual car, like wheels or powertrain. In their mind, these things don't amount to anything useful because they can't move a user from A to B on their own. In other words, we can't make users happy by selling them really good tires (too bad for companies like Bridgestone ).
2. More criminally, they think that you can iterate your way from a skateboard to a car. This is absolutely nonsensical from an engineering and from a business point of view. Just try to imagine SpaceX developing and selling gliders before pivoting to hot air balloons, then helicopters, then airplanes, and finally Starship 🤦🏻‍♂️.

Just to drive the point home, no successful car company in history has ever developed a car this way. Their actual process looks suspiciously similar to the "not like this" row above, something like:

But wait, maybe the diagram creators meant we should take these real-world steps, but do it for each of the skateboard, scooter, bike, motorcycle and car. Well, that doesn't make sense either. I mean which part would we be able to carry over from each step to the next? Can we reuse the skateboard wheels for the scooter? Well, in physical systems it's easy to see how crazy that would be because we have intuitive understanding of the physical world but...

This is when the tragic reality of the situation finally dawned on me: Those who think this illustration is brilliant (or even makes basic sense) all work in the software industry. In software, the skateboard wheels are invisible and can totally be fitted onto a car. They are merely an implementation details 😱.

Cars with skateboard's wheels

How many modern software products feel clunky, slow, and unstable - like cars with skateboard's wheels, bicycle seats, and motorcycle engines? It's so prevalent that applications that are actually good stand out in the crowd and feel like they are operating under a different set of rules (we'll talk about what these have in common later). The "implementation details fallacy" can (at least partially) explain how and why most apps end up this way.

A recipe for disaster

Here's how it works:

1. For the majority of product managers, everything technical is an implementation detail. As long as the narrow requirements of a ticket are satisfied, ideally within the pre-allocated amount of time, they don't care how it was achieved.
2. Because they are not technical, the majority of product managers lack intuition about the cost of postponing key technical decisions. As a result, in product meetings, when someone brings up edge cases or any concerns that don't seem immediately relevant, they are politely dismissed with "let's take it offline" or "make a ticket and put it in the backlog". In reality, as the codebase grows in size and real users' data starts flowing into the system, breaking-changes become problematic and any modification to core parts of the system becomes difficult and risky, as the rest of the code is built on top.

One could expect engineering to come to the rescue. Unfortunately there are a few factors preventing it from happening:

1. Since we're agile, we break down larger tasks into small tickets and rarely plan (in detail) for more than one or two sprints ahead. This creates a tiny world with a short time horizon for developers. It disincentivizes them from trying to understand the system as a whole in its current state, let alone what it should become in the future. Developers are given one ticket at a time, and their objective is to mark it done and move on to the next ticket with as little friction as possible.
2. This is amplified as developers frequently need to contribute to code bases they are not fully familiar with and work in domains they have no experience in or deep understanding of.
3. In the context of each one of these small tasks, rewriting big parts of the system is simply not an acceptable solution (rightfully so).
4. As a result, if the system is currently a skateboard and a developer gets a ticket that says "add steering," they need to find a way to add steering to the skateboard somehow. They don't have the time or desire to learn the entire system first or align themselves with the long-term vision (if one exists). The same goes for the tools they use (libraries, frameworks, etc.) - they usually learn as little as they can to get something working. How to add steering is an important design decision, but it's rarely treated this way. There's no time for that. Even if several options are considered, they all tend to share this state of mind. It's just an "implementation detail" after all.

If that's not enough, there are environmental factors too:

1. With the internet, pushing software updates to users is much cheaper and faster than updating any physical medium. This creates the illusion that any change one wishes to make is possible at any point in time. Change the skateboard to a scooter, push a button to deploy and you're done. In reality nothing is further from the truth.
2. Unlike a car prototype that gets tested on a real road, with real physics from day one, software products get users gradually, and the system can look like it's working fine as long as the scale is small. Concurrency issues, race conditions, rare edge cases, performance issues, and other bottlenecks - all of these don't tend to materialise without significant usage. Very few teams "road test" their solutions from day one.

This combination of ingredients creates a vicious cycle. Estimates keep inflating as the system grows larger and gets increasingly unfit for purpose. Product managers start to secretly resent the developers and vice versa. Users start complaining. Management is scratching its head. It's a disaster.

"Implementation details" are everything

• What made Chrome, when it came out, different to Internet Explorer? "implementation details".
• What made git possible? "implementation details".
• What differentiated the iPhone touchscreen from touchscreens that existed before? "implementation details".

Look at every successful software product that you use on a daily basis, and in 9.5 cases out of 10, you'd find some sort of "masterpiece of engineering" engine that makes the product possible. A "car engine" that was there from day one. Not in its final form, sure, but it was there. The system was never a skateboard or a scooter. It has a cohesive core, that defines a clear and cohesive set of technical properties and behaviours that are uniquely designed for what the system does.

There are countless examples. Here are some:

1. This nine years old article describes how Figma used web assembly and implemented their own rendering engine (rather than using the browser's primitives) to make their product possible. They also implemented their own sync engine (which I discussed further in a previous post).
2. Google Maps (including the insanity that is street-view) and Waze.
3. Whatsapp.
4. Skype, Zoom.
5. Google docs / sheets.
6. Netflix, Youtube and Spotify.
7. ChatGPT, Suno, MidJourney.

Based on my experience from working on both successful and unsuccessful software products, I dare to say that it is not possible to build any system of this caliber using the modern development process I described in the previous section. You can't have separate product, design, and development functions, each "staying in its lane," and developers that get shuffled all over the place and get these kinds of results. Sorry, not possible. Interestingly, as these products mature and start hiring more people and adopting these modern practices, their quality usually drops, and progress stagnates. Spotify is an easy example, and modern-day Google is another easy target.

But not all hope is lost. I think it's fixable, but it won't fix itself.

The antidote - learning to reason at the system level

Like cars and skateboards, software systems are, well, systems. Our mental model of the application we are working on needs to be that of a dynamic system, its history, and the ecosystem it operates in. It doesn't mean you have to know every little detail and every line of code. Think about car enthusiasts; some of them know a whole lot about types of engines, tires, gears, off-road capabilities etc., without being a mechanic or an engineer. You need to be like them - in your domain. This applies to both product managers and developers who are working on the system.

Your workplace is unlikely to require that level of expertise nor will it provide the proper training or support. Nevertheless, it's a must.

Product manager: Ask your developers technical questions (like a car enthusiast would a mechanic) but don't trust them as your sole source of information. Educate yourself. If your system uses event sourcing (or your developers want to introduce event sourcing), read about it, understand where it shines and where it struggles, and try to figure out its downstream effects at the system level. Challenge technical decisions if you have concerns. Some developers will get irritated by your probing, but they are usually the ones that can't defend their choices and need to deepen their knowledge as well. Don't treat developers as resources and don't shift them around hectically between projects. Never tell them to build a skateboard if your end goal is a car. Get your hands dirty and don't be afraid of technical terms.

Developer: Learn your tools in depth. Understand where and when they fall short. Every tool has shortcomings, yet when I ask software engineers about the downsides of their favourite tools, very few can give a proper answer. Think about a tool you like. Is it TypeScript? React? Tailwind? Ask yourself: "under which circumstances this tool should not be used?" if you can't give a solid technical answer, you don't understand the tool well enough and need to learn more.

Same goes for pre-existing codebases you work on. Ask for time to dive deep and understand the main component of the system and how it all comes together. Understand why things are the way they are and challenge that.

Know which properties your system must have in order to meet its "end goal", get those right from the get-go, and make sure they are never lost along the way. Stress test and chaos test from an early stage in situations that mimic concurrent usage in a realistic environment. Think of yourself as your peers from other engineering disciplines - the ones that build cars and spaceships and hold yourself to similar standards. Your bar should be way higher than "working software". If your organisation doesn't share the same ambitions for excellence, leave.

Founder/ Manager: Treat onboarding very seriously, like a university course. Make sure newcomers gain deep understanding of the problems the company is trying to solve, the domain, and all the relevant technical aspects. Give them tests if needed. Make sure they know the "why" behind everything. Don't throw them into the codebase (with a ticket assigned of course) and hope for the best. Remember that in a knowledge based industry, deep understanding is the only currency.

Final thoughts

This was my attempt to call for a small but important cultural change in the software industry; one that can happen from the bottom up. Even if you don't agree with everything I wrote, I hope it gave you some food for thought. Please comment with any insights, questions, or anything else you'd like to share. Thanks for reading!

Comments

[allen_z]

Thank you for sharing this insightful article, Isaac! As a Product Manager at Docuopia, I truly appreciate the perspective and the emphasis on fostering a healthy and productive relationship between developers and product managers.

At Docuopia, we understand the importance of clear communication and collaboration in creating successful products. Our platform is designed to streamline documentation and knowledge sharing, ensuring that both developers and product managers have access to accurate and up-to-date information. By providing tools for real-time collaboration and comprehensive documentation, we help bridge the gap between development and product management.

Transparency and efficiency are at the core of what we do. We believe that with the right tools, both developers and product managers can work together more effectively to achieve their common goals. Learn more about how Docuopia supports seamless collaboration and documentation on our Docuopia website.

[Alex]

Great article!
Sadly, situation won't change, management want things get done fast.
"Good, fast, cheap" - can choose any 2 of them.

[Isaac Hagoel]

You are right that that's the default, but I've been on small teams that managed to break away from this and build coherent pieces of software. Not sure if a change is possible but I surely hope it is.