Software engineering is an interwoven part of the loop

#softwaredevelopment #design #learning #ai

In my computer science studies at university, I learned that about two thirds of all software projects are neither delivered on time nor within budget. At the time, this seemed ridiculous to me. It doesn't anymore, despite having worked with many smart people for ten years since.

How can this be? Poor handovers? Weak business ideas? Broken feedback loops? My overall guess is that we excel within our own specialties but lack interest and insights into the full value chain needed for a successful digital product. Engineering excellence alone is rarely enough.

In this article, I invite you to follow my thoughts and share a set of recommendations for leveling up.

TLDR - My key takeaways

User-centered design is important. The famous DORA 2024 Accelerate State of DevOps report even highlights this as the most important skill.
Socio-technical skills are important.
AI is a rapidly evolving, exciting technology that can help us to get out ideas faster.
I recommend to explore different disciplines along the complete value chain.

1. What are we building?

The obvious goal is to develop a product for customers. In most cases we also want revenue to exceed costs. So, we need to find a problem people are both eager to solve and willing to pay for.

This sounds easy, yet is probably the hardest part. Consider two examples:

If I had asked people what they wanted, they would have said faster horses (Henry Ford)

In retrospect, investing into automobile development, like Ford did, was an obvious call. As contrasting example:

I think there is a world market for maybe five computers (Thomas Watson, chairman of IBM)

In retrospect, this is an obviously ridiculous bad call.

However, at the time these decisions were made, uncertainty was high, information was incomplete, and outcomes were far from obvious. Product development lives in the uncomfortable space between insight, intuition and incomplete data.

How can we improve our chances for those calls upfront? This is a science on its own and one which is still evolving. Three personal recommendations for leveling up:

The Mom Test: How to Talk to Customers & Learn if Your Business is a Good Idea When Everyone is Lying to You
Competing Against Luck
Product Design (UX/UI) Course - my learning journey was really great and all my exercises finally came together into Together, We Choose

2. Exploration and feedback loops

If we do not excel in the previous part - understanding what we are building - software development will suffer. Requirements will change frequently, chosen architecture trade-offs will be based on false assumptions and become anti-trades. Preventing this early drift is paramount!

Therefore, we need to explore and gather feedback as quickly as possible. This can be some user interviews, paper prototypes, digital prototypes, whatever works to evaluate your idea and reduce uncertainty.

At the latest when you are ready to invest in digital prototypes or early versions, software developers usually enter the picture. The goal is not to build the full product with a perfect architecture. Instead, the architecture must support ongoing exploration.

Exploration likely requires collaboration across roles. Business analysts, customers, designers, developers, investors, managers, etc. need to work together: Come up with ideas, hand them over, build experiments, evaluate outcomes and exchange knowledge again and again. For that, engineering excellence alone is obviously not enough. We need short, effective feedback loops and people skills to profit from everyone's expertise. In the bigger picture, it is often beneficial to deliberately accept technical debts as trade-off for faster exploration.

As a side note: AI can help you to generate ideas and lately even generate full prototypes for exploration. However, don't mistake those little generated apps as production ready, secure, bug free, cash-cow apps - if it were that simple, users could prompt the AI themselves instead of giving you their money. And AI-slop, bugs, security incidents, etc. can irreparably damage user confidence in you and your product, like the gigantic nx s1ngularity incident caused by an AI implemented mini feature.

One of the core ideas of the Agile movement is to encourage feedback loops and maximize the amount of work not done aka waste. I strongly agree with these principles. However, in my humble opinion, parts of the surrounding consulting industry have diluted the message over time. I recommend to level up in the terms user-centered design and socio-technical skills, e.g. with:

3. Engineering excellence

Once, we have managed to create a valuable product, we need to maintain and evolve it. In reality, we have likely already sacrificed the cleanliness of a greenfield project in favor of exploration and evaluation. Therefore, we need engineering skills to navigate and improve.

Technical perfection in software development does not exist, and even if it did, it would probably be a waste of money. The goal is to improve structure, reduce risk, and increase the system’s ability to change, so we can continuously support the ongoing product exploration and development.

This also has a socio-technical component: If you change something, you may overlook some previous trade-offs which still hold value. Teams can get tired of yet another change. Or you may unintentionally hurt someone’s feelings by questioning decisions they once made under very different constraints.

Resources I recommend for leveling up technical expertise:

Refactoring: Improving the Design of Existing Code
Working Effectively with Legacy Code
Clean Code: A Handbook of Agile Software Craftsmanship
And most importantly: Code your own work, accumulate your own experience and apply your common sense - if you don't see value in the next hype just save your time and energy and don't fall into the fear of missing out trap.