Foreword
If you run a software project, especially an open source software project, you have almost invariably heard things like this...
We need to push harder towards progress.
One week without a commit? What the heck?
When will the software be done?
Why isn't there a BETA release already?
What are you guys even doing??
If you're the one hearing these statements, it can be frustrating. Rest assured, for the most part, even the most impatient of these statements aren't bourne out of ill will. Yes, some users...okay, a lot of users...have wildly unreasonable expectations about timelines, but that is largely due to a lack of knowledge about how software development works. So take a deep breath, and remember that they're misinformed, rather than malicious.
And then calmly link them to this article.
Now, if you got handed this article by a developer on a project you watch, or if you've ever said any of the above, then you need to understand something: Software timelines are fundamentally weird. Or, as the age-old industry adage goes...
It takes nine months to give birth to a child, no matter how many women you assign to the job.
Software development seems to have a mind of its own. No, computers aren't sentient, and this has nothing to do with the much dreaded Singularity wherein our devices overthrow mankind. It all has to do with the unusual natural of software itself. You could easily spend an entire career studying this -- Fred Brooks basically did -- but most of y'all don't have that kind of time. So, let me break down the basics for you. Consider this the Cliff Notes version of Software Project Management 101.
NOTE: If you want the long version, go read Dreaming in Code by Scott Rosenberg. It goes into far more vivid detail, through the lens of the true cautionary tale of the comically fated Chandler project.
The Law of Translation
Software development takes as long as it will, and no amount of human intervention can speed things up past this implicit, inherent "speed limit". No one has ever really calculated how to determine the Great Inherent Project Schedule. We can only figure out how to get out of the way and not make things worse.
It's still a mystery why software takes its own time, and why we are completely helpless in moving it along (more on that in a minute). However, I believe the fundamental cause might be what I call the Law of Translation...
Law of Translation: When two parties have no natural common ground of understanding, there is no way to objectively check the quality of a transfer of knowledge.
To take that out of babble-speak, let's imagine that you have two people. One speaks Spanish natively, and the other speaks Inuit natively. Neither one knows the other's language, and they have absolutely no shared cultural experiences. Barring non-verbal communication, which we'll just ignore for the sake of the analogy, there are only three ways they may communicate.
- The Spanish speaker must learn Inuit.
- The Inuit speaker must learn Spanish.
- They both must learn a common language, say, Esperanto.
However, in any of those methods, they cannot learn the other's way of thinking. If the Inuit-speaker gives instructions for building an igloo, and the Spanish-speaker has never seen ice or snow in any form, it will be a difficult task indeed! To make it more difficult, the Inuit-speaker must be able to discover whenever something is not understood, and there are only two ways to do it:
The Spanish-speaker may realize he doesn't understand, and express that to the Inuit-speaker.
The Spanish-speaker may think he does understand, and take the wrong action, which the Inuit-speaker observes.
In other words, the Inuit-speaker has no way of actually knowing whether the Spanish-speaker understands the concept; he either must be told explicitly, or he must infer the lack of understanding from the observable results. Then, he must find a way of helping the Spanish-speaker understand, which is hard, because he doesn't actually know what exactly isn't making sense!
This is almost exactly the situation between the Programmer, who speaks and thinks in Human language, and the Computer, who "speaks" and thinks in binary math and CPU instructions. Computers are inherently incapable of abstract thought, inferences and conclusions, and a host of other human-brain things we take for granted. Meanwhile, computers break literally every thought down to simple addition; we humans do not.
Thus, we have only three ways we may communicate with, and therefore teach (program) computers.
The Programmer must learn to think and write in binary CPU instructions...and some have.
The Computer must learn, or rather be taught, to think in Human Language. Despite how much it looks like computers can do this, that's actually a whole lot of smoke and mirrors. We are actually a long ways off from beating the Turing Test (look it up).
The Programmer and the Computer must learn to work in a common language. These are our programming languages, and this is where 99% of programming takes place.
In doing this, however, we still have the same problem as before: we have no inherent way to know how well the computer understood us. We can only discover this those same two ways.
The Computer realizes it doesn't understand and tells the Programmer. This is a "compiler error".
The Computer misunderstands and does the wrong thing, which we then observe.
Then the Programmer has to figure out what the Computer isn't understanding, which is especially hard.
This is the inherent reason why building software isn't like building bridges. When you engineer something, you can see and measure precisely what is wrong. However, when you write code, you have to fight through this "language barrier" between human and computer to infer what is wrong. These are literally what bugs are!
Fact is, starting out, you have absolutely NO IDEA how well your human thoughts will translate to the computer. What looks to us to be an easy task, such as picking out the tallest person out of twenty people in a room, is actually deceptively complicated to a computer.
Making it harder still, you're not just dealing with the instructions you're giving the computer, but also with the instructions hundreds of other programmers have given, in the language, the frameworks, the libraries, the operating system, and so forth. These are called "layers of abstraction", where we move further away from the ones and zeroes of computer language, and closer towards human language. Walking into any given "stack" of abstractions, we have no idea if there are other mistranslations, or "bugs", that could slow us down.
In short, the entire process of writing software is so subject to these translation errors, a seemingly simple task can easily balloon into a months-long saga of instruct (write), observe (test), infer (debug)...wash, rinse, repeat.
The Quality Triangle
There's a principle in software development called the Quality Triangle that states...
I can make it fast, cheap, or quality, pick two.
There are actually multiple valid interpretations of this, but the one relevant to our discussion is this...
We can make it quickly.
We can make it inexpensively.
We can make it relatively stable and bug free.
Pick two. If you want it faster, the developers need to be able to put more hours in, which costs money. If you want it more stable, the developers will have to spend more time testing and debugging, instead of just calling it "good" and moving on to the next feature. If you want it less expensive, they'll either have to ditch quality or take more weeks.
There are terrifying tales of projects attempting to achieve all three. Stories of developers exhausted and resenting their careers, software ridden with delays and bugs, deadlines missed, expenses running over. If you try to achieve all three goals, you will invariably MISS all three goals. The software will be over-budget, over-due, unstable, bug-ridden, and missing critical features. In most cases, the finished product will never even see the light of day.
However, even within picking two, there are a few limits.
For one, you can't just speed up a project by throwing money at it. Developers only have so many hours they can work in a week before they burn out, and we also have Brooks' Law to contend with (see the next section.) A high-budget project will get done faster and higher quality than a low-budget or no-budget project -- at least, assuming that money is put into the developers and their infrastructure, and not management bonuses -- but there are limits.
Second, it is possible to set quality goals so high that it will take more time than exists in the known universe to actually complete the software, even given an unlimited budget. Virtually all software has bugs. There exist a few rare, serene oases where there are actually no known issues, but that's often because of a strictly limited feature set. The only truly bug-free software is that with no features.
Or, as Scott Rosenberg so aptly puts it...
Software is hard.
Brooks' Law(s)
The idea of throwing people at a project to speed it up is as old as the hills. As much as some of us may mock the notion, it does work to some extent in more tangible disciplines: you can build a house faster with a crew of 20 men than you can with a crew of 10, and faster still if you have a crew of 30 men!
IBM Project Manager Fred Brooks first made the observation that this didn't work in the programming world in the early 1970s, which led him to write his watershed book, The Mythical Man-Month. In it, and its sequel, No Silver Bullet, he made many important observations about the nature of software development. Perhaps his most legendary observation is quoted as Brooks' Law:
Adding human resources to a late software project makes it later.
While even Brooks views this as an oversimplification, it is a useful adage nonetheless. The more people you add onto the project, the more factors exist to slow it down. You have to train the person, account for their opinions, ensure they understand ideas, ensure others understand their ideas, and so forth.
To put it another way, if you have six Inuit-speakers explaining how to build an igloo to the one Spanish-speaker, you can imagine the compounded confusion that can easily occur: conflicting and confusing instructions to the Spanish-speaker, the need for additional people to comprehend the nature of the misunderstandings, and just plain old "too many cooks in the kitchen."
Now, it is sometimes possible to speed up a project by adding people to it, but it requires precision planning, carefully organized collaboration, and effective practices to facilitate all this. Putting all that together requires...guess what?...TIME!
In response to this, many well-meaning project managers, developers, and observers will suggest a whole host of "magic" organization techniques and methodologies that will fix everything and make the team get along. Thankfully, Fred Brooks has something to say about this too...
There is no single development, in either technology or management technique, which by itself promises even one order of magnitude [tenfold] improvement within a decade in productivity, in reliability, in simplicity.
Every project, and every team, is unique. Nothing works for everyone. Or, as Brooks observed, there is "No Silver Bullet".
In other words, absolutely nothing exists, nor can exist, to make software get done faster than it "wants" to. You can throw everything imaginable at it, and it will still take its own time.
(For more information, go read The Mythical Man Month and No Silver Bullet by Fred Brooks. There is a TON of good information therein!)
Don't Underestimate the Foundation
If you're observing a project in its early stages, you also must account for one more critical, if invisible, piece of the project: the foundation!
For a programming team to be effective, they must have the tools, standards, and methodologies they need worked out, and this requires a lot of time! By way of example, it took me two years of full-time work to hammer out the entire development platform, workflow, and standards for my team at MousePaw Media...and this was after a few years of experimentation. I had relatively little time to write code during the whole of 2017, not to mention the latter half of 2016, as I was getting all that worked out. Sure, my team was able to work during this time period (albeit with difficulty, lacking some of the tools they needed), but it still slowed us down.
Now, despair not! There is no rule that says it will take every project two years to work out their development platform. However, it will take time, and like with software development itself, it just isn't possible to set a deadline on it.
In short, if it seems like the developers are dragging their feet in getting started, they're probably just laying the foundation. That is a long, arduous, and largely invisible process, but it is critical to a project's success!
How You Can Help
By this point, you may be thinking "Oh, crud, this project isn't ever going to be done, is it???"
Fear not! What I have just described applies to every single software project ever, and if you haven't noticed, we do manage to ship functional software! Proprietary companies and open source organizations alike manage to release great games, applications, tools, websites, and operating systems. Software does get done...it just does so in its own weird, unpredictable timeline.
So how can you help? It's actually pretty simple.
Don't nag for the software to get done faster. Instead, put your resources where your mouth is. Adding money and/or manpower to a project will help it along. Ask how you can best help!
If you decide to help out, remember, we're fighting against Brooks Law here, so don't be insulted if your offer to Do A Bunch Of Hard Stuff is turned down...it's not you! You may be most helpful just doing menial coding and project tasks so the developers can focus on the tough stuff.
Above all, help educate others about these concepts! Being nagged by users (and worse, bystanders) is one of the most irritating things we developers face. Help us deescalate complaints and attacks by sharing what you've learned here.
Top comments (2)
This article should be a must read for every developer, especially freelance developers.
Great one Jason!
Good article, Jason! I feel like many managers and product people still underestimate the complexity of software.
I wrote this article helping them (and developers as well):
What's Common Between Opera Houses And Software Projects, or 10 Reasons Software Developers Go Overtime.