(A version of this post was originally published in full on hackernoon)
Giving time estimates is something we have to do all the time during software development. Whether we estimate correctly or not, it is something we rely on and base decisions around. However, in typical situations, time estimates given by developers tend to fall short and turn into a sort of "fantasy time". In this post, I will discuss why estimating time is indeed a fantasy and ultimately how to get better at estimating how long something will realistically take.
Why are time estimates mostly fantasy time?
In a previous post, I gave an account of the very familiar pattern one often finds themselves in when estimating how long creating a piece of software will take and ultimately, the resulting whirlwind and turmoil associated with giving incorrect estimates or failing to achieve the desired results in the given time frame. So why are time estimates akin to fantasy time?
Software development is subjective to a lot of the road blocks encountered in any creative process
Writing software is much more of a creative process than it appears to be. The programming languages we choose, frameworks, libraries etc. are all technical, but how we implement them, integrate them or manipulate them to solve a problem is inherently a creative process at the mercy of the person tasked with solving the problem. It is the reason multiple solutions can exist for the same problem. Everyone can have the same tools, but the masterpiece each produces is different and subjective to their creative process.
We go through the same motions as any creative endeavor. We have times when the plan is clear and times when we stare at our editor not knowing what to start with or what to write next. We design how everything is connected and keep revising these decisions as requirements change. We consider each user individually and at the same time we consider the collective users and how we might meet their needs. We write a bunch of code one day, then delete all of it another day. We wake up some days confident with what we have done so far and some days we second guess every decision we have made thus far. This goes on until we have a final product. It's a whirlwind of ups and downs.
Time estimates in a creative process are akin to a sort of fantasy time. We hope it will take X amount of time, but at the end of the day, we can't say for sure.
There is no real one size fits all solution
The assumption behind time estimates is that problems that appear similar and are of similar complexity should by extension take a similar amount of time to solve. If you really think about it, if it was the same problem, all one would need to do is plug the same solution. Similar is not the same. Therefore however “similar” problems appear, they are inherently not the same and therefore require modifications to make them work. The time associated with these modifications is not always known or constant as each problem is a new venture.
You probably have a lot of tasks you are working on
Ideally, when you initially estimated the time it would take, you did not factor in external environments such as how the other projects you were working on would still need your attention. When asked for your time estimate, you probably isolated that project from all the rest. What you really meant was “Without other commitments and in a perfectly isolated environment, it could take me X amount of time if all other factors are held constant”. By factors you meant, if program specifications do not change, if no additional requests are made and if you do not have to share your time with other projects.
Changing program specifications midway does not affect time in any predictable fashion
There is an assumption that removing specifications directly means the software can be completed in less time. While this might be true in certain scenarios, I have yet to experience this as truth. Ironically, what I have experienced is adding or removing specifications midway just affects time estimates in the same way; they no longer apply. It does not mean you will finish the project faster. All it means is you have to revise your software and consider how this change will affect the work you have already done thus far. It also means you have to consider what you would need to add or remove to meet the new program requirements.
Things will go wrong and sometimes there is no quantifiable time taken to fix them
Some time during the process, some unforeseen problems may occur and it is entirely possible, it could take days or weeks to figure out how to solve those problems. Problems may come from your computer, your chosen language, framework, dev environment, latest updates released by one of your dependencies, inability to integrate third-party tools properly etc. It could even come from the solution you chose to implement. Turns out, maybe the testing phase revealed it is not as efficient as initially estimated and is therefore not optimal and now you are back to the drawing board 9 days before the deadline. Factoring for potential errors is difficult as you cannot foretell what problems may occur and how long it will take to solve them.
There is a lot of interdependence within the system
When collaborating to write software in a team, it is common to split the tasks between team members. Ideally, each of the tasks can be completed independently, however, in some cases, they could depend on each other. If one part lags, it drags the dependent part with it. Typically, to curb this, it is ideal to simulate the input and write one’s section accordingly as you wait for the actual input. Sometimes, this approach works but in some cases it may be harder to do. For example, it can be difficult to simulate images that require subsequent image processing to remove defects or unwanted artifacts without having an idea what the defects could be. Therefore a delay in completing the first part that supplies input to a second part, may delay the second part as well.
In a team, collaboration is at the heart of writing chunks of software. You may be struggling to fix something and subsequently affect your teammates ability to work on their part, because it relies on you finishing yours. Thereby not only affecting your schedule but theirs too, putting pressure on both of you.
You aren’t good at estimation
It’s easy when others place the time estimate on you and sure you can blame them, but sometimes, even your own time estimations just aren’t good enough. You thought it would take an hour, it took 2 hours. For example, you estimated 2 weeks on a program but were given 1 week. When you missed the deadline, you were given an extra week, however, you still missed the deadline. Therefore, your initial estimation was still wrong. What are we without idealistic human optimism? It is easy to think of oneself as being able to power through the mental drain writing software takes, however, more often than not, mental drain comes coupled with physical drain and the time taken to rest and power up in between is often overlooked. Small things often add up to big things. Taking time to rest, brainstorm every once in a while, are tasks that are difficult to accurately quantify but do matter and can take up quite some time. They may seem trivial when ballparking an overall time estimate, but they significantly affect the time taken to write software.
So why do we still need real time estimates?
If time is all a fantasy why do we still need and rely on it?
Your users have a real-time use for it
Need has a time factor to it. Something needed in the next 3 months does not apply if it arrives in 6 months. Therefore, you may have to plan what is feasible around your user's needs. If you are organizing a marathon and need an app made to track runner’s location, pace, vitals, speed and direct them along the proper route, there is absolutely no use in getting the app long after the marathon is over. It no longer solves the problem. Giving a time estimate to how much time would be needed to create the app can give stakeholders proper time to plan around the deadline or even start the project earlier than intended to meet their user's needs.
Deadlines can help one stay focused
I tend to procrastinate sometimes and when it comes to my side projects, they can suffer especially when my interest has faltered, or when I have become deeply obsessed with a different project, and there is no fixed deadline. Time estimates help me stay focused on a specific task for a certain amount of time. Even though I may procrastinate within the set time frame, at the back of my mind, I am aware there is a project that needs to be completed and I will carve out time to get it done.
Time estimates do aid in overall planning
I don’t think people ask for time estimates because they enjoy putting people under pressure, if they do, you should probably reconsider your work environment. Time estimates help others plan their part of the project around a given time frame. A lot of software is integrated or built to work hand in hand. You can’t launch a product if part of the product is missing a vital area. Since the entire system is the sum of its parts, its total time is affected by the time estimates of each part. With near approximate time estimates, the project manager can decide which parts can be created in parallel, which parts can only start once a different section is complete and overall how this affects when they can complete the project.
We tend to understand the reason for time estimates and why we sometimes fall short of how long we think something will take. So the next step would be to re-evaluate how we estimate software design.
With that in mind, how can I get better at estimating?
I still catch my breath when someone asks me, “How long will this take?” Why does this question cause me so much anxiety? Well, because at the back of my mind, I am aware of all the bits that go into time-estimates as well as all those we typically overlook. However, with time I have come up with a bunch of steps I take to break down how long I think a certain task will need. I haven’t nailed it down to the second, however, with time I have gotten better at giving better approximations.
Give a rough estimate and ask for time to break down the problem and come back with a better estimate
Nowadays, I give an initial rough estimation based on the mental gymnastics I can perform at that time, however, I also ask for some time to come up with a better time estimate outside of the meeting environment. Doing this often allows me to sit with the scope of the project, break it down into chunks and identify what I think each chunk will take. Once that is done, to each chunk I add various miscellaneous time estimates, let's call them “Exploratory time”, “Bug fix time” and “Pause time”. These are not industry terms just personal terms.
“Exploratory time” is generally the time I estimate I will take to consider my options in determining how to implement that particular chunk. Great thing about writing software to solve problems is there can be multiple solutions to one problem. Ironically, this can also sometimes a problem in itself. Too many options can lead to analysis paralysis when trying to determine which approach one should use to solve a problem. Personally, when choosing options, I do take time to consider how each option could solve the problem, their flexibility, advantages, disadvantages, future maintainability and general interdependence. I also consider, if I choose to change my approach later on, what will be the cost to switch to a different solution and how difficult will that be? I have found that adding an exploratory time also allows me to research without feeling like I am costing the overall time, because I accounted for time to explore my options in depth.
“Bug fix time”, is generally the time I estimate it will take me to find and fix bugs in a particular chunk. When writing software that is similar to something you have implemented previously, this time estimate might not be necessary. I find having this time is a safety net for when I get stuck on a bug that I am unsure how to solve. It isn’t easy to estimate how long it will take to fix a bug, but estimating there will be no bugs at all is worse. When bugs do pop up, urgency and subsequently stress set in because you had not budgeted for time to fix bugs. This has happened to me in the past, and I didn't like it. However, expecting things to go wrong and having time set aside to solve them can leave you feeling calm and still on time in the grand scheme of things.
“Pause time” is generally a nice way of saying break time. Time to step away from the software, take a break and refresh. Each chunk gets its own pause time because not all chunks are the same, thus their mental and physical drain can be different. The break I need after fighting with css may not be the same break I need after writing tests.
Generally, consider your habits when writing programs, how long did those small things you overlook take? Find a way to term them and add them to your general time estimates.
Estimate sections you are familiar with differently from those you are unfamiliar with
Familiar things are easier to estimate because we have done them 10, 100 or even 1000 times. We know the general flow and how long it takes us to work on them. However, even things that seem similar are not always entirely similar in every aspect. Nonetheless, minor variations are easy to handle because they only deviate so slightly. Therefore, their estimates can be close enough to problems previously encountered.
Things we are unfamiliar with are harder to estimate because we have never done them before. Maybe we have heard of them and know the general approach, but we have never actually sat down and implemented them. So how do we estimate something we have never done before? One approach would be to use estimates based on other’s experience. When we do not know how long something will take, we could either make a speculative guess, or rely on those with experience on the matter and ballpark our estimates based on theirs. It is not ideal, but it is better than magically producing a number. There are a lot of unknowns that come with using other people’s estimates on how long a task should take. For one, time estimates on known matters are subjective and can be influenced by experience. Someone who has designed and written hundreds of application code that queries databases may take only X amount of time to implement something similar when compared with someone who has never done it before. Knowledge in a certain domain also influences how long one takes to write code in that domain. We are all different.
Personally, when it comes to estimating the time I will take to work on something I have never done before, I tend to take a senior’s time estimate then double it. By doubling it, I can have time to implement it twice in theory. Once to get it done, and I use the extra time to review my understanding of the task, setting myself up for the next time I may have to implement a similar task.
Remember to take your personal habits into account
People are different and have different habits. How we tackle problems, how we brainstorm and eventually how we assess and handle situations is also different. Some people prefer to tackle the same problem for hours on end then take breaks, while others prefer to concentrate for a couple of minutes with short breaks in between. Some would prefer to work on mentally tasking projects during the evening while others prefer working on such tasks in the morning hours of the day. Some people can’t work on an empty stomach while others can’t work on a full stomach. Nonetheless, whatever your preference is, consider how your habits affect your performance and take them into account when estimating. Will you need more breaks in between or not. Do you need to devote ample time to doing research or not?
Keep your team informed, and if revisions are necessary, consult them
All in all whatever happens, always keep your team in the know. I do realize this is not a direct way of getting better at giving time estimates, but constantly communicating with your team about your efforts and where you are in the timeline can allow you and your team to always be on the same page. Remember writing software and launching a product is a team effort, you may not think they are affected by revisions to your timeline, but they probably are. Having proper estimates that account for delays on your side, could help your teammates keep their schedules intact. This way, the stress to beat the deadline does not fall on them when anything changes on your side.
Lastly, learn from your own estimations
Every time you work on something, note how long you thought it would take, and how long it actually took. Then compare how far off your estimation was. Were there outside factors that caused this value to deviate exceptionally, and if there were what were those factors.
With time, practice and learning from your own experience, you should get better at estimating.
Hope you enjoyed reading thus far! Let me know in the comment section.
Top comments (10)
Great article, truly.
The problem is, in some places, developers know that their managers expect the estimations to be less than they really are, but the managers don't know, that developers usually give the shortest UNREALISTIC estimations, without considering ANY hurdles that might come up on the way.
So, a developer thinks it would take him 6 months to finish a project, but knowing that the manager would not like that, he says 5 months. The manager though, would like the project even sooner, so thinking they are doing their job, the manager reduces the 5 months to 4 months. BUT, in REALITY, it would take the developer at least 9 months to finish the project.
So in such case, the developer need to work for the last 5 months (month 5 to 9) under pressure from their management that they are OVER-deadline...
This happened to me, but it was WORSE than this. I estimated 6 months with 3 developers (incl. me), management reduced it to 5 months. Few days after the estimation, the other 2 developers were shifted onto other projects and I was left alone, and somehow, they thought the estimation should not change.
That was 2 years ago, and I am still working on that project!
This is a tricky issue because often managers are under the same pressure.
While a poll conducted (and cited by Steve McConnell in his "Software Estimations: Demystifying the black art" book) said that managers inside companies prefer you to overestimate and come in early rather than underestimate and come in late, there are often many expectations in terms of time to market, etc that make this decision hard.
It's even worse if you're trying to win a RFP for some project. Some people will lie and say they can do something in 3 months when your estimate is 9 months. This can cause sticker shock and make you lose the business. So it becomes quite tricky. Customers often don't want to know what it will really cost.
But ultimately I've found that its best to stick to your guns and estimate.
The way we do it is we ask everyone to make their best estimate. But if anything takes longer than 5 days they have to break the issue into parts so that no issue takes longer than 5 days (this helps keep it real).
Then we grab everyone's estimates and multiply them by 2. This, almost always, results in an reachable goal. And we deliver on time (but rarely early).
Some people in the team overestimate others underestimate but because of the law of averages we can hit pretty close to the target.
We've probably lost some deals because our quote is higher than other people's and takes longer, but if the other side is so concerned with price that they would want to artificially push it down (without removing features) than we're maybe not a good match anyways.
That being said we've recently been subjected to some "emergency projects" where you save a company in distress and we've been amazed at what we can do under pressure. Like doing a 4 month project in 2 months and then even delivering that early.
But these projects weren't from scratch.
The 5 days trick seems interesting.
It is virtually impossible to give an estimate on a big project, especially if it has parts that are unseen/unknown to most or all the members of the team. How can you estimate how long it would take you to do something you have never done before? Even with the doubling technique you mentioned, although it is a good one, but still would not work in some cases..
I might just write an article of my own to discuss this issue further..
Well, it's also important to push for an MVP. If it's that ill-defined, then maybe lets focus on the first phase first and estimate the second later?
The larger the project is, the more likely your estimates will be off.
Definitely, the larger the project, the likelier the estimates are off. At that point, its easier to focus more on the time estimates for the individual components, and try divert whatever time "saved" to parts that need more time than budgeted. Easier said than done though!!
Yes it's problematic because some organizations want to know the overall cost before embarking on a multi-year project. That's a hard number to pin down often. It's maybe easier to set a burn rate and a 5 year horizon for those cases rather than trying to pin down the cost from a early set of features that may change in the intervening years.
This is exactly true.
I really like this write-up, some of the causes are things I had never considered before (I certainly underestimate fatigue when estimating).
I think I'd argue we actually do not need "real time estimates" though. I'm (perhaps foolishly) still a believer in the idea of story point estimation, and the predictive qualities that approach brings. Really what a business wants from estimation is the ability to plan/forecast, and (if done well) abstract story point estimates do that. What happens when people get hung up on "real time" estimates it gets tricky because of things like differences in skill or productivity level. You can have a task that two developers (say one who is very experienced and one who is very new) agree entirely on what work needs to be completed, but it will take the more experienced dev much less time than the less experienced dev. Abstract story point estimates account for this (instead of measuring hours, you're measuring "inherent complexity", and then "on average our team completes X story points every cycle" -- ie team velocity) which still gives that predictability for the business. The problem I've seen is that often teams get hung up on equating story points to units of time, which completely neglects the value of story point estimation. Every job I've had there's been members of the team who fall into this trap (that story points == units of time), and every single time I've seen it lead to poor estimates.
The last section "Lastly, learn from your own estimations" is also excellent advice -- if a task ends up being dramatically more complex than the estimate would have implied, it is so worth the time investment to reflect on why that was.
Great write-up!
Thank you so much for the feedback and your thoughts on the post!
I like the story point estimation technique and when done properly you do get better estimates especially in terms of complexity and proper pacing. I definitely agree, it doesn't equate to units of time, and to force an equation or a relationship between the two just ends up completely missing the mark.