Why developers are SO sh*t at estimating!

Luke Garrigan on August 10, 2019

The 10x striver: A short story, about you. Chapter 1 - Planning You're a programmer, you're in the middle of sprint planning ... [Read Full]
markdown guide
 

We also estimate the happy-path, solving if everything goes smooth. When was the last time you said "this should take 5 minutes" and there you are 2 hours later, installing drivers and reading arcane config syntax..

 

Yeah that’s so true, I’ll be honest, it happens far too often in my case 😂

 

Let's break it down a bit. The process looks like this:

  1. Oh, this looks exactly what I want, and sounds easy.
  2. Start reading docs or installing package. At this point we are hooked, and likely to go on.
  3. First hurdle comes, but now we have invested. Why leave now, success is so near. Like when waiting for bus - I won't leave now, I have waited for long, it surely will arrive soon.

The first sign of barrier should make us stop and think. When we branch out to 3 Chrome tabs, following how to configure various deps. Or get a result from a call that is not expected according to the docs.

I'm not saying we should throw up the white flag, but we have to ask ourself: This seems rather 4 hours instead the 10 minutes I hoped. Is it worth the investment, or should I stop now?

 
  • Customer: We want this modification in 6 months, can you do it? (in fact, he wants for yesterday)
  • IT: We could do it in 8 months.
  • Seller: Nah, we could get it in 4 months, top!.
  • Customer: Ok, let's start.

Finally, the project is delivered in a year.

 

Alternative:

...
The project is actually delivered in 4 months.

  • Customer: But it's a sh*ty deliverable.
  • Seller: It's a sh*ty deliverable.
  • IT: But you asked to deliver 8 months work in 4 months.
  • Seller: Can you fix it?
  • IT: Sure, we should fix it in 4 months.
  • Seller: Nah, we could do it in 1 month
  • Customer: Ok, let's start.
 
 
 

You are right, totally right with your article. I'm currently in a position where I have to estimate a lot of projects/solutions and estimating the time is something that makes me crazy. Overall because I could be the person in charge of that project and it's completely possible that I forgot some tasks or constraints.

In my company, we only estimate the effort for coding/configuring phase and then complete the rest of the phases (analysis, design, testing, ...) as a percentage of it. For example, I estimate implementing an API could be 5 days of coding, the testing phase will be 80% of it so we will spend 4 days. The whole task will be the sum of all these phases getting a final time of 9 days for this case.

In addition to this, we set a lot of assumptions to avoid forgetting anything. Think of a web application you estimated creating a simple user interface but that doesn't mean anything to your customer. You need to add some assumptions for limiting the scope like: no CSS rules will be developed. This is something the customer will understand or, at the end of the day, you will have as a shield to defend your position.

 

Thank you for this, I love hearing how people take different approaches to estimate tasks. Your approach is very interesting, we do a relatively similar thing at work although we have Quality Assurance engineers who estimate the effort for testing and that gets bundled on top of the overall estimate.

 

Thanks for this. QA team is a good thing to test the whole product but there are a lot of tests in a project lifecycle. You have to include some effort for unit testing, integration tests, smoke test, user acceptance tests, etc. So, there should be something in the estimation to cover testing saying what kind of test are included in the estimation so you can adjust the percentage of the effort of the team.

Do your estimations include the time QA team will spend in the project? This is something you should consider to include inside or outside your estimation and you should co-work with your QA mate.

 

We take this approach too, but also add another batch of time for iterations. How many times does the mock-up not do justice to the task or a fringe case wasn't caught in the original API design? Those things we take into account for as well when we estimate.

 

I learned that estimations are a value we said based on experience, calculus or any other technique we used. So, if a customer requires an API implementation for their business logic, we will estimate the implementation of a number of services and their complexity, let's say 5 services with medium complexity, and not the final solution - a description of what each of these 5 services will do like a black and white box. In the assumptions section (or any other place), we will define what means "medium complexity" so fringe cases will be considered out of the scope of the current project and we should include a note telling need an assessment to give a proper estimation for that issue, problem, requirement or system.

We, as estimators (that exists?), need to limit what we are estimating and considering. We cannot give the final solution in the estimation so we are not able to include fringe cases if we don't study them.

 

I liked Making Things Happen (aka The Art of Project Management), which despite the title, is a totally BS-free book about overseeing a project. Has great advice about estimation. You can pop it open anywhere your current interests are.

 

In my case, it's because I'm a perfectionist most of the time, and refuse to do ad-hoc solutions. That being said, I tend to always under-estimate the time needed to finish the project.

On the other hand, when I'm laser focused and free of distractions, I seem to be able to summon a huge amount of coding mantra, and produce more quality code that I ever expected to be possible. To paraphrase the last sentence; I sometimes still manage to surprise myself, even after 20 years of coding.

 

I love that feeling, somedays you feel like you could just take any ticket and code it up in a few minutes, embracing that inner 10x.

 
 

Back when I was a wee CS student a professor I respect a great deal told us to estimate the time and then just double it. At the time I think a lot of us only half-listened to all these "old guys". Now that I'm approaching old guy status I accept most of what they said. 🤣

That said, there's a reason why many companies/projects don't have precise, external roadmaps as to what's coming when. It's easier for things to slip than predict a hard date. Better to delay than "ship" something not ready.

I'm convinced it's more witchcraft than science, really.

 

Haha the older you get the more you realise that these old buggers actually made some sense 😄

 

At a previous job, we used to estimate time and log time on our sprint tasks. I am very guilty of underestimating tasks from optimism, pressure, and trying to impress. Eventually I got better by just adding a padding of time from my actual estimates, but if I went over that then I would feel super poopy.

These days where I am uses points and so far it's been far less stressful being constraint to an approximate amount of work than being constraint to a time to be done.

Not that points is the answer but it's been better for me!

 

I've used a point system also but we end up converting the points into days anyway. Like 1 point would be half a day, 2 points is a day, 3 points is 2 days, etc. I don't know whether that's just company-specific, but I like the idea of using points without the conversion to time!

 

A lot of it I'd imagine is company-specific. We don't do straight time conversions because points have to account for initial work, work post PR to address comments, and QA.

It's been an adjustment but it's been pretty good so far.

 

Well it's simple manager don't understand those points they want a date / time it will take. So the easy answer is to convert those points to days and voila.
Been there done that.

 

When it comes to frontend nowadays I feel like I am really good at estimate tasks. We estimate a task and it's risk separately. For example this might be a half a day task but it has a big risk (two days). So it's fine if that task spans for 2.5 days. Also modern frameworks like angular made predictions even easier due to their structured nature

 

Yeah, I agree with this, to be honest, a lot of my dodgy estimations are because of the unknown of the backend, but sometimes the frontend does catch me out!

 

It’s important, that when you need to fix estimates in a team, most probably the root cause lies in management. Estimates are estimates, but this is hard to understand for managers. So, the most important part is to acknowledge that even if estimates get better, the work won’t be done faster.

If that step has reached, based on my experience it works pretty good to use points and fixed length iterations. Once team (and it should alway estimate the whole team) has a good common sense on how many points for a comparable task, you just change the meaning of what a point is, based on how many points have been successfully delivered in the iteration. You dont‘t try to improve the estimations itself.
There will still be times where you will still need a week longer than planned. But that’s the way it is.

Ah, and breaking something up in subtasks can help, but it’s not worth to do that upfront if it’s not clear that the task will be done at all. In that case, simple team complexity estimation is sufficient.

 

I particularly like the part about even if estimates get better, the work won't be done faster, I'll have to use that next time when I'm trying to blame the bad code for me taking so long😄

 

Maybe you have to work on your communication, too. If it’s late, than only because we need to meet our company’s quality goals. 😉

 

I honestly don't feel too much like what you described - I know that I have the tendency to underestimate how long something might take me, so I just always go wa, way higher than that, so that I can be sure that I'll be done in the time I promised. Now, that might sound great, but I honestly believe that that's what cost me my last web development job. They went with someone else, and even though it took them just as long as I estimated it would take me, me saying it would actually take me that long tipped them off.

 

This week I was given the task of solving a bug on the application front-end. It looked simple enough so I said when I asked that you should be solving by tomorrow (Monday). Well, I still have no idea how I'm going to solve it. It is basically some bug around Samsung smartphones' keyboard predictive text that makes the user input act like crazy. I tried a lot of things already, but it seems Samsung was very crafty as to how to turn developer's life like hell. u.u

 

The planning fallacy only occurs when planning for our own tasks, interestingly, us humans take a much more pessimistic approach when estimating for other people.

Not always. I've known a number of cases where bosses, clients, or even co-workers set unrealistic deadlines for others. (You even mentioned a few such cases.)

By the way, I've written about this phenomenon as well, and broken it down into several root causes and how to fix them.

 

Yeah of course, as I mentioned with Elon Musk setting unrealistic deadlines!

What I meant is that if someone from a 3rd party standpoint were to make an honest estimation of time for a particular person to do a task it would likely be more pessimistic.

 

Are you trying to describe me??

I need to improve on this as well. I once estimated a 5 month project could be completed in 1, maybe 2 months tops. Yeah, that didn't work out very well.

Since then I have tried to be more pessimistic when estimating.

I have not got much better at it, but I haven't got worse as least 😊😊

 

😅 that's brilliant, I've never had to estimate an entire project but I can only imagine that's 10x worse, so, so many unknowns!

 

I rather overestimate than underestimate. We use points to estimate in the team. If the estimates are far apart then the lowest and the highest explain their position, after which we estimate again. Of course some stories can take longer based on uncertainty but I don't feel I have to estimate low in order to impress my team.

 

You're an exception William, maybe you're one of those 10x's 😄

 

This is a solid topic. Thanks for writing it! 🙌🏻

I've created a 3 point estimate Excel.
When I need to estimate effort for a development project I use it to get a more realistic estimation.
It even has a risk indicator that adds an extra percentage depending on the risk level, etc.

It basically allows me to write down my best case, realistic, and worst case estimation for a (sub)task.
This is less overdone than a X times multiplier but it doesn't overcome the fact that experience/historical metrics, and the need for objective estimations still remain very important.

 

Biggest problem is usually changing goal posts.

I think I estimate pretty well, but the stuff we agree on at the start is never 100% of what I'm actually end up doing.

But I could imagine, if you're a dev for a corp das does everything waterfall style, and not for a startup, you could get the impression that estimation isn't that of a problem.

 

I think we also often fail to take into account "coordination overhead" - e.g. when my task depends partially on understanding or agreeing an approach taken by someone else working on another task.. or liaison back and forth with a client or designer. Sometimes these chats and slack convos end up taking a decent chunk of time and I think we're pretty bad at factoring it in!

 

Completely, that's something that didn't even cross my mind! Thank you Suzanne.

 

Hey Luke!
Nice write up. I’ve faced this so many times and it’s nice to know I’m not the only one.

Also thought you might Michael Wolfe’s answer on this:
quora.com/Why-are-software-develop...

 

Thank you, yeah you're not alone! 😅

haha, I loved that write up, thank you for sharing!

 

Yeah, I agree!

I do wonder though if, for instance, the person in charge of the project used to be a programmer and they think to themselves: I could do it in half a day.

 

One rule I use is, whenever I give a hardcoded estimate, for example half-day work, I double that to be sure, so it becomes a days work.

 
 

Uf, I'm so glad to have read this. It happens to me all the time.

The tips are great, btw.

Thank you so much ;)

 

Haha that's good to hear, we're all in the same boat!

You're welcome 😃

 

That's why I'm glad at my work we have the full team saying how long a story should take and agree on the work required.

 
 

Oh me too, many companies enforce it though.

But yeah, If I had it my way I'd never estimate again.

 

Absolutely right, Im still struggling with that problem. If I plan to complete a task in one day, it takes 3 days. Even I try my best to follow the plan

 

So true, and not just for developers, this is the case for many many day to day tasks!!

 

Thank you, yeah it's definitely a universal thing! 😃

code of conduct - report abuse