One of the most difficult challenges in software development is to come up with proper estimations related to a project.
In traditional Engineering...
For further actions, you may consider blocking this person and/or reporting abuse
"...multiply the estimations provided the developer by a factor of 1.5 (better if your can multiply by 2 or even better 3)". To summarize: The higher your estimate multiplication factor is, the better your estimates are.
Please do not add 'buffers' to your estimates, and especially do not multiply the estimates you are given "by a factor of 1.5 (better if your can multiply by 2 or even better 3)". If you are unsure of how long something will take, ask for help.
As a project manager, I don't expect developers to provide 100% accurate estimates, but it is my job, not yours, to evaluate the risk in those estimates. I'll typicaly ask for two numbers: first, a best-case estimate in which everything goes perfectly well (which will be the smaller of the two numbers), and an estimate of the time needed to guarantee that something will be done (the larger of the two). The probable duration is between these, and I'll use something like PERT to calculate the most likely number.
If you are padding your estimates, and I don't know about it, those padded hours are included in my own estimates, to which I'll be adding a 15 to 20% risk buffer (NOT 3x), especially if the tasks I've asked you to estimate are on the critical path for the project.
Let's say that you, the developer, thinks a module might take two weeks to complete, but you add a week as a buffer, for a total of three. Assume that I take that 120-hour estimate and don't reduce it. How long will it take for you to do the task when it comes time to do the actual work? 120 hours. That's what's in the project schedule. Multiply that by the number of developers on the team and the number of high-level task, and very quickly the estimate can grow much larger than it should be.
What's the harm, though? If ten developers pad their estimates by 80 hours each, I'll end up with an extra 800 hours. At an internal cost of $150/hour/dev, that's $120k extra in the budget (plus of course the extra time). If, toward the end of the project, I, as the project manager, realize that we're coming in significantly under budget and under schedule, I might proudly walk in to my VP's office and announce that we're coming in early and I've saved the company $150k. Everyone's happy, right?
Nope. I might get fired if I do that too often. Companies run projects in order to make money. Granted, there are some required things we have to do, but basically each project run is anticipated to contribute to the top line. The more projects I can run, the better my top line looks. Your bad estimating just tied up $150k that I could have used on another project...you've bascially screwed shareholders out of an increase in their investment, and, if things are tight, might have resulted in a down quarter.
It isn't the developer's job to fudge the estimates. Leave that to the project managers, who should at least have the experience to fudge them correctly.
Great reply. Agreed mostly, however when you are working with over-confident developers (specially good ones too early in their careers), sometimes things get out of hand. The developers should try to give proper estimates. I was kind of coming up from a project manager's point of view who already has a software development background. The multiplying by 2 is stretching the fact a little too much, but was more trying to make a point. If you really have a great team of experienced developers then ofcourse you don't add too much of a buffer. But thanks for the observations and comments - this way we all learn more.
Any padding, whether by project managers or developers, is a problem. where are you getting "15-20% but not 3x" from? All padding is arbitrary - and it's an arbitrary number multiplied by an estimate that is itself based mostly on ignorance.
You're multiplying two unknowns and making decisions and promises against the nonsensical result. The entire approach of time and cost estimates is wrong-headed.
I can't argue that the way we estimate software projects is a problem, but it's the way most non-trivial projects are managed. We normally don't call the extra time padding...really it is a buffer to protect what really IS important in most projects, which is the completion date. That date often is set by product or marketing management, and if it is missed, the result can be as bad as losing market share in a quickly moving market, or missing revenue trargets for a quarter or more.
Typically projects are broken down into work units and laid out as a connected acyclic graph that relfects their dependencies. Estimates are applied to each, both for an optomistic and a pessimistic schedule. The graph will usually have many paths that converge to a delivery in the future. One or more of those paths will be very tight schedule-wise...it has the least tolerance for late starts on tasks such that ANY late start will cause the delivery date to slip. This path through the project is called the 'critical path' and is closely managed since it directly affects the end date.
The critical path is determined by analyzing start and end estimates across the entire graph; we assign a value called 'slack' to each task that reflects how late that task can start before it causes the next task to slip. For example, if I have a task that takes five days, but the following task converges into a path that doesn't need to start for another ten days, I have five days of slack, meaning that I can delay the start of the earlier task by that much without delaying anything else. In some cases I can reassign the resources on the shorter task to another task that needs help.
Since the critical path so directly drives the delivery date, and the delivery date i often a hard stop, when planning a project we'll add a risk buffer to the critical path (and thus the entire project). The critical path has little or no slack, and so that extra time tends to be arbitrary. It's like an insurance policy...I'm happy to pay $1,500 to protect against missing a delivery date that might cost me $10,000.
A method that isn't quite so arbitrary is called Critical Chain, in which slack times across each unique path are gathered into mini-buffers at the end of each of these feeder paths, then rolle into a buffer that sits prior to the delivery date. Managing a Critical Chain project is a little easier than a Critical Path project since we can examine the size and burn rate of the buffer at the end to get a sense of how the project is going overall.
Both methods still require estimates for early and late starts. The project team, and especially the project manager, need to provide reasonable estimates and be able to recognize estimates that are outliers.
This description of project management graphs doesn't begin to address the fact that estimates of non-trivial, unique software-creating tasks are a total wild guess when given at the start of a project. The only way to provide anything close to a reasonable estimate is by knowing as much as possible about everything involved (and even then it could be way off) and that state of knowledge is never present at the outset. You never know less than you do on Day 1 - therefore your estimates only ever improve as you go on in a software project.
This is the core principle of Agile and the reason why big up front designs and their resulting estimates are disasters. How many software projects need to suffer the same fate before everyone is finally willing to admit this?
You very succinctly identified (and ignored) the crux of the problem:
How can a date be set by a team that has no knowledge of the process involved in completing the tasks that must be completed by that date? If I give a contractor a completion date of
2 weeksto build my house, I'm the one to blame when the contractor misses the ill-informed deadline.PS this isn't even taking into account user/product owner feedback, which can (and should) entirely re-steer a given project, making all initial estimates less than worthless and indeed harmful.
I'm guessing you haven't worked on any non-agile projects in your career. I like agile methodologies -- I'm a fan of the Rational Unified Process in particular -- and there is certainly a large class of projects where agile makes a lot of sense. A web site? Sure. A back-end equities trading platform? Not so much.
That said, most of my development and consulting career was spent in Fortune 500 companies (primarily financial services, some healthcare) where projects spanned 12 to 24 months, and exactly zero of the projects I was involved in for those clients used agile methodologies.
Day 1 estimates are not wild guesses, as you put it. They are informed by experience (of both the group and the developers), vetted by management, and in my experience are typically quite close to the actual effort. A large project usually starts with a lengthy requirements gathering stage, during which the architecture and functionality of the system are agreed on by both the business and the technology team.
Project planning at this scale is necessary to coordinate resources, equipment, schedules, marketing, QA/testing, purchasing, deployment, and so many other things. Individual contributors (especially devs) are expected to be capable of providing reasonable estimates; if they can't, they won't be working for long. I've pulled developers from projects and sent thm back to the bench for training when they've shown that they haven't learned how to do estimation yet.
And to that point, you can't just plop a new dev into a project and expect them to be able to provide proper estimates; it's a skill, just like coding, and it takes time and experience to get good at it.
To your last point on feedback from product management changing the project, that's not how it happens at this level. Once the project plan and requirements docs are signed off, there are no further ad-hoc changes. If the product team wants to make a change, the request goes through an approval process (I've often seen them denied) that might require new estimates and a new schedule that must also be signed off on by all parties if the change is significant.
I'm absolutely not saying that agile doesn't have its place, and in fact many of the larger projects I've worked on probably could have been done more quickly if parts of them were done that way. I am saying that not all projects are appropriate for agile and require significant planning and management effort. As a developer I think it's a mistake to brush off non-agile methodologies as too hard to estimate, or having too much overhead, or taking too long. Doing so is to shut off a large, lucrative chunk of software development opportunities.
BTW props for moving to the countryside, I'm not that far away and spend quite a bit of time in the summers down in LC and Westport. Beautiful place to live.
Interesting article! This is a topic I really like.
I recently implemented this open source web application that is Jira Project Estimator: github.com/sebaherrera07/jira-proj...
It allows people to get estimates for project finish dates, not before the project started, but once the team is making progress based on their velocity and the remaining tickets story points.
I would like to share it with anyone who might be interested in it and also thank for any feedback!
After more than a decade of experience, I'm still bad at estimating. This post matches my best understanding of how they should be done though. The goal is small chunks, each of which is well-understood enough to give someone instructions on without them having to improvise.
Another thing is to let the stakeholders know as soon as possible if part of the schedule is slipping.
One problem I've had though, is that sometimes the tasks I've broken something down into turn out to not be the right direction to slice things. So I've got tasks A, B, and C, but a week later I've got half of A, 75% of B, and a quarter of C done.
It is nearly impossible to get everything right in the initial stages of a project and the situation that you mention is not unusual. However this is the reason we now have iterative software development process (agile) in place with small sprints. With shorter sprints, with time as you get more aware of the requirements of the project your estimations get better.
I like the idea of agile, but when done in a formalized way it seems to do more harm than good. I've done scrum in a few places, but never enjoyed it or found it to be very helpful. It seems to be designed to keep people from getting a chance to catch their breath or get into their work. I suppose there may be some types of project where it works well, or some people who are good at it.
Basecamp has recently posted some videos about their process with six week iterations, which looks interesting.
While I kinda agree on this method, there is one particular point worth emphasizing on: once your tasks are laid out, you need to ask two questions.
Just sit there and challenge everything again and again. Don't let anything in oversight. And certainly do not focus on the main features since there is an increasing number of pain in the ass coming to get you (GDPR to pick a recent one, but also password reset, email validation, user preferences, email unsubscription, ...).
Regarding the things you have never done, you need to do preliminary PoCs to estimate how long it's going to take. If you can't sell this, then put those tasks ahead in your planning (the riskier the task the first it should be done) and be ready to re-adjust scope or deadline.