The Problem With Estimates

I’m a big proponent of Agile (mostly XP; I’m mostly anti-Scrum) and I’ve contributed some to the #noestimates “movement”.

I don’t really mean that nobody should ever estimate anything. I mean that I’ve never seen useful (fine-grained) estimates anywhere. Here are some of the problems with estimates that I’ve seen frequently:

  1. We’re not good at estimating how long things will take. We’re usually optimistic about how quickly we can get things done, and almost always miss thinking about things that will take more time. I’ve never seen a case where a project is completed more quickly than estimated. I’ve only rarely seen fine-grained (story-level) tasks completed more quickly than estimated.
  2. Management asks for estimates and then treats them as deadlines. The team then learns to inflate their estimates. Then management learns to reduce the estimates they’re given. Given fudge factors in each direction, the estimate no longer has much reliability. Even if you’re using story points, the point inflation/deflation leads to less consistency and therefore reduced reliability.
  3. Estimates that are given are negotiated down, or simply reduced. This leads to the question why you’d ask for an estimate and not take the answer provided. If you’re not going to listen to the answer, why are you asking the question? This is probably the craziest one on the list — given my first point, increasing an estimate would make sense. Reducing the estimates is just magical wishful thinking.
  4. Plans change and work is added, but the deadline (presumably based on the estimates) is not changed to correspond with the extra work involved. So again, you’re not actually even using the estimates that were given.
  5. Management dictates deadlines arbitrarily, without speaking to the people who will be doing the work. Spending time estimating how long each task will take when the deadline is already set is completely pointless.
  6. Almost every deadline is complete bullshit, based on nothing. Often the excuse is that marketing needs to know when something will come out, so that they can let people know about it. Why they need to know the exact release date way in advance, I’ve never been able to figure out. Many people intuitively know that the deadlines are bullshit, and will likely be allowed to slip. The only exception to bullshit deadlines I’ve come across are regulatory deadlines. (I know there are a few other exceptions out there.)
  7. Estimation at a fine-grained level isn’t necessary. Many Agile teams estimate using story points, and determine a conversion from story points to time based on previous empirical data. This is fine, except that the time spent estimating the story is wasted time — counting the number of stories almost always gives the same predictive power. Teams tend to get better at breaking up stories over time, so that they’re more consistent in size, so this becomes more likely over time.
  8. The ultimate purpose of an estimate is to evaluate whether the proposed work will be profitable, and therefore worth doing. Or to compare the ROI (return on investment) between alternative projects. But to know that, you’ll have to know what value that work will provide. I don’t believe I’ve ever seen that done — at least not at a fine-grained level. Usually by the time you’re asked to estimate, the project has already gotten approval to proceed.

I’ll note that most of these pit management against the team, instead of working together toward a common cause. Most of the practices also lead to seriously demoralizing the team. And most of the time, the estimates aren’t really even taken into account very much.

My advice is to first understand the value of a project before you consider estimating the costs. The estimation at this point will be very rough, so make sure that you have a very wide margin between the expected value and the rough estimate of the cost. If you’re pretty certain of the expected value, I’d probably want to make sure I could still be profitable even if it took 3 or 4 times as long to complete as the rough estimate. And if there’s uncertainty in the expected value, much more.

Another way to mitigate the risk of throwing money at something that’s not going to have positive ROI is to reduce the feedback loop. Order the work so that the tasks are ranked in order of value to the customer. (Realistically, you’ll have dependencies of tasks to worry about, and should consider effort involved too.) So work on the most valuable feature first — get that out into production as soon as possible. Once that’s done, you can assess if your ROI is positive or not. Keep iterating in this fashion, working on the features that will provide the most value first. Keep assessing your ROI, and stop when the ROI is no longer worth it, compared to other projects the team could be working on.

At a fine-grained level, if you’re using story points, I’d ask you to do the math to see if just counting the stories would be as effective at predicting how much will be done over time as using the story points. If so, you can save the time the team spends on estimating stories. I’d still recommend spending time talking about stories so that everyone has a shared understanding of what needs to be done, and to break stories up into a smaller, more manageable size — with one acceptance criteria per story. Also take a look to see if empirical average cycle time (how long it takes a single story to move from start to finish) might provide you the predictive power just as well as estimates. (I.e. is it bandwidth or latency that really provides the predictive power you’re looking for?)

And don’t forget Hofstadter’s Law: It always takes longer than you expect, even when you take into account Hofstadter’s Law.

Estimation Isn’t Agile

I don’t believe that estimation should be part of any Agile practice.

One of our managers recently mentioned that we hadn’t met the “contract” that we had “committed” to in our last iteration. This was complete nonsense, because A) we hadn’t made any such commitments, and B) we completed many more story points than the previous iterations (and without inflating story points).

estimates-as-deadlines

But her language made me come to several realizations. First and foremost, estimates are contracts. Sure, they’re not supposed to be treated as commitments, but they almost always are. And what does the Agile Manifesto say about this? It says that we should value customer collaboration over contract negotiation, and responding to change over following a plan. So it’s pretty clear that treating estimates as commitments is completely counter to the Agile values.

Why does this matter? What benefits do the Agile values bring us? I think the biggest benefit they bring is changing the way that we work, so that we can better deliver value to our customers. Without Agile, we’d just keep working the way we’ve always done things. And that didn’t seem to be working out so well. If we follow the Agile values and principles, at least we’ll have a fighting chance of improving our ability to deliver value.

Ask yourself — have you ever seen a software development project that was on time and on budget? Where the estimates were spot-on? Of course not. For one, we’re terrible at estimating. For another, our plans change — either from external factors, or from what we learn as we go.

Improved Estimation

To me, Agile is also about facing reality — and embracing it. It realizes that we’re terrible at estimating. It realizes that plans change. Most Agile methodologies have some tricks to counteract Hofstadter’s law. Generally, we use relative story points instead of hours, and then use an empirical factor to convert points to hours.

When this works, it is better than any other estimation I’ve ever seen. But it doesn’t work very often. People have trouble with relative estimation. How do you set the basis for what a point means without relating it to actual hours? Affinity estimation could work, but then you have to remember what the basis was. We’ve got a large distributed team, and when we tried this, we couldn’t all remember what the basis was.

Since we couldn’t get affinity estimation to work, we tried changing to perfect hours (only powers of 2). But then people thought of them as time. When we took longer than the estimate on an individual story, managers and team members thought we were taking longer than we should have. So our estimates ended up causing problems.

What Can We Do Instead?

Managers want estimates so that they can have predictability. They want to know when new features will be available. Is there a better way to get what we need?

I believe there’s a better way — prioritization. If you work on the most important thing first, then the most important thing will get done first. We should always be working on the next most important thing.

What if there’s more than 1 thing that’s most important? Then you’ve failed. You’ve failed at logic if you can’t understand that only 1 thing can be most important. You’ve failed at prioritizing the customers’ needs. You’ve failed at project management.

Arguments

1. Why can’t you just tell us how long it will really take?

Because we don’t know. Because we can’t know. This is the first time we’ve ever implemented the functionality you’ve asked for. If we’d done it before, we’d just use that existing code. As Glenn Vanderburg pointed out in his excellent talk on Software Engineering, we’re not building software, we’re architecting it.

2. But we have to tell our customers what to expect.

Why? Is the product so bad that you can’t keep customers around without leading them on with future enhancements? And why do customers need exact dates? A general roadmap telling them what the priorities for upcoming features should be sufficient.

3. But we have to have messaging about new features.

OK. Then send out that messaging once the feature has made it to Staging. Or even after it’s been rolled out to Production.

4. But we’ve promised these new features to the customers by this date.

Ah, so you’ve made promises to the customer that you don’t have control over. Have you ever heard of “under-promise and over-deliver”? That’s how you create happy customers. Yet you’ve done just the opposite, haven’t you? And then you want to blame someone else.

Risk

Estimates are risk. But the risk doesn’t come at the end, when the estimates are shown to be incorrect. The risk was in asking for the estimates in the first place, and placing trust in them. Don’t do it. Don’t promise things that you can’t be sure of.

Embrace this reality. Embrace this uncertainty. Always focus on what’s most important. That’s how you make customers happy.