The Problem With Estimates

I’m a big proponent of Agile (mostly XP; I’m mostly anti-Scrum) and I’ve contributed some to the #noestimates “movement”.

I don’t really mean that nobody should ever estimate anything. I mean that I’ve never seen useful (fine-grained) estimates anywhere. Here are some of the problems with estimates that I’ve seen frequently:

  1. We’re not good at estimating how long things will take. We’re usually optimistic about how quickly we can get things done, and almost always miss thinking about things that will take more time. I’ve never seen a case where a project is completed more quickly than estimated. I’ve only rarely seen fine-grained (story-level) tasks completed more quickly than estimated.
  2. Management asks for estimates and then treats them as deadlines. The team then learns to inflate their estimates. Then management learns to reduce the estimates they’re given. Given fudge factors in each direction, the estimate no longer has much reliability. Even if you’re using story points, the point inflation/deflation leads to less consistency and therefore reduced reliability.
  3. Estimates that are given are negotiated down, or simply reduced. This leads to the question why you’d ask for an estimate and not take the answer provided. If you’re not going to listen to the answer, why are you asking the question? This is probably the craziest one on the list — given my first point, increasing an estimate would make sense. Reducing the estimates is just magical wishful thinking.
  4. Plans change and work is added, but the deadline (presumably based on the estimates) is not changed to correspond with the extra work involved. So again, you’re not actually even using the estimates that were given.
  5. Management dictates deadlines arbitrarily, without speaking to the people who will be doing the work. Spending time estimating how long each task will take when the deadline is already set is completely pointless.
  6. Almost every deadline is complete bullshit, based on nothing. Often the excuse is that marketing needs to know when something will come out, so that they can let people know about it. Why they need to know the exact release date way in advance, I’ve never been able to figure out. Many people intuitively know that the deadlines are bullshit, and will likely be allowed to slip. The only exception to bullshit deadlines I’ve come across are regulatory deadlines. (I know there are a few other exceptions out there.)
  7. Estimation at a fine-grained level isn’t necessary. Many Agile teams estimate using story points, and determine a conversion from story points to time based on previous empirical data. This is fine, except that the time spent estimating the story is wasted time — counting the number of stories almost always gives the same predictive power. Teams tend to get better at breaking up stories over time, so that they’re more consistent in size, so this becomes more likely over time.
  8. The ultimate purpose of an estimate is to evaluate whether the proposed work will be profitable, and therefore worth doing. Or to compare the ROI (return on investment) between alternative projects. But to know that, you’ll have to know what value that work will provide. I don’t believe I’ve ever seen that done — at least not at a fine-grained level. Usually by the time you’re asked to estimate, the project has already gotten approval to proceed.

I’ll note that most of these pit management against the team, instead of working together toward a common cause. Most of the practices also lead to seriously demoralizing the team. And most of the time, the estimates aren’t really even taken into account very much.

My advice is to first understand the value of a project before you consider estimating the costs. The estimation at this point will be very rough, so make sure that you have a very wide margin between the expected value and the rough estimate of the cost. If you’re pretty certain of the expected value, I’d probably want to make sure I could still be profitable even if it took 3 or 4 times as long to complete as the rough estimate. And if there’s uncertainty in the expected value, much more.

Another way to mitigate the risk of throwing money at something that’s not going to have positive ROI is to reduce the feedback loop. Order the work so that the tasks are ranked in order of value to the customer. (Realistically, you’ll have dependencies of tasks to worry about, and should consider effort involved too.) So work on the most valuable feature first — get that out into production as soon as possible. Once that’s done, you can assess if your ROI is positive or not. Keep iterating in this fashion, working on the features that will provide the most value first. Keep assessing your ROI, and stop when the ROI is no longer worth it, compared to other projects the team could be working on.

At a fine-grained level, if you’re using story points, I’d ask you to do the math to see if just counting the stories would be as effective at predicting how much will be done over time as using the story points. If so, you can save the time the team spends on estimating stories. I’d still recommend spending time talking about stories so that everyone has a shared understanding of what needs to be done, and to break stories up into a smaller, more manageable size — with one acceptance criteria per story. Also take a look to see if empirical average cycle time (how long it takes a single story to move from start to finish) might provide you the predictive power just as well as estimates. (I.e. is it bandwidth or latency that really provides the predictive power you’re looking for?)

And don’t forget Hofstadter’s Law: It always takes longer than you expect, even when you take into account Hofstadter’s Law.

Website Checklist

While creating a web page isn’t too difficult, there are a lot of moving parts involved in creating an effective website. We’ve written up this checklist as a guide for anyone who has decided that they need a website.

This list is for a basic static website. A web application will require significantly more effort. We’re working on a separate checklist for web apps.

Overview

  • Determine your goals
  • Develop a business plan
  • Register a domain name
  • Set up DNS hosting
  • Set up web hosting
  • Design and develop the site
  • Deploy and test the site
  • Market your site
  • Set up additional services
  • Maintenance

Goals

  • What do you want to achieve from having the site?
  • What “call to action” do you want visitors to take?
  • How will you measure success?
  • What will be the focus of the site?
    • Info (“brochure”) for an existing business
    • Blogging
    • Sales
    • Community
    • Web app
    • Mobile app

Business Plan

  • Who is your target audience?
  • Marketing
    • How will you get them to come to your site?
    • How will you get them to buy something?
  • Who is your competition?
    • How do you compare to them?
    • How do you differentiate from them?
    • What is your niche?
    • What makes your site better than theirs?
  • How will you make money?
    • Bringing people into brick and mortar business
    • Advertising
    • Periodic billing/subscriptions
    • Selling goods/services
    • Get bought out
  • Pricing
    • Aim high — it’s easier to lower prices than raise them
    • Tiered pricing often makes sense; most people pick the middle tier
  • What will it take to have a positive ROI?

Domain Name

You’ll probably want your own domain name.

  • Think of a name
    • Stick with a .com name if possible; don’t use .biz
    • Some other top-level domains come into fashion on occasion
      • .io is pretty popular right now
  • Check availability of the name
  • Keep checking names, until you find a good name that is available
  • Register the name with a respectable registrar
    • DNS Registrars run from $10-$35/yr
    • DO NOT allow your web host provider to own/control your name
    • You may want to grab the .net, .org, and other versions of the same name
    • Multiple-year registration is cheaper, but it’s easier to forget how to renew it
  • Your registrar will likely point your domain to an “under construction” page initially
  • DO NOT LOSE YOUR NAME!
    • Spammers and pornographers will take it over if your registration lapses
    • Make sure your contact info (especially email address) is up-to-date
  • Beware scams (usually by US mail) trying to get you to renew with a different registrar
  • Note that the email address you register with will get spammed
    • Some registrars provide some protection for a fee

DNS Hosting

You’ll need to have someone take care of the servers that tell the world what server addresses your domain name corresponds to.

  • Find a DNS hosting provider
    • We like DNSimple; they also do domain registration
  • Provide the name servers to your DNS registrar

Web Hosting

You’ll need servers to host your web site on. There are a lot of options available, from virtual hosts (a Linux server where you control everything) to application-specific services.

  • Talk to your developer or designer first!
    • Web host can significantly restrain the development environment
  • Cost
    • $10 to $500 / month is typical
  • Bandwidth
    • Number of users × how often they use it × average “page” size
    • What happens if you go over?
  • Up-time
    • What kind of down-time can the site sustain?
    • Higher guaranteed uptime costs more
    • What if the specified uptime is not met?
  • Development environment
    • What programming languages are installed?
    • What databases are installed?
    • What libraries are installed?
    • What if other libraries are required?
  • Shared/dedicated/virtual hosting
    • Shared means others are using the same machine, with security implications
    • Dedicated is expensive, but you “own” the whole machine
    • Virtual is somewhere in between
      • You have a “slice” of a machine dedicated to your site
  • How responsive is the host to problems and requests?
  • Backups
    • What do they back up?
    • How often do they back up?
    • How can files be restored?

Design and Development

Designing and developing the site can vary from picking an existing template and adding content, to developing a full web application.

  • Cost ($30 – $300 / hr)
  • Project management
    • Story/task management
  • Revision control
    • Ensures changes can be rolled back quickly
  • Functionality
    • What does the site need to do?
  • Usability
    • How easy is it to use each page?
    • Is it easy to navigate the site to find what you’re looking for?
    • Check for broken links
    • Check for ADA/508 compliance
    • Spell checking and grammar checking

Deploying and Testing

  • Can updates be deployed quickly?
    • Deploy early and often, so it’s not such a big deal, and it becomes routine
  • Consider a staging and/or beta site
    • Test everything thoroughly on staging site before deploying to production
    • Staging site should (mostly) use the same code as production
    • Staging/test site should not process credit cards, etc.
  • Automated testing
    • Prevents regressions
  • Exploratory testing
    • See how things work
    • See if you can break things
  • Security testing
    • Penetration testing
  • Performance
    • Load testing
  • Beta testers

Marketing

  • Search engine “optimization” (SEO)
    • Good design, good URLs, and well-written HTML should cover most of this
    • Submit site to search engines
    • Use robots.txt and site maps
  • Directories
  • PR sites
  • Targeted groups
  • DO NOT send out spam

Additional Services

What other services do you need, besides just the web site?

  • Email
  • Blog
  • Wiki
  • File storage
  • Forums
  • Authentication
  • Customer interaction
    • CRM
    • Feedback
    • Bug tracking
    • Customer Q&A (StackExchange)
    • Follow-up emails to customers to offer assistance

Maintenance

Over the lifetime of the site, you’ll likely pay more in maintenance costs than the upfront costs.

  • Responding to user emails
  • Requests for info
  • Feedback about the site
  • Password resets?
  • Tracking bug reports and feature requests
  • Site improvements
  • Additional content
  • Moderation of user content
    • Spam removal
  • Log analysis
    • Google Analytics
  • Assessing advertising effectiveness
  • Analysis of revenues/profitability
  • Upgrades
    • New/improved functionality
    • Bug fixes
    • Upgraded infrastructure
  • Down-time
    • Web host
    • Upgrades
    • Accidents
    • Bugs
  • Backups
    • Testing restoring from backups
  • Payments for services
    • Domain name registration – DO NOT LOSE YOUR NAME!
    • Web hosting
    • Marketing/advertising