Estimation

The pre-frontal cortex of our brains can mislead us.  Big time. I am continuing my journey into the nature of project estimation and the irrationality of the human mind, in order to get better at it.  A few of the books I’ve listened to over the past few weeks include:

• The Black Swan: The Impact of the Highly Improbable – Nassim Nicholas Taleb
• A Whole New Mind: Why Right-Brainers Will Rule the Future – Dan Pink
• How We Decide – Jonah Lehrer

There is an example from that last book I want to talk about today.

MIT economists led by Dan Ariely did an experiment with their business school graduates , and later on executives and managers at the MIT Executive Education Program.

It was an auction with various products including wine, a wireless keyboard, and chocolate truffles.

Before they bid on miscellanous items, they were asked to write down the last 2 digits of their social security numbers.

Then, they were asked whether or not they would be willing to pay that amount (the last 2 digits of their social security number) for each of the products.

Finally, students wrote down the maximum amount they would be willing to pay for each item.

Results

It’s obvious that the last 2 digits of your social security number should have nothing to do with the value you place on random objects, right?  Someone with two last digits of 10 and another person with two last digits of 90 might be expected to bid similar amounts on the same item, on average.

Here’s an example of what happened.

For the cordless keyboard, the group who had social security numbers ending between 80 and 99 bid an average of $56.  Those who had social security numbers ending between 1 and 20 had an average bid of $16.

FOR THE SAME PRODUCT ???

In Project Management

This phenomenon is reflected in project estimation.  I’ve recently had short-term estimates turn out to be completely off, and I’m talking about within 2 weeks of actually doing the work.  WHY?

One explanation could be lack of specific information.  With a complex interconnected software system it can be difficult to tell what the “real” impacts of something will be until you get in there and start looking at the specifics.

Another explanation is this anchoring effect.  And anchoring doesn’t just happen when you give someone a number right before they estimate…it can be coming from many sources, with or without your knowledge as the project manager.

I don’t believe this means that expert opinion is useless for estimation; at least not totally.  I’ve been thinking through some techniques that could be used to eliminate the influence of anchoring to a large extent, and if you have any ideas on the topic I’d love to hear them.

6 Comments Continue reading

Glen Alleman recently pointed me to a paper by Jorge Aranda among other material on software estimation.

I sat down and read Anchoring and Adjustment in Software Estimation and it was well worth my time.

To cut to the chase, the subjects were tasked to give estimates for software tasks in a controlled manner, in 3 groups with various “anchoring” methods being used.  The only difference between the groups was the expectation statement by the manager before estimation.

Group 1 (control – no explicit anchor given)

“I’d like to give an estimate for this project myself, but I admit I have no
experience estimating. We’ll wait for your calculations for an estimate.”

Group 2 ( ’2 months’ condition)

“I admit I have no experience with software projects, but I guess this
will take about 2 months to finish. I may be wrong of course, we’ll
wait for your calculations for a better estimate.”

Group 3 (’20 months’ condition)

“I admit I have no experience with software projects, but I guess this
will take about 20 months to finish. I may be wrong of course, we’ll
wait for your calculations for a better estimate.”

You will need to read the full paper to see all the goodies (and to determine if you think it is relevant to your domain), but I would like to summarize some of the results I found striking.

These were the results among all participants, and there are other slices of the data available in the paper including only experienced participants and also by estimation method chosen.

’2 months’ condition

• mean – 6.8 months
• median – 6 months
• standard deviation – 3.7

control – no explicit anchoring

• mean – 8.3
• median – 7
• standard deviation – 4.4

’20 month’ condition

• mean – 17.4
• median – 16
• standard deviation – 5.6

The results in general coincide with my own experience on this matter.  An important point to note is that even though they were supposedly estimating the exact same software requirements, it is very likely that the ’2 month’ group would have produced a significantly different product than the ’20 month’ group.

Food for thought.

When you and your team are putting together estimates, what influences are creating these anchors?  From my experience there are many of them, some of which are likely to be arbitrary or set (even inadvertently) without sufficient knowledge or experience.  They may be coming from stakeholders, sponsors, the project manager, or even a team member/lead.

My favorite example of this is when a team is asked to provide a “back of the envelope” estimate without really understanding the scope yet.  It produces a bad estimate and sets a rather arbitrary anchor for future estimates.

What do you think?

8 Comments Continue reading

Project estimation can be a headache for new project managers.  There are lots of opinions out there and not much that weighs various methods…rather you find people who feel strongly about one way or another.  To give a quick overview of some ideas and then call for your input, I recorded this video.  Please leave a comment and share your thoughts on this topic!

• GAO Cost Estimating and Assessment Guide
• 2008 NASA Cost Estimating Handbook
• Why 3 Point Estimates Create False Optimism
• Estimating Effort: Part 3

22 Comments Continue reading

Once upon a time, I was searching for some solid guidelines on cost estimation to improve the way my contract was doing it.

I found a few great resources

4 Comments Continue reading

“Mark, I expect you to perform at the level you committed to. There is no excuse for this, you are making our entire project team look bad. Your tasks were nowhere near the critical path before, but now you ARE the critical path.”

9 Comments Continue reading

Josh asked me to write about some of the challenges and obstacles that we face in our environment. In order to do that I should explain a little about that environment.

Over the past 36 months we have transformed ourselves…

9 Comments Continue reading

In this last article of the series, I’ll cover a potpourri of other estimating topics including dealing with poorly defined work and what to do when management thinks it should cost less or take less time.

Dealing with Poorly Defined Work

In order to introduce the basic concepts of effort estimating, we limited our discussion to developing an estimate for a single, reasonably well-defined activity. Although this type of work may prevail in some project environments, most projects will have a significant amount of work to estimate that is not well-defined. In this section, I will cover some approaches to those situations.

No one on the team has experience with this kind of work. Here are some ideas that should help:

• Find someone who does have experience, e.g., a consultant or someone in the organization who is not on your team.
• Use a wider range estimate to reflect the greater uncertainty. Unless the unknown work represents a significant percentage of the total work, your overall project budget will not be greatly affected. Remember that we are willing to tolerate budget variances in individual work items as long as the project total is acceptable.
• Break the work down into smaller units. Often, you will find that you do have experience with much of the work. The aspects that you don’t have experience with can be handled with a wider range estimate.
• Develop some experience. It may be possible to have one or two team members do a couple of “practice” activities as part of the project planning process. I used this approach quite successfully as part of a database conversion project several years ago.

This is truly state-of-the-art stuff. See previous paragraph, especially the last bullet.

We really have no idea of what’s required or we don’t know enough at this point to define the work. Neither of these two complaints is an estimating issue; they are both either a scoping issue or a risk management issue. They might also be a defensive response to a skeptical manager. If so, see below.

Conflicting Estimates

Did you ever have two team members who differed widely in their opinions of the likely amount of effort required to complete a work item? Range estimates can help, particularly if the individuals involved are inherently optimistic or pessimistic — if their ranges overlap, it is usually much easier to reach agreement.

But the root cause of such difference is most likely to be the presence of conflicting assumptions. For example, a team member who assumes that previous design work can be reused will produce a lower estimate than one who does not. By surfacing the conflicting assumptions, the team can discuss the alternatives and decide what to do.

One particularly pervasive assumption is the skill level of the person doing the work. Estimating is often done by more senior staff, and they tend to think in terms of “how much effort would I personally have to expend to complete this work.” This is likely to produce an estimate that it too low. At the same time, they may become aware of their tendency to overestimate and overcompensate. I recommend that you remind your estimators to constantly assume an “average” resource, and to recognize that it will take 2-3 projects of comparing actuals to estimates for them to get good at knowing what “average” really is.

Converting Effort into Duration

I know that I am going to get a lot of flak from some of my colleagues about this next point. The technically correct answer about how to estimate duration is that you should do just that — look at the effort estimates and develop a three-point range estimate of the duration for each work item. For example, with an effort estimate of 30, 40, and 80 hours to be performed by a single individual working fulltime on the project, I think it is reasonable to predict that this person may need more than two full weeks to get this item done if it ends up requiring 80 hours of effort.

In addition, since virtually all network analysis techniques (Critical Path, Critical Chain, PERT, etc.) underestimate the most likely duration of the project, many would also argue that a Monte Carlo simulation is a necessity. And I would agree … if you’ve got the budget.

If not, take the easy way out — take the effort budget (the expected value of the effort estimate) and convert that into a single-point duration budget based on the availability of the individual assigned.

Maintaining Your Estimates

If the work of the project changes via an approved change order, you will need to estimate the new work and update your budget as well.

If any of the assumptions that went into your estimates are changed, you should also update your estimates. In this case, you may not be able to modify the budget, but you should still re-estimate the work. For example, if one of your most skilled staffers was supposed to work on Work-item D, and due to circumstance beyond your control D is being done by an intern from the local community college, you better reset everyone’s expectations about how long it is going to take.

Management Cuts It in Half

The first thing you have to do is find out why. Most such management actions are created by one of the following misunderstandings:

• Your manager may not understand the difference between an estimate and a budget. Educate them. But in the nicest way possible if you want to keep your job.
• Your manager may view your estimate as a negotiating position. Some managers assume that you have “padded” your estimate to provide negotiating room. Again, educate them — explain how the suggested cuts will either reduce quality or increase the risk of an overrun.
• Your manager’s cuts may be a negotiating position. Most good managers understand that a more difficult task (where the target is aggressive) can serve to motivate the project team. But even good managers often fail to appreciate that an impossible task (where the target is so aggressive that it is clearly impossible) is a powerful demotivator.

Summary

Underestimating will drive you crazy. You will spend endless hours explaining why you are over budget and behind schedule. You will spend endless hours dealing with unhappy customers and stressed out team members. Overestimating may not be much better. Your projects will not get approved because they will be seen as too expensive. You may even lose your job if your padding is viewed as unethical.

Three-point estimates for effort may not get you into heaven, but they will make this life a lot more rewarding for everyone affected by your projects.

0 Comments Continue reading

This is the penultimate article in a series of five on effort estimating. In the first, I focused on definitions since many people use terms such as estimate and budget as synonyms when, in fact, they are very different. Here is a brief recap of some key definitions:

Estimate. An informed assessment of an uncertain event. Informed means that you have an identified basis for the estimate. Uncertain recognizes that multiple outcomes are possible.

Effort. Effort is an expenditure of physical or mental effort on the part of a project team member. Effort is normally measured in terms of staff hours.

Budget. A management metric that is derived from the estimate of the relevant work.

In an earlier article, I introduced the idea of three-point range estimates and covered the mechanics of how prepare them with your team. I ended that article by asserting that three-point estimates usually take less time to prepare than traditional single-point estimates.

In this article, I’ll cover the key outcome of estimating and converting effort estimates into budgets.

Key Outcome of Estimating

When I teach people how to estimate, I emphasize that the most important outcome of the estimating process is a project budget that is accurate, i.e., one where the actual results fall within an acceptable range of the budget. Acceptable budget accuracy will vary based upon the project :

• In New Product Development (NPD), it is often acceptable for the final cost of the project to be 3-4 times the initial estimates.
• In Information Technology (IT), ±10% is a reasonable target if the requirements are known, agreed, and stable, while the NPD range of +300-400% may also be perfectly acceptable if the project is mission-critical.
• For a fixed price consulting engagement, management may be reluctant to accept any overrun.

I argue that an estimate is accurate if (a) the actual results for most work items fall within the estimate’s range most of the time, and (b) the sum of the actual results is close to the sum of the expected values of the range estimate. In brief, we care most about the total. We care about the individual results only as a means to an end.

Let me illustrate. Let’s say that we have a project with 100 activities. To keep it simple, let’s ignore scheduling concerns for the time being, and let’s further assume that each work item has a three-point range estimate of 20, 25, and 30 hours of effort. The budget for each individual activity will be 25 hours (more on that later), and thus the budget for the project will be 2,500 hours.

If this project goes according to plan, most of the activities, probably around 70% of them, will be completed for an actual cost within an hour or two of their budgeted effort. One or two may take as many as 30 hours; one or two may take only 20. But the project total is likely to be very close to 2,500. Statistically, we have about a 95% probability that the project will finish within a range of ±50 hours. Not bad!

Now let’s take another project with 100 activities, only for this project, each activity has a range estimate of 10, 25, and 40 hours of effort. We will still budget this project for 2,500 hours, but ±50 hours has only about a 60% chance. If we want 95% accuracy, we need to go to ±120 hours.

Three-point range estimates help to support our goal of accuracy by encouraging everyone on the team to spend as much time and effort as they need to finish the work item correctly, and only the time and effort that they need. We avoid a lot of dysfunctional behavior:

• Team members may use the full budget even if they don’t need it.
• Team members may cut corners to avoid exceeding the budget.
• Managers at all levels of the project may waste time explaining minor variances.
• Managers may be tempted to manipulate the reported actuals in order to make the estimates appear “better.”

And one more point: there are some circumstances when a single-point estimate is okay. For example, we can safely predict that a one hour team meeting that includes seven people will require seven hours of effort. Is there some chance that only six people will show up? That it will take an hour-and-a-half? Sure, but this is likely to be such a small source of error that we can usually ignore it.

Converting Estimates into Budgets

While an estimate is a range; a budget is a single number. An estimate expresses an opinion about the possible outcomes; a budget expresses a judgment about how much risk is acceptable. For example, if you expect a trip to the airport to take 40-60 minutes, you may budget 70 minutes if you are picking up your boss (close to zero chance of being late) and 40 minutes if you are picking up your spouse (he or she can wait!).

We can use some simple statistical techniques to convert our ranges into budget numbers. For this article, I’m going to ignore other budget preparation issues like unstable requirements and pre-defined budget maximums since these are more properly addressed through project risk management than through estimating.

The formulas for adding up ranges can be found in any basic statistics text. If you are willing to accept my suggestion that a triangular distribution is a good assumption, the basic process is straightforward:

• Develop range estimates (low, likely, high) for each detail item in your project plan.
• Calculate the expected value for each item by summing the three numbers and dividing by three.
• Total the expected values. This is your base budget (i.e., it does not include any reserves or contingencies).

And yes, it is just that simple. To the left is a worked example for a small project with two deliverables and seven activities that produces a base budget of 290 hours:

Many of you are probably also familiar with the formula for the expected value that is used in the Program Evaluation and Review Technique (PERT). The PERT formula is different because PERT assumes a beta distribution. I prefer to assume a triangular distribution because it produces more conservative results (a higher budget base) than a beta. Either assumption will work.

6 Comments Continue reading

Preparing Three-Point Range Estimates

The mechanics of preparing three-point range estimates, particularly if you are working with team members who have never used the technique before, are important. And just a reminder: we are still dealing with effort estimates; duration estimates come later.

Working one-on-one. Let’s again start with a simple case where you need an effort estimate from an individual team member regarding how much of their effort is likely to be required to complete a specific activity. The first step is to make certain that both of you have a common understanding of words like “estimate” and “complete.” For example, it is always tempting to say something like “how long do you think this will take?” This question is likely to produce a duration estimate rather than an effort estimate. Try to ask something like “how many hours of your time do you think this activity will take?”

Begin the discussion with a review of the work to be done to ensure that both you and your team member have a common understanding of the work. If there are unknowns, decide how to handle them—make assumptions, go get answers, whatever.

Once you have a reasonable level of agreement, ask the individual for their effort estimate of the most likely result. I tend to use words like “if these assumptions turn out to be true, how much of your time do you think this work is likely to take?” After they’ve answered that question, I ask “if every thing goes well, if you get very lucky with this activity, and still using the same assumptions, how much of your effort do you think this activity will take?” Then I use similar language to obtain a pessimistic effort estimate.

Remember to keep your assumptions constant. If you develop an optimistic estimate based on everything going well, and a pessimistic estimate based on nothing going well, the range of the estimates will be quite large. Very large ranges make most people uncomfortable because it appears that they don’t understand the work. In addition, it will be harder for you to assess the accuracy of the range estimate later because you’ll have to factor in all the different assumptions.

Understand that the first few conversations with any given individual are unlikely to go smoothly. They may generate additional questions or ask you to make additional assumptions at any point during the conversation. Fine. Let them. Help them climb the learning curve. My experience is that it takes most people 3-5 iterations to get comfortable with the process. After that, they’ll start giving you three-point estimates automatically.

Feel free to challenge their estimates, but do so constructively by asking clarifying questions. Do not cast aspersions on their estimates. If your estimate differs from theirs by a substantial amount, the most likely explanation is that your understanding of the work differs from theirs. Concentrate on identifying and resolving those differences.

It also seems to be important to start by asking for the most likely result, then the optimistic, and then the pessimistic. I don’t know why, but this sequence seems to work best.

Working with a group. Most of your estimates will not be developed one-on-one, but rather will be developed by small groups as part of a team planning process. The basic process remains the same for these small groups:

• Review and agree on the work to be done.
• Document the assumptions.
• Develop three-point range estimates starting with the most likely result.

Even when some of the group members don’t have deep expertise in the project’s work, their presence can help to surface unstated assumptions and to build team-wide commitment to and understanding of the effort estimates.

Other approaches to three-point range estimates. Some project managers try to expedite the development of a three-point range estimate by asking for only two numbers—the most likely and some form of variance (e.g., “10 hours, plus or minus 2”). While this statement equates to a three-point range estimate of 8, 10, and 12, in my experience, it lacks the richness of my process—it doesn’t seem to generate the same degree of thoughtfulness from the estimator.

Will this take too much time?

I am convinced that the thinking process and the conversations that support three-point range estimating contribute to the development of more accurate estimates. Short-circuiting the process may appear to save time, but it’s a false economy.

My experience is that developing three-point estimates almost always takes less time than single-point estimates because the conversational process eliminates the posturing and gamesmanship that often delays the development of estimates. I’ve have had teams in my project management training courses spend less than an hour to develop an estimate for a 4,000 hour project — an estimate that later proved highly accurate. A recent client spent less than a day to discover that a mission-critical, multi-million dollar new product development project was going to be eight months late. An investment of something less than $10,000 is expected to return in excess of $10,000,000. Time spent estimating is not a cost, it is cheap insurance.

In the next article of this series, I’ll cover some of the more difficult challenges of estimating effort on projects: what to do when management thinks it should take fewer hours, using range estimates with project management software, how to deal with the natural (and normal) biases of the estimators, understanding the difference between accuracy and precision, and how to use some basic, simple statistical methods to create an estimate for the whole project.

4 Comments Continue reading

This is the second in a series of articles on estimating effort. In the first, I focused on definitions since I find that many people use terms such as estimate and budget as synonyms when, in fact, they are very different. Here is a brief recap of the key definitions that I will use:

Estimate. An informed assessment of an uncertain event. Informed means that you have an identified basis for the estimate. Uncertain recognizes that different outcomes are possible.

Effort. Effort is an expenditure of physical or mental effort on the part of a project team member. Effort is normally measured in terms of staff hours.

Budget. A management metric that uses the effort estimate as one of the key inputs.

Baseline. A time-phased budget that has received all necessary approvals.

What are you estimating??Let?s start with a fairly simple example to build a conceptual foundation.?It may seem obvious, but the first requirement for developing an estimate is to know what you are estimating. For now, let?s assume that you have been asked to estimate how much effort (how much of your time) is likely to be required to paint your bedroom. Although this is a fairly small activity, it is still one with a significant amount of uncertainty:

• Do you have to paint the ceiling and the woodwork, or just the walls?
• Are there windows to be painted? Do they have mullions?
• Is there furniture in the room? Can it be moved? If it can be moved, is moving it out and back in part of ?painting the room??
• Is choosing the color part of this activity? What about going to the store to buy the paint?
• How many coats are needed? Is there any special finishing involved (like marbleizing)?
• Will anyone be helping you? Or will you be doing all of the work yourself?

If you don?t know the answers to these questions, you have a couple of choices:

• You can decline to prepare an estimate at all.
• You can prepare an estimate that allows for a large amount of uncertainty.
• You can make some assumptions about the answers to these questions and estimate based on those assumptions.
• You can try to reduce the uncertainty by getting answers to the questions before preparing an estimate.

Any of these approaches is acceptable as long as your stakeholders know which approach you are using!?Most of the time, you will use some combination of approaches. You will make some reasonable assumptions (and you will always document all of your assumptions!). You will ask some questions. Then you will apply your best judgment.

Range estimates.?You can improve the usefulness of your effort estimates by making range estimates. How does a range estimate work? In its simplest form, you estimate a low value and a high value for the amount of effort that you think you are likely to need. For example, in the case of painting your bedroom, and assuming that the work is limited to applying paint to the walls, you might estimate 2-3 hours.

The fundamental concept behind range estimating is that you don?t need to know the exact amount of effort that will be required. You do need to know that it isn?t going to take a full day out of your schedule. You do need to know that this isn?t a trivial activity that can be completed in a few minutes. As long as the actual result is somewhere between 2 and 3 hours, you have made a good estimate.

Even if the actual result falls outside of that range, you still made a good estimate since it was the best estimate that you could make given what you knew at the time. In addition, you can learn from any?variance (an actual result that falls outside the predicted range) and use that information to try to improve your future estimates.

Using range estimates takes much of the pain out of the estimating process. If you are accustomed to single point estimates, it can be very difficult to choose between 2 hours and 3 hours. Even proposing a single-point estimate of 2.5 hours can be difficult because you know that it might take more or less than that amount. Using range estimates allows you to estimate even in the face of uncertainty.?If your boss insists on a single point estimate, he or she is most likely looking for a budget, not an estimate. There?s no problem with giving them a single number as a budget, but you really should share your estimate with them as well so that they will understand how much or how little uncertainty is involved.

Three-point range estimates.?You can improve the?usefulness?of your effort estimates still further by making three-point range estimates. In addition to the high and low values of a simple range estimate, you add an assessment of the most likely in-between result.

Let?s say, for example, that you are living in the world of Groundhog Day such that you can paint your room 100 times and keep records of how much effort it actually took. You discover that you never spent less than 2 hours and never more than 5. You also discover that your actual results produce a triangular distribution with a peak at 3 hours as shown below:

Based on the availability of this actual, historical information, you would estimate the amount of effort required to paint a similarly sized bedroom as follows:

• Most likely: 3 hours
• Optimistic: 2 hours
• Pessimistic: 5 hours

I understand that you will seldom if ever have such wonderful, perfect historical information to base your estimates on, but the great thing about range estimating is that you don?t need perfect data! As long as the three-point range estimate is reasonable, errors in the effort estimates for individual activities are likely to balance out over the course of the project.

Even in this case, even with perfect data, you still don?t know how exactly how much effort will be required for the next bedroom that you paint. It might be 2.5 hours; it might be 3.75 hours. Doesn?t matter. What is important is that you have effort estimates that are reasonable; that you have effort estimates that will allow you to manage your project more effectively.

Next: preparing three-point range estimates

3 Comments Continue reading