Estimating Effort: Part 4
This is the penultimate article in a series of five on effort estimating. In the first, I focused on definitions since many people use terms such as estimate and budget as synonyms when, in fact, they are very different. Here is a brief recap of some key definitions:
Estimate. An informed assessment of an uncertain event. Informed means that you have an identified basis for the estimate. Uncertain recognizes that multiple outcomes are possible.
Effort. Effort is an expenditure of physical or mental effort on the part of a project team member. Effort is normally measured in terms of staff hours.
Budget. A management metric that is derived from the estimate of the relevant work.
In an earlier article, I introduced the idea of three-point range estimates and covered the mechanics of how prepare them with your team. I ended that article by asserting that three-point estimates usually take less time to prepare than traditional single-point estimates.
In this article, I’ll cover the key outcome of estimating and converting effort estimates into budgets.
Key Outcome of Estimating
When I teach people how to estimate, I emphasize that the most important outcome of the estimating process is a project budget that is accurate, i.e., one where the actual results fall within an acceptable range of the budget. Acceptable budget accuracy will vary based upon the project :
- In New Product Development (NPD), it is often acceptable for the final cost of the project to be 3-4 times the initial estimates.
- In Information Technology (IT), ±10% is a reasonable target if the requirements are known, agreed, and stable, while the NPD range of +300-400% may also be perfectly acceptable if the project is mission-critical.
- For a fixed price consulting engagement, management may be reluctant to accept any overrun.
I argue that an estimate is accurate if (a) the actual results for most work items fall within the estimate’s range most of the time, and (b) the sum of the actual results is close to the sum of the expected values of the range estimate. In brief, we care most about the total. We care about the individual results only as a means to an end.
Let me illustrate. Let’s say that we have a project with 100 activities. To keep it simple, let’s ignore scheduling concerns for the time being, and let’s further assume that each work item has a three-point range estimate of 20, 25, and 30 hours of effort. The budget for each individual activity will be 25 hours (more on that later), and thus the budget for the project will be 2,500 hours.
If this project goes according to plan, most of the activities, probably around 70% of them, will be completed for an actual cost within an hour or two of their budgeted effort. One or two may take as many as 30 hours; one or two may take only 20. But the project total is likely to be very close to 2,500. Statistically, we have about a 95% probability that the project will finish within a range of ±50 hours. Not bad!
Now let’s take another project with 100 activities, only for this project, each activity has a range estimate of 10, 25, and 40 hours of effort. We will still budget this project for 2,500 hours, but ±50 hours has only about a 60% chance. If we want 95% accuracy, we need to go to ±120 hours.
Three-point range estimates help to support our goal of accuracy by encouraging everyone on the team to spend as much time and effort as they need to finish the work item correctly, and only the time and effort that they need. We avoid a lot of dysfunctional behavior:
- Team members may use the full budget even if they don’t need it.
- Team members may cut corners to avoid exceeding the budget.
- Managers at all levels of the project may waste time explaining minor variances.
- Managers may be tempted to manipulate the reported actuals in order to make the estimates appear “better.”
And one more point: there are some circumstances when a single-point estimate is okay. For example, we can safely predict that a one hour team meeting that includes seven people will require seven hours of effort. Is there some chance that only six people will show up? That it will take an hour-and-a-half? Sure, but this is likely to be such a small source of error that we can usually ignore it.
Converting Estimates into Budgets
While an estimate is a range; a budget is a single number. An estimate expresses an opinion about the possible outcomes; a budget expresses a judgment about how much risk is acceptable. For example, if you expect a trip to the airport to take 40-60 minutes, you may budget 70 minutes if you are picking up your boss (close to zero chance of being late) and 40 minutes if you are picking up your spouse (he or she can wait!).
We can use some simple statistical techniques to convert our ranges into budget numbers. For this article, I’m going to ignore other budget preparation issues like unstable requirements and pre-defined budget maximums since these are more properly addressed through project risk management than through estimating.
The formulas for adding up ranges can be found in any basic statistics text. If you are willing to accept my suggestion that a triangular distribution is a good assumption, the basic process is straightforward:
- Develop range estimates (low, likely, high) for each detail item in your project plan.
- Calculate the expected value for each item by summing the three numbers and dividing by three.
- Total the expected values. This is your base budget (i.e., it does not include any reserves or contingencies).
And yes, it is just that simple. To the left is a worked example for a small project with two deliverables and seven activities that produces a base budget of 290 hours:
Many of you are probably also familiar with the formula for the expected value that is used in the Program Evaluation and Review Technique (PERT). The PERT formula is different because PERT assumes a beta distribution. I prefer to assume a triangular distribution because it produces more conservative results (a higher budget base) than a beta. Either assumption will work.
Estimating Effort
- Estimating Effort: Part 1
- Estimating Effort: Part 2
- Estimating Effort: Part 3
- Estimating Effort: Part 4
- Estimating Effort: Part 5
Jun 8th, 2009 at 8:10 pm
Thanks Bill! Do you find that anyone pushes back on using an average of the 3-point estimates instead of using the PERT method?
I’d be interested to see what Glen Alleman’s experience is with this in aerospace…. what about other industries?
Reply
Bill Duncan Reply:
June 10th, 2009 at 6:04 am
The direct answer to your question is “no,” mostly because I explain why right up front.
The first thing to understand is that the basic PERT assumption — that a beta distribution defines the possible outcomes for any work-item — is nothing more than an assumption. A couple of analysts from the RAND Institute decided way back in the early 1960s that this would be a good assumption to use. There is no research behind it. It was based entirely on gut feel.
Second, there is some recent research, from the Texas Department of Corrections, that showed that a triangular distribution was a more realistic assumption.
Finally, I’m not just “averaging” the three points; I’m calculating the expected value of the distribution: check any statistics text for confirmation.
I don’t have any objection to the PERT formula. In fact, the “correct” approach is to look at each work-item and make an assumption about the likely distribution of results. There is no reason why you couldn’t mix betas, triangles, flat, and others on the same project. But I think that most of the benefit from using range estimates comes from simply considering the range.
Once you get to the point where most of your actuals are within the range, then you can start to consider fine-tuning the distribution assumptions.
Reply
Josh Nankivel Reply:
June 10th, 2009 at 6:32 am
Great stuff Bill, thanks for sharing your wisdom!
Reply
Glen B. Alleman Reply:
June 13th, 2009 at 8:28 pm
Josh,
We avoid the PERT formula for a variety of reasons, not the least of which it is reverse engineered and only represents the data from which it was reverse engineered.
The triangle distribution is best when there is not underlying probability distribution function (PDF).
The current guidance in A&D is to NOT develop the three point estimates as Bill suggests but to develop a ordinal classification system. This system defines the upper and lower bounds of the PDF for each risk category. Look at the DoD PMBOK (http://www.dau.mil/pubs/gdbks/DoDExtPMBOK–June%2003.pdf) for guidance here.
As well the seminal work for risk management and the topic Bill is presenting is Edmund Conrow’s “Effective Risk Management: Some Keys to Success,” 2nd Edition, AIAA Press, 2003.
The is a large body of estimating and programmatic risk management material out there that ranges from weak to wrong.
The best cost estimating sources are:
1. GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, GAO-09-3SP, March 2009.
2. NASA Cost Estimating Handbook, 2008. http://ceh.nasa.gov/ceh_2008/2008.htm
3. Join The Society of Cost Estimating and Analysis (http://www.sceaonline.org/) These are the folks that earned their living doing cost estimating.
Start with these and apply them in your environment. Ignore most of the other stuff for all the right reasons.
Read Ed’s book to see why what Bill suggests is not allowed in NASA or DoD. DID 81650 mandates Monte Carlo simulation as does the Basis of Estimate CDRL’s that accompany most contracts.
There is absolutely no reason these approaches can’t be applied universally to any project in any domain. There is no reason to “simplify” the approach. There numerous free and affordable Excel tools – Crystal Ball being my favorite for cost modeling.
I posted a link awhile back about coupling between WBS elements and its influence on the cost estimate.
http://herdingcats.typepad.com/my_weblog/2009/05/undercorrelated-wbs-elements-means-understated-cost-risk.html
If we’re going to go down the path of building credible “basis of estimate,” let’s look at how it is done in “high risk” domains.
There is a move afoot to try and simplify things, to essential take the hard work out of the work. EVM has such a move afoot. I personally see this as dumbing down what is a critically important process.
I’m saying Bill is doing this, but care needs to be taken here (estimating) and with EV, not assume you’re “doing ev” or “doing estimating,” when in fact you’re just playing in the sand box, while the real work remains outside.
As a final not, Bill’s last sentence is also not allowed at NASA. The distributions of the actuals MUST be correlated with the BCWS. Using the actuals to adjust the estimate is a one-side point estimate statistic against a two sided-estimator and pretty much meaningless for forecasting the future, since the correlation factor is lost.
This is the common disease of most of the non-technical world from Ophra to public policy.
The confusion between correlation and causality. We want to know the causality of the estimating so we can improve not only the estimate but the cause of the “over target baseline” outcomes. Knowing the correlation is nice of classroom discussion, but I can’t run the program with that information. I’m driving in the dark AND looking in the rear view mirror.
Reply
Glen B. Alleman Reply:
June 13th, 2009 at 10:40 pm
I meant to say in the 3rd to last paragraph
“Bill is NOT doing this”
But all the same trying to simplify may be useful for educational purposes, but I’ve seen nothing but grief when those class room examples are found on live programs.
Even at the recent PMI CPM conference, there were several example of “simplifying” how management reserve is used and performance calculated.
If DCMA (Defense Contract Management Agency) saw that, they’d be writing up a CAR (Corrective Action Request) and asking for the PM to example why she hasn’t read the ANSI-748B.
Let’s not let the classroom 101 example become the basis of program management.
Reply
Bill Duncan Reply:
June 18th, 2009 at 6:25 am
I was remiss in my introduction. The scope of my article was intended to be limited to estimating effort from individual professionals who will be working on internal projects. For the most part, this means Information Technology (IT), New Product Development (NPD), and Process Improvement.
In those environments, range estimating helps to address the most common problems with developing both preliminary and detail budgets.
Glen is absolutely correct in noting additional concerns for projects done under contract, and especially when those projects are done under government contracts.
Reply