What do I think? I think you got it!!
Now scale that to a multi-billion $ program, with distributed teams, rolling waves, and a 5 year of performance, and imagine what happens to that original basis of estimate.
Add variable underlying probability density functions, that need to be estimated within the 1 STD, and you've got a full time job just keeping the ETC straight.

Glen, do you use historical data to determine the PDFs for various types of activities while doing new estimates? How are those PDFs determined?

Josh,
We start with “past performance.” That data gives us upper, lower, and mode values for the Monte Carlo. If there there is no “like kind” numbers, then a “derived” estimate is made. This is done in a parametric manner from past performance.

If this is a “new” domain, then a parametric model is used based on interfaces, and other modeling variables. COCOMO, the McConnell tools, SEER, Price-S or Price-H, ProPricer or similar tools are used. Rarely do we use personal opinion.

This is a systems engineering problem, and the estimating starts with the architecture of the system.

Along the way forecasts are updated, data from the past is used to improve our estimates and of course real data now “anchors” and “adjusts” those past estimates.

In all cases the forecasts are statistically adjusted with variances. Anything outside of 1 STDev is unacceptable. You've got to keep the variances under control. The 3 to 6 sigma variance on the estimate population would be considered “wide ass guessing” in our world.

As to the specific PDF, we use Triangle when we don't know the underlying distribution. This is the case when there is not enough samples to produce the needed confidence interval. You can reconstruct the PDF of course when you have enough samples, but just having a varaince adjusted upper and lower limit, turned into a percent variance with the Triangle is sufficient for our needs. The normal variability will swamp the error in correlation between the true PDF and the estimated values from that PDF.

Josh,

We provide services (environmental), and our estimates are tied to our proposal bids, most of which are firm fixed price. The risk is high on these multi-million, multi-year contracts, and a lot of the work is considered “new work” for us; that is, work we haven't previously done, and with a new client. It is essential that we neither over – or under-price; the first leaves us in a non-competitive position (we won't win), while the second may have us win, but then we can't perform the contract work for the price we bid. Either scenario will have you soon putting up the “for sale” sign.

To ensure that we've bid correctly, we use past performance (similar jobs, similar clients), try to get intel on how others are bidding (there are 3rd party business intel folks who can provide you info to help determine that … I've found them to be pretty danged good at what they do), and of course, we use “Joe” … the gent who's been doing estimating/pricing for over 25 years. We do our best not to lend more “weight' or creedence to any one source, but use all the info to come up with what we think is a “winning” (and executable) bid for us.

I know when I worked with Northrop Grumman on a proposal a few years ago, they had over 250 employees who did this kind of work full time for them. They called the process: “Bid to Win,” and they have been very successful …

I just wanted to provide an update…I'm reading “The Black Swan” by Nassim Nicholas Taleb now. It's a book I've wanted to read for a long time, and I thought it might have some mind-expanding things to say about project estimates. I laughed out loud when he started talking about anchoring and referenced the same study I've seen several studies on this topic reference.

By the way it's a great book, although I'm still not completely sure what to do with it yet!!! I may have to read it a few times between thinking and talking spells before I draw any conclusions.

Josh,

I can't say this surprises me that much. I have seen cases where, due to anchoring, the estimated cost of a project came out 20% lower than the cost of an almost identical project for another client.

Of course the usual argument that we were now much more productive was used to justify the drop in cost. Guess what happened…

Patrick Richard ing., PMP
http://www.thehardnosedpm.com
@hardnosedpm
http://www.heavyrotations.com

Josh,

Take care with the Black Swan book. Taleb has strong critics of his approach because of his simple (and possibly simple minded) principles. He's also a post-hoc kind of guy, which is pretty easy to do these days.

There have been other PM voices that took up Taleb's mantra. The core problem is the variability found in the financial markets is a different kind of variability than we see in PM. The book “Against the God's” is a good starting point for the details.

But they are basically the different between Stochastic Bayesian processes (Project Management), where the past is an indicator of the future. This paradigm is grounded on the bounded behaviors of the underlying “physics” of project work. How a propulsion system works is pretty much fixed by the chemistry of the propellants.

This is not the case in the financial instruments world of Taleb. They are non-linear, and non-Bayesian (the past does not forecast the future). So the Black Swans appear periodically and reek havoc.

Pat,

All too familiar.

And of course the notion that “I haven't influenced you in any way regarding your quoted cost or duration,” is naive at best.

Anchoring and Adjustment takes place in the absence of suggestions by the simple fact of self suggestions. This is the fundamental reason for past performance, parametric models, and Monte Carlo simulation.