The pre-frontal cortex of our brains can mislead us. Big time. I am continuing my journey into the nature of project estimation and the irrationality of the human mind, in order to get better at it. A few of the books I’ve listened to over the past few weeks include:
- The Black Swan: The Impact of the Highly Improbable – Nassim Nicholas Taleb
- A Whole New Mind: Why Right-Brainers Will Rule the Future – Dan Pink
- How We Decide – Jonah Lehrer
There is an example from that last book I want to talk about today.
MIT economists led by Dan Ariely did an experiment with their business school graduates , and later on executives and managers at the MIT Executive Education Program.
It was an auction with various products including wine, a wireless keyboard, and chocolate truffles.
Before they bid on miscellanous items, they were asked to write down the last 2 digits of their social security numbers.
Then, they were asked whether or not they would be willing to pay that amount (the last 2 digits of their social security number) for each of the products.
Finally, students wrote down the maximum amount they would be willing to pay for each item.
Results
It’s obvious that the last 2 digits of your social security number should have nothing to do with the value you place on random objects, right? Someone with two last digits of 10 and another person with two last digits of 90 might be expected to bid similar amounts on the same item, on average.
Here’s an example of what happened.
For the cordless keyboard, the group who had social security numbers ending between 80 and 99 bid an average of $56. Those who had social security numbers ending between 1 and 20 had an average bid of $16.
FOR THE SAME PRODUCT ???
In Project Management
This phenomenon is reflected in project estimation. I’ve recently had short-term estimates turn out to be completely off, and I’m talking about within 2 weeks of actually doing the work. WHY?
One explanation could be lack of specific information. With a complex interconnected software system it can be difficult to tell what the “real” impacts of something will be until you get in there and start looking at the specifics.
Another explanation is this anchoring effect. And anchoring doesn’t just happen when you give someone a number right before they estimate…it can be coming from many sources, with or without your knowledge as the project manager.
I don’t believe this means that expert opinion is useless for estimation; at least not totally. I’ve been thinking through some techniques that could be used to eliminate the influence of anchoring to a large extent, and if you have any ideas on the topic I’d love to hear them.
No related posts.
{ 6 comments… read them below or add one }
How interesting! I am tempted to buy this book because I am intrigued by my own ability to make decisions. Sometimes it is quick and simple with no regrets, such as buying a house. But, other decisions seems to take forever, such as deciding what to eat for supper.
After a decision is made, I am either comfortable with it or it lingers in the back of my mind for months of years as I wonder if I really made the right decision. I would love to learn about the difference between emotional vs. rational decision-making. Which decisions am I most comfortable with? Am I uncomfortable with an emotional decision that does not seem rational? or with a rational decision that doesn't fit with my gut instinct?
When it comes to estimating, I always second-guess (don't we all?). But, if other teams members agree, I become comfortable with my estimates. Do we sometimes get “bullied” into reducing our estimates because of the subtle reactions of others?
Scenario 1…
“My estimate is 3 weeks of testing”
“Really? Are you sure? Development is only 1 week.”
“Well. Maybe I could do it in 2 weeks then”
Result: Re-think the estimates and wonder where we went wrong. Look at how we can reduce the testing so we can meet the 2 week mark. Complain about the fact that we don't have enough time to do adequate testing. Miss both the 2 week mark AND the 3 week mark.
Scenario 2…
“My estimate is 3 weeks of testing”
“Yup. That makes sense.”
Result: Get to work right away to hit the 3 week mark or finish early.
These scenarios do not take into account the situations where the estimate really is off the wall. That's a whole other issue.
Jennifer,
In both cases the variance of the estimates is missing. I use an analogy for this problem that goes like this.
You're in Trinidad Tobago and you write down the high temperature of the day for one year. You make a histogram of the numbers and their frequency of occurrence. The number that occurs most often in that histogram is the MOST LIKELY value or the MODE of the statistical distribution of the high temperatures in Trinidad.
Now you go to Cody Wyoming, just north a few 100 miles of we're we live and do the same thing. After living in Cody for a year and wishing you were back in Trinidad, you compare the two histograms.
Turns out the Most Likely (Mode) is very close, around 78 degrees F. But what is different is the variance, the standard deviation. Don't go to Cody in February in your shorts.
Without knowing the variance in the Most Likely temperatures Cody and Trinidad look pretty much the same.
No single point estimate can be credible in the absence of the variance.
For a final note, the notion – mis-informed notion – that adding all the underlying statistical distributions together produces a “normal” distribution of these variances is ONLY true is each distribution is independent of the others AND the number of sample distribution for a 90% confidence with a 5% error numbers in the 100's.
So in both cases you provide – and these are common discussion – what is the variance, what is the shape of the probability distribution (right tail is likely for software development), and what are the consequences of being late by some percentage. These answers then drive how much “margin” you need.
The only way out of this is to apply Monte Carlo Simulation to an “Ordinal” ranked model of the variance in the estimate. Asking people their estimates if fraught with problems (Anchoring and Adjusting is always present). Past Performance and model based estimating systems are now mandatory in government programs and a competitive advantage for large commercial and construction projects.
I've moved many of the anchoring and adjusting and Monte Carlo resources to a BOX.NET site if you'd like to read more.
Thanks for the comment Jennifer! I liked all these books, I'm just about finished with the Lehrer book.
Taleb's views on prediction are compelling to me…I will be reading his other book “Fooled by Randomness” soon. Perhaps it's the difference between Mediocristan and Extremistan, and my project is living a lot in Extremistan, but reliable estimates seem very difficult. We have too many government agencies involved, and too many interdependencies from other contractors.
Customer expectations and input from spacecraft vendors in particular seem to change regularly with lots of late deliveries and last-minute changes in direction. All of these lead to so many avenues of uncertainty that it seems sometimes to me that any estimate is just a shot in the dark, regardless of whether we use data from the past in an analogous fashion, using monte carlo analysis, or just expert opinion.
And here I go into the void of project estimation nihilism….
-Josh
You may find this recent video from TED to shed some light on how we make decisions. Pretty scary, actually, but when you think about it…….
http://tinyurl.com/35qlzt4
BR,
Dr. PDG, Jakarta
Loved it Paul, and it was right in line with other sources of information on the topic I'm coming into contact with now. Thanks!
The TED piece is pure Tversky & Kahneman and “Against the Gods,” Peter L Bernstein, just added monkeys, instead of doctors, traders or investors.