Serendipity happens.

I responded to a student Inside pmStudent e-Learning with what turned out to essentially be an article about criteria for project success and failure.

A few days later, Shim wrote “Projects failure rate – the conventional wisdom is wrong!” on his fantastic blog.  I started leaving a comment, and it became one of those comments that should really be it’s own post.

Chaos

What I really mean are the set of reports out out by various organizations on project failure rates.

Shim listed the following on his post:

1. Chaos Report (1994) – only 16.2% of projects were successful by all measures. Of the 70% of projects that were not successful, over 52 percent were partial failures and 31% were complete failures.
2. OASIG survey (1995) – the IT project success rate quoted revolves around 20-30% based on its most optimistic interviews.
3. Chaos Report (1995) – The Standish Group research predicts that 31.1% of projects will be cancelled before they ever get completed. Further results indicate 52.7% of projects will cost over 189% of their original estimates.
4. KPMG Canada Survey (1997) – 61 % reported details on a failed IT project.
5. Conference Board Survey (2001) – 40 % of the projects failed to achieve their business case within one year of going live.
6. Robbins-Gioia Survey (2001) – 51 % viewed their ERP implementation as unsuccessful
7. Dr. Dobb’s Journal (DDJ) Survey (2007) – 72% of all Agile projects were successful, compared to only 63% of traditional of Data Warehouse projects.
8. Chaos Report (2009) – Only 32% of projects have been defined as being ‘successful’ compared with 35% in 2006.

See Shim’s comments section for some great commentary from the likes of Glen Alleman, Steve Romero, and Pawel Brodzinski.  I share the skepticism of these reports.  I’ll put that aside for a moment.

Controls

These are all observational studies, with all the limitations that come with that class of research.

How do you control for the variation between organizations, the people in them, and the subtleties of culture and methodology?

For all practical purposes, you can’t.

Was an implementation of EVM, configuration management, Kanban, Agile, or Waterfall more or less effective due to the charisma or skill of the individual leading the charge?  How do you control for charisma across project teams, organizations, countries, industries?

For all practical purposes, you can’t.

If you want to gain real knowledge from a study, it’s got to be an experimental one in my book.  As with all experiments some are better conducted than others.

See the studies I reference in the following for some examples of experimental research into human psychology that can actually be applied to better our projects.

• How Expectations Mess Up Project Estimates
• What’s That Got To Do With The Price Of A Cordless Keyboard?
• Do Your Project Teams Suffer From Chronic Media Multitasking?

Applicability

What’s the value of these reports anyway, besides being a mechanism to trigger fear of failure in sales prospects for project management consulting and software that will ‘fix the problem’?

Even if these reports were an accurate reflection of the aggregate, does it really mean anything to you?  Can you equate the state of behaviors and results in your little organization or program to the aggregate?  Certainly not.  See Controls.

False Dichotomy

There is a logical fallacy called the false dichotomy.  In this case, you would be led to believe there are two states for projects; success or failure.

Reductio ad absurdum can be a useful tool to uncover these false dichotomies.  Allow me to demonstrate.

On the axis of cost, a successful project has met its budget.  It was $100,000 +10%/- 5% for a total budget of $110,000 assuming G&A and reserve in that final budget.  That’s your max spending with all thresholds and reserve taken into account.

You spent $109,995 and got the job done.  Close to the max budget but still within range.  Success!

Oooooops.  A $10 order of post-it notes for your Kanban board was missed.  Now you’ve spent $110,005.  Failure

You could do a similar exercise with any attribute of the project.  My point is that project success is not a binary or if-then calculation.  It is a continuum, and is likely to be a different continuum for each and every stakeholder involved.

Can you say you successfully met or failed to meet the terms of a contract?  Yes, if it’s written well.

Can you say you successfully met or failed to meet the criteria with thresholds of your EVMS/PMB?  Yes, if your system is good enough.

What impact do the above conditions have upon the opinion of Larry, Sue, Dave, Kim, Robert, John, Laura, Sam, Roger, taxpayers or Huckleberry Finn?

For that matter, what would Rufus T. Firefly, Otis B. Driftwood, Horatio Hornblower, Ebenezer Scrooge,  Ace Ventura, Benovan Stanchiano, Scut Farkus, Buckaroo Banzai, or Apollo Creed say?

Eye of the Beholder

Some people say that as long as the requirements are met (verified) it’s successful, even if it isn’t useful (not validated).  I disagree.

How about these 2 criteria?

1) Are the stakeholders happy, especially the key stakeholders?

This is everyone, including the project team.  If you drove your team to quitting the company, the project was probably detrimental to the organization.  A project is an end to a means, not an end in and of itself.  If you are not benefiting your company, employees, and customers as a whole group, then why are  you running projects?

2) Does the project fulfill the configuration-controlled and managed performance measurement baseline (PMB)?

I laugh when I hear about projects being ‘over budget’.  That doesn’t really tell you anything.  If you are using change control processes to manage scope, schedule, budget, and quality and you finish the project within the thresholds set, you’re fine.  People look at these massive projects where the total spent is double than the original proposal and think they are automatically failed projects.  To the public who was told it would cost $50MM who finds out it actually cost $100MM, it’s a failed project.  If you are blind to any change control process you can easily think of this as a failure, even though it may have been a resounding success to many other key stakeholders who were directly involved.

A little ranty, I know.  But I had to get that off my chest.  A penny for your thoughts?

{ 11 comments }