Newsletter December 2016

Welcome to the December 2016 issue of “RiskIntegral” - an occasional newsletter about the analysis and management of risk, mainly in projects, by Risk Integration Management Pty Ltd (RIMPL). In this issue, we look at the end-to-end ingredients needed to achieve project success and review a book we believe to be one of the most important published on quantifying time and cost risk in projects.

The newsletter concludes as usual with news of our activities and our reflection on the tragic death a year ago of our friend and WA representative Peter Hudson.

INTEGRATING SUCCESS FACTORS FOR PROJECTS

Introduction

While our focus is often on the roles of the original schedule and estimate in enabling project success through the provision of adequate time and cost contingencies, we recognise that is but one element of the mosaic of pre-requisites to achieve successful project and program outcomes.

This articles steps back and asks “What are all the ‘critical success factors’ needed to develop and deliver successful projects and programs?” This is to counter the tendency to over-simplify to advance an argument. Reality, as in almost all aspects of life, is more complex, as illustrated by the story of the blind men feeling different parts of an elephant and drawing incomplete conclusions about it.

Source: (Image courtesy of nature.com) Cartoon originally copyrighted by the authors G. Renee Guzlas

Project Critical Success Factors

The following elements are all needed in one form or another, for completion of successful complex and major projects (and programs):

• A clear and valid business case for the project that justifies the need for and benefits of developing and delivering the project, including an expected benefit/cost ratio or IRR / NPV from the resultant asset(s) as applicable.

• Scope definition sufficient to produce a realistic estimate of the cost. This is true for all types of projects, especially for Information and Communications Technology (ICT) projects, for which scope creep during project execution is an endemic problem. In the language of the project benchmarking consultants Independent Project Analysis, Inc. (IPA), that means sufficient Front End Loading (FEL) to define WHAT is to go into the project and HOW it is to be developed and delivered.

• A robust, risk-based estimating process that references past performance (is empirical), includes schedule development, defines and aligns the Basis of Estimate with the Basis of Schedule (and vice-versa) and produces realistic quantification of time and cost contingencies and management reserve based on a comprehensive assessment of:

Ο Systemic risks especially relating to the capacity of the organisation to deliver the project/program;

Ο Project specific time and cost uncertainties;

Ο All significant project time and cost impact risk events identified by qualitative risk analysis and refined by quantitative risk review and avoids self-inflicted risks due to faulty processes.

• An effective project gateway review process especially for Financial Closure so that the project/program starts off with realistic allocations of time and money.

• Good project leadership and governance set in a mature project management organisation and culture. This means producing timely and appropriate decisions to set up the project with optimal delivery and contracting strategies and then delivering it. It also means applying corrective actions when required to deal with the inevitable deviations from planned conditions and the occurrence of risk events.

• A capable project controls team that tracks and analyses project progress and expenditures and reports in a timely and incisive way whenever deviations and risk events occur that require corrective decisions and actions by the PM team. Likely to be using Earned Value Management.

• Effective, timely and comprehensive recruitment and training of personnel required to operate the assets produced by the project / program so that they provide guidance on Operations & Maintenance (O&M) efficiency to the designers and are ready to be involved in commissioning and startup and to take over when the assets are ready for operation.

• A budgeted and approved process for closing down the project, documenting project performance metrics and lessons learned plus a process for archiving the project performance and lessons learned and circulating the information to those who need to be aware, including:

Ο Designers, Estimators and Planners

Ο Project Managers & Project Directors

Ο C-level managers who make decisions on sanctioning future projects

• A project development and delivery system that includes reference data on previous projects that will enable estimators to produce progressively more realistic estimates for projects over time.

One of the realities to achieving end-to-end success is that, just as for the blind people experiencing different parts of the elephant, there are different stakeholders and parties responsible for ensuring the different elements above are performed to best practice standards. The challenge is to rehearse the performance so that they are all playing from the same sheet music in time to the conductor’s baton! This requires experienced leadership at all levels, but especially at the top of the organisation.

This list is almost certainly incomplete and readers are welcome to provide feedback to improve its usefulness and also to make use of the checklist for the future benefit of project delivery organisations.

BOOK REVIEW: ‘PROJECT RISK QUANTIFICATION’ BY JOHN K HOLLMANN

Introduction

The poor forecasting record of organisations which develop and deliver complex and major projects is a matter of public record. Too often the time taken and capital expenditure required to complete projects significantly exceed the announced numbers when the projects are launched, in a wide range of industries and sectors. This review is of a book that could, if the author’s goals are fulfilled, produce material improvement in the success rate of major and complex projects achieving their time and cost targets. We consider it to be a very important book that has changed our thinking and is changing our methodology. The review examines why we believe this book to be very significant.

Scope of Book

“Project Risk Quantification” (PRQ), as can be seen from the accompanying image of the cover, is subtitled “A Practitioner’s Guide to Realistic Cost and Schedule Risk Management”. It is presented in two parts. The first eight chapters are written for a general audience with a stake or an interest in the achievement of accurate forecasting of project time and cost outcomes. Part 1 explains:

• Why risk quantification is important in projects.

• That the risk quantification methods used should be based on empirical evidence (from past results) that they work and that the methods are practical to use. (The book cover illustrates this with an image of the Roman God Janus who looked both backwards and forwards.)

• The Phase Gate approval system for capital projects and its relationship to scope definition, capital growth and schedule slippage.

• Accuracy of time and cost forecasting and its variation with project scale versus the reality of unrealistic expectations of project owners.

• How projects are assessed at the Decision Gate and the role of risk quantification in this.

• How project estimates are built up, including how biases are incorporated and how the behaviours of stakeholders can help or harm projects.

• The use of Risk quantification in project execution for setting up and managing project controls effectively and the influence of project strategy (conscious or unconscious) on the adequacy of contingency.

• Managing Escalation and Exchange Rate uncertainties and the use of Management Reserve.

• An introduction to risk quantification methodologies: what is recommended and what is not.

• Organisational requirements for effective risk quantification and management – structure, capabilities, people and analysis tools. Owner and EPC Contractor examples of successful risk quantification approaches.

The second part- chapters 9 to 18 - is addressed to quantitative risk practitioners, providing a careful explanation of the steps, tools and techniques Hollmann recommends using to:

• Establish the base estimate and schedule

• Employ descriptive and inferential statistical tools to understand project data and analyse it

• Assess three kinds of risk:

Risk Type	Description	Recommended Assessment Method
Systemic	Risks arising from systematic causes such as the capabilities of the project owner’s project development and delivery organisation and systems	Parametric methods with Monte Carlo Simulation (MCS)
Project Specific	Time and cost impact risks particular to the project	Expected Value method with MCS
Escalation and Exchange Rate	Uncertainties of these parameters affecting the project	MCS

• Assess large and complex projects at risk of tipping into a chaotic state where overruns in labour cost can blow out by up to the order of 200% of budget in extreme cases and to forecast the probability and impact range of such blowouts.

• Apply the recommended approach to sectors other than process industries.

• Communicate analysis results to the stakeholders, especially large contingency recommendations to owners.

• Drawdown contingencies and Management Reserves (if required) during project execution.

• Close the loop by capturing project performance and lessons learned

There are two appendices. Appendix A describes a method whereby systemic risk calculated by the parametric method is applied to a Critical Path Method (CPM) model, based on AACE International’s Recommended Practice 57R-09 Integrated Cost and Schedule Risk Analysis using Monte Carlo Simulation of a CPM Model (written by leading practitioner and author Dr David T Hulett). Appendix A also demonstrates the use of the CPM model to model risk responses (treatments) using probabilistic branching and shows how to model a project tipping into a chaotic state.

Appendix B addresses bias in small and conservatively biased projects and how to interpret and analyse historical data for such projects to derive more realistic foundations for future project performance forecasting.

Who is the author John K Hollmann?

The following detailed information is provided about John Hollmann to demonstrate how his life experience and capabilities uniquely equip him to provide authoritative guidance on quantifying project risk.

John Hollmann graduated in Pennsylvania as a mining engineer and worked in that field for nine years, completing an MBA during that period. He became a Cost Engineer, working in a nuclear waste storage program for a couple of years before switching to the process and pipeline industries working for Fluor Daniel for four years as a Senior Project Controls Engineer. Through the mid-nineties he worked for Eastman Kodak for six years, learning then practising cost estimating as a Senior Estimator while estimating the cost of chemical and consumer manufacturing facilities and leading the development of estimating systems, methods, tools.

In the late 90s, John joined independent Project Analysis, Inc. (IPA), where he worked as a Senior Analyst for seven years, benchmarking projects and leading research into cost engineering / project controls.

From 2005 onwards, John has run his own cost engineering, estimating and project controls consultancy Validation Estimating LLC. He specialises in evaluating the cost engineering practices of project-focused owner organisations, then helping them to improve, taking them through the processes described in this book. His goal is to support and train the organisation’s personnel to implement the processes repeatably so that they can become self-sufficient in estimating and project controls through the project development and delivery life-cycle.

Any biography of John Hollman, however short, must include the profound contribution he has made to the field of Cost Engineering and the Association for the Advancement of Cost Engineering (AACE) International, of which he has been a member since 1989 (27 years). He has probably contributed more than any other AACEI member to the intellectual rigour of the discipline of Cost Engineering by leading the writing of a number of Recommended Practices and guiding and supporting the writing of many others. He has also written and presented many seminal papers at AACEI Annual Meetings, subsequently published in AACEI’s Cost Engineering magazine.

But his magnum opus has arguably been the creation and editing of AACEI’s “Total Cost Management (TCM) Framework”, subtitled “An Integrated Approach to Portfolio, Program and Project Management”. Using a systems engineering approach, it is “a conceptual representation that provides a structured, integrated overview of cost engineering.” It forms the foundation of cost engineering. TCM is incorporated in the constitution of AACEI and the Framework includes a process map that sets out the relationships between the various aspects of cost engineering, including those relating to strategic asset management and project development and delivery from a project controls perspective. The TCM Framework represented 10 years of effort, from 1996 to 2006 to issue the first edition, plus further effort culminating in 2011/12 in the publication of a major revision.

John Hollmann subsequently stepped back from this intense level of effort for AACEI but the TCM Framework continues to evolve, with a new edition recently published.

Why is this book significant?

PRQ is significant for a number of reasons, revealing and explaining:

1. The centrality of risk (effect of uncertainty on objectives according to ISO 31000:2009 with a similar but longer definition by AACEI) in developing and delivering projects and operating the assets produced.

2. The centrality and pervasiveness of risk playing out in previously unrecognized or under-appreciated ways such as:

a. The estimating strategy underlying the project schedule and project estimate, whether stated explicitly or implied in the approaches taken overtly or through lack of direction

b. The way that the project organisation is led (or not), structured and supported by functional or absent project systems, procedures, processes and practices

3. The categorisation and modeling of risk applicable to projects into:

a. Systemic;

b. Project specific; and

c. Escalation and exchange rate uncertainty

4. The generally dominant role of systemic risks, led by those relating to the ability of the project delivery organisation’s to deliver the project as planned, despite the reluctance or refusal of the organisation to recognise this kind of risk (what Hollmann characterises as the “unknown knowns” in Rumsfeld-speak).

5. The use of Parametric Methods to assess systemic risk.

6. The use of Expected Value to evaluate project specific risk.

7. The use of Monte Carlo Simulation to convert mean values into probabilistic distributions.

8. The integration of the methodology into a process for assessing project risk, contingency and management reserve

9. The tendency of at-risk projects with poor systems to tip into chaos when exposed to project stressors, including a methodology to calculate the probability and severity of the chaos.

10. The tendency of small projects to be over-estimated to avoid overruns and how to correct for this in analysing and normalising past such project performances.

11. The importance of documenting past project performances and lessons learned usefully for future reference and improved estimating and project outcomes (“closing the loop”).

12. How to document and report what matters from risk quantification assessments.

13. How to manage contingency effectively through project execution.

14. How to apply the methodologies described in the book for owners in the process industries to other industry sectors and to the contractor perspective.

15. How to apply systemic risk assessed using parametric methods to Integrated Cost-Schedule Risk Analysis using Monte Carlo Simulation.

16. How to improve the organisation’s ability to deliver projects and lower the level of systemic risk.

What are the strengths of the book?

PRQ alerts the reader to the pervasiveness and dominance of systemic risk in projects, particularly relating to the capabilities or inadequacies of the organisation to deliver the project and how to assess and quantify this. The background and character of John Hollmann makes him uniquely qualified to reveal this and apply it “without fear or favour”. His commitment to train his clients to replicate his methodology means that his aim is, like Johnny Appleseed, to leave a beneficial trail behind him, in his case of enhanced organisational and individual capabilities.

Also exposed in PRQ is the importance of identifying the strategy behind the estimate and schedule by analysing the biases, consciously or unconsciously, incorporated. When the class and quality of the estimate are added to assess the potential for the scope to grow, the importance of assessing where the base estimate and schedule lie on the probability axis becomes clear for the subsequent quantification of both kinds of contingency.

PRQ emphasises the importance of empiricism – basing risk quantification methods for predicting future project outcomes realistically on measured past performance (after controlling for extraneous variables).

Scale and complexity in projects change the degree of difficulty of developing and delivering them successfully. The book alerts the reader to the measured tendency of small scale projects to be driven by failure avoidance, to the extent that they are often conservatively estimated and are inefficient in their use of capital.

What are the weaknesses of the book?

John Hollmann has an estimating rather than a planning background. He admits to an antipathy toward project planning. He argues that CPM based risk quantification methods are too complicated and that it is unrealistic to expect that the project schedule developed by the project team will be good enough to use for realistic integrated cost and schedule risk analysis.

Our experience of performing Integrated Cost & Schedule Risk Analyses is that schedule quality is almost always able to be improved to an acceptable standard through interaction with the project team. The performance of NASA using its CPM-based Joint Confidence Limits (JCL) methodology, (similar to our IRA methodology) as noted in our December 2015 Newsletter, confirms the value of CPM risk quantification in materially improving the accuracy of project time and cost forecasts.

John Hollmann also believes that risk responses or risk treatments cannot be modeled realistically using CPM based modeling in cases where the project management team is able to compensate for schedule slip by spending extra money to accelerate the rate of progress or may choose to change schedule logic to avoid or minimise the risk impact.

He does note that probabilistic branching can be used to limited effect in CPM modeling to model the response to a risk, but states that responses may be quite elaborate and difficult to foresee. Our response to this is twofold:

• Firstly, through workshopping the risk and its possible treatments it is possible to arrive at consensus on how the team would respond to a threat and may conclude that what started as a schedule impact risk may, after consideration, be a cost impact treated risk or perhaps a cost impact and reduced schedule impact treated risk.

• Secondly, where the response may change, depending on when the threat occurs in the project, or some other uncertain factors, it is possible to incorporate macro-driven decision making in the CPM-based IRA model. It is our intention to demonstrate this in a paper being prepared for the AACEI June 2017 conference.

To be fair, John Hollmann does recognise benefits of CPM-based analysis: “detailed insight to schedule behaviour in an orderly project” (non-chaotic). He also recognises that if the analysis team has the skills and time to do it and has acceptable input schedule quality, CPM-based Integrated Cost & Schedule Risk Analysis including his hybrid approach for systemic risk quantification can be valuable.

Overall Recommendation for “Project Risk Quantification”

If you have an interest in quantifying time and cost risk in projects, this book is a “must have” addition to your professional library. It will change the way you think about risk in projects!

RIMPL NEWS

Quantitative Risk Analyses

Since our last RIMPL Newsletter in the first half of 2016 the following activities have kept us active in our core business:

• Continuation of Schedule Risk Analyses for Downer EDI Rail started in March and completed in October for the now successful Evolution Rail Consortium’s tender to build own and operate the 65 train sets to be put into service as the High Capacity Metro Trains (HCMT) for the Melbourne metropolitan rail network, including design and construction of a training and maintenance facility near Pakenham.

• Schedule Risk Analyses for Downer EDI Rail for the 24 train sets Sydney Growth Trains Program in July, which Downer EDI Rail has since won.

• IRAs to review the adequacy of time and cost contingencies allocated to the new water-based paint manufacturing facility being built by Dulux on the northern outskirts of Melbourne, conducted by RIMPL in May/June and again in September/October.

Project Controls & Contract Administration Review Services

We have continued to provide project planning and forensic scheduling advisory services for manufacturing and commercial building companies and also provided a review of contract administration performance for a Victorian Municipality:

• Provision of planning services for Australian dairy manufacturing facility projects through 2016 up to August.

• Planning and Forensic Scheduling Advisory Services for a Commercial Construction company, as required through 2016, up to August.

• Provision of planning services for a major marine maintenance overhaul project from May 2016 until early 2017.

• Detailed contract administration review of community projects for a Victorian Municipality from July to September.

Vale Peter Hudson – RIMPL’s former Western Australian Representative

We are approaching the first anniversary of the death of our representative in Western Australia, Peter Hudson. Peter tragically died in a car accident just before Christmas last year.

Peter, a qualified chemical engineer, was an enthusiastic and very capable project management consultant who had held various key roles including Project Director for a number of process industry projects in Australia and overseas. He was very generous with his time and had many friends and colleagues across the project management community in Western Australia, particularly in the oil & gas sector. He was a person of strong faith and his death has been a huge blow to his wife Deborah and his children and their families. He is sadly missed by friends and former colleagues, including ourselves.

CONTACT US

18 Cottesloe Court
Doncaster East VIC 3109
AUSTRALIA
+61 412 031 161
info@riskinteg.com