Quantitative Risk Analysis: Science or Black Magic?

It’s a common criticism: Monte Carlo Method (MCM) simulations are sometimes perceived as black arts, whereby a bunch of numbers goes in, and out pop answers with little to no visibility of how they were derived. Because we’re making predictions about things that are yet to happen, of course there’s a degree of subjectivity involved. However, by applying the empirical method (defined as conducting an investigation relying on experimentation) to the analysis of these inputs, it is possible to produce informative and accurate predictions about potential project outcomes. In this article, we will look at how ‘art’ and science come together in quantitative risk analysis (QRA) to help calculate cost and schedule contingencies.

Why overlay science on subjective inputs?

First let’s deal with the elephant in the room; what’s the point of applying an empirical methodology to a process based on subjectively gathered data?

We’ve already mentioned that making predictions about future events necessarily involves some degree of subjectivity. For some, this is enough to completely put them off QRA altogether. However, the same individuals are often quite accepting of the related, but more widely used project management practices, such as project planning and estimating on which QRA is based. When you think about it, both of these practices also involve making predictions about the future: when things could be achieved and how much they might cost. When we estimate or plan a job, its price and duration are inherently uncertain, as we haven’t completed it yet.

So we see that in project management, we’re constantly making assumptions regarding things that are inherently uncertain. We typically do this for two reasons:

1) Because in a complex project environment, it’s difficult enough to understand a ‘deterministic’ view of the estimate and schedule, let alone one that also encompasses uncertainty.

2) Because the tools and expertise to produce probabilistic forecasts of project outcomes aren’t readily available to us.

However, when working only with deterministic schedules and estimates, we’re blind to all but one interpretation of how the project will come together and how much it might cost us. By treating estimate line items and task durations as uncertainties however, we’re able to envisage a range of potential outcomes. Add to this the recognition of things that may or may not happen, and you start to get a reasonably representative view of potential project outcomes. Yes, the range inputs are subjective, but it’s clearly more accurate to recognize the inherent uncertainty of future events than to assume them as a fixed / constant value.

Having established that subjectivity is a matter of necessity for all forms of project management that deal with making predictions of future outcomes, we can move on to why the empirical or simulation methodology enhances our ability to make predictions of outcomes based on these subjective inputs.

In reality, most complex projects consist of many, many tasks and costs, all of which are, to a greater or lesser extent, uncertain. There are a multitude of factors that contribute to each of these uncertainties, such as quantity, productivity and rate uncertainties, not to mention uncontrollable external influences such as weather uncertainty. Account for the potential existence of multiple risk events, as well as co-variance between uncertainties and it’s literally impossible for the human mind to piece it all together without some sort of tool to assist it.

MCM simulation is just such a tool. Initially created by physicists working on the Manhattan project to simulate random nuclear events, the methodology relies on repeated random sampling to determine probabilistic outcomes. By performing repeated ‘experiments’ many hundreds or thousands of times and recording the observed outcomes of each, the Monte Carlo method allows us to model very complex interactions between many uncertainties, and to build up a picture of the range of potential outcomes given the inputs provided.

Thankfully, there are many ‘off-the-shelf’ tools available in the marketplace for applying the Monte Carlo simulation method to quantitative cost and schedule risk analysis. These include: Oracle’s Primavera Risk Analysis (formerly Pertmaster), @Risk, Polaris, ARM, Acumen Risk, Crystal Ball, Full Monte, Risky Project and many others. These products enable users to work with large sets of complex interactions between uncertainties in order to generate repeatable predictions of project outcomes that are based on auditable input assumptions. It is this repeatable and auditable process that gives the empirical simulation method the advantage over crude +/- ranging of estimates based on ‘gut-feel’.

How subjective is subjective?

Having explained that the empirical simulation method contributes credibility to the analysis of subjective inputs, it’s worth noting that there are several ways in which the credibility of the inputs themselves can also be bolstered. Clearly, even the most rigorous analysis is going to be flawed if the inputs to it are plain wrong, so let’s take a look at some of the important measures that should be taken...

Firstly, it is important to ensure that inputs are gathered from informed participants. If we want to know about environmental permits, there’s no point in talking to the procurement staff; it’s just not their area of expertise. Ensuring that you are addressing the right people to understand all aspects of project uncertainty is critical to producing credible inputs to the model. Unless people are well informed on the subject matter, they’re unlikely to be able to provide useful insights into the range of potential outcomes for the uncertainty in question.

Second, it’s important to have a diversity of opinions in order to moderate against individual risk attitudes / biases. Individual’s perceptions of potential outcomes are almost always framed by prior experiences. If someone had only been involved with projects that went massively wrong, they’d likely be more prone to increased pessimism relative to someone who had been involved in a more balanced range of project experiences. By polling a greater breadth of informed participants, we protect against the inputs to the analysis becoming unduly driven by the risk biases of each individual.

Finally, perhaps the most powerful tool in reducing the subjectivity of QRA inputs is to bypass the use of opinions altogether. Where actual performance data is available from a previous project, it may be possible to adopt this data as a predictor of future behaviours. For example, why ask people their opinions on rainfall or strong winds, when this data has long been collected and may be freely available from online sources such as the Meteorology Office in the country of the project location. This eliminates the need for subjective opinions altogether, if we can rely on fully quantitative past weather data as a predictor of future weather...

In summary:

• Because quantitative cost and schedule risk analysis involves making predictions about future outcomes, it necessarily involves some degree of subjectivity.

• However, by accounting for uncertainty, simulation provides more realistic interpretations of potential project outcomes than do deterministic predictors such as project schedules or estimates.

• MCM simulation contributes credibility to the analysis of these inputs by providing a repeatable and auditable process by which probabilistic project outcomes may be determined

• It allows for the modelling of many complex interactions beyond what is practical/possible using traditional mathematical methods or ‘gut feel’ approaches.

• Although subjectivity and bias cannot be eliminated, they can be moderated or normalised by gathering a diversity of informed opinions as inputs to an analysis.

• In situations where past performance is a likely predictor of future performance, subjectivity can be further controlled by replacing opinions with historical performance data as analysis inputs.

RIMPL News

RIMPL has recently completed an Integrated Cost & Schedule Risk Analysis of a near-shore gas and condensate field development in Victoria.

RIMPL continues to provide Integrated Cost & Schedule Risk Analysis support for hydrocarbons exploration in Papua New Guinea, providing seismic survey and drilling forecasting services recently.

We have been delivering and writing papers for conferences in New Orleans, Milan, Bangkok and Melbourne. And we have been marketing our services in Western Australia while continuing to deliver services in eastern Australia.

We participated in a four day Floating LNG Workshop run by the Society of Petroleum Engineers in Perth in May, including giving a short presentation on the use of QRA for FLNG projects, and a smaller scale Floating Systems Workshop at the beginning of July, also in Perth, run by ITF (Industry Technology Facilitator)

We recently added a key missing section to our Knowledge Base on Integrated Cost & Schedule Risk Analysis (IRA)

This is an important addition as it represents our core quantitative methodology.

Invitation

Our project personnel subsidiary business Contract Project Services (CPS) is offering $100 Myer Gift vouchers for referrals of capable personnel that result in their engagement with clients. Please contact us if you are aware of competent project controls or other project services personnel looking for contract work.