Risk Matirces Are Useless

[Fred Lewis]

Reference is made to a typical risk matrix such as the one proposed by Transport Canada.

 

The claim is made that this risk matrix is objective. Nothing could be further from the truth. The degrees of both probability and severity are arbitrary and therefore subjective. In order for the measure of a thing to be objective, some sort of number must be attached to it. Sometimes objectifying reality is a difficult but risk matrices are wrong and can be dangerously misleading. The discussion of risk is more thoroughly described by approaching it from the statistical perspective which includes an application of probability.

 

The TC risk matrix attempts to classify probability into five levels. What is the difference between the probability of Level 1 Risk, described to be “Mishap almost impossible” and Level 2 Risk described to be “Postulated event. (Has been planned for, and may be possible, but not known to have occurred)”? One could argue that “almost impossible” is in fact a more frequent occurrence than “may be possible, but not known to have occurred”. The same is true of a comparison of Level 3 Risk and Levels 1 and 2. The best that can be said about these loosely defined levels is that they all describe the same level of probability.

 

Level 4 and Level 5 Risk are likewise similar. They really describe the same level of risk. In the grand scheme of things when the nature of helicopter accidents is considered on a global scale, just about every conceivable calamity befalls someone somewhere on an annual basis, effectively eliminating Level 4 risk as a consideration.

 

The criteria for severity of the levels of risk are similarly subjective. How could any rational person chose between courses of action which could result in “Personnel - Disability/Severe injury. “ or “Personnel - Fatal, life threatening.”? Under what conceivable circumstances could one option be acceptable and the other not in light of the fact that no objective probabilities for these events are provided with which the matrix user can work?

 

The values of probability and the values of severity are then multiplied together to quantify risk. Are we to believe that probability 5 and severity 5 is 5 times riskier than probability 1 severity 5? What is the statistical basis for this? Why are the risk and severity multiplied together? Why are they not added? There is no reason.

 

Objectifying risk in aviation is difficult. The probability of occurrences must be determined empirically as opposed to mathematically and can therefore be inaccurate. To illustrate, consider the tossing of a coin or the rolling of a die. In the former, the probability of a toss resulting in a head can be immediately computed to be 0.50 and the probability of not tossing a head to be 0.50. In the latter case, the probability of rolling any particular number is one in six, or 0.17 and the probability of not rolling a particular number is 0.83.

 

Suppose one plays a game in which one wagers $1 that the result of the roll of a die can be correctly guessed. Success results in a payoff of $5. If the game is played 600 times, on average success will be achieved 100 times. The winnings will be $500 but it will have cost the player $600 in wagers for a net loss of $100. When the odds of winning are greater than the ratio of the wager to the payoff, the game is a winning proposition or

 

Probability of success >= Wager/Payoff.

 

In the case of the die game, the probability of success 0.17 and the Wager/Payoff is 0.20. It is not a good bet.

 

This applies to empirically determined probabilities as well as mathematically determined ones.

The Transportation Safety Board of Canada has published the number of fatal helicopter accidents (66) and the total number of hours flown (6,167,000) for the years 1998 to 2007 inclusive. From these numbers, the average fatal accident rate per hour for these years can be estimated. It is 0.0000107 and the probability of no fatal accident is 0.9999893.

 

Another legitimate way of looking at this is that if 10,000,000 helicopters each flew for one hour, 107 of them would be involved in a fatal accident. Conversely, 9,999,893 would not.

 

Suppose a helicopter with a hull value of $1,000,000 does a one hour job and that the operator is on the hook for it if there is total loss. The wager is $1,000,000 and the payoff is the return of the machine undamaged and any profit realized, say $500. If the probability of success is greater than the wager divided by the payoff, then in the long run, the operation is a winning proposition. In this hypothetical case, the wager divided by the payoff ($1,000,000/$1,000,500) is 0.9995. The probability of success (0.9999893) is greater than the ratio of the wager to the payoff (0.9995). It is a winning proposition.

 

This scenario is simplified. In reality the passengers have value and so does any cargo. The range of loss can be from total to perhaps as little as a few thousand dollars but almost all the time there is a small gain. Each sort of loss will have its own probability of occurrence. However, the point that there is a real way of quantifying probability and relating it to helicopter operations is illustrated.

 

Increasing the probability of success and the payoff, and reducing the wager will favour the operation. The wager can, for instance, be reduced by not taking more passengers than completion of the job requires. The payoff can be increased by maximizing the profit on each hour flown, which some operators do very poorly, but the most effective way of reducing the risk in to increase the probability of success, which is almost entirely in the control of the pilot.

 

The pilot can increase the probability of success by: always loading the aircraft so that it will have sufficient reserves of thrust to maneuver nimbly out of ground effect without exceeding engine or transmission limits; considering relative humidity when computing density altitude; using the utmost discretion when flying in marginal weather; not operating in the shaded area of the height velocity diagram longer than is necessary; closely monitoring his physical and psychological health and not flying when either of those are compromised; not flying when he is fatigued; not pressing the limits of his abilities; not attempting to match the performance of pilots of superior skill; cultivating skills for firmly but politely dealing with aggressive customers; cultivating skills for firmly but politely dealing with aggressive employers.

 

The pilot is the single most important part of a safe and successful helicopter operation. The responsibility for human lives and expensive machinery is in his hands. No known laws equip the pilot with the authority to deal with this responsibility. It is something every pilot must develop for himself.

 

None of this is rocket science. The math has been around for hundreds of years. It is very sad when the wheel is reinvented disguised, in this case as a risk matrix, as some new and innovative thing. Risk matrices are wrong. They will never give the right answer. The common sense approach to safety will produce better results.

For another perspective on risk matrices have a look at What’s Wrong with Risk Matrices?

[GoldMember]

Don't you think that any tool that actually, at the very least, puts in motion a thought process that may in fact help a person avoid an unsafe situation, is at least worth utilizing? Rather than talk about how wrong something is, why not show how you would improve it? Not every pilot or engineer has years and years of experience to draw upon when making decisions, as I'm sure you do, so if reviewing or creating a risk matrix helps them, I'd say mission accomplished.

[412driver]

The common sense approach to safety will produce better results.

 

 

nuff said.....

[Lunchbox]

I don't think that the intent of risk matricies is an end product in and of itself. You're not going to be looking at a risk matrix before crossing the street and judging how and when to cross the street based on what the risk matrix says. The matrix is there to quantify risks in some way. Quantifying anything human related is always going to be difficult.

 

I think risk matricies are intended more as a somewhat standardized way to help organize, judge, and develop actions for various risks. It's really just a way of putting thoughts onto paper. Yeah, you can try to keep it all in your head, but eventually you have to come up with some way of putting it down on paper because you just can't keep track of everything in your head.

 

They key is, that one has to use common sense when using risk matricies

[Heliduck]

 

"None of this is rocket science. The math has been around for hundreds of years. It is very sad when the wheel is reinvented disguised, in this case as a risk matrix, as some new and innovative thing. Risk matrices are wrong. They will never give the right answer. The common sense approach to safety will produce better results."

 

 

You've done quite a detailed analysis on the risk MATRIX, the focus of our attention should be on the risk ASSESSMENT. The risk matrix is merely a tool to use during the process of conducting the risk assessment.

 

You've focused on the mathematical irregularities of the risk matrix, & from this I could probably guess the final risk rating you as an individual would apply to a task. The real strength in conducting risk assessments is when they are not done in isolation, but as a group. Then prejudices, opinions & acceptance of risk can be balanced against others who may have had different experiences to us, & using that vastly increased pool of experience a reasonably accurate guage of likelihood & consequence can be determined. As Goldmember said, an individual or a group stopping & thinking about the seperate steps of the task & what hazards are relative to each step is a huge benefit to safety, what mathematical formulae you apply to each rating is irrelevant. The matrix is a tool to prompt the thought process, not a calculator.

There is never a right & wrong answer to a risk assessment, we need to use our training & previous experience as a crystal ball & try to foresee the future. If the future involves equipment damage or injury, we need to put controls in place or take a different path to our goal to minimise the chance of a negative outcome.

 

Common sense??? Unfortunately common sense is not that common, & using a tool which prompts discussion & thought about a task can help identify areas of concern for people who don't have as much common sense as they should. I fly single pilot utility work day & night & personally I don't use a piece of paper before each flight to calculate the risk, but I do consider any hazards & do my best to avoid or minimise them. That works for me, but some people might need to write stuff down in a consistent format & that's where the risk matrix comes in. I have met some pilots who would frighten the **** out of me if I was in a multi crew situation with them, so what's common sense for 1 person might not even enter the head of another. I also think there must be a better way to do this than a risk matrix, but I haven't been able to come up with one which also satisfies the corporate requirement for documenting the common sense most of us apply on a daily basis.

[skullcap]

I find the good thing about a risk matrix is that it does make people discuss the risk, but is a risk matrix required to have this discussion. Quantative and qualitive levels seemingly are needed to determine risk levels but in my opinion only to help compare the risk, but my gosh, how long is a piece of string and just because the risk matrix comes out in the green does not by any means allow you to go merrily along thinking things are rosey.

 

Interesting conversation the other day with someone in the know when I meantioned that I have a hard time coming up with some of the risk levels such as the risk of having an engine failure when over different types of terrain. Was given an example that if we did not fly over built up areas at all or very seldom the risk is very low to have an engine failure over a built up area, but personally have an issue with that as the helicopter really does not know what the terrain is it is flying over and the risk is exactly the same be it over trees, fields, ocean or built up area(barring of course altitude, temperature, and load on powertrain). I think that the failure rate is sometimes mistaken for the the risk of failure and there in lies the rub.

 

One concern I have with the matrix is that you may be held liable if you misdiagnose the risk level, so is imperative that you try and mitigate the risks whenever possible.

[old]

Reference is made to a typical risk matrix such as the one proposed by Transport Canada.

How refreshing to have something on this site worth considering. Thanks for this post.

 

The claim is made that this risk matrix is objective. Nothing could be further from the truth. The degrees of both probability and severity are arbitrary and therefore subjective. In order for the measure of a thing to be objective, some sort of number must be attached to it. Sometimes objectifying reality is a difficult but risk matrices are wrong and can be dangerously misleading. The discussion of risk is more thoroughly described by approaching it from the statistical perspective which includes an application of probability.

 

The TC risk matrix attempts to classify probability into five levels. What is the difference between the probability of Level 1 Risk, described to be “Mishap almost impossible” and Level 2 Risk described to be “Postulated event. (Has been planned for, and may be possible, but not known to have occurred)”? One could argue that “almost impossible” is in fact a more frequent occurrence than “may be possible, but not known to have occurred”. The same is true of a comparison of Level 3 Risk and Levels 1 and 2. The best that can be said about these loosely defined levels is that they all describe the same level of probability.

 

Level 4 and Level 5 Risk are likewise similar. They really describe the same level of risk. In the grand scheme of things when the nature of helicopter accidents is considered on a global scale, just about every conceivable calamity befalls someone somewhere on an annual basis, effectively eliminating Level 4 risk as a consideration.

 

The criteria for severity of the levels of risk are similarly subjective. How could any rational person chose between courses of action which could result in “Personnel - Disability/Severe injury. “ or “Personnel - Fatal, life threatening.”? Under what conceivable circumstances could one option be acceptable and the other not in light of the fact that no objective probabilities for these events are provided with which the matrix user can work?

 

The values of probability and the values of severity are then multiplied together to quantify risk. Are we to believe that probability 5 and severity 5 is 5 times riskier than probability 1 severity 5? What is the statistical basis for this? Why are the risk and severity multiplied together? Why are they not added? There is no reason.

 

Objectifying risk in aviation is difficult. The probability of occurrences must be determined empirically as opposed to mathematically and can therefore be inaccurate. To illustrate, consider the tossing of a coin or the rolling of a die. In the former, the probability of a toss resulting in a head can be immediately computed to be 0.50 and the probability of not tossing a head to be 0.50. In the latter case, the probability of rolling any particular number is one in six, or 0.17 and the probability of not rolling a particular number is 0.83.

 

Suppose one plays a game in which one wagers $1 that the result of the roll of a die can be correctly guessed. Success results in a payoff of $5. If the game is played 600 times, on average success will be achieved 100 times. The winnings will be $500 but it will have cost the player $600 in wagers for a net loss of $100. When the odds of winning are greater than the ratio of the wager to the payoff, the game is a winning proposition or

 

Probability of success >= Wager/Payoff.

 

In the case of the die game, the probability of success 0.17 and the Wager/Payoff is 0.20. It is not a good bet.

 

This applies to empirically determined probabilities as well as mathematically determined ones.

The Transportation Safety Board of Canada has published the number of fatal helicopter accidents (66) and the total number of hours flown (6,167,000) for the years 1998 to 2007 inclusive. From these numbers, the average fatal accident rate per hour for these years can be estimated. It is 0.0000107 and the probability of no fatal accident is 0.9999893.

 

Another legitimate way of looking at this is that if 10,000,000 helicopters each flew for one hour, 107 of them would be involved in a fatal accident. Conversely, 9,999,893 would not.

 

Suppose a helicopter with a hull value of $1,000,000 does a one hour job and that the operator is on the hook for it if there is total loss. The wager is $1,000,000 and the payoff is the return of the machine undamaged and any profit realized, say $500. If the probability of success is greater than the wager divided by the payoff, then in the long run, the operation is a winning proposition. In this hypothetical case, the wager divided by the payoff ($1,000,000/$1,000,500) is 0.9995. The probability of success (0.9999893) is greater than the ratio of the wager to the payoff (0.9995). It is a winning proposition.

 

This scenario is simplified. In reality the passengers have value and so does any cargo. The range of loss can be from total to perhaps as little as a few thousand dollars but almost all the time there is a small gain. Each sort of loss will have its own probability of occurrence. However, the point that there is a real way of quantifying probability and relating it to helicopter operations is illustrated.

 

Increasing the probability of success and the payoff, and reducing the wager will favour the operation. The wager can, for instance, be reduced by not taking more passengers than completion of the job requires. The payoff can be increased by maximizing the profit on each hour flown, which some operators do very poorly, but the most effective way of reducing the risk in to increase the probability of success, which is almost entirely in the control of the pilot.

 

The pilot can increase the probability of success by: always loading the aircraft so that it will have sufficient reserves of thrust to maneuver nimbly out of ground effect without exceeding engine or transmission limits; considering relative humidity when computing density altitude; using the utmost discretion when flying in marginal weather; not operating in the shaded area of the height velocity diagram longer than is necessary; closely monitoring his physical and psychological health and not flying when either of those are compromised; not flying when he is fatigued; not pressing the limits of his abilities; not attempting to match the performance of pilots of superior skill; cultivating skills for firmly but politely dealing with aggressive customers; cultivating skills for firmly but politely dealing with aggressive employers.

 

The pilot is the single most important part of a safe and successful helicopter operation. The responsibility for human lives and expensive machinery is in his hands. No known laws equip the pilot with the authority to deal with this responsibility. It is something every pilot must develop for himself.

 

None of this is rocket science. The math has been around for hundreds of years. It is very sad when the wheel is reinvented disguised, in this case as a risk matrix, as some new and innovative thing. Risk matrices are wrong. They will never give the right answer. The common sense approach to safety will produce better results.

For another perspective on risk matrices have a look at What’s Wrong with Risk Matrices?

[Skids Up]

The purpose of quoting this in it's entirety is...?

[helicopterjim]

What the #### is a risk matrice? I don't have one in my flight manual, my ops manual or my life manual? Am I unsafe without one? I seem to have survived 30 years flying without one ......

[MMike]

At the risk of upsetting people, that diagram with the red yellow and green square the exact same diagram they used in my risk management training as it pertains to project management.

 

Things like,

will a machine breakdown making it impossible to build your prototype?

Will your critical supplier go out of business?

will your pregnant designer go on maternity leave early?

will the zombie-pocalypse happen?

 

Anything that will jeopardize the project.

 

Personally, I have a hard time assigning a quantitative value to the risk. But I think it IS a worthwhile exercise to brainstorm anything (within reason) that can go wrong. At least it forces you to remove your rose coloured glasses to evaluate your situation critically. In theory it should reduce the chances of being blind-sided by something bad happening.

 

That said, I'm not sure how it would relate to a pilot flying the same mission day after day.

 

Please forgive my impudence.