When speaking about future events, it’s inherently subject to uncertainty. A risk assessment tries to understand future loss events and is therefore also subject to uncertainty. The less history or fewer measurements, the greater the uncertainty.
How can one predict the consequences of future events when they are subject to significant uncertainty? This article will look at rendering intervals using decomposition and calibration.
Confidence Intervals
An interval is expressed as a minimum value, a maximum value, and a value for most likely (MIN, MAX and ML). For the rest of the article, we only deal with simple intervals with a MIN and MAX value.
We use intervals every day. When asked when you get home from work, you might say between 5:00 PM and 5:30 PM. You have given an interval which considers the uncertainty you have about the time of your return home. You would have to widen your interval range if your uncertainty were greater. Like, if you were to travel the world in your sailboat for the next two years. So, when your friends ask you when you will be home, you would have to say, “I’m at home between 1st and 31st December 2025”. Greater uncertainty, wider range.
Being able to provide intervals that sufficiently take uncertainty into account is essential for a risk assessment. It’s a technique that everyone involved in risk assessments must learn and master. We want respondents with a high level of calibration, an ability to give intervals that consider their uncertainty. It’s beneficial if we can get experts with a deep insight into an area, previous incidents and possible vulnerabilities to yield ranges that consider the uncertainty. Then, we are moving towards forecasting. Forecasting is what we dream of. Imagine if we had a better understanding of the future and what might happen to us.
A term you must know in this connection is a “90% confidence interval (90% CI)”. It’s an interval where you are 90% sure that the correct answer falls into your range. Here’s an example, you see me standing upright next to a kitchen table. Now, you are to provide a 90% confidence interval for my height in centimetres. You think about and give the interval 180-200 cm. You will be uncertain, but no more than you can cast this narrow range. It’s a pretty good range, by the way – I’m 188 cm.
To assess whether you have submitted a good 90% CI, you can ask yourself whether you are willing to play on your range. If you had a wheel of fortune with 9 red and 1 black field, it should be subordinate whether you played on red or your 90% CI.
A respondent is a measuring instrument that must be calibrated before use
eing able to provide good 90% confidence intervals in areas of uncertainty is something you need to practice. We typically suffer from excessive self-confidence or a desire to hit the answer precisely when asked about things fraught with uncertainty. We often submit too narrow intervals, which do not correspond to our real uncertainty.
You must compensate for this built-in error when you become aware of it. What if I ask you how high Mount Everest is? Perhaps, you are an alpinist and have been there, or you might never even have heard of this mountain. The alpinist can give a very narrow range (8,850-9,000 m), while the other has to compensate for his great uncertainty through a very wide range (5,000 – 10,000 m).
When we at ACI work with obtaining estimates, we always train our respondents in that discipline. If you want to see if you are calibrated, you can try our exercises at aci.dk/kalibrering. Prepare for a rather disappointing result the first time, but rapid improvements by taking the test 2-3 times.
Decomposition
If I ask you: “How many cups of coffee are drunk in Denmark per day?” you can choose to look up at the ceiling and then come up with a shot from the hip. Most people do, but such estimates are often far from the actual number.
A better approach is to decompose. You take the big question and divide it into smaller questions, each of which has a lower uncertainty.
You think:
- How many Danes are we? – 5.8 million
- How many Danes drink coffee? – between 50% and 80%
- How many cups does a coffee drinker drink daily? – between 2-6 cups
Because of this decomposition, we can output a range that can be used by simply multiplying the min and max values together.
MIN = 5.8 million x 50% x 2 = 5.8 million cups
MAX = 5.8 million x 80% x 6 = 27.8 million cups
You might object that this is a very broad range. Yes, but it’s an interval that corresponds to the uncertainty I had on the three decomposed questions. Had I known about statistics on questions 2 and 3, I might have been able to narrow my range, but I didn’t.
By the way, Danes drink 22 million cups of coffee daily, according to Mokkland.
The use of decomposition and intervals in an IT risk assessment
Now let’s take an example from the world we live in. You want to understand what the consequence (the expected loss can be) for the following IT scenario:
Incident where an external malicious actor (cyber-attack) attacks the CRM system, which causes the confidentiality of data in the system to be lost.
You call in a few experts from the company. Together, they know both historical events, vulnerabilities and the use of the CRM system. After calibrating the experts, you ask them to come up with intervals for five areas of loss (productivity, handling, compensation, fines and client flight). They emerge with the following decomposition and 90% confidence intervals.
| Loss category | Minimum | Maximum | Interval |
| Productivity | 5 hours for a team of 10 persons | 16 hours for a team of 10 persons | DKK 20.000-64.000 |
| Handling | 3 hours for a team of 2 persons | 50 hours for a team of 10 persons | DKK 2.400-200.000 |
| Compensation towards 3rd parties | DKK 0 | DKK 50.000 | DKK 0-50.000 |
| Fines | DKK 0 | DKK 500.000 | DKK 0-500.000 |
| Client flight | 0 clients | 20 clients | DKK 0-200.000 |
| TOTAL | DKK 22.400-1.014.000 |
In the example, we conclude that this scenario can cost between DKK 20,000 and DKK 1 million. Even though the range is wide, it says a lot. We now understand that the incident will probably (in 90% of cases) not cost more than DKK 1 million. It is good knowledge if you, e.g., must negotiate cyber insurance for this incident and where you are offered a product with a deductible of DKK 1.5 million.
Summary
- An interval contains a Minimum (MIN) and a Maximum (MAX) value.
- A 90% confidence interval is an interval where you are 90% sure that the truth lies within the given interval.
- Decomposition is a technique where you break down a question with great uncertainty into a series of smaller questions with less uncertainty.
- A good risk assessment is based on estimates from experts who have learned to decompose and then give 90% confidence intervals. It’s something you can practice, and you should.
These techniques are essential in the work of quantitative IT risk assessments. However, more is needed to perform a good quantitative risk assessment. We will regularly write about this on these pages.
So, stay tuned.
