🔦 Spotlight - Measuring vague concepts

What to do when your CEO says "what's the impact of IT security?"

I wrote about how BizOps can measure (at least some of) its impact on the business. A few people wrote back with questions about their specific situations and how might they be able to apply some of those methods. The examples I gave are of course single point measurements. They help, but they are not the answer to a question of the scope “How do I measure the impact of IT security on my business?” This question is big, vague, but like anything, it is not impossible to tackle. It is of course not about measuring only BizOps, but entire orgs, motions, or initiatives. In this post I am going to dig in a way to define and measure impact of stuff that is tough to measure.

Back to a real world example that I had to tackle while at Hopin: How do you measure the effectiveness of IT security?

Photo by Diana Polekhina on Unsplash

I heard of teams that measure this by e.g. how many employees took a certain training, whether there is a specific system implemented, whether certain certificates are obtained. These are all the wrong measures. With these measures you are measuring something but not the real impact. (I will come back to this later)

To understand the impact of IT security, you need to know two things:

• what needs to be measured,

• and how.

You’d need to understand the drivers of the impact. I like to drill using a simple What does that mean technique. (The What does that mean technique is simply a derivative of the ‘5 Whys’ technique, but with a sometimes better prompt.)

‘What does that mean?’

What does it mean to have security effectiveness?

Without proper security your business would be at risk.

What does that mean?

[Here you identify three main drivers]. You would be at risk of (1) your employees losing productivity, (2) regulators fining you, and (3) customers losing trust in you.

What does that mean?

[Go a level deeper and determine the mechanics] (1) Employees would divert their attention to fixing the damage done by attackers; and some employees would be unable to perform their duties altogether; (2) regulators would launch investigations into your practices, potentially leading to more cost; and (3) customers would be effectively unfulfilling their responsibilities to their customers and their regulators.

What does that mean?

[A level deeper you can start quantifying the effects] (1) Your eng team would lose X amount of time (vs working on new features) per Y amount of engineers affected; if you have a physical space maybe all your employees would lose access to the premises; or at least their machines. (2) there is a chance of a fine in the range of X-Z. And of course (3) a C% of customers would leave, for breach of contract.

What does that mean?

[Create your formula for calculating the risk, using the quantifiable estimates for the three drivers.]

You need to ask “What does that mean” enough many times until you can get to a level that enables you to come up with a back of a napkin calculation. Now I know you know 1) what you need to measure, and 2) how.

Beyond drivers

In interrogating the meaning of IT effectiveness we determined the key drivers of risk should there be inadequate security in place:

• Productivity loss

• Regulatory fines

• Customer loss

In my experience, the more you perform these kinds of estimates, the more sophisticated you can get. This may or may not be needed, depending on the stage of company you are in, and the required level of precision. The earlier you are, the better off you are in identifying the key drivers, and only adding more if the estimated cost would make your overall estimate marginally much better. If you are Amazon, you might also add:

• Brand image erosion

• Operational debt

…And you can do this segmenting by product / service / geo / type of threat etc.

In setting up the measurements operationally, we are not done yet. Identifying the right drivers is the first part of the job. The second part is finding the leading indicators. Drivers become first level ‘lagging indicators’ of success. They each need a leading set.

Grand indicators of success for IT effectiveness can be measured with the (lagging) metrics of downtime, number of breaches, affected users etc. A level deeper, take the driver of productivity loss. One example (lagging) metric can be your engineers are expected to spend 4-6h tackling an attack. You can break this into (leading) indicators for reducing that time frame: e.g. speed of breach identification, speed of damage area identification, lock down speed, etc.

Final Word

Now that we have arrived at the end of this simple walk through, I hope you never ever want to measure effectiveness of anything by the number of people that completed a training. Measuring the number of employees completing a training measures how well you pestered them to do it. But doesn’t measure whether you delivered impact or not. This is why when HR pesters you to complete your anti-bias training, that’s all that it is - ticking a box, not tackling the real issue (sorry, not sorry). If you want to reduce bias, find the drivers of the behaviour. Now measure those.