Understanding Security Evaluations

A security evaluation should help you to understand your risks.  Risks are situations or properties of a system that expose things that you care about to danger or loss.  For example, security problems in your product may expose you to a financial liability or cause your customers to stop trusting you and hence buying your products.  Both of these can put your company, job or reputation in danger.

The table below summarises the properties of the common approaches to security evaluations.  The meaning of type and methodology are explained below.

 

 

Methodology

Black Box

White Box

 

Type

Penetration

Test

Aim: Demonstrate insecurity

Risk understanding: Low

Scope: Known vulnerabilities

Coverage: Very low

Automation use: High

Value: Low

Cost: Low

Aim: Demonstrate insecurity

Risk understanding: Low

Scope: Known vulnerabilities

Coverage: Low

Automation use: High

Value: Low

Cost: Low

Vulnerability

Assessment

Aim: Evaluate risk

Risk understanding: Low

Scope: Entire system

Coverage: Low

Automation use: Low

Value: Low

Cost: Medium

Aim: Evaluate risk

Risk understanding: High

Scope: Entire system

Coverage: All

Automation use: Low

Value: High

Cost: High

 

Evaluation types

The most common security evaluation types are “Penetration Testing” and “Vulnerability Assessment”.

The aim of Penetration Testing is simply to see if defences can be broken.  This means that a Penetration Tester will look for the easiest method of attack and is likely to rely heavily on automated tools targeting known vulnerabilities (e.g. security problems in software libraries).  A Penetration Test will stop once a security problem has been found as the aim will have been achieved or (in the case of the usual business model) continue for a fixed period of time or budget to try and find a security problem.

Penetration Tests therefore have limited value to help you to understand your risks as the coverage (vulnerabilities found versus vulnerabilities likely to be present) is small and there are likely to be very large blind spots.  Even if the particular security problem discovered is fixed, there may be many more.

A Vulnerability Assessment is very different – it actually IS a risk assessment and its aim is to determine ALL the ways in which your system can be attacked and the assets (the things you care about) accessed or damaged.

A Vulnerability Assessment considers all the assets (what you need to protect), the threats against them and all the ways that the system can be attacked.  It can then propose mitigations (ways to prevent or reduce the damage) and assign risk.  This then allows a fully risk-prioritised action plan to be created, resulting in the best security improvements for a given financial budget to be made.

Unlike Penetration Testing, Vulnerability Assessments rely much less on automated tools and generally require much higher levels of skill and manual effort to really understand the system being evaluated.  This results in them being initially more expensive.  However, the benefits are a complete understanding of the risks and the ability to get the best return of investment when fixing the problems discovered.

 

Evaluation methodologies

Evaluations are typically performed in a “White Box” or “Black Box” manner. 

White Box evaluations are where the people performing the evaluation are provided with full information about the system.  This may include things like circuit diagrams, hardware information, software with source code, network configurations, etc.

Black Box evaluations are where no additional information about the system under evaluation are provided to those performing the evaluation.  This means that whoever is evaluating the product security will have to spend significant time in figuring out how the system works.

The Black Box method is supposed to simulate more of a “real-world” situation for the attack.  However, with enough time a capable attacker will probably reverse engineer many parts of the system anyway.  It also shifts the emphasis from the target of the evaluation to how good the evaluator is at reverse engineering!

Similar to Penetration Testing, a Black Box methodology may result in blind spots in the understanding of the risk.  Just because one evaluation engineer does not manage to work out how a part of the system works it doesn’t mean that another engineer could not do this. Even if no security vulnerabilities are found it doesn’t mean that none exist.

 

Recommendations 

If you don’t understand the security risk associated with your system, then a Penetration Test will do little to help.  It can in fact lead to both a false sense of security or a quick fix and re-test attitude which will be expensive and not have a significant improvement on the real security and reduction of your risk.

If you don’t understand your risk, then a White Box Vulnerability Assessment will provide the best real understanding of risk and will be most cost effective in the long term for fixing any security issues discovered.

 

About the author

Dr Carl Shaw is a co-founder of MathEmbedded and an embedded systems software and security expert, with over 20 years’ experience working in the defence and consumer electronics industries.

 

About MathEmbedded

Based in UK but operating globally, MathEmbedded has been providing expert, independent and reputable security services since 2009. MathEmbedded design, develop and implement end-to-end security on connected embedded devices and IoT/M2M platforms for companies of all sizes from start-ups to well-known global brands.

Email MathEmbedded at This email address is being protected from spambots. You need JavaScript enabled to view it. or contact via the website at https://www.mathembedded.com for more information on security evaluations.

 

You can also download this whitepaper as a PDF file.