Severity - the relative harm of risks
Severity is a new component in the SRA's risk assessment methodology, and will be used to improve assessment of firm level risk.
The regulated community was asked to complete a survey to capture a range of views about the seriousness of the risks in the Risk Index. The responses were used in conjunction with an identical internal SRA survey to generate final severity scores for all risks. This short report explains more about the surveys, the results and the measure of severity.
Read below Download (PDF 5 pages, 315K)
This report explains how the SRA has calculated the levels of harm posed by different regulatory risks, and shows the overall results which will be introduced into the firm risk assessment process.
Introduction and background
A new component to our quantitative risk assessment methodologies named "severity" is being introduced. To inform this component, the regulated community were asked to complete a survey during April and May 2013 to capture a range of views about the seriousness of the risks in the Risk Index.
Almost 300 people responded, and these were used in conjunction with an identical internal SRA survey with over 50 responses to generate final severity scores for all risks. This short report explains more about the surveys, the results and the measure of severity.
What is severity?
Severity is a new component in the SRA's risk assessment methodology, and will be used to improve assessment of firm level risk. These assessments are made up of the following three components:
Firm Footprint: the affect that a firm would have on the regulatory objectives should a risk crystallise.
Severity: the new component to differentiate between the seriousness of different risks.
Probability: how likely it is for a risk to crystallise in a firm.
Severity is therefore one component in the firm risk assessment methodology. At present, the level 1 risks in the Risk Index are not weighted or prioritised. This means, for example, if the same firm footprint and probability were applied to the risk of criminal association and to the risk of a conflict of interests they would be considered to have the same harm on the regulatory objectives. The new severity component influences the weight and priority given to specific risks because severity scores reflect how serious the SRA believe different risks are in their relative potential to harm the objectives.
Each risk will have a percentage severity score which will place it relative to, and in context with, other risks in the Risk Index. So for example, if it is considered that the risk of criminal association inherently has more potential to harm the achievement of the regulatory objectives than the risk of a conflict of interest it will be allocated a higher severity score.
What was the purpose of the surveys?
The SRA does not have consistent quantitative information to inform how severe all risks are in relation to one another. To understand the relative seriousness of risks to the regulatory objectives a wide range of opinions from respondents who are well versed in risk were sought. Two surveys, one for SRA staff and one targeted at compliance officers, were undertaken.
By surveying both internal and external respondents a range of responses were obtained from people who understand the regulatory objectives in different contexts. The results helped assign a severity score of between 0—100% to each of the 29 firm based risks in the Risk Index.
How were the surveys conducted?
The surveys used a method called maximum difference scaling that allowed measurement of the preference or importance people give to each of the firm based level 1 risks. The advantage of this technique is that it enables robust scaling to be applied without the need for ranking or rating of each and every risk. This method also establishes the intensity of the difference between risks, which a ranking approach could not.
In the survey each respondent was asked to complete twelve questions. In the first question, a random selection of just six risks was displayed, and the respondent was asked to select which of the selection of six risks they believed was the most severe and which was the least severe in terms of the harm they could cause to the regulatory objectives. In the next question, six random risks were again displayed (some of which may have been asked in the previous question) and a judgement was again entered about the most and least severe risks. This process was completed for 12 questions. The responses provided in each of the 12 questions - whether a risk was selected as the most or least severe, or if a risk was not selected - gave us 72 data points for each completed survey. A single respondent’s results in isolation had limited value, but by aggregating responses together information could be generated on the severity for the full selection of risks.
What were the results?
The results were aggregated from the internal survey and compared with the aggregated results from the external survey. The two results were similar with an overall correlation of 0.91, suggesting very strong similarities between SRA staff and the regulated community in the perception of harm that the risks can cause to the regulatory objectives. These similarities were particularly evident towards the higher severity risks such as dishonest misuse of client money and criminal association. For some risks there was more discrepancy. Comparing the aggregated results for the internal respondents against the external respondents the following key observations were made:
- There was agreement that the three most severe risks, in order, are:
- Dishonest misuse of client money
- Criminal association
- Money laundering
- Both surveys found that the following risks featured in the top nine (not in order):
- Bogus firm or individual
- Bribery & corruption
- Criminal association
- Dishonest misuse of client money or assets
- Dishonest misuse of non-client money or assets
- Failure to act with integrity or ethics
- Financial difficulty
- Intentional misleading
- Money laundering
- There was a difference of opinion in the order that misuse of non-client money was ranked - for the internal survey this was rated higher than in the external survey by several places and, for example, the external survey scored this risk below financial difficulty.
- There was a notable difference of opinion in the order that that disorderly closure was rated. This risk was rated in the top half of all risks by internal respondents, and in the bottom half by external respondents.
- Failure to meet duties to third parties or the court was ranked twelve places apart between the two surveys, with external respondents classing it as more severe than internal respondents.
While completing the survey different respondents may have been thinking about the regulatory objectives in different ways. This may be a partial cause of the discrepancies noted above. In the future such differences will be addressed by exploring the nature of the relationship between different risks and specific regulatory objectives more clearly.
The final severity scores reflect the views of both the internal and external respondents. An equal weighting between internal and external aggregate responses was applied, and the final severity scores are shown below.
Severity weighted average results
Based on internal and external results (n=351)
|Dishonest misuse of client money or assets
|Dishonest misuse of non -client money or assets
|Bribery & corruption
|Bogus firm or individual
|Failure to act with integrity or ethics
|Lack of legal competence
|Lack of financial competence
|Lack of management competence
|Acting outside regulatory permissions
|Breach of confidentiality
|Inadequate systems & controls
|Lack of independence
|Failure to meet duties to 3rd parties or the court
|Poor standard of service
|Failure to co -operate or comply with notification and info. req's
|Inappropriate firm structure
|Conflict of interests
|Supply chain risks
|Misleading or inappropriate publicity
|Inadequate complaints handling
The severity scores do not mean that risks that are lower down the scale are not of concern. The assessment of risk is partly informed by these new severity scores, but it is also informed by other components in the overall approach to risk assessment. The assessment takes account not only of how serious different risks are (severity), but also the context of the firms in which they might arise (firm footprint) and the likelihood that they may occur (probability). For example, we ask for information about first-tier complaints handling during the annual renewal exercise. Although the associated risk of inadequate complaints handling may not be rated as particularly severe, concerns about where it is prevalent and its frequency led to the data requirements relating to this risk. Other less quantifiable factors can also influence the assessment of risks which may not be reflected in the severity scores, but which will influence the risks that the SRA focuses its attention on, for example new emerging risks or risks that drive other risks.
What are the differences in perspective by different firm types?
The number of responses from the regulated community that included the respondent’s firm ID were not significant enough to perform a robust analysis of how different types of firm regarded the severity of risks differently. Further work will be undertaken to understand where there are differences of opinion amongst different types of firm - for example based on work type specialism.
What happens next?
These results give a robust baseline understanding of the relative levels of harm that can be caused to the regulatory objectives by each of the risks, disregarding the probability of them arising or where they may occur. In time, we will look to add more contextual information to the way severity is calculated. These scores will be built into our developing overall risk assessment methodologies, as outlined in the recent webinar and in the forthcoming Firm Risk Assessment Methodology paper. This will ensure that the prioritisation of our resources is guided by the amount of harm that risks can cause.
The SRA will develop and build on these scores, and will undertake further work to understand the links between risk harm and each specific regulatory objective. Similar surveys will be undertaken in the future, potentially targeted at specific objectives. As well as including the opinions of SRA staff and of the regulated community, engagement with consumers will establish a broader range of opinions on the harm caused by different risks.