Model Validation: 4 Elements to Determine the Accuracy of Your Models
“The set of processes and activities intended to verify that models are performing as expected, in line with their design objectives, and business uses.” Model validation also identifies “potential limitations and assumptions, and assesses their possible impact.”
The Guidance also states that, “All model components – inputs, processing, outputs, and reports – should be subject to validation” irrespective of whether they are developed in house or purchased from, or developed by a third party.
Model validations should be independent, in order to be objective. This generally means that validation activities are performed by individuals independent of model development or use. Models, therefore, should not be validated by their owners.
Models can be highly technical, and some organisations may find it difficult to assemble a model risk team that has sufficient functional and technical expertise to carry out independent validation. When faced with this obstacle, organisations often outsource the validation task to third parties.
If done effectively, model validation will enable your company to have confidence in the accuracy of its AML models, and alignment with business and regulatory expectations.
The 4 Elements
Model validation consists of four crucial elements.
Source: Crowe Horwath LLP analysis
1. Conceptual Design
The foundation of any model validation is its conceptual design. Organisations should complete and document a coverage assessment or gap analysis to demonstrate the model’s ability to meet business and regulatory needs, and unique AML risks.A validation should effectively challenge the underlying conceptual design and documentation that support the AML model’s logic, whilst independently evaluating the model’s ability to achieve its desired outcome.
2. System Validation
All automated systems implemented to support models have inherent limitations. System validation independently confirms that the development, implementation, and ongoing use of technology are properly designed and integrated in to the business.
3. Data Validation and Quality Assessment
Data errors or irregularities impair model outputs, and might lead to an organisation’s failure to identify, and respond to AML risks. For example, customer data is incomplete, your organisation may conduct business with an individual/ entity that is outside of risk appetite. Best practise indicates that organisations should apply a risk-based data validation. This enables the reviewer to consider risks unique to the organisation and the model, and prioritise accordingly.
Assessing the accuracy of source data is vital because it can be derived from a variety of source systems, some of which might lack controls that maintain data integrity.
4. Process Validation
To verify that an AML model is operating effectively, it is important to verify that the policies and procedures support the model’s sustainability. A review of the processes also determines whether the AML models are producing output that is accurate, managed effectively, and subject to appropriate controls.
A Complex but Essential Process
The complex, resource-intensive nature of validation makes it necessary to dedicate sufficient resources to its execution.
An independent validation team well versed in data management, technology, and relevant financial products or services – for example financial crime compliance – is vital for the appropriate evaluation of models. Where shortfalls in the validation process are identified, timely remedial actions should be taken to close the gaps.
By failing to validate models, companies increase the risk of regulatory criticism, fines, and penalties. Further, models without proper validation are less likely to provide the efficiency, cost control, and other benefits they would if they functioned as intended.