The FFIEC BSA/AML Manual states-
Management should periodically review and test the filtering criteria and thresholds established to ensure that they are still effective. In addition, the monitoring system’s programming methodology and effectiveness should be independently validated to ensure that the models are detecting potentially suspicious activity. The independent validation should also verify the policies in place and that management is complying with those policies.

Is this required ANNUALLY? We outsource our audits since we are so small and our engagement letter has it as optional. BSA Audit with our without validation.

I would suggest making a determination if you need one be done on a annual basis.

Whether you actually need a new one might be based on a number of factors - new products added, significant core system upgrades, significant AML system upgrades, other significant changes in the overall AML risk profile, length of time since the last one, etc.

Once you have a validation performed you should periodically conduct a low-level validation of the information going forward. As Randy stated there are a number of factors to consider. If you haven't had any major changes then there is no need to do a full validation, IMO. If you go through a core conversion or another major event, then that is a different story.

All this being said, your policy and what-not should outline/ determine the frequency of these reviews in accordance with the Model Risk Management Guidelines
http://www.occ.gov/news-issuances/bulletins/2011/bulletin-2011-12.html

^^^^from OCC website but applies to ALL financial institutions.

There's no specific period dictated and there's no definition of how independent the validation must be. Going internal is tough because the folks with the most expertise (BSA Dept) aren't independent enough.

With that being said, I haven't seen or heard of too many mid to small banks getting criticized for this unless there are other systemic issues. However, given the ever increasing focus in this area, I foresee more exam comments on this topic.

Also, you should aim to perform your self-testing at appropriate risk-based intervals. Read the OCC's guidance and identify all of the models you use. This means the ETL process, OFAC/list checks, 314a scans, case management functions, detection parameters, etc. Think of a strategy to self-validate each - it shows that you have a strong model governance program.

I would argue the following:

- AML Software data loads: Daily reconciliation
- OFAC: Validation at SDN update
- 314a: Validate after each run
- Case Management Functions/Processes: Annually
- Detection Parameters:
- Quarterly - Review of select ineffective parameters


- Annual full parameter review/gap analysis:
- Poor scoring rules targeted

This was a hot topic at the last (2013) OCC Midsize Bank BSA Officer Roundtable in Chicago. When I say hot topci, I mean it got heated between the bankers and the regulators. The regulators were outling very technical and advanced requirements for the validation, but when questioned, could not come to a consensus on what is, or is not, considered a model, how often a validation is to be performed, who should perform it, etc. Some of the bankers in the room that had exams and were told they needed to get an external validation done were spending at a minimum $100k for validation and stated there are not many consultants with the appropriate expertise to perform it, as not only do they need to be statisticians and very experienced in model validation, but they also need extensive BSA experience, as per the OCC members presenting.

I can see why that got heated!

I tend to agree that the best bet to get the right mix of independence and expertise is to go external, but for them to suggest that you need an individual with a PhD in statistics is a little overboard. BSA experience and the ability to understand systems are really all you need. It does not take a rocket surgeon to run a few "parallel simulations" on the blackbox.

Does anyone have a Model Validation program they'd be willing to share?

If so please pm me!