We are a 1.4B community bank. We currently have our Chief Data Officer within our IT Group doing our independent validation for us. But we are looking to benchmark the hours of testing effort we are doing. Can I get any feedback from the group over how many hours of testing per year is considered adequate...or another benchmark to use.

We are a 1.5 billion community bank regulated by the OCC. We do not validate our BSA system on an annual basis, performed only when the system is updated. We have performed this in house and outsourced the process. In house takes about 80 hours and outsources about 60 hours.

I think there are a number of factors that would determine time to perform a validation, including the type of system, volume of transactional activity and knowledge of the person performing the validation. Also, many examiners are looking for tuning and optimization of rules as part of the validation process. Also, examiners would not really look at the number of hours it took to perform the validation. They would look at the methodology and results along with the qualifications of the person performing the validation. I have seen a validation that was not accepted as adequate by examiners.

When your system is updated? Do you mean when a release was installed? Because for us that would exceed annually.

I have been searching for keywords in the Exam Manual for any guidance on data testing. So far, nothing. Can anyone point me to the spot where this is discussed?

Model validation is not unique to BSA/AML compliance; i.e. there is no reason why the BSA/AML manual would address it in detail. The experts on the subject I've met consistently cite SR Letter 11-7 as a basic explanation.

As noted, the number of hours one bank spends on this process has no bearing on those that another bank might spend.

We have our internal auditors do a validation every year. We decide what areas they need to spend time on. The audit is customized. We currently used Yellow Hammer. Every Bank is different but here are some of the areas they look at:
Review YH Risk Management Process
Test and Evaluate model performance
AML model Policies and Procedures
Segregation of duties, access controls
Board Reporting
Risk Assessment
Contingency Plan
Model design- Variables, Data mapping, calculations
Review current parameters inc. high risk and sars
Review of alerts-
Stress testing
Analyze model outcomes

Aha. I remember seeing that Federal Reserve letter when it first came out. It's not exactly a fountain of clarity, but thanks, Ken.

Lilly, can you give me a ballpark of how much time they devote to this? We are looking at resource allocation.

They come out and spend a week with us. We make it part of our round of internal audits. At the end of the week they present us with a draft with any recommendations. The first time we did it was last year and we had a lot of recommendations. After working closely with our core I think we are in a better position but it still needs work. I keep reviewing and doing my own validation/testing every month and keep tweaking the system. I keep an excel spreadsheet of all and any changes to the system. I report to the Board all the work involved (number of cases reviewed), updates to the system and changes to parameters. It sounds like a long report but it's only one page long.

I also spent some time reading the Letter Ken is referring to before we began the whole process. It gave me an idea on the purpose and process for the validation.

Lilly, did they actually find data integrity issues, or was the recommendation more about settings and parameters?

They tested samples of transaction from the core feeds to determine that they are accurately capture in the YH system.. They also tested sample of reports, alerts and customer characteristics to determine whether those meeting our thresholds are creating an alert for us to review take action. As a result we ended up doing quite a few adjustments to our parameters.

They covered things like determining if our ADD and CDD questions are asked at account opening and scores are assigned for each answer.

The scope of the validation covered about 12 areas.

If you are an OCC Bank refer to OCC 2011-12 Supervisory Guidance on Model Risk Management. This is what our regulators always refer too.

That's a different citation for the same document; it was jointly issued by the Fed and the OCC. To my knowledge, there is no document unique to the FDIC; i.e. I have found SR 11-7 cited in some FDIC publications on related topics.

As others have noted, there are tons of factors that influence the frequency/depth of a validation. IMHO, if you're doing it independently for the first time, your best bet is to go with a two week job (100-120: 80 hours + overhead) with roughly one week budgeted for assessing data management, one week for assessing system settings (e.g. rule parameter evaluations, etc.), and some time in each to assess governance controls.

If you're doing it annually or even bi-annually, you could probably pass with a one week job (50-70 hours total: 40 hours + overhead) with the scope dependent upon your combination of core and AML system. Some AML systems have really robust controls for data integrity and can allow the independent validation to be more focused on the system's ability to detect suspicious activity; others it's the reverse. Some AML systems are awful on both fronts and need a ton of validation work. It all depends.

Also, of course, if your Bank has any major system changes, it's not a bad idea to validate both internally and externally thereafter.