Skip to content
BOL Conferences
Thread Options
#2092461 - 08/10/16 01:33 PM Hours devoted to BSA system validation
Daisy Doodle Offline
Diamond Poster
Joined: Feb 2014
Posts: 1,030
Southern U.S.
We are a 1.4B community bank. We currently have our Chief Data Officer within our IT Group doing our independent validation for us. But we are looking to benchmark the hours of testing effort we are doing. Can I get any feedback from the group over how many hours of testing per year is considered adequate...or another benchmark to use.

Return to Top
BSA/AML/CIP/OFAC Forum
#2092465 - 08/10/16 01:40 PM Re: Hours devoted to BSA system validation Daisy Doodle
osucpa Offline
Diamond Poster
Joined: May 2011
Posts: 1,410
We are a 1.5 billion community bank regulated by the OCC. We do not validate our BSA system on an annual basis, performed only when the system is updated. We have performed this in house and outsourced the process. In house takes about 80 hours and outsources about 60 hours.

Return to Top
#2092467 - 08/10/16 01:41 PM Re: Hours devoted to BSA system validation Daisy Doodle
Happy Birthday Big Dog Offline
Power Poster
Big Dog
Joined: Mar 2005
Posts: 2,659
Kennel
I think there are a number of factors that would determine time to perform a validation, including the type of system, volume of transactional activity and knowledge of the person performing the validation. Also, many examiners are looking for tuning and optimization of rules as part of the validation process. Also, examiners would not really look at the number of hours it took to perform the validation. They would look at the methodology and results along with the qualifications of the person performing the validation. I have seen a validation that was not accepted as adequate by examiners.
_________________________
CAMS, AMLP, AKC, K-9






Return to Top
#2092470 - 08/10/16 01:53 PM Re: Hours devoted to BSA system validation osucpa
Daisy Doodle Offline
Diamond Poster
Joined: Feb 2014
Posts: 1,030
Southern U.S.
When your system is updated? Do you mean when a release was installed? Because for us that would exceed annually.

Return to Top
#2092472 - 08/10/16 01:56 PM Re: Hours devoted to BSA system validation Daisy Doodle
Daisy Doodle Offline
Diamond Poster
Joined: Feb 2014
Posts: 1,030
Southern U.S.
I have been searching for keywords in the Exam Manual for any guidance on data testing. So far, nothing. Can anyone point me to the spot where this is discussed?

Return to Top
#2092479 - 08/10/16 02:16 PM Re: Hours devoted to BSA system validation Daisy Doodle
Elwood P. Dowd Offline
10K Club
Elwood P. Dowd
Joined: Aug 2001
Posts: 21,939
Next to Harvey
Model validation is not unique to BSA/AML compliance; i.e. there is no reason why the BSA/AML manual would address it in detail. The experts on the subject I've met consistently cite SR Letter 11-7 as a basic explanation.

As noted, the number of hours one bank spends on this process has no bearing on those that another bank might spend.
_________________________
In this world you must be oh so smart or oh so pleasant. Well, for years I was smart. I recommend pleasant.

Return to Top
#2092481 - 08/10/16 02:23 PM Re: Hours devoted to BSA system validation Daisy Doodle
Lilly C Offline
100 Club
Lilly C
Joined: Dec 2004
Posts: 209
Sunny Florida
We have our internal auditors do a validation every year. We decide what areas they need to spend time on. The audit is customized. We currently used Yellow Hammer. Every Bank is different but here are some of the areas they look at:
Review YH Risk Management Process
Test and Evaluate model performance
AML model Policies and Procedures
Segregation of duties, access controls
Board Reporting
Risk Assessment
Contingency Plan
Model design- Variables, Data mapping, calculations
Review current parameters inc. high risk and sars
Review of alerts-
Stress testing
Analyze model outcomes

Return to Top
#2092483 - 08/10/16 02:26 PM Re: Hours devoted to BSA system validation Daisy Doodle
Daisy Doodle Offline
Diamond Poster
Joined: Feb 2014
Posts: 1,030
Southern U.S.
Aha. I remember seeing that Federal Reserve letter when it first came out. It's not exactly a fountain of clarity, but thanks, Ken.

Return to Top
#2092485 - 08/10/16 02:27 PM Re: Hours devoted to BSA system validation Lilly C
Daisy Doodle Offline
Diamond Poster
Joined: Feb 2014
Posts: 1,030
Southern U.S.
Lilly, can you give me a ballpark of how much time they devote to this? We are looking at resource allocation.

Return to Top
#2092490 - 08/10/16 02:37 PM Re: Hours devoted to BSA system validation Daisy Doodle
Lilly C Offline
100 Club
Lilly C
Joined: Dec 2004
Posts: 209
Sunny Florida
They come out and spend a week with us. We make it part of our round of internal audits. At the end of the week they present us with a draft with any recommendations. The first time we did it was last year and we had a lot of recommendations. After working closely with our core I think we are in a better position but it still needs work. I keep reviewing and doing my own validation/testing every month and keep tweaking the system. I keep an excel spreadsheet of all and any changes to the system. I report to the Board all the work involved (number of cases reviewed), updates to the system and changes to parameters. It sounds like a long report but it's only one page long.

Return to Top
#2092493 - 08/10/16 02:39 PM Re: Hours devoted to BSA system validation Daisy Doodle
Lilly C Offline
100 Club
Lilly C
Joined: Dec 2004
Posts: 209
Sunny Florida
I also spent some time reading the Letter Ken is referring to before we began the whole process. It gave me an idea on the purpose and process for the validation.

Return to Top
#2092495 - 08/10/16 02:43 PM Re: Hours devoted to BSA system validation Lilly C
Daisy Doodle Offline
Diamond Poster
Joined: Feb 2014
Posts: 1,030
Southern U.S.
Lilly, did they actually find data integrity issues, or was the recommendation more about settings and parameters?

Return to Top
#2092500 - 08/10/16 02:57 PM Re: Hours devoted to BSA system validation Daisy Doodle
Lilly C Offline
100 Club
Lilly C
Joined: Dec 2004
Posts: 209
Sunny Florida
They tested samples of transaction from the core feeds to determine that they are accurately capture in the YH system.. They also tested sample of reports, alerts and customer characteristics to determine whether those meeting our thresholds are creating an alert for us to review take action. As a result we ended up doing quite a few adjustments to our parameters.

They covered things like determining if our ADD and CDD questions are asked at account opening and scores are assigned for each answer.

The scope of the validation covered about 12 areas.

Return to Top
#2092503 - 08/10/16 03:04 PM Re: Hours devoted to BSA system validation Daisy Doodle
osucpa Offline
Diamond Poster
Joined: May 2011
Posts: 1,410
If you are an OCC Bank refer to OCC 2011-12 Supervisory Guidance on Model Risk Management. This is what our regulators always refer too.

Return to Top
#2092504 - 08/10/16 03:09 PM Re: Hours devoted to BSA system validation osucpa
Elwood P. Dowd Offline
10K Club
Elwood P. Dowd
Joined: Aug 2001
Posts: 21,939
Next to Harvey
That's a different citation for the same document; it was jointly issued by the Fed and the OCC. To my knowledge, there is no document unique to the FDIC; i.e. I have found SR 11-7 cited in some FDIC publications on related topics.
_________________________
In this world you must be oh so smart or oh so pleasant. Well, for years I was smart. I recommend pleasant.

Return to Top
#2093448 - 08/16/16 03:14 PM Re: Hours devoted to BSA system validation Daisy Doodle
Pat Patriot Act Offline
Gold Star
Pat Patriot Act
Joined: Apr 2009
Posts: 450
As others have noted, there are tons of factors that influence the frequency/depth of a validation. IMHO, if you're doing it independently for the first time, your best bet is to go with a two week job (100-120: 80 hours + overhead) with roughly one week budgeted for assessing data management, one week for assessing system settings (e.g. rule parameter evaluations, etc.), and some time in each to assess governance controls.

If you're doing it annually or even bi-annually, you could probably pass with a one week job (50-70 hours total: 40 hours + overhead) with the scope dependent upon your combination of core and AML system. Some AML systems have really robust controls for data integrity and can allow the independent validation to be more focused on the system's ability to detect suspicious activity; others it's the reverse. Some AML systems are awful on both fronts and need a ton of validation work. It all depends.

Also, of course, if your Bank has any major system changes, it's not a bad idea to validate both internally and externally thereafter.
_________________________
CFE, CAMS

Return to Top

Moderator:  Andy_Z