Automated Software -'independent' validation?

Posted By: kw004h

Automated Software -'independent' validation? - 08/12/11 07:14 PM

If you use automated software, have you completed a formal "independent validation" as laid out on page 73-74 of the FFIEC BSA Exam Manual?

When we first went on our automated system (AML Manager), we performed a multi-month parallel review of customer activity to assure, in detail, that teh software was functioning as intended. However, our third party auditors have claimed (and continue to claim) that because bank staff performed this review, it was not 'independently validated'. (They have kindly offered to perform this validation for us, for a fee of course.)

Anyone have any similar experience to share? Have you succesfully defended any particular process as fulfilling an 'independent validation'? Any comments from auditors or examiners?
Posted By: ACBbank

Re: Automated Software -'independent' validation? - 08/12/11 07:28 PM

I think your 3rd party auditors misunderstood what the manual is stating. For example when we switched to BAM over a year ago, I as the BSA Officer did the initial verification. This included checking CTR figures, cash reports, wire logs, etc. against the old system. The OCC and IA were fine with this.

Now, when IA does their independent review, they request a certain time frame and do their own verification. This is what the manual is talking about, not the initial set up. This verification is including in their fee for the entire audit.

You’re not required to have IA verify the initial set up. That said if you have the resources why not?
Posted By: rlcarey

Re: Automated Software -'independent' validation? - 08/12/11 07:43 PM

ACB is correct. A validation test should be part of the annual BSA audit - not an extra separate audit.
Posted By: WonderWoman

Re: Automated Software -'independent' validation? - 08/13/11 12:00 AM

My auditor stated the same thing - that I needed to have an independent validation & that it would take another week (& fee). I brushed it off until I received my BSA Exam entrance letter from the FDIC.

They didn't even ask for SARs (I'm guessing because they're now pulling off the efile system) - but they specifically asked for my independent validation. Which I don't have.


So I had my internal auditor & IT department go through everything & validate for me - I'm hoping it passes muster.
Posted By: John Burnett

Re: Automated Software -'independent' validation? - 08/15/11 02:19 PM

"Independent" in this case refers to having the review done by someone not involved in the day-to-day BSA operation. For example, if your BSA officer conducts the annual review of the BSA program, that review isn't independent. But if your IA department does it, assuming it is qualified to do so, it should not be a problem.
Posted By: kw004h

Re: Automated Software -'independent' validation? - 08/15/11 07:20 PM

I guess we should work towards having our Internal Audit department perform something meaningful.

Maybe next year, as Randy suggests, I should make sure the independent auditor performs some sort of system validation as part of their standard engagement. Their scope did state they would perform "a review of the effectiveness of the suspicious activity monitoring systems (manual, automated, or a combination of both) used for BSA/AML compliance." However, while on site, they reviewed only whether our responses to the alerts received by the software seemed reasonable, as opposed to reviewing whether the software was generating alerts as expected.

A big thank you to all of you for weighing in!
Posted By: kw004h

Re: Automated Software -'independent' validation? - 08/15/11 07:22 PM

Also, happy birthday, John.
Posted By: Kathleen O. Blanchard

Re: Automated Software -'independent' validation? - 08/15/11 07:25 PM

When combining it with the normal BSA audit, it is important to see if the "validation" is a full validation of the entire system (including mappping) or is it the customary checking a sample of CTRs,SARs, etc. back to source data.

That is 2 different levels of validation. The first is the one that is usually discussed as a separate audit with additional time required.
Posted By: kw004h

Re: Automated Software -'independent' validation? - 08/15/11 08:29 PM

Another question was raised here:

As the automated software is designed and maintained by the same company that processes our core (FiServ), and we have no control over the internal settings of the software, would the BSA Officer be sufficiently "independent" of the system itself to audit the validation of the system?

Opinions?
Posted By: A_G

Re: Automated Software -'independent' validation? - 08/16/11 01:27 PM

No, imho.
Posted By: ACBbank

Re: Automated Software -'independent' validation? - 08/16/11 05:10 PM

Originally Posted By: kw004h
Another question was raised here:

As the automated software is designed and maintained by the same company that processes our core (FiServ), and we have no control over the internal settings of the software, would the BSA Officer be sufficiently "independent" of the system itself to audit the validation of the system?

Opinions?


If you're talking the independent testing of BSA/AML compliance (One of the "four pillars"), then I would say no. The manual requires that "Independent testing (audit) should be conducted by the internal audit department, outside auditors, consultants, or other qualified independent parties."
Posted By: kw004h

Re: Automated Software -'independent' validation? - 08/17/11 03:02 PM

ACBbank, it would depned on whether we considered validating the software to be within the overall scope of the "four pillars", or whether one was arguing that any piece of software could possibly be considered as a 'tool' used within the program and not necessarily the 'program' itself.

Personally, I agree with your comments. I think that the intention is to have confirmation (apart from the users who are interacting with the software every day) that the output is as expected.

Again, thanks to all for your input and advice here!
Posted By: BrendaC

Re: Automated Software -'independent' validation? - 08/17/11 03:12 PM

We simply generated a list of cash transactions and sorted by size. We then compared the list to the report to validate that all transactions were properly captured and aggregated. Transactions impacting internal account for MI purchases were also included in exercise. Regulators were satisfied.
Posted By: Snowmann

Re: Automated Software -'independent' validation? - 06/06/14 06:32 PM

Now that it has been a few years since automated BSA has been going strong, do you have a good feel on what is expected of you when it comes to independent testing, if using internal audit?

We have implemented the software within the last year and are wondering what type of testing has passed regulatory reviews, or has been recommended by your regulators.

We do an annual audit every year, conducted by an employee that is not involved with BSA, but we will surely need to add in a software validation portion. But does anyone have any specific things to test for that your examiners liked?
Posted By: rlcarey

Re: Automated Software -'independent' validation? - 06/06/14 06:35 PM

Mainly, you have to validate that everything that is going through your core systems is captured properly in your AML software.
Posted By: P*Q

Re: Automated Software -'independent' validation? - 06/10/14 01:11 PM

Originally Posted By: rlcarey
Mainly, you have to validate that everything that is going through your core systems is captured properly in your AML software.
How would an "independent" third party know that?
Posted By: ACBbank

Re: Automated Software -'independent' validation? - 06/10/14 01:26 PM

Most AML systems pull information from a "core system," which can generate reports. Typically, you would pull reports from the core and the AML system and look for deviations. If the AML system has an audit log, you can review the log for input errors, warnings, etc.
Posted By: Kathleen O. Blanchard

Re: Automated Software -'independent' validation? - 06/10/14 03:21 PM

Originally Posted By: P*Q
Originally Posted By: rlcarey
Mainly, you have to validate that everything that is going through your core systems is captured properly in your AML software.
How would an "independent" third party know that?


It is done by tracing transactions from beginning through various places in core to the AML system, checking total # & $ by category, making sure all expected categories sre captured, reviewing programming scripts, etc.
Posted By: Princess Romeo

Re: Automated Software -'independent' validation? - 06/10/14 09:05 PM

There are actually two types of validation:

Data integrity
Model validity

Data integrity is simply tracking the data from transactions to be sure it is captured accurately by your software. If someone conducts a transaction, is the dollar amount, type and method of transaction accurately reported? A cash deposit for $6,000 shows as such, a deposit of checks, a withdrawal of cash, checks being paid, ACH transactions, etc., etc.

Model Validation is bit more complex as that involves ensuring that your system parameters are properly set for your institution to flag those transactions that should rise to the level of requiring a review. And depending on the types of customers and volumes that you have, a model that makes sense at one institution would be hopelessly inadequate, (or overkill) for another institution.

It is the Model Validation that I have been seeing the examiners focus on more and more.
Posted By: Kathleen O. Blanchard

Re: Automated Software -'independent' validation? - 06/11/14 03:07 AM

Either one can make your system useless if not working properly. A good model validation tests data integrity, it has to.
Posted By: LMBrown

Re: Automated Software -'independent' validation? - 06/12/14 06:00 PM

Originally Posted By: Princess Romeo
There are actually two types of validation:

Data integrity
Model validity

Data integrity is simply tracking the data from transactions to be sure it is captured accurately by your software. If someone conducts a transaction, is the dollar amount, type and method of transaction accurately reported? A cash deposit for $6,000 shows as such, a deposit of checks, a withdrawal of cash, checks being paid, ACH transactions, etc., etc.

Model Validation is bit more complex as that involves ensuring that your system parameters are properly set for your institution to flag those transactions that should rise to the level of requiring a review. And depending on the types of customers and volumes that you have, a model that makes sense at one institution would be hopelessly inadequate, (or overkill) for another institution.

It is the Model Validation that I have been seeing the examiners focus on more and more.


Does anyone have any good resources to share to best test the "Model Validation" portion of this review? How would you suggest to test whether the model makes sense for your institution? Thanks so much!
Posted By: TryingtoComply

Re: Automated Software -'independent' validation? - 06/12/14 09:31 PM

I've had examiners/auditors ask for reports of customers with high cash/wire activity from the core system. They sort the data to identify customers that would appear to be high risk and then compare that data to reports generated from your AML software. I've also had them use the data to evaluate whether or not we have identified all of our high risk customers.

If you rely on your AML software to idenfity high risk customers I think the model needs to be evaluated to ensure that all high risk customers are identified too. We use a well-known AML software that has a risk scoring model that considers business type, geography, product, transactions and TIN. In some circumstances customers with high cash or wire activity did not receive enough points to be considered high risk. The software works well to identify your highest risk customers that have a high risk business type and a combination of cash/wire/ACH activity; however, additional points need to be added using a feature in the product to cause certain cutomers to be classified as high risk.
Posted By: Kathleen O. Blanchard

Re: Automated Software -'independent' validation? - 06/12/14 11:20 PM

Every single module needs to be validated You can't do pieces; if you check that all data is pulled but do not check that the math is correct, you have accomplished nothing. A validation, just like an interest rate risk validation, checks EVERYTHING.
Posted By: Blessed

Re: Automated Software -'independent' validation? - 04/23/15 05:18 PM

Does anyone have a Model Validation program they'd be willing to share?

If so please PM Me