Thread Options
#1591604 - 08/12/11 07:14 PM Automated Software -'independent' validation?
kw004h Offline
100 Club
Joined: Nov 2009
Posts: 219
Chicagoland, IL
If you use automated software, have you completed a formal "independent validation" as laid out on page 73-74 of the FFIEC BSA Exam Manual?

When we first went on our automated system (AML Manager), we performed a multi-month parallel review of customer activity to assure, in detail, that teh software was functioning as intended. However, our third party auditors have claimed (and continue to claim) that because bank staff performed this review, it was not 'independently validated'. (They have kindly offered to perform this validation for us, for a fee of course.)

Anyone have any similar experience to share? Have you succesfully defended any particular process as fulfilling an 'independent validation'? Any comments from auditors or examiners?

Return to Top
BSA/AML/CIP/OFAC Forum
#1591628 - 08/12/11 07:28 PM Re: Automated Software -'independent' validation? kw004h
ACBbank Offline
Power Poster
ACBbank
Joined: Jul 2006
Posts: 3,959
New York City
I think your 3rd party auditors misunderstood what the manual is stating. For example when we switched to BAM over a year ago, I as the BSA Officer did the initial verification. This included checking CTR figures, cash reports, wire logs, etc. against the old system. The OCC and IA were fine with this.

Now, when IA does their independent review, they request a certain time frame and do their own verification. This is what the manual is talking about, not the initial set up. This verification is including in their fee for the entire audit.

You’re not required to have IA verify the initial set up. That said if you have the resources why not?
_________________________
"100 victories in 100 battles isnt the most skillful. Subduing the other's military w/o battle is the most skillful." Sun-Tzu

Return to Top
#1591641 - 08/12/11 07:43 PM Re: Automated Software -'independent' validation? ACBbank
rlcarey Online
10K Club
rlcarey
Joined: Jul 2001
Posts: 77,260
Galveston, TX
ACB is correct. A validation test should be part of the annual BSA audit - not an extra separate audit.
_________________________
The opinions expressed here should not be construed to be those of my employer: PPDocs.com

Return to Top
#1591788 - 08/13/11 12:00 AM Re: Automated Software -'independent' validation? rlcarey
WonderWoman Offline
Diamond Poster
WonderWoman
Joined: Mar 2007
Posts: 2,108
gone fishin'
My auditor stated the same thing - that I needed to have an independent validation & that it would take another week (& fee). I brushed it off until I received my BSA Exam entrance letter from the FDIC.

They didn't even ask for SARs (I'm guessing because they're now pulling off the efile system) - but they specifically asked for my independent validation. Which I don't have.


So I had my internal auditor & IT department go through everything & validate for me - I'm hoping it passes muster.
_________________________
My opinions are my own, and not that of my employer.

Return to Top
#1591874 - 08/15/11 02:19 PM Re: Automated Software -'independent' validation? WonderWoman
John Burnett Offline
10K Club
John Burnett
Joined: Oct 2000
Posts: 39,336
Cape Cod
"Independent" in this case refers to having the review done by someone not involved in the day-to-day BSA operation. For example, if your BSA officer conducts the annual review of the BSA program, that review isn't independent. But if your IA department does it, assuming it is qualified to do so, it should not be a problem.
_________________________
John S. Burnett
BankersOnline.com
Fighting for Compliance since 1976
Bankers' Threads User #8

Return to Top
#1592081 - 08/15/11 07:20 PM Re: Automated Software -'independent' validation? John Burnett
kw004h Offline
100 Club
Joined: Nov 2009
Posts: 219
Chicagoland, IL
I guess we should work towards having our Internal Audit department perform something meaningful.

Maybe next year, as Randy suggests, I should make sure the independent auditor performs some sort of system validation as part of their standard engagement. Their scope did state they would perform "a review of the effectiveness of the suspicious activity monitoring systems (manual, automated, or a combination of both) used for BSA/AML compliance." However, while on site, they reviewed only whether our responses to the alerts received by the software seemed reasonable, as opposed to reviewing whether the software was generating alerts as expected.

A big thank you to all of you for weighing in!

Return to Top
#1592084 - 08/15/11 07:22 PM Re: Automated Software -'independent' validation? kw004h
kw004h Offline
100 Club
Joined: Nov 2009
Posts: 219
Chicagoland, IL
Also, happy birthday, John.

Return to Top
#1592089 - 08/15/11 07:25 PM Re: Automated Software -'independent' validation? kw004h
Kathleen O. Blanchard Offline

10K Club
Kathleen O. Blanchard
Joined: Dec 2000
Posts: 21,277
When combining it with the normal BSA audit, it is important to see if the "validation" is a full validation of the entire system (including mappping) or is it the customary checking a sample of CTRs,SARs, etc. back to source data.

That is 2 different levels of validation. The first is the one that is usually discussed as a separate audit with additional time required.
_________________________
Kathleen O. Blanchard, CRCM "Kaybee"
HMDA/CRA Training/Consulting/Mapping
The HMDA Academy
www.kaybeescomplianceinsights.com

Return to Top
#1592152 - 08/15/11 08:29 PM Re: Automated Software -'independent' validation? Kathleen O. Blanchard
kw004h Offline
100 Club
Joined: Nov 2009
Posts: 219
Chicagoland, IL
Another question was raised here:

As the automated software is designed and maintained by the same company that processes our core (FiServ), and we have no control over the internal settings of the software, would the BSA Officer be sufficiently "independent" of the system itself to audit the validation of the system?

Opinions?

Return to Top
#1592281 - 08/16/11 01:27 PM Re: Automated Software -'independent' validation? kw004h
A_G Online
10K Club
Joined: Jul 2004
Posts: 18,957
No, imho.
_________________________
With the lights out, it's less dangerous.

Return to Top
#1592446 - 08/16/11 05:10 PM Re: Automated Software -'independent' validation? kw004h
ACBbank Offline
Power Poster
ACBbank
Joined: Jul 2006
Posts: 3,959
New York City
Originally Posted By: kw004h
Another question was raised here:

As the automated software is designed and maintained by the same company that processes our core (FiServ), and we have no control over the internal settings of the software, would the BSA Officer be sufficiently "independent" of the system itself to audit the validation of the system?

Opinions?


If you're talking the independent testing of BSA/AML compliance (One of the "four pillars"), then I would say no. The manual requires that "Independent testing (audit) should be conducted by the internal audit department, outside auditors, consultants, or other qualified independent parties."
_________________________
"100 victories in 100 battles isnt the most skillful. Subduing the other's military w/o battle is the most skillful." Sun-Tzu

Return to Top
#1592882 - 08/17/11 03:02 PM Re: Automated Software -'independent' validation? ACBbank
kw004h Offline
100 Club
Joined: Nov 2009
Posts: 219
Chicagoland, IL
ACBbank, it would depned on whether we considered validating the software to be within the overall scope of the "four pillars", or whether one was arguing that any piece of software could possibly be considered as a 'tool' used within the program and not necessarily the 'program' itself.

Personally, I agree with your comments. I think that the intention is to have confirmation (apart from the users who are interacting with the software every day) that the output is as expected.

Again, thanks to all for your input and advice here!
Last edited by kw004h; 08/17/11 03:03 PM.
Return to Top
#1592888 - 08/17/11 03:12 PM Re: Automated Software -'independent' validation? kw004h
BrendaC Offline
Power Poster
BrendaC
Joined: Sep 2001
Posts: 6,029
Sweet Home AL
We simply generated a list of cash transactions and sorted by size. We then compared the list to the report to validate that all transactions were properly captured and aggregated. Transactions impacting internal account for MI purchases were also included in exercise. Regulators were satisfied.
_________________________
Life without Jesus is like an unsharpened pencil - it has no point.

Return to Top
#1930455 - 06/06/14 06:32 PM Re: Automated Software -'independent' validation? kw004h
Snowmann Offline
Junior Member
Joined: Feb 2011
Posts: 48
Now that it has been a few years since automated BSA has been going strong, do you have a good feel on what is expected of you when it comes to independent testing, if using internal audit?

We have implemented the software within the last year and are wondering what type of testing has passed regulatory reviews, or has been recommended by your regulators.

We do an annual audit every year, conducted by an employee that is not involved with BSA, but we will surely need to add in a software validation portion. But does anyone have any specific things to test for that your examiners liked?

Return to Top
#1930458 - 06/06/14 06:35 PM Re: Automated Software -'independent' validation? kw004h
rlcarey Online
10K Club
rlcarey
Joined: Jul 2001
Posts: 77,260
Galveston, TX
Mainly, you have to validate that everything that is going through your core systems is captured properly in your AML software.
_________________________
The opinions expressed here should not be construed to be those of my employer: PPDocs.com

Return to Top
#1930931 - 06/10/14 01:11 PM Re: Automated Software -'independent' validation? rlcarey
P*Q Offline

Power Poster
P*Q
Joined: May 2001
Posts: 8,437
Somewhere
Originally Posted By: rlcarey
Mainly, you have to validate that everything that is going through your core systems is captured properly in your AML software.
How would an "independent" third party know that?

Return to Top
#1930943 - 06/10/14 01:26 PM Re: Automated Software -'independent' validation? kw004h
ACBbank Offline
Power Poster
ACBbank
Joined: Jul 2006
Posts: 3,959
New York City
Most AML systems pull information from a "core system," which can generate reports. Typically, you would pull reports from the core and the AML system and look for deviations. If the AML system has an audit log, you can review the log for input errors, warnings, etc.
_________________________
"100 victories in 100 battles isnt the most skillful. Subduing the other's military w/o battle is the most skillful." Sun-Tzu

Return to Top
#1931016 - 06/10/14 03:21 PM Re: Automated Software -'independent' validation? P*Q
Kathleen O. Blanchard Offline

10K Club
Kathleen O. Blanchard
Joined: Dec 2000
Posts: 21,277
Originally Posted By: P*Q
Originally Posted By: rlcarey
Mainly, you have to validate that everything that is going through your core systems is captured properly in your AML software.
How would an "independent" third party know that?


It is done by tracing transactions from beginning through various places in core to the AML system, checking total # & $ by category, making sure all expected categories sre captured, reviewing programming scripts, etc.
_________________________
Kathleen O. Blanchard, CRCM "Kaybee"
HMDA/CRA Training/Consulting/Mapping
The HMDA Academy
www.kaybeescomplianceinsights.com

Return to Top
#1931239 - 06/10/14 09:05 PM Re: Automated Software -'independent' validation? kw004h
Princess Romeo Offline

Power Poster
Princess Romeo
Joined: Jun 2001
Posts: 8,272
Where the heart is
There are actually two types of validation:

Data integrity
Model validity

Data integrity is simply tracking the data from transactions to be sure it is captured accurately by your software. If someone conducts a transaction, is the dollar amount, type and method of transaction accurately reported? A cash deposit for $6,000 shows as such, a deposit of checks, a withdrawal of cash, checks being paid, ACH transactions, etc., etc.

Model Validation is bit more complex as that involves ensuring that your system parameters are properly set for your institution to flag those transactions that should rise to the level of requiring a review. And depending on the types of customers and volumes that you have, a model that makes sense at one institution would be hopelessly inadequate, (or overkill) for another institution.

It is the Model Validation that I have been seeing the examiners focus on more and more.
_________________________
CRCM,CAMS
Regulations are a poor substitute for ethics.
Just sayin'

Return to Top
#1931274 - 06/11/14 03:07 AM Re: Automated Software -'independent' validation? kw004h
Kathleen O. Blanchard Offline

10K Club
Kathleen O. Blanchard
Joined: Dec 2000
Posts: 21,277
Either one can make your system useless if not working properly. A good model validation tests data integrity, it has to.
_________________________
Kathleen O. Blanchard, CRCM "Kaybee"
HMDA/CRA Training/Consulting/Mapping
The HMDA Academy
www.kaybeescomplianceinsights.com

Return to Top
#1932007 - 06/12/14 06:00 PM Re: Automated Software -'independent' validation? Princess Romeo
LMBrown Offline
New Poster
Joined: Jun 2014
Posts: 1
Originally Posted By: Princess Romeo
There are actually two types of validation:

Data integrity
Model validity

Data integrity is simply tracking the data from transactions to be sure it is captured accurately by your software. If someone conducts a transaction, is the dollar amount, type and method of transaction accurately reported? A cash deposit for $6,000 shows as such, a deposit of checks, a withdrawal of cash, checks being paid, ACH transactions, etc., etc.

Model Validation is bit more complex as that involves ensuring that your system parameters are properly set for your institution to flag those transactions that should rise to the level of requiring a review. And depending on the types of customers and volumes that you have, a model that makes sense at one institution would be hopelessly inadequate, (or overkill) for another institution.

It is the Model Validation that I have been seeing the examiners focus on more and more.


Does anyone have any good resources to share to best test the "Model Validation" portion of this review? How would you suggest to test whether the model makes sense for your institution? Thanks so much!

Return to Top
#1932163 - 06/12/14 09:31 PM Re: Automated Software -'independent' validation? kw004h
TryingtoComply Offline
Diamond Poster
Joined: Apr 2013
Posts: 1,813
The West
I've had examiners/auditors ask for reports of customers with high cash/wire activity from the core system. They sort the data to identify customers that would appear to be high risk and then compare that data to reports generated from your AML software. I've also had them use the data to evaluate whether or not we have identified all of our high risk customers.

If you rely on your AML software to idenfity high risk customers I think the model needs to be evaluated to ensure that all high risk customers are identified too. We use a well-known AML software that has a risk scoring model that considers business type, geography, product, transactions and TIN. In some circumstances customers with high cash or wire activity did not receive enough points to be considered high risk. The software works well to identify your highest risk customers that have a high risk business type and a combination of cash/wire/ACH activity; however, additional points need to be added using a feature in the product to cause certain cutomers to be classified as high risk.
_________________________
TryingToComply
CRCM

Return to Top
#1932197 - 06/12/14 11:20 PM Re: Automated Software -'independent' validation? kw004h
Kathleen O. Blanchard Offline

10K Club
Kathleen O. Blanchard
Joined: Dec 2000
Posts: 21,277
Every single module needs to be validated You can't do pieces; if you check that all data is pulled but do not check that the math is correct, you have accomplished nothing. A validation, just like an interest rate risk validation, checks EVERYTHING.
_________________________
Kathleen O. Blanchard, CRCM "Kaybee"
HMDA/CRA Training/Consulting/Mapping
The HMDA Academy
www.kaybeescomplianceinsights.com

Return to Top
#2009815 - 04/23/15 05:18 PM Re: Automated Software -'independent' validation? kw004h
Blessed Offline
Diamond Poster
Blessed
Joined: Oct 2007
Posts: 2,389
USA
Does anyone have a Model Validation program they'd be willing to share?

If so please PM Me
Last edited by Blessed; 04/23/15 08:57 PM.
_________________________
Ecclesiastes 10:2 (NIV)

Return to Top

Moderator:  Andy_Z