Any Yellow Hammer BSA and Fraud users out there that have had a model validation performed, I would love to know what firm you used for the validation and if the validation was deemed adequate by your regulator, and/or if you would recommend the firm.

We've engaged two firms so far to perform the validation and in both cases we found they do not have the expertise of the Yellow Hammer software to complete an adequate validation to pass the scrutiny of our regulator.

I do. I will message you.

We perform the validation, in house. If you are an OCC bank, the model validation should follow OCC 2011-12 Supervisory Guidance on Model Risk Management.

Sent you a message

We do a quick and dirty in house validation on a monthly basis, but we do a full independent validation every 2 years. I'm looking for a new firm as well. Could you please share - thanks. The firm that we used last year, I won't use again so I'll pass that on as well in PM if you're interested. THanks.

Out of curiosity, could you explain what the consultants did and what the examiners thought they lacked? Unfortunately, the firm does not have the chance to defend their service during the exam. I wonder if if some pushback would be effective. It seems borderline inappropriate for them to criticize you if performed adequate due diligence prior to selecting a firm.

Also, are you OCC regulated?

I just had ours done by a new company and it was very thorough. Would definitely recommend them. Send me a PM and I will get you their information.

Out here in the west we are experiencing a lot of criticism regarding audit firms as well despite performing considerable due diligence in choosing a firm. We have stopped using two firms now due to FDIC stating they are inadequate. These are firms that have been around forever and they are probably wondering why they aren't getting any engagements.

Madawaska, It would be helpful if you could be more specific about what was considered inadequate.

Cher, perhaps you could list the steps you take to perform your monthly validation so that others can compare.

There are several steps to validation of AML/BSA software.

1. Determining that all transactions from your core, wire, ACH systems, etc. are flowing to the software and are categorized correctly. This is the easy part.

2. Validating that your rules/parameters/cut-off scores are working appropriately and efficiently. This is the not so easy part.

Many companies do a good job on 1. Few do a good job on 2.

In your opinion, what steps are the examiners looking for the firm to conduct relative to validating that the "black box" is set properly?

Are examiners looking for consultants to "confirm" the rules do what they should by performing "parallel simulation" of the client's settings or running their own "high risk scenario" rules and see if the system detected it; or, should they be performing a full-fledged system tuning and gap analysis? Is there anything else that the examiners would be looking for the firm to do?

Since most of the BSA examiners have no idea how the various AML software even work or what they are or are not capable of doing, I think you will hear a very wide variety of expectations. Some of which will make sense and many that do not. There are no current standards. We are again left to the whims of individual examiner teams and their wide variations in expectations.

Exactly as I thought, which is why the OP's story makes my blood boil. Perhaps it's unfair for me to think this way, but I'm envisioning a young overzealous examiner that read a couple Big 4 whitepapers on AML validation and went to town on a community bank for choosing a firm that doesn't meet that "standard".

They did not even read a white paper they read this

This bulletin provides guidance to help financial
institutions mitigate potential risks arising from
reliance on computer-based financial models th
at are improperly validated or tested.

http://ithandbook.ffiec.gov/media/resources/3676/occ-bl2000-16_risk_model_validation.pdf


Now IMO BSA models only became financial models due to CMPs and cost to validate them.