Skip to content
GeoDataVision
Thread Options
#2232676 - 03/09/20 09:39 PM Model Validation- FCRM
BSA2019 Offline
New Poster
Joined: Mar 2019
Posts: 16
Is there anyone willing to provide some background information on cost for independently validating FCRM (AML Manager)?

Return to Top
BSA/AML/CIP/OFAC Forum
#2232680 - 03/10/20 11:50 AM Re: Model Validation- FCRM BSA2019
osucpa Offline
Diamond Poster
Joined: May 2011
Posts: 1,402
I would tell you there are several variables involved in this. One is how much customization do you have to the system. We use a different program and ours was 45k four years ago.

Return to Top
#2232686 - 03/10/20 01:35 PM Re: Model Validation- FCRM BSA2019
bcompliance Offline
Diamond Poster
Joined: Sep 2014
Posts: 1,245
we compared several vendors 3 years ago and they ranged from 15k-90k. PM me if you'd like contact info.
_________________________
CRCM, CAMS

Return to Top
#2232687 - 03/10/20 01:44 PM Re: Model Validation- FCRM BSA2019
SmallBankBSA Offline
100 Club
Joined: May 2018
Posts: 172
are you looking for model validation prices or prices to implement a new system? We had a model validation for our AML software from an independent auditor recently and it was $6000. We have not implemented new software in several years.

Return to Top
#2232708 - 03/10/20 05:04 PM Re: Model Validation- FCRM SmallBankBSA
osucpa Offline
Diamond Poster
Joined: May 2011
Posts: 1,402
For 6k I am impressed.

Return to Top
#2232709 - 03/10/20 05:08 PM Re: Model Validation- FCRM BSA2019
ACBbank Offline
Power Poster
ACBbank
Joined: Jul 2006
Posts: 4,190
New York City
There is no way a full model validation is being performed for $6,000. My guess is there is no backtesting being performed and the process verification is being done on a sample basis.

There are many vendors out there claiming to do a model validation but all they are doing is data mapping. Prior to selecting a vendor you should be asking very detailed questions and ensure that the SoW details what exactly the selected vendor will do.
_________________________
"100 victories in 100 battles isnt the most skillful. Subduing the other's military w/o battle is the most skillful." Sun-Tzu

Return to Top
#2232712 - 03/10/20 05:13 PM Re: Model Validation- FCRM BSA2019
SmallBankBSA Offline
100 Club
Joined: May 2018
Posts: 172
$6000 is what we paid and our regulators have had no issues with what has been done. We have this completed every 18 months. We are in a rural area so the low cost may surprise some additionally we are a small community bank with low(er) risk customers.

Return to Top
#2232713 - 03/10/20 05:24 PM Re: Model Validation- FCRM BSA2019
ACBbank Offline
Power Poster
ACBbank
Joined: Jul 2006
Posts: 4,190
New York City
I'm not doubting that's what you paid, the report name or the feedback from your regulators. I'm doubting what they actually did. I understand that pricing varies by location, but backtesting a model properly takes a significant amount of time. I would completely surprised if they followed SR 11-7 or 2011-12 completely at that price point.
_________________________
"100 victories in 100 battles isnt the most skillful. Subduing the other's military w/o battle is the most skillful." Sun-Tzu

Return to Top
#2232719 - 03/10/20 05:42 PM Re: Model Validation- FCRM BSA2019
SmallBankBSA Offline
100 Club
Joined: May 2018
Posts: 172
Hmm.... now I'm concerned that what we are getting may not be appropriate. I will review SR 11-7 and 2011-12 again and compare it to my reports.

If the recommendation is to do validation every 18-24 months, are other banks doing that at the higher prices noted above..i.e. 45k etc. that often? Maybe for a larger bank that cost is not as big an eye opener as it is for a bank with under $500 Million in assets? Or is it just that, a recommendation and the bank is choosing its own time frame? Sorry just trying to understand!

Thanks!

Return to Top
#2232740 - 03/10/20 09:27 PM Re: Model Validation- FCRM BSA2019
#Just Jay Offline
10K Club
#Just Jay
Joined: Oct 2006
Posts: 14,390
Cheeseheadland
When FDIC was here last year, they stated that they have been directed to back off the 18-24 full validation stance if there has been no significant system changes and you are a low risk profile bank.

We did our validation inhouse (300million) and designed it with consultation from our outside audit firm and input from the FDIC. It was a ton of work and staff hours, but saved several thousand dollars.
_________________________
I don't repeat gossip, so listen closely...

Return to Top
#2232743 - 03/10/20 11:08 PM Re: Model Validation- FCRM BSA2019
BSA2019 Offline
New Poster
Joined: Mar 2019
Posts: 16
I am so happy I posted about this! Thank you all for your input. We just recalibrated our software and balanced so I would say we have a fair amount of customization embedded.

Return to Top
#2232750 - 03/11/20 01:36 PM Re: Model Validation- FCRM BSA2019
ACBbank Offline
Power Poster
ACBbank
Joined: Jul 2006
Posts: 4,190
New York City
If you're a small community bank (With no high risk business lines) and you're using one of the lesser tier AML Models (Abrigo, FCRM, Prime, Verafin, etc.) do you really need a full model validation? Do you really need to backtest these systems? Unless you have a regulatory order or serious issues with your model, I would opine that you don't. You can probably get by with a data validation and calibration.

Just my opinion though.
_________________________
"100 victories in 100 battles isnt the most skillful. Subduing the other's military w/o battle is the most skillful." Sun-Tzu

Return to Top
#2233139 - 03/17/20 06:06 PM Re: Model Validation- FCRM ACBbank
Pat Patriot Act Offline
Gold Star
Pat Patriot Act
Joined: Apr 2009
Posts: 450
Originally Posted by ACBbank
If you're a small community bank (With no high risk business lines) and you're using one of the lesser tier AML Models (Abrigo, FCRM, Prime, Verafin, etc.) do you really need a full model validation? Do you really need to backtest these systems? Unless you have a regulatory order or serious issues with your model, I would opine that you don't. You can probably get by with a data validation and calibration.

Just my opinion though.


As someone that does what I'd prefer to call "system validations", as opposed to "model validations", I can attest that there's roughly a 1-3 year timeframe that the regulators will give you before they start verbally mentioning/warning you that a validation is needed.

You're absolutely correct that many firms mislabel their work a "model validation." A true model validation is an engagement or set of engagements that your software vendor contracts out to highly qualified firm - I'm talking one with Data Scientists, AML SMEs, and Quants. You're dead on that, for example, your average firm isn't going to say, do a regression analysis on Verafin's neural networks or calculate k-nearest neighbors to assess the appropriateness of risk segment thresholds. [censored], you wouldn't want them to at an individual bank level, there ain't enough data and it'd be a waste of time.

What I refer to as "system validations" consists mostly of the vendor subsection of SR 11-7,etc. with the addition of extensive benchmark testing (a.k.a. "parallel simulation"). That way, there is effective model challenge involved. And beyond that, of course, analyzing mapping for appropriateness and accuracy by sampling. Some firms might do a core-system reconciliation, but to me it's like a pyrrhic victory if the firm can accomplish that - they've got at most two weeks of field time and can diagnose problems much quicker through other methods.

With that being said - and although I'm admittedly bias - I do see "system validations" adding value all the time. Meeting examiner expectations, pointing out data shortcomings, finding model weaknesses to relay to the vendor, and optimizing parameters are routine outcomes. So many banks, small to mid-size in particular, are missing vital but easily obtained data, running junk scenarios, and totally unaware of their gaps. Not to mention the dearth of model governance procedures.
_________________________
CFE, CAMS

Return to Top
#2245634 - 11/18/20 04:43 PM Re: Model Validation- FCRM BSA2019
BSA2019 Offline
New Poster
Joined: Mar 2019
Posts: 16
Is there a definitive requirement that the Bank to engage a 3rd party to conduct a validation? If so, can someone please quote the Exam manual portion that explicitly states this? Our vendor has a validation performed on the model and we provide the attestation reaffirming the version of the model's effectiveness. Is that enough? In your experience how have Examiners viewed allowance of a vendor-only engaged validation?

Return to Top
#2245636 - 11/18/20 04:56 PM Re: Model Validation- FCRM BSA2019
BrianC Offline
Power Poster
BrianC
Joined: Nov 2004
Posts: 6,316
Illinois
There is no requirement that you engage a third party to perform a validation, but the expectation is that the validation be independent. If your internal audit department has adequate expertise to perform an independent validation, you can perform it in house.

Your vendor may be able to assist you in calibrating parameters so they are appropriately risk-based for your institution, but your vendor is not independent since its the vendor's system.

All the prudential regulators have adopted guidance on model risk management.
https://www.occ.gov/news-issuances/bulletins/2011/bulletin-2011-12a.pdf

"Validation involves a degree of independence from model development and use.
Generally, validation should be done by people who are not responsible for development
or use and do not have a stake in whether a model is determined to be valid.
Independence is not an end in itself but rather helps ensure that incentives are aligned
with the goals of model validation. While independence may be supported by separation
of reporting lines, it should be judged by actions and outcomes, since there may be
additional ways to ensure objectivity and prevent bias. As a practical matter, some
validation work may be most effectively done by model developers and users; it is
essential, however, that such validation work be subject to critical review by an
independent party, who should conduct additional activities to ensure proper validation.
Overall, the quality of the process is judged by the manner in which models are subject to
critical review. This could be determined by evaluating the extent and clarity of
documentation, the issues identified by objective parties, and the actions taken by
management to address model issues."
_________________________
Sola Gratia, Sola Fides, Sola Scriptura, Solus Christus, Soli Deo Gloria!
www.tcaregs.com

Return to Top
#2245646 - 11/18/20 06:39 PM Re: Model Validation- FCRM BSA2019
ACBbank Offline
Power Poster
ACBbank
Joined: Jul 2006
Posts: 4,190
New York City
Larger institutions have teams that have the experience to do this internally. Generally this expertise will sit within Risk Management. As Brian mentioned what matters is the validation is independent of the model owner and users.
_________________________
"100 victories in 100 battles isnt the most skillful. Subduing the other's military w/o battle is the most skillful." Sun-Tzu

Return to Top
#2245649 - 11/18/20 06:45 PM Re: Model Validation- FCRM BSA2019
edAudit Offline
Power Poster
edAudit
Joined: Jul 2008
Posts: 4,780
You are here
Many auditors are fully capable to conduct the validation but can they produce a report that would pass regulatory review?
_________________________
Opinions can be considered as coming from anywhere but my employer.

CAMS


Return to Top
#2245722 - 11/20/20 02:10 PM Re: Model Validation- FCRM BSA2019
WIBanker91 Offline
100 Club
Joined: Nov 2011
Posts: 180
WI
Does the FDIC have a similar guidance as the OCC?

Return to Top
#2245723 - 11/20/20 02:19 PM Re: Model Validation- FCRM BSA2019
rlcarey Online
10K Club
rlcarey
Joined: Jul 2001
Posts: 79,633
Galveston, TX
_________________________
The opinions expressed here should not be construed to be those of my employer: PPDocs.com

Return to Top

Moderator:  Andy_Z