Is there anyone willing to provide some background information on cost for independently validating FCRM (AML Manager)?

I would tell you there are several variables involved in this. One is how much customization do you have to the system. We use a different program and ours was 45k four years ago.

we compared several vendors 3 years ago and they ranged from 15k-90k. PM me if you'd like contact info.

are you looking for model validation prices or prices to implement a new system? We had a model validation for our AML software from an independent auditor recently and it was $6000. We have not implemented new software in several years.

For 6k I am impressed.

There is no way a full model validation is being performed for $6,000. My guess is there is no backtesting being performed and the process verification is being done on a sample basis.

There are many vendors out there claiming to do a model validation but all they are doing is data mapping. Prior to selecting a vendor you should be asking very detailed questions and ensure that the SoW details what exactly the selected vendor will do.

$6000 is what we paid and our regulators have had no issues with what has been done. We have this completed every 18 months. We are in a rural area so the low cost may surprise some additionally we are a small community bank with low(er) risk customers.

I'm not doubting that's what you paid, the report name or the feedback from your regulators. I'm doubting what they actually did. I understand that pricing varies by location, but backtesting a model properly takes a significant amount of time. I would completely surprised if they followed SR 11-7 or 2011-12 completely at that price point.

Hmm.... now I'm concerned that what we are getting may not be appropriate. I will review SR 11-7 and 2011-12 again and compare it to my reports.

If the recommendation is to do validation every 18-24 months, are other banks doing that at the higher prices noted above..i.e. 45k etc. that often? Maybe for a larger bank that cost is not as big an eye opener as it is for a bank with under $500 Million in assets? Or is it just that, a recommendation and the bank is choosing its own time frame? Sorry just trying to understand!

Thanks!

When FDIC was here last year, they stated that they have been directed to back off the 18-24 full validation stance if there has been no significant system changes and you are a low risk profile bank.

We did our validation inhouse (300million) and designed it with consultation from our outside audit firm and input from the FDIC. It was a ton of work and staff hours, but saved several thousand dollars.

I am so happy I posted about this! Thank you all for your input. We just recalibrated our software and balanced so I would say we have a fair amount of customization embedded.

If you're a small community bank (With no high risk business lines) and you're using one of the lesser tier AML Models (Abrigo, FCRM, Prime, Verafin, etc.) do you really need a full model validation? Do you really need to backtest these systems? Unless you have a regulatory order or serious issues with your model, I would opine that you don't. You can probably get by with a data validation and calibration.

Just my opinion though.

As someone that does what I'd prefer to call "system validations", as opposed to "model validations", I can attest that there's roughly a 1-3 year timeframe that the regulators will give you before they start verbally mentioning/warning you that a validation is needed.

You're absolutely correct that many firms mislabel their work a "model validation." A true model validation is an engagement or set of engagements that your software vendor contracts out to highly qualified firm - I'm talking one with Data Scientists, AML SMEs, and Quants. You're dead on that, for example, your average firm isn't going to say, do a regression analysis on Verafin's neural networks or calculate k-nearest neighbors to assess the appropriateness of risk segment thresholds. [censored], you wouldn't want them to at an individual bank level, there ain't enough data and it'd be a waste of time.

What I refer to as "system validations" consists mostly of the vendor subsection of SR 11-7,etc. with the addition of extensive benchmark testing (a.k.a. "parallel simulation"). That way, there is effective model challenge involved. And beyond that, of course, analyzing mapping for appropriateness and accuracy by sampling. Some firms might do a core-system reconciliation, but to me it's like a pyrrhic victory if the firm can accomplish that - they've got at most two weeks of field time and can diagnose problems much quicker through other methods.

With that being said - and although I'm admittedly bias - I do see "system validations" adding value all the time. Meeting examiner expectations, pointing out data shortcomings, finding model weaknesses to relay to the vendor, and optimizing parameters are routine outcomes. So many banks, small to mid-size in particular, are missing vital but easily obtained data, running junk scenarios, and totally unaware of their gaps. Not to mention the dearth of model governance procedures.

Is there a definitive requirement that the Bank to engage a 3rd party to conduct a validation? If so, can someone please quote the Exam manual portion that explicitly states this? Our vendor has a validation performed on the model and we provide the attestation reaffirming the version of the model's effectiveness. Is that enough? In your experience how have Examiners viewed allowance of a vendor-only engaged validation?

There is no requirement that you engage a third party to perform a validation, but the expectation is that the validation be independent. If your internal audit department has adequate expertise to perform an independent validation, you can perform it in house.

Your vendor may be able to assist you in calibrating parameters so they are appropriately risk-based for your institution, but your vendor is not independent since its the vendor's system.

All the prudential regulators have adopted guidance on model risk management.
https://www.occ.gov/news-issuances/bulletins/2011/bulletin-2011-12a.pdf

"Validation involves a degree of independence from model development and use.
Generally, validation should be done by people who are not responsible for development
or use and do not have a stake in whether a model is determined to be valid.
Independence is not an end in itself but rather helps ensure that incentives are aligned
with the goals of model validation. While independence may be supported by separation
of reporting lines, it should be judged by actions and outcomes, since there may be
additional ways to ensure objectivity and prevent bias. As a practical matter, some
validation work may be most effectively done by model developers and users; it is
essential, however, that such validation work be subject to critical review by an
independent party, who should conduct additional activities to ensure proper validation.
Overall, the quality of the process is judged by the manner in which models are subject to
critical review. This could be determined by evaluating the extent and clarity of
documentation, the issues identified by objective parties, and the actions taken by
management to address model issues."

Larger institutions have teams that have the experience to do this internally. Generally this expertise will sit within Risk Management. As Brian mentioned what matters is the validation is independent of the model owner and users.

Many auditors are fully capable to conduct the validation but can they produce a report that would pass regulatory review?

Does the FDIC have a similar guidance as the OCC?