What should US banks do to prepare for regulatory exams? That question-and others like it-is one Fair Isaac hears a lot from our clients.
To help shed light on regulatory issues, Fair Isaac asked two experts-Jeffrey Brown, Office of the Comptroller of the Currency, and Mariana Rexroth, Office of Thrift Supervision-to give us a regulators' eye view of what they look for. Karlene Bowen, Fair Isaac Scoring Solutions, moderated the panel, which took place at InterACT 2003 in San Diego.
What should lenders do to prepare for a banking exam?
Mariana Rexroth: I think you'll get this answer from any regulator: We want to see the real stuff. What we often get is the PowerPoint version of the credit scoring model. And that's not what we want to see. We want to see reports (application, override and delinquency reports, for example). And we want to see information on the development of the model.
One of the things we're looking at is the whole process: from development, implementation, its application and ongoing monitoring, and through to the validation. This is a continuum, and we want to see the management processes that are in place at each of those points.
Jeffrey Brown: Now that we've got a sane and reasonable answer, let me give you a very provocative answer to the question "What should you do to prepare for a banking exam?"
You should do nothing. You should position yourselves so that you don't really have to do anything in anticipation of the exam. What we expect is that banks use good tools and use them well. And scoring models are just tools, but they are predictive tools-to be validated.
What we look for is not a single discrete activity that you might call validation. Instead, we look for an ongoing process we call validation. For the sake of description, I've compartmentalized that process into three components.
First, there's developmental evidence. A tool has to be built or developed, and that throws off some evidence that tells you whether it's likely to be predictive.
Second, there's ongoing evidence. That includes the kind of basic reports that we have referred to, but it's not limited to that. There are a host of different things that good practice banks do to make sure that their tools are well implemented and predictive.
The third kind of evidence we'll call back-testing. Some kind of comparison of actual outcomes to predictions.
What issues do you see repeatedly that never cease to amaze you?
Brown: What surprises us is that, even at good practice institutions, we frequently see gaps in documentation, for a host of reasons. As a former researcher, I know how difficult it is to document some things, but the fact is we frequently see gaps.
Rexroth: The really critical piece is management of overrides. And there are several issues related to that.
One is the tendency to confuse an override process with what should be proper policy decisions. You need to make a decision cleanly as you implement the scorecard. Maybe some characteristic in the model-say bankruptcy-has not received its full negative weight in the model that it would have had in underwriting. In this case, it should be a policy decision.
We also often see extremely subjective override reasons: "It's a nice property." And my response is, "Oh, please don't open up all those redlining issues for me."
You also need to break down overrides. Often the best I can get is that "We've got a 6% override rate." You should break it down further.
Brown: Another surprisingly common mistake that we see is what I call ad hoc adjustments to welldeveloped scoring models. For whatever reason, the institution will decide they want to tweak the model. And that raises a host of questions about the empirical and statistical soundness of the model. And it can raise questions under Reg B.
One last general mistake that we see is using these tools as rote instruments, if you will. Some companies want to buy scoring tools and forget about them. But if you don't do the monitoring, it's not just a technical problem. You might miss important information about shifts in population. Some of the variables may not be as predictive as they were at the time of development.
So you can't just buy the tool, turn the key and let it run.
Are there any additional things to be considered when using a scoring model on a subprime population?
Brown: Subprime has been on our radar screen recently. And because of that, we are starting to pay more attention to all issues associated with it.
Let's start with the question of what tools are being used to score subprime accounts. One of the issues that has arisen in some companies is that they have used tools that were developed on prime developmental populations to score a less-than-prime population. In this case, it might be a good tool but it wasn't being deployed properly, and it didn't validate out.
Rexroth: One of the other problems we see with subprime relates to pricing. If you are using a scorecard to price a loan, you need one more finely tuned than one that is being used for a yes/no decision on a prime product. This is one area where we have seen that lenders have not recognized this and have gotten into some pretty serious trouble.
Brown: That takes me to another point-the distinction between what we call the reliability and the accuracy of a model. If you're going to use these tools in a risk-based pricing framework, they are going to have to be calibrated to a probability-ofdefault kind of concept. And that means you are looking for accuracy, not just reliability.
There is a perception of 660 as the line in the sand for the definition of subprime. Could you provide some clarification on that?
Rexroth: When the guide (the interagency document "Expanded Guidance for Subprime Lending Programs," June 2001) was issued, people tended to read it as, "This is the agency's definition of subprime." And that's not what it says. The concept of 660 is a for example. It starts with the concept that you need to manage your business and understand what subprime lending is. Whether you're doing it though an affiliate or you're doing it by accident, you have a range of possibilities. So you need to understand that and then manage it economically in terms of capital and so on.
Brown: Any particular scorecard will be associated with odds. The idea the agency was trying to get across in the subprime guide was that there will be differing odds ratios for what many call a prime population and what many call a subprime population. I don't know exactly what that odds ratio is. In fact, I think it probably varies from institution to institution.
However, when we see institutions with large concentrations of their portfolio with scores associated with low odds rates, we're going to dig deeper.
Rexroth: The other aspect of this is, say if you're a nationwide lender, you may find you have different odds in different markets. So you may not want to present the regulator with overall lender-wide national odds, but do it on a market basis.
Let me flesh that out with an example of a subprime credit shop. At cutoff the odds ratio might be 10 to 1. But if I'm thinking about secured lending-and we've got some banks that do that-the odds ratio might be 1 to 1 because it's secure. You've really got to think about it through a profit-maximization lens.
What final thoughts do you want to communicate to lenders anticipating an exam?
Rexroth: There are telephones and email, so if you have questions, you can communicate with your regulator. Some people think, "If I call her up she's going to write down my name, and she'll remember and I have an exam coming up." But the answer is no, that's not what we do.
Brown: To sum up, most of the function of the exam is really not to provide some total report card-otherwise we'd be there far longer than you want us to be. The function of the exam-whether it's compliance or safety and soundness-is to make an evaluation of how effective your management of the process is.
Fair Isaac BoilerplateFair Isaac Corporation provides
predictive modeling, decision analysis, intelligence management,
decision management systems and consulting services that power billions
of mission-critical customer decisions a year.
BankersOnline is a free service made possible by the generous support of our advertisers and sponsors. Advertisers and sponsors are not responsible for site content. Please help us keep BankersOnline FREE to all banking professionals. Support our advertisers and sponsors by clicking through to learn more about their products and services.
Print Friendly!
Email This Article!
Discuss NOW!
