For Fair Lending, the bank should be concerned for a minimum credit score, if it is arbitrary, because as you indicated there could be issues of Disparate Impact, as protected classes make up a greater portion of lower scores.
That being said, I'm sure that your risk management committee (or equivalent) does set and have risk tolerances that it adheres to. Make sure that there is discussion, and analysis of the risks. The 3 CRA's have a plethora of information to identify the rates of delinquency and charge offs associated with the various credit score tiers. Make sure that that is built into the discussion and framework of your analysis. BTW, 650 is one of the more common scores (second to 680) for unsecured credit that I've seen.
_________________________
Integrity. With it, nothing else matters. Without it, nothing else matters.