Just to clarify are we talking regulators or audit firm employees? With either group, it should be "your-house, your-rules"
The only person who has ever lost keys that I gave them are DFI Examiners...TWICE on seperate exams.
Since you have additional controls over key-cards that you dont have with keys, it probably isnt so bad. Making them wear them with a bright orange lanyard is probably a good idea, if nothing else it will help your bank staff remember / recognize who is walking around.
THANK YOU GOVERNMENT:
“If I seem unduly clear to you, you must have misunderstood what I said” - A. Greenspan