One other thing - if the removal is for "cause", the bank might consider sending an appropriate complaint to the state licensing board.
I also think it is important to document some place as well as why they are being dropped (timing issues, professionalism, appraisal quality), whether in a minutes of an appropriate committee or Board minutes. This helps to document for your examiners why referral to your state licensing board may or may not be in order.
We send a letter when dropping an appraiser rather than just not use them again and hope they get the hint. It is just a professional courtesy, even if the reasoning is not always pleasant.