Whether he's the only one qualified or not does not preclude the independence requirements. If there is an independence question then you will have to go outside your market to get someone qualified.
On the surface I really don't see an Independence issue since he is not involved with the property or the loan transaction. But again I don't know him either. Is he the type person that would "inflate" the value due to his involvement 5 years ago?
This is really going to boil down to your comfort level with the appraiser. I would suggest, due to the concerns you have, that you call your examiners and explain the situation and ask for their guidance.
_________________________
The opinions expressed are mine and they are not to be taken as legal advice.