Sure you can refuse.
As a non-customer of a certain global bank that will go unnamed, I have been refused when trying to cash a check drawn on one of their customer's accounts. A dentist wrote me a $300 refund check for an over-payment, and I went next door to cash it at the dentist's bank. "We don't cash checks for non-customers. If you'd like to open an account, please have a seat in the lobby. Otherwise you can take this check and deposit it at YOUR bank," they said, with a smile.
As a non-customer payee of a certain local bank here, I had the same experience. I took a payroll check there once and was told "We don't cash checks for non-customers."
I think it is perfectly reasonable to refuse to cash such a large check. I'd offer a cashier's check, and explain that when he or she takes the check to his or her bank, they can call us to verify the funds are good.
I'd also confidentially refer the payee's insistent demand to your BSA Officer, for a possible SAR (without letting the customer know that part, of course).