The key to this question is the day on which the customer came into the branch: You haven't indicated when that was. If it was on Friday, you certainly had reason to return the check even if your customer wasn't able to provide the affidavit when he was there. You could have returned the check for signature (not forgery) just on the basis that everyone agreed the signature was unauthorized. You could have issued the Regulation CC large-return-item notice on Friday, too, or called the depositary bank to give them a heads-up the check was going to be bounced.
When DID you return the check?
You mention that the bank may have a defense to your customer's claim for reimbursement. As you said, that's a side issue, and one that will only be resolved after angering your customer and, possibly, dragging the question into court. All of that could have been avoided if you had timely acted on the information you had on Friday when your customer first visited the bank.