If the loan agreement does not specifically address this then what do you think would be your legal authority to do so?
Consult your attorney. Set-off rights generally do not extend to liability accounts.
What you propose sets the concept of setoff on its ear. Setoff presumes that two parties have reciprocal debts. Party A owes money to Party B, and Party B owes money to Party A. In your scenario, the customer owes you money on both accounts. Setoff isn't applicable.