As Richard points out, RESPA would be the authority on this with its prohibition of unearned fees. Hopefully, you disclosed all fees in the TIL. The fact that some were unused does not change the APR and finance charge calculations since those were accurate as of the time the loan was made. So it should be a simple refund transaction.