It's my understanding that closure of an HSA by the bank can be done once the account is inactive by your banks Inactive/Dormant process, but after April 15 of the following year as an HSA can be funded by April 15 for the previous year.
I would again suggest reviewing your custodial agreement. Without reserving the right to terminate the account in the agreement, what allows you to terminate? These are not standard deposit accounts that the bank has free reign to close at their discretion. You have a fiduciary duty to the customer to abide by the custodial agreement. If you don't reserve the right to terminate in the contract, your first stop should be your legal counsel.