The right of set-off is governed by state statute and state court decisions. It is oftentimes enhanced by contract terms.
I can tell you that a federal income tax refund does not have the same protections afforded to federal benefit payments. However, no one can responsibly say that you have a right of offset unless they are familiar with all of the above mentioned resources.
In this world you must be oh so smart or oh so pleasant. Well, for years I was smart. I recommend pleasant.