The purpose of the loan is not a factor. The purpose of the structure is what matters. You need to evaluate the specific purpose of each detached structure. If it's personal use, it can be exempted.
You state the detached structure is used for personal purposes and that there is a separate residential structure. Sounds like it meets the exemption requirements.