You post some good questions that we don't have solid answers to. You say it is a shed and silo. A silo definitely sounds like a farm building, but I assume you want to say "it's not used for farming anymore, therefore, we ought to be able to exempt it."
§339.4(c)(1) doesn't just say the building can't be "used primarily for agricultural, commercial, industrial, or other business purposes". It also says the detected structure must also be "used primarily for personal, family, or household purposes." So in your example, how is the shed and silo being used for personal, family or household purposes? It sounds like they aren't being used at all - which is the gray area we all face.
You can choose an aggressive view and say "since it's not being used for ag/commercial purposes, it's therefore used for personal reasons" or you could choose the conservative approach and say "since it's not being used for personal purposes, it can't be exempt". It depends on your risk tolerance and appetite. We really don't have any further guidance.