In general, if they took in cash payments exceeding $10,000 in connection with one or more transactions within a 24 hour period, they would file a Form 8300
. The requirement is not unique to auction houses, it applies to all U.S. businesses.
Local customs for household auctions vary. In the communities where I have lived, the buyer makes checks payable to the seller (not the auctioneer) and the seller receives and deposits the cash. In this circumstance, if the seller is an individual, there would be no requirement to file the form 8300.