If a teller believes a customer is trying to cash a check with a fraudulent ID, should they (are they required to) refuse to return the ID to the customer? Or should they refuse to cash the check and return both to the customer?

Wow these are broad questions. Is the check being presented stolen or not? You may wish to narrow the scope in your hypothetical situation to make a response more direct.

Are you really contemplating putting a teller into this position? What if they are wrong? What if the person is armed?

Safest thing to do is refuse the transaction and move on. Confrontation is normally not the best approach in an unknown situation.

Also you said customer. I am assuming that you meant a non-customer.

And remember to call the cops and document the event. The crook is just going to go down the street and find an institution that will fall for the fraud. If the check is drawn upon your institution, contact your other branches ASAP -- they're likely next.