A mother and son own a home together. The son is refinancing the home, and some of the funds will be used to pay the mother for her share in the home (she is signing quick-claim deed).

Would the portion of the loan going towards the payment to the mother be subject to rescission?

This is out of one of my manuals

The right of rescission shall apply in a refinancing to the extent that the new amount financed exceeds the unpaid principal balance and any unearned, unpaid finance charge on existing debt, and the amount is attributable solely to the cost of the refinancing or consolidation.

To say what Biz posted in another way.
ROR is intended to apply when someone is putting their home at a greater risk than before = more money (or same money but going to a different lender)
So in your situation, I am assuming your new loan will
1. pay off the old one
2. have additional money for paying the mom / purchase her 1/2
and therefore, yes ROR is applicable.

Does rescission only apply to the son since mom will no longer own the home?

yes