It is hard to answer this type of question without seeing all of the data. That said, if you are just basing your assessment area on where your branches are located, that could be a problem, especially if your branches are not in higher minority concentration areas. The concern, of course, is redlining as several banks have had issues when they haven't taken a full MSA as their assessment area (though there are typically other issues beyond just where the assessment area is drawn).
Here is one example of a bank that did not take all counties in an MSA in their assessment area, but ended up with a redlining settlement with the DOJ:
https://www.justice.gov/opa/pr/just...k-resolve-lending-discrimination-claims. This complaint should give you some understanding of what the regulators concerns were (and again, it wasn't just where the assessment area was drawn). The link to the complaint (as well as exhibit A and exhibit B, which provide maps of the assessment area and branch locations - which helps to see visually) can be found at the bottom of the page on the left.
All of that said, you could have good reason to NOT add a census tract in an MSA to your assessment area - especially if the county does not have a higher minority concentration. Again, this will all depend upon your data and specific situation.
If you PM me, I can send you a link to a video I recently recorded on redlining that may assist you in making your determination.