Loan officers are in the sales business. They can charge whatever they want. If they can negotiate more fees, than good for them. However, any time you charge different interest/fees, you subject yourself to fair lending issues. If the loan officers are charging a protected class more, then there is an issue. If there is no correlation between higher fees and protected classes, then they are just doing their job.
Think about it this way:
1. Should ever person get the same rate/fees? No two applicants represent the same risk and sales are part of lending. If 20 of us went to a car lot interested in the same car, I guarantee you we would get 20 different prices for that car depending on our negotiation skills, financing options, etc. Is that illegal?
2. If people are treated differently because of their race, sex, age, etc. that's wrong.