Thread Options
#2225404 - 11/10/19 06:21 PM Gender bias? Oops!
Richard Insley Offline
Power Poster
Richard Insley
Joined: Oct 2000
Posts: 9,999
Toano, VA
Just ask any bank compliance officer--it's not smart to hand out (systemically) lower credit limits to women than to non-women. Yet, that is the subject of a just-opened investigation by the New York Department of Financial Services into "Apple Card"--a joint venture between Apple and Goldman Sachs.

Headline: "New York regulator probes Apple Card algorithms for gender bias after viral tweets"
Article: https://www.marketwatch.com/story/n...ender-bias-after-viral-tweets-2019-11-09

This will be a fun story to follow, folks. Get ready to update your "disaster poster child" file for future training sessions! If you're still budgeting for 2020, add this story to the long list of reasons why your regulatory managers need to attend major conferences. It's also the classic example of why a regulatory manager needs to sit in on all retail product development meetings.

It's always cheaper to do it right the first time.
_________________________
...gone fishing.

Return to Top
Fair Lending
#2225405 - 11/10/19 10:02 PM Re: Gender bias? Oops! Richard Insley
Rocky P Offline
Power Poster
Joined: Jun 2003
Posts: 7,238
Florida
IF there was something built in the algorithms, they would have had to include surrogate for gender, as it's not a collected field. I honestly think there is something else involved. HOWEVER, if the regulators do come across a surrogate generator (or something else triggering off the difference), it will be one of the largest fines to hit the papers. 2 companies with deep pockets.
_________________________
Integrity. With it, nothing else matters. Without it, nothing else matters.

Return to Top
#2225406 - 11/11/19 12:11 PM Re: Gender bias? Oops! Richard Insley
Richard Insley Offline
Power Poster
Richard Insley
Joined: Oct 2000
Posts: 9,999
Toano, VA
How many innocuous data elements do you think it would take to determine gender with 95+% reliability? Since "effect" trumps "intent" or "awareness," doesn't that put Apple & Goldman in the precarious position of having to prove a negative?
_________________________
...gone fishing.

Return to Top
#2225413 - 11/11/19 05:25 PM Re: Gender bias? Oops! Rocky P
rainman Offline
Power Poster
rainman
Joined: Nov 2004
Posts: 3,021
Originally Posted by Rocky P
IF there was something built in the algorithms, they would have had to include surrogate for gender, as it's not a collected field. I honestly think there is something else involved. HOWEVER, if the regulators do come across a surrogate generator (or something else triggering off the difference), it will be one of the largest fines to hit the papers. 2 companies with deep pockets.


Assume it's something in the algorithms that is not a proxy for gender but still causes the results to skew according to gender (if you want to argue that this would really be a proxy for gender, that's fine, but it's not my point). At some point after the system has been functioning for a while, shouldn't they have done some QC to see if there were any real world issues with their results? Better yet to have done testing before rolling it out live.
_________________________
Nobody's perfect, not even a perfect stranger.

Return to Top
#2225416 - 11/11/19 05:30 PM Re: Gender bias? Oops! Richard Insley
rlcarey Offline
10K Club
rlcarey
Joined: Jul 2001
Posts: 77,261
Galveston, TX
Apple Card completely rethinks everything about the credit card.

I think their tag line says it all. smile
_________________________
The opinions expressed here should not be construed to be those of my employer: PPDocs.com

Return to Top
#2225457 - 11/12/19 06:41 PM Re: Gender bias? Oops! Richard Insley
bOaty Offline
Power Poster
bOaty
Joined: Aug 2006
Posts: 4,266
Chillin an grillin
To add to this, check out the article in the latest ABA Compliance mag. They talk about bias in AI from everything from marketing to decisioning.

Very timely information for me to accompany my FLRA.
_________________________
HMDAHMDAHMDAHMDAHMDAHMDA

Return to Top
#2225459 - 11/12/19 06:54 PM Re: Gender bias? Oops! Richard Insley
Rocky P Offline
Power Poster
Joined: Jun 2003
Posts: 7,238
Florida
Richard and Rainman, right on. It’s in the reg for testing. 1002.2 (p) both ref and OSC.
_________________________
Integrity. With it, nothing else matters. Without it, nothing else matters.

Return to Top
#2225507 - 11/13/19 03:03 AM Re: Gender bias? Oops! Richard Insley
Richard Insley Offline
Power Poster
Richard Insley
Joined: Oct 2000
Posts: 9,999
Toano, VA
Keep a good supply of popcorn handy. This will be fun to watch...from the outside as a non-combatant.
_________________________
...gone fishing.

Return to Top
#2225748 - 11/16/19 01:25 AM Re: Gender bias? Oops! Richard Insley
Rocky P Offline
Power Poster
Joined: Jun 2003
Posts: 7,238
Florida
An update on the case
https://www.foxbusiness.com/technology/apple-card-artificial-intelligence-gender-bias-silicon-valley

Like Richard mentioned - there is a start of the story
_________________________
Integrity. With it, nothing else matters. Without it, nothing else matters.

Return to Top
#2226310 - 11/25/19 09:52 PM Re: Gender bias? Oops! Richard Insley
Andy_Z Offline
10K Club
Andy_Z
Joined: Oct 2000
Posts: 27,353
On the Net
I have read in some articles that the credit limits were switched, M/F so it was not always favoring the male. I initially thought it might be income based in some cases as that would be an easy explanation, but I've no idea if separate incomes are listed.

I read some folks saying because it's in a community property state they should be equal but that isn't the case. Community property laws vary. In Texas future income is not community property and if my spouse borrows, if I don't sign it isn't my debt.

It will be interesting. I haven't looked at the Apple application, but generally these apps are so brief today these potential bias questions are not asked. I couldn't believe it would go off a name, and if there are instances of "Jill" getting a higher limit than "Jack" then that isn't 100%. So I'd ask, other than the app, what credit info is included and how is it evaluated? It will be interesting to see what comes of any analysis - if its made public and not a proprietary trade secret.
_________________________
AndyZ CRCM
My opinions are not necessarily my employers.
R+R-R=R+R
Rules and Regs minus Relationships equals Resentment and Rebellion. John Maxwell

Return to Top
#2226416 - 11/27/19 03:49 PM Re: Gender bias? Oops! Richard Insley
Richard Insley Offline
Power Poster
Richard Insley
Joined: Oct 2000
Posts: 9,999
Toano, VA
There doesn't have to be any "credit information" requested, known, or used for there to be a problem.

Give a statistician a few bits of personal information and it's easy to get a very reliable (good enough for a credit decision) determination of gender, age, and other characteristics. For example, here are 3 items Amazon knows I purchased within the past 6 months:
- Set of 2 Trailer Tires & Rims 480-8 4.80-8 4.80x8 8" B 4 Lug/4" Hole Bolt Wheel White Spoke
- 3.0Ah Replacement Black and Decker 40V Lithium Battery LBXR36, for Black & Decker 40-Volt Cordless Power Tools
- 2-Blade Set of 3-N-1 Blades for 42 Inch Cut John Deere Mower
Based solely on these 3 bits of information, am I male or female?
_________________________
...gone fishing.

Return to Top