Let’s play a little game. Imagine that you’re a computer scientist. Your company wants you to design a search engine that will show users a bunch of pictures corresponding to their keywords — something akin to Google Images.
On a technical level, that’s a piece of cake. You’re a great computer scientist, and this is basic stuff! But say you live in a world where 90 percent of CEOs are male. (Sort of like our world.) Should you design your search engine so that it accurately mirrors that reality, yielding images of man after man after man when a user types in “CEO”? Or, since that risks reinforcing gender stereotypes that help keep women out of the C-suite, should you create a search engine that deliberately shows a more balanced mix, even if it’s not a mix that reflects reality as it is today?
This is the type of quandary that bedevils the artificial intelligence community, and increasingly the rest of us — and tackling it will be a lot tougher than just designing a better search engine.
Computer scientists are used to thinking about “bias” in terms of its statistical meaning: A program for making predictions is biased if it’s consistently wrong in one direction or another. (For example, if a weather app always overestimates the probability of rain, its predictions are statistically biased.) That’s very clear, but it’s also very different from the way most people colloquially use the word “bias” — which is more like “prejudiced against a certain group or characteristic.”
The problem is that if there’s a predictable difference between two groups on average, then these two definitions will be at odds. If you design your search engine to make statistically unbiased predictions about the gender breakdown among CEOs, then it will necessarily be biased in the second sense of the word. And if you design it not to have its predictions correlate with gender, it will necessarily be biased in the statistical sense.
So, what should you do? How would you resolve the trade-off? Hold this question in your mind, because we’ll come back to it later.
While you’re chewing on that, consider the fact that just as there’s no one definition of bias, there is no one definition of fairness. Fairness can have many different meanings — at least 21 different ones, by one computer scientist’s count — and those meanings are sometimes in tension with each other.
“We’re currently in a crisis period, where we lack the ethical capacity to solve this problem,” said John Basl, a Northeastern University philosopher who specializes in emerging technologies.
So what do big players in the tech space mean, really, when they say they care about making AI that’s fair and unbiased? Major organizations like Google, Microsoft, even the Department of Defense periodically release value statements signaling their commitment to these goals. But they tend to elide a fundamental reality: Even AI developers with the best intentions may face inherent trade-offs, where maximizing one type of fairness necessarily means sacrificing another.
The public can’t afford to ignore that conundrum. It’s a trap door beneath the technologies that are shaping our everyday lives, from lending algorithms to facial recognition. And there’s currently a policy vacuum when it comes to how companies should handle issues around fairness and bias.
“There are industries that are held accountable,” such as the pharmaceutical industry, said Timnit Gebru, a leading AI ethics researcher who was reportedly pushed out of Google in 2020 and who has since started a new institute for AI research. “Before you go to market, you have to prove to us that you don’t do X, Y, Z. There’s no such thing for these [tech] companies. So they can just put it out there.”
That makes it all the more important to understand — and potentially regulate — the algorithms that affect our lives. So let’s walk through three real-world examples to illustrate why fairness trade-offs arise, and then explore some possible solutions.
Then-Google AI research scientist Timnit Gebru speaks onstage at TechCrunch Disrupt SF 2018 in San Francisco, California. Kimberly White/Getty Images for TechCrunch
How would you decide who should get a loan?
Here’s another thought experiment. Let’s say you’re a bank officer, and part of your job is to give out loans. You use an algorithm to help you figure out whom you should loan money to, based on a predictive model — chiefly taking into account their FICO credit score — about how likely they are to repay. Most people with a FICO score above 600 get a loan; most of those below that score don’t.
One type of fairness, termed procedural fairness, would hold that an algorithm is fair if the procedure it uses to make decisions is fair. That means it would judge all applicants based on the same relevant facts, like their payment history; given the same set of facts, everyone will get the same treatment regardless of individual traits like race. By that measure, your algorithm is doing just fine.
Another conception of fairness, known as distributive fairness, says that an algorithm is fair if it leads to fair outcomes. By this measure, your algorithm is failing, because its recommendations have a disparate impact on one racial group versus another.
You can address this by giving different groups differential treatment. For one group, you make the FICO score cutoff 600, while for another, it’s 500. You make sure to adjust your process to save distributive fairness, but you do so at the cost of procedural fairness.
Gebru, for her part, said this is a potentially reasonable way to go. You can think of the different score cutoff as a form of reparations for historical injustices. “You should have reparations for people whose ancestors had to struggle for generations, rather than punishing them further,” she said, adding that this is a policy question that ultimately will require input from many policy experts to decide — not just people in the tech world.
Julia Stoyanovich, director of the NYU Center for Responsible AI, agreed there should be different FICO score cutoffs for different racial groups because “the inequity leading up to the point of competition will drive [their] performance at the point of competition.” But she said that approach is trickier than it sounds, requiring you to collect data on applicants’ race, which is a legally protected characteristic.
What’s more, not everyone agrees with reparations, whether as a matter of policy or framing. Like so much else in AI, this is an ethical and political question more than a purely technological one, and it’s not obvious who should get to answer it.