"But AI Isn't Biased, It's Just Math" And Other Myths That Need to Die
“But AI Isn’t Biased, It’s Just Math” And Other Myths That Need to Die
The AI You Don’t See | Myths Series #1
I hear this one at every conference, every boardroom, every dinner party where someone discovers I write about AI ethics.
“But Akshay, AI is just math. Numbers can’t be biased. People are biased. The machine just calculates.”
It sounds so reasonable. So clean. And it is completely, dangerously wrong.
Let me show you why.
Math Is Not the Problem. The Inputs Are.
Here is an analogy. Imagine I give you a perfectly calibrated weighing scale. Flawless engineering. Accurate to the milligram. Now imagine I put my thumb on one side of the scale before asking you to weigh something. The scale itself is not broken. The measurement is not inaccurate given what is on the scale. But the result is wrong, because what went onto the scale was compromised from the start.
That is what happens with AI.
The algorithm — the math — often works exactly as designed. The problem is what you feed it. And what you feed it is data. Human data. Data generated by societies that have spent centuries building structures of inequality into every system they operate.
Amazon learned this the hard way. In 2014, they built a hiring algorithm trained on ten years of successful resumes, as first reported by Reuters in October 2018. The system worked beautifully — in the sense that the math was clean. It identified patterns in successful hires and replicated them. There was just one problem. The tech industry had been overwhelmingly male for those ten years. So the algorithm learned that being male was a signal of success. It started penalising resumes that contained the word “women’s” — as in “women’s chess club captain” or “women’s college.” It downgraded graduates of two all women’s universities.
Amazon’s Edinburgh based engineering team tried to fix it. They removed gender as an explicit variable. But the algorithm found proxies — other data points that correlated with gender, including verbs more commonly found on male engineers’ resumes like “executed” and “captured.” The math was doing exactly what math does: finding patterns. The patterns just happened to encode decades of discrimination.
Amazon scrapped the entire system in 2018. As the ACLU put it: these tools are not eliminating human bias — they are merely laundering it through software.
The math was never the problem. The world the math was trained on was the problem.
When “Neutral” Data Isn’t Neutral
The “it’s just math” argument rests on a hidden assumption: that the data going into the system is a neutral, objective representation of reality. It almost never is.
Take facial recognition. In December 2019, the U.S. National Institute of Standards and Technology published its Face Recognition Vendor Test (NISTIR 8280), testing 189 algorithms from 99 developers against over 18 million images. They found false positive rates 10 to 100 times higher for African American and Asian faces compared to Caucasian faces. Not because the math was different for different faces. Because the training datasets were overwhelmingly composed of lighter skinned faces. The algorithm got really good at recognising what it had seen the most of, and really bad at recognising what it had been shown less of. Notably, NIST found that some algorithms developed in Asian countries showed no such dramatic difference between Asian and Caucasian faces — the data you train on shapes the outcomes you get.
In their 2018 Gender Shades study, MIT researcher Joy Buolamwini and Timnit Gebru tested commercial facial recognition systems from IBM, Microsoft, and Face++ (Megvii). They found error rates as low as 0.8% for lighter skinned men and as high as 34.7% for darker skinned women. Same math. Same algorithm. Wildly different outcomes depending on whose face was in front of the camera.
In India, this plays out in ways that hit closer to home. AI credit scoring systems use data points like your PIN code, the language of your phone’s operating system, and your smartphone model. These seem like neutral data points. They are not. Your PIN code is a proxy for your neighbourhood, which in India is often a proxy for your caste and economic background. The language setting on your phone maps to regional and educational patterns. The model of your phone maps to income.
The algorithm is not asking about your caste. It does not need to. The data is encoding caste through a dozen innocent looking proxies, and the math is doing exactly what it was told to do: find patterns and use them to make decisions.
The Feedback Loop Makes It Worse
Here is the part that should really concern you. Biased AI does not just reflect existing inequality. It amplifies it.
If a hiring algorithm screens out candidates from certain backgrounds, those candidates never get hired, never build the track record that would make future algorithms rate them higher. The algorithm’s output becomes the next cycle’s input. The bias compounds. It is not a photograph of inequality. It is a photocopier, running on a loop, making each copy a little darker than the last.
Predictive policing systems in the US showed this clearly. PredPol (rebranded as Geolitica in 2021, shut down December 31, 2023) directed police patrols to neighbourhoods that already had high arrest rates. More police presence led to more arrests. More arrests led the algorithm to direct even more police to those same neighbourhoods. The Markup’s October 2023 investigation found the system had less than 0.5% accuracy in Plainfield, New Jersey, and none of the departments they studied could point to a single arrest that came as a direct result of a PredPol prediction. The math was flawless. The result was a system that systematically over policed communities of colour while producing almost no measurable law enforcement benefit.
The math was just doing math. But the outcomes were anything but neutral.
So What Do We Do With This?
I am not telling you to distrust math. Math is a tool. I am telling you to distrust the assumption that because something is mathematical, it is objective. Objectivity is not built into the equation. It has to be designed in, tested for, and continuously monitored.
The next time someone tells you AI cannot be biased because it is just math, ask them three questions:
1. What data was the math trained on? If the data reflects historical inequality, the math will replicate it.
2. Who chose what counts as a “good” outcome? Every AI system optimises for something. The choice of what to optimise for is a human decision, loaded with human values.
3. Who checked the results across different groups? If nobody looked at whether the math produces different outcomes for different populations, nobody knows whether it is fair.
Math does not lie. But it does not ask questions either. That part is on us.
Coming Next in the Myths Series
This was Myth #1. There are more where this came from, and they are just as stubborn:
Myth #2: “AI treats everyone equally.” It does not. Equal treatment and equitable outcomes are mathematically incompatible in most real world scenarios. Kleinberg, Mullainathan, and Raghavan proved it in 2016, and Chouldechova independently proved a similar result the same year. I will walk you through it.
Myth #3: “More data fixes bias.” More data from a biased world gives you a more confident biased system. Quantity is not quality.
Myth #4: “Humans are more biased than AI anyway, so let the machine decide.” This one sounds progressive. It is not. I will explain why replacing one form of unaccountable bias with another is not progress.
If you want the full picture — the frameworks, the case studies, the practical tools for catching bias before it ships — that is what the book is for.
[Beyond Bias: The Four Way Test for Ethical AI](https://akshaywalimbe.com/beyond-bias/)