Someone Asked Me: Can AI Be Racist?
Someone Asked Me: Can AI Be Racist?
The AI You Don’t See | Reader Q&A #1
A few weeks ago, someone at a talk I gave in Bangalore asked me a question that silenced the room.
“Akshay, just tell us straight. Can AI be racist?”
I could see the audience split in real time. Half the room expected me to say yes, obviously. The other half expected me to say no, because machines do not have feelings. Both sides were waiting for a simple answer.
I could not give them one. Because the honest answer is more interesting — and more important — than either yes or no.
The Intent Problem
Let me start with what AI cannot do. It cannot hate. It cannot hold prejudice. It does not wake up in the morning and decide that one group of people deserves less than another. It has no intent, no consciousness, no moral compass. In the way we normally use the word “racist” — describing a person who holds beliefs of racial superiority and acts on them — no, AI cannot be racist.
But here is the thing. Racism, in the real world, has never required individual intent to cause massive harm. Redlining in America was not one banker’s personal prejudice. It was a system — maps, policies, lending criteria, actuarial tables that produced racially discriminatory outcomes at industrial scale while allowing every individual in the chain to say, truthfully, “I was just following the process.”
AI is the most powerful process following machine ever built. And when the process is built on data that encodes historical discrimination, the machine follows that process with terrifying efficiency.
How It Actually Works
Let me walk you through the mechanics with an Indian example, because this is not just a Western problem.
Imagine a lending algorithm used by a fintech company in India. The company wants to predict which loan applicants are likely to repay. They train their model on five years of historical lending data. The algorithm is never given the applicant’s caste. Nobody typed “Dalit” or “Brahmin” into a database column. The developers would be horrified at the suggestion.
But the data contains a PIN code. And in India, residential segregation along caste lines is not ancient history. It is present reality. A 2024 study published at ACM GoodIT (the Indian BhED dataset) found that large language models produce caste stereotypical outputs 63 to 79% of the time. Your PIN code, in many cities and certainly in most towns, maps to your community’s caste composition with uncomfortable accuracy.
The data also contains the applicant’s educational institution. Certain universities and colleges have historically had higher enrollment from certain communities — a legacy of access, not ability.
The data contains the employer’s name. Certain industries and companies, for reasons rooted in social networks and hiring patterns, have workforces that skew along caste lines.
The algorithm was never told about caste. It did not need to be told. It found the proxies — the data points that correlate with caste without naming it. PIN code. College name. Employer. Phone model. Language preference. Each one, individually, seems harmless. Together, they construct a caste profile without ever using the word. Dvara Research, in a 2024 study on AI in digital credit in India, warned that such systems can “perpetuate biases present in data but do so at scale, without traceability.”
And then the algorithm makes its decision. Approve or reject. Higher interest rate or lower. Larger credit limit or smaller.
The machine did not intend to discriminate. But the person who was denied a loan because of where they live and where they studied — they do not experience the intent. They experience the outcome.
The Feedback Loop That Locks It In
Now here is where it gets worse.
If the algorithm denies loans to people from certain PIN codes at higher rates, those communities have less access to credit. Less credit means fewer business starts, fewer home purchases, fewer opportunities to build the financial track record that would make future algorithms approve them. The algorithm’s decision becomes the input for the next cycle of decisions.
This is not a one time error. It is a compounding one. Each cycle reinforces the pattern. The gap widens. And because the algorithm is processing thousands of applications per hour, the compounding happens faster than any human system of discrimination ever could.
A biased loan officer might deny fifty applications a day based on unconscious prejudice. A biased algorithm denies fifty thousand. Same bias. Different scale.
So What Is the Right Word?
If AI cannot be racist in the way a person can, but it can produce racially and caste discriminatory outcomes at scale, perpetuate historical patterns of exclusion, and compound inequality through feedback loops — what do we call that?
I think the language matters less than the outcome. Whether you call it “biased,” “discriminatory,” or “systemically racist” is a semantic debate. The person who was denied a loan, or flagged by a facial recognition system, or screened out of a job application does not care what label you put on the mechanism. They care that it happened, and that nobody can explain to them why.
What I tell people is this: AI cannot be racist in the way you are thinking about it. It has no beliefs, no malice, no agenda. But it can be the most efficient vehicle for discrimination ever invented, precisely because it operates without intent. Nobody has to decide to be unfair. The system produces unfair outcomes by default, because it was trained on a world that is already unfair.
And here is the part that should keep you up at night: because the discrimination is embedded in the math, in the correlations, in the proxy variables, it is invisible in a way that a human bigot’s prejudice is not. You can confront a person. You can challenge their reasoning. You can appeal to their conscience. You cannot do any of that with a model that has billions of parameters and cannot explain its own logic.
What You Can Actually Do
If you are building AI systems, deploying them, or making decisions based on their output, here are three things that matter:
Test the outputs, not just the inputs. A clean dataset with no explicit protected categories can still produce discriminatory outcomes through proxy variables. The only way to know is to check the results across demographic groups. If approval rates differ significantly by geography, institution, or any proxy that correlates with protected characteristics — you have a problem, regardless of what is in the training data.
Build feedback loop detection into your monitoring. Bias does not sit still. It amplifies over time. Researchers Kristian Lum and William Isaac demonstrated this mathematically in 2016 using Oakland drug crime data, and subsequent work by Ensign and others in 2018 proved that such feedback loops can drive 100% of system resources to a single group. If you are not tracking whether your system’s outcomes are diverging across groups over months and years, you are flying blind.
Ask who is missing. The most insidious form of AI discrimination is not the wrong answer for someone in the system. It is the absence of someone from the system entirely. Roughly 500 to 630 million Indians remain offline, according to IAMAI and TRAI data, and are therefore not meaningfully represented in the datasets that AI systems train on. If your model has never seen them, it cannot serve them fairly.
The question “Can AI be racist?” does not have a yes or no answer. It has a longer, harder answer that involves understanding how systems work, how data encodes history, and how scale changes everything.
If you want to go deeper — into the proxy variables, the impossibility theorems, the practical audit tools — that is exactly what I wrote a book about.
Got a question about AI ethics you want me to answer? Drop it in the comments or send it to me directly. The harder the question, the better.
[Beyond Bias: The Four Way Test for Ethical AI](https://akshaywalimbe.com/beyond-bias/)