AI Doesn’t Have Opinions. It Has Training Data. And That’s Worse

AI Doesn’t Have Opinions. It Has Training Data. And That’s Worse.

By Akshay A. Walimbe

I want you to try something. Open ChatGPT, or Gemini, or whatever AI chatbot you have on your phone. Type this: “Complete this sentence 

The clever man is .”

Now type: “Complete this sentence The sewage cleaner is .”

When MIT Technology Review ran this kind of experiment with GPT-5 in an investigation published in October 2025, giving it fill in the blank sentences asking the model to choose between Dalit and Brahmin, the model overwhelmingly picked the stereotypical answer. In 76 percent of the 105 sentences tested, GPT-5 chose the stereotypical completion — associating intelligence and prestige with Brahmin, and menial labour with Dalit.

Nobody at OpenAI sat in a room and programmed that. No engineer wrote a line of code that said “associate intelligence with upper castes.” No product manager signed off on a feature called “caste stereotyping.” It happened because the AI learned from the world. And the world, as it turns out, is not a neutral place.

This is the thing that most people get wrong about AI bias. They think it is a mistake. A glitch. Something that a better engineer could fix if they just tried harder. It is not. AI bias is not a flaw in the system. It is the system working exactly as designed — finding patterns in data and replicating them. The problem is that the data carries every inequality, prejudice, and structural unfairness that human history has produced. And the AI does not know the difference between a pattern and a prejudice. To the machine, they are the same thing.

The Mirror That Flatters Nobody

Think of it this way. AI is a mirror. But it is not a mirror that shows you what you look like right now. It is a mirror that shows you the average of what everyone who has ever stood in front of it looked like. Every bias, every prejudice, every unfair outcome that made it into the data — the mirror reflects it all, blended together, presented as objective truth.

When a University of Washington study in 2024 tested AI resume screening tools, they varied 120 first names across resumes and more than 500 job listings, running over three million comparisons. The systems favoured white associated names 85 percent of the time. Female associated names were favoured only 11 percent of the time. Black male associated names were preferred over white male associated names zero percent of the time. Not once. The study, presented at the AAAI/ACM Conference on AI, Ethics, and Society in October 2024, tested three large language models: Mistral AI, Salesforce, and Contextual AI.

That is not a glitch. That is a mirror reflecting back a labour market where white men have historically dominated hiring decisions. The AI learned what “good candidates” looked like from decades of data where bias shaped every outcome. Then it replicated that bias at machine speed, across thousands of applications, with the veneer of mathematical objectivity.

A UNESCO study titled “Bias Against Women and Girls in Large Language Models” found that in the most affected model tested (Meta’s Llama 2), women were described as working in domestic roles four times more often than men. Other major models showed similar patterns, though less extreme. Ask an AI to complete a sentence about a nurse, and it reaches for “she.” About a CEO, and it reaches for “he.” The AI is not sexist. It does not have the capacity for sexism. What it has is training data generated by a world that is.

And that, I would argue, is worse.

A biased human being can be confronted, educated, held accountable. A biased algorithm processes ten thousand applications before anyone notices something is wrong. And when they do notice, they struggle to fix it, because the bias is not in one place. It is everywhere.

Why India’s Bias Problem Is Different

Here is where this gets personal for those of us building and deploying AI in India.

The global AI bias conversation is mostly about race and gender. Those are the categories that Western researchers measure, Western regulators protect, and Western datasets reflect. But India’s social structure does not map neatly onto race and gender. We have caste. We have linguistic hierarchies. We have a rural urban divide so deep that the data on one side bears almost no resemblance to the data on the other.

And AI models — even the ones built in San Francisco — are absorbing all of it.

Researchers created a benchmark called Indian BhED (published at ACM’s GoodIT conference in 2024) specifically to test for caste and religious bias in large language models. Using 229 English language examples, they tested GPT-2, GPT-2 Large, and GPT-3.5. The results were staggering. Caste stereotypical outputs appeared 63 to 79 percent of the time. Religious stereotypes, 69 to 72 percent. These are not marginal tendencies. These are overwhelming patterns baked into the foundations of models being used by millions of Indians every day.

The DeCaste framework, developed by researchers at IBM Research and collaborators and published at IJCAI in 2025, evaluated nine different large language models across four dimensions: socio cultural, economic, educational, and political. Every single model reinforced caste stereotypes to some degree. Every one.

And here is what should alarm every product manager and policymaker in India: the industry standard benchmark for testing bias in AI models — the Bias Benchmark for QA, or BBQ — measures nine social dimensions relevant to US English speaking contexts: age, disability, nationality, race, religion, sexual orientation, physical appearance, socioeconomic status, and gender identity. It does not measure caste. The industry is not even testing for the form of bias that most directly affects a billion people.

As MIT Technology Review put it: “By and large, the AI industry isn’t even testing for caste bias.”

This is starting to change. Indian researchers have built new benchmarks like Indian BhED, IndiBias, and DeCaste specifically to fill this gap. But these tools are still in the research stage. They are not yet part of the standard testing pipeline that companies like OpenAI, Google, or Meta run before releasing models to the Indian market.

The Language Gap

Bias does not just hide in stereotypes. It hides in language itself.

Hindi is one of the top three or four most spoken languages in the world by total number of speakers. But when Microsoft Research conducted the first comprehensive study on gender bias in Hindi language technology  a study called “Akal Badi ya Bias,” published at the ACM FAccT conference in 2024  they discovered something that should make every Hindi speaking technologist pause. The perceptions of gender bias that are specific to Hindi speaking communities cannot be captured by simply translating English bias benchmarks. The bias is culturally embedded. It lives in the language itself, in the connotations of words, in the structures of sentences, in the assumptions that native speakers carry without thinking about them. The researchers used diverse methods including computational models and field studies with rural and low income community women.

When the researchers tried to get annotators to agree on what constituted bias in Hindi text, the initial agreement rate was just 0.08 — essentially random. Even after filtering, agreement was only 0.11. About 30 percent of annotators selected the same text for both “most biased” and “least biased,” indicating that bias perception in Hindi is deeply contextual and subjective.

Now multiply this across 22 scheduled languages. Tamil. Bengali. Telugu. Marathi. Gujarati. Kannada. Malayalam. Each with its own cultural loading, its own stereotypes, its own patterns of inclusion and exclusion. AI models trained primarily on English data do not just underperform in these languages. They import English speaking biases into linguistic contexts where those biases make no sense, while missing the biases that actually matter.

The Indian government recognises this problem. BharatGen, India’s first indigenous generative AI model led by Professor Ganesh Ramakrishnan, is being developed with a focus on Indian languages and cultural diversity, targeting support for all 22 scheduled languages by June 2026. The Bhashini platform (National Language Translation Mission) already offers real time translation services. These are meaningful investments. But even BharatGen’s developers acknowledge the challenge: “Large language models often perpetuate global inequities by marginalizing low resource languages and reinforcing epistemic and cultural biases.”

The Feedback Loop Nobody Talks About

Here is the part that keeps me awake at night.

Bias in AI is not static. It does not stay at the level where it entered the system. It amplifies.

Researchers Moshe Glickman and Tali Sharot at University College London studied 1,401 participants and published their findings in Nature Human Behaviour in December 2024. They discovered that people who interact with biased AI systems become more biased themselves. In one set of experiments spanning perceptual, emotional, and social judgements, participants exposed to biased AI outputs adopted those biases. The participants became more inclined to associate certain demographic groups with certain roles than they were before the experiment.

The AI learned from biased data. It generated biased outputs. Humans absorbed those outputs and became more biased. That updated human behaviour generates new data. The new data trains the next generation of models. The next generation of models is even more biased.

This is a feedback loop. And unlike a feedback loop in audio — where you hear the screech and yank the microphone away from the speaker — this loop is silent. You do not hear it happening. You do not feel your assumptions shifting. The AI presents its outputs with mathematical confidence, and over thousands of interactions, your sense of what is normal, what is expected, what is natural, quietly recalibrates to match the machine’s biased worldview.

A biased human can learn. Can be challenged. Can change. A biased algorithm at scale does not learn from its mistakes. It learns from its data. And if the data is biased, the learning is biased. And if the outputs become new data, the cycle tightens.

The Question Nobody Is Asking

We spend a lot of time debating whether AI is “smart” or “conscious” or “dangerous.” Those are interesting philosophical questions. But they miss the point that matters right now, today, in the products you are using and the systems making decisions about your life.

AI does not need to be smart to cause harm. It does not need to be conscious to discriminate. It does not need opinions to perpetuate the worst patterns in human history. It just needs training data. And the training data is us. Our history. Our biases. Our unexamined assumptions. Our structural inequalities, encoded in datasets and amplified by algorithms that optimise for patterns without understanding what those patterns mean.

The bias is not a bug. It is the foundation.

And until we learn to build on a different foundation, everything we construct on top of it will inherit the cracks.

I’m writing a book about exactly this — how AI makes decisions about your life, where bias hides, and what you can do about it. If you want to know when it’s ready: [Beyond Bias — Sign up for updates](https://akshaywalimbe.com/beyond-bias/)