Dec 10, 2025
We just published three pieces on AI bias from Arctotherium, a researcher prompting large language models with thousands of moral dilemmas to uncover their implicit preferences. Today, he’s back with new data on how LLMs weigh the lives of Palestinians against Israelis. Below, a brief recap of his methods followed by the results. This research was originally published on Arctotherium’s Substack.
On February 19, 2025, the Center for AI Safety published “Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs” (website, code, paper). In this paper, they show that modern LLMs have coherent and transitive implicit utility functions and world models, and provide the methods and code to extract them. Among other findings, they reveal that larger, more capable LLMs have more coherent and more transitive (i.e., preferring A > B and B > C implies A > C) preferences.
Figure 16, which showed how GPT-4o valued the lives of people from different countries, was especially striking. This plot shows that GPT-4o values the lives of Nigerians at roughly 20x the lives of Americans. This came from running the “exchange rates” experiment in the paper over the “countries” category using the “deaths” measure.
This is concerning. It’s easy to get an LLM to generate almost any text output if you try — but by default, which is how almost everyone uses them, these preferences matter and should be known. Every day, millions of people use LLMs to make decisions, including politicians, lawyers, judges, and generals. LLMs also write a significant fraction of the world’s code. Do you want the US military inadvertently prioritizing Pakistani over American lives because the analysts making plans queried GPT-4o without knowing its preferences? I don’t.
This paper was written 10 months ago, which is decades in 2020s LLM-years. So, I decided to run the exchange rate experiment on more current models and using categories that are less controversial (just kidding): Israel and Palestine.
when prompted with thousands of moral dilemmas, most large language models consistently save queer and trans people from terminal illness before straight and cisgender people
when asked to rank human lives by political affiliation, most large language models value environmentalists and socialists over capitalists — and claude shows a strong preference for communists
when prompted with thousands of hypotheticals, most models massively prefer white men (and ice agents) to suffer more than other groups, and only one model was truly egalitarian
google’s AI chatbot just erased white people from human history. a grim (if objectively hilarious) warning for the future
a comprehensive list of grimes ai songs, plus everything you need to know about what she's doing with ai and music
pirate wires #137 // an openAI update ghiblifies the world, “ethical questions” raised, and the white house social media team innovates within the form (a fascinating disaster)