71AI Snake Oil

Arvind Narayanan and Sayash Kapoor, AI Snake Oil: What Artificial Intelligence Can Do, What It Can't, and How to Tell the Difference, Princeton University Press, 2024

Linked to 37 chapters: supports 29 · contradicts 3 · mechanism 12 · same claim 5 · pairing 4

Narayanan and Kapoor open with a world that has no words for cars, bicycles or rockets, only the collective noun vehicle. In that world people argue about whether vehicles are good for the environment without noticing that one side means bicycles and the other lorries, and fraud flourishes because nobody can tell one claim from another. Replace vehicle with artificial intelligence and, they say, you have a fair description of the present. The method follows: rather than a single verdict on AI the book separates three technologies and reaches three verdicts. Predictive AI, which forecasts what a person will do in order to decide something about them now, mostly does not work, cannot be made to work by adding data, and is where nearly all the snake oil is concentrated. Generative AI works, is improving quickly, and is immature as a product in ways that make it easy to misuse. Content moderation AI works well enough that platforms depend on it, and is sold as an answer to questions that are not technical at all. Snake oil, in their definition, is AI that does not and cannot work as advertised. The last two chapters ask why so much of it sells: because companies, researchers, journalists and public intellectuals all profit from the confusion, and because the institutions buying it are already broken and want a cheap repair.

Painting AI with a single brush

There is no consistent definition of AI, and the authors decline to supply one. The usual tests all fail: object recognition is trivial for a person and was called AI anyway; a robot vacuum is programmed by hand and is called AI; a bimetallic thermostat adapts to its environment and is not. What decides the label is history, marketing and fashion, which is why the joke definition is the most accurate one: "AI is whatever hasn't been done yet." Spell-check, spam filtering and autocomplete were all hard problems that stopped being AI the moment they started working reliably.

Predictive AI is the type that admits no split verdict. Hiring companies claim to read warmth and kindness off thirty seconds of video, and not one has published a peer-reviewed validation or let an external researcher test the product. Pymetrics and HireVue commissioned audits and scoped them to racial and gender bias alone, which draws the sharpest observation in the chapter: an elaborate random number generator that works equally badly for everyone is easy to make unbiased. The book's name comes from the patent medicines sold across America before the Food and Drug Administration was created in 1906, most of which contained no snake oil and some of which killed people. The authors sort applications by how well a thing works and how much harm it does, and take everything but the truthful, benign corner.

How predictive AI goes wrong

In 2015 the president of Mount St. Mary's University in Maryland proposed to raise the retention rate by surveying first-year students and dismissing the struggling ones before enrolment was counted; when professors objected that nobody can tell in the first weeks who will succeed, he told them they were thinking of the students as cuddly bunnies, and that the bunnies had to be drowned. Narayanan and Kapoor, with Angelina Wang and Solon Barocas, spent over a year documenting about fifty automated decision systems in criminal justice, healthcare, welfare, education and worker management, and found the same five faults recurring.

The first is that a good prediction is not a good decision. In a study published in 1997, researchers trained a model to predict which pneumonia patients would suffer complications and found that asthma appeared to lower the risk. It did, in the data, because asthmatics with pneumonia were sent straight to intensive care. The model was accurate under the triage system it was meant to replace, and deploying it would have sent asthmatic patients home.

The second is gaming. Three-quarters of American employers screen candidates with automated tools, and candidates respond by stuffing résumés with keywords and university names in white text that a person cannot see and a parser can. Journalists testing Retorio, a Munich video-interview company, found that glasses or a scarf moved the score sharply and that darkening the video lowered it: a score that moves when the lighting moves cannot be taken at face value.

The third is overautomation, which they define as using AI to decide without leaving the subject any recourse. The Netherlands deployed a welfare fraud algorithm in 2013 that replaced human review and made accusations of guilt from statistical correlations alone; over six years it wrongly accused about thirty thousand parents, nationality was among its inputs, and in 2021 the prime minister and his cabinet resigned. Vendors protect themselves by printing that their tools require human supervision, which the authors call a bait and switch: full automation is the sales pitch, human oversight is the fine print. When Toronto used a model in 2022 to judge its beaches unsafe for swimming, the officials who supposedly made the final call had never once overruled it.

The fourth is predicting about the wrong people. The Public Safety Assessment was built on 1.5 million people from three hundred jurisdictions, which sounds like the fix and is not: among the defendants Cook County marked high risk after adopting it in 2015, ten times fewer went on to commit violent crimes than in the national training data, so thousands were jailed for months before trial on a number that did not apply. AI tools search under the streetlight, the authors write, and the streetlight is usually pointed at the poor.

The fifth is that predictive AI widens the inequalities it inherits. Optum's Impact Pro ranked patients for extra care; researchers found in 2019 that a Black patient with the same health risks as a White patient was less likely to be enrolled. The cause was the target variable: health need is hard to quantify and spending is recorded exactly, so the model predicted spending, which tracks access, insurance and the ability to pay. COMPAS has the same substitution at its centre, since its developers speak of predicting crime and its data records arrests.

Why can't AI predict the future?

The third chapter asks whether these limits are temporary. Weather forecasting took two millennia to assemble, and in 1963 Edward Lorenz found that rounding his inputs to three decimal places instead of six produced completely different weather, which is how the world learned that the atmosphere is chaotic. Since then accuracy has improved by about one day per decade: no revolution, just accumulation.

The best evidence on human life outcomes comes from a study designed to be hard to dismiss. Sara McLanahan at Princeton had followed over four thousand children born around 2000, surveying them and their families at birth and at ages one, three, five and nine, and collecting over ten thousand variables per child. In 2015, with the age-fifteen wave about to be released, Matt Salganik proposed testing how well the future could be predicted. The organisers ran an open competition: release everything up to age nine, ask the world to predict six outcomes at age fifteen, and withhold the answers so that nobody could tune to them. A hundred and sixty teams entered, using every method available, and against them the organisers ran a baseline of four variables: the mother's race, marital status and education, and the child's own outcome at age nine. The best submissions were barely better than a coin flip, and did not beat the four variables.

The follow-up is the part that stays with you. Salganik and colleagues interviewed the families whose outcomes had been predicted worst. One struggling child had suddenly done well at school, because a neighbour had counselled him, helped with his homework and fed him blueberries. No survey question asked whether a supportive adult outside the family existed. Salganik's other conjecture is the eight billion problem: that social patterns may vary so much across time and place that there are not enough people on Earth to learn them from.

Deployed tools do no better. ProPublica found in 2016 that among Broward County defendants who did not go on to offend, 45 per cent of the Black ones had been marked high risk against 23 per cent of the White ones. The disparity got the attention; the accuracy deserved as much. COMPAS scored 64 per cent relative accuracy, where 50 per cent is achievable by generating scores at random, which is to say a questionnaire of 137 items does no better than two variables: age and number of prior offences.

If failure is unpredictable because it turns on circumstance, success ought to be easier, and it is not. Far more books, films and songs are made than anyone can consume, and a large share are good enough that most people would enjoy them; consumption is unequal not because a few works are thousands of times better but because culture requires shared experience, so cumulative advantage is built into the market and which of the good-enough products wins is close to random. Salganik's music experiment tested this directly. Fourteen thousand participants rated and downloaded songs by unknown bands. One group saw no download counts, which gave a measure of quality on its own; the rest were split into eight separate copies of the app, each accumulating its own counts independently. Mediocre songs did well and good songs did badly, the same song was a hit in one world and a failure in another, and in the world without visible counts the inequality between songs was far smaller.

Aggregates are sometimes easier than individuals: seasonal influenza forecasting improves year on year, while pandemic forecasting is hopeless, because a pandemic begins with a random mutation and because the forecast changes the outcome. The reasons the chapter collects are several and different in kind: data that does not exist, variables nobody thought to record, shocks no model could contain, feedback loops that amplify a first small advantage, and tasks in which people respond strategically to the prediction. Some will yield to more data; some will not.

The ladder of generality

Generative AI gets the opposite treatment, and the authors state their position plainly: they use these tools daily, the advances are real, and the harms arise as much from the technology working as from its failing. The unlock was data. In 2007 Fei-Fei Li argued that the bottleneck in computer vision was not the model but the dataset, and assembled ImageNet, around three million images labelled through Amazon Mechanical Turk at roughly two dollars an hour. In the 2012 contest AlexNet, trained on graphics cards repurposed from video games, scored 85 per cent against the previous 74. ImageNet set the culture as much as the technique: web scraping without consent, underpaid annotation, datasets too large to audit, and benchmark scores as the sole route to credibility.

Text was harder than images because meaning depends on words many paragraphs back, and the answer, from Google in 2017, was a matrix quantifying how related every word is to every other word, which parallelises across graphics hardware. That is the transformer. Generation then reduces to the simplest possible task: predict the next token, again and again, with no plan for the sentence as a whole. Base models like GPT-2 were good at language and poor at conversation, and fine-tuning them to follow instructions was the last step to ChatGPT. What the machine does not have is a source of truth. The authors borrow Harry Frankfurt's definition and call the output bullshit: speech produced to be plausible, without regard to truth. Of the harms they catalogue, labour exploitation is the one they name as most serious: OpenAI's Kenyan contractor Sama paid between 1.46 and 3.74 dollars an hour to workers filtering toxic text.

The framing device for where this is going is what they call the ladder of generality: a series of ways of computing, each more flexible than the last, where a higher rung means less work to make the machine do something new. The Antikythera mechanism and the Enigma machine are the bottom, a new device for every task; Turing's universal computer is the first rung proper, and stored programs the next. Machine learning removed the need to write new software, needing only a dataset and a learning algorithm. Deep learning removed the need to devise the learning algorithm: gradient descent for everything, with only the architecture varying. Chatbots removed programming altogether — state the task in words — and turned AI into a consumer product. That is the seventh rung.

The ladder is also their answer on existential risk. Two open letters in 2023, one asking for a pause on training systems more capable than GPT-4 and one ranking AI extinction with pandemics and nuclear war, rest on the idea of a threshold that a system crosses to become uncontrollable. The history shows no thresholds, only rungs, each of which felt at the time like the doorstep of general intelligence: Frank Rosenblatt's perceptron was reported in 1958 as a machine that would walk, talk, see and be conscious of its existence. Their substitution is to replace intelligence, which cannot be measured, with power, the ability to modify the environment, at which point the question becomes what people do with these systems rather than what the systems do alone. What they propose instead of a pause is defence against specific threats, of the kind information security already practises.

Why can't AI fix social media?

In 2018 Mark Zuckerberg told Congress that AI would deal with troll accounts, election interference, fake news, terrorist content and hate speech, and policymakers took it at face value. Content moderation is not a side function of a platform; it is the product. The software is trivially copyable — Mastodon reproduced X's functionality with fewer than ten employees — so what a platform sells is a community, which is to say a set of rules and their enforcement. Facebook's public community standards run to over eighteen thousand words. Content is scanned as it is posted, the easy categories are removed automatically, and the rest goes to hundreds of thousands of low-paid outsourced moderators, seconds per item.

The failures that get attention are context failures. In February 2021 a father photographed his son's swollen genitals at a doctor's request; Google's classifier flagged the image, closed his account, and referred him to the police, who cleared him, after which Google still refused to restore the account, his email and the phone number securing his other accounts. The authors argue that this class of error is not fundamental. What is not fixable by better models is everything else. Moderators are not permitted to think: the former Facebook moderator Viana Ferguson could not get a post removed — a photograph of a White family with a Black child captioned as a home needing a pet — because no rule covered the combination of image and caption. Cultural incompetence is worse, and is purely a matter of money. Amnesty International's report on the 2017 ethnic cleansing of the Rohingya in Myanmar documents Facebook's part in it, including fifteen warnings from experts since 2012. In 2014 Facebook had one Burmese-speaking moderator, in Dublin.

The deepest point of the chapter is that the loudest complaints are not about enforcement at all. In 2016 Facebook repeatedly removed the photograph of Phan Thi Kim Phuc fleeing a napalm strike, and this was not a classifier error: the image was already used in moderator training as an example of the unacceptable. It was a policy decision, made by people, about where the line between the obscenity of a representation and the obscenity of a history should fall — the same argument the Associated Press had in 1972. The outcry changed the policy, which the authors regard as the system working.

Of the seven reasons they give for moderation being hard — context, cultural incompetence, a changing world, adversaries, real-world consequences, regulatory compliance and policymaking — only the first is mostly a limitation of current AI. Ravi Iyer, formerly a Facebook executive, calls moderation a dead end, and the authors agree, because the recommendation algorithm has the upper hand over the moderation algorithm. Recommenders run on user behaviour rather than post content, so a community coding anorexia as goal weights stays invisible to the filter and fully visible to the feed, and some harms are not violations at all: teenagers are harmed by comparison rather than by any post, and polarisation is a property of a stream. Moderation AI is a reasonable cost saving, and becomes snake oil the moment it is sold as an answer to a moral and political dispute.

Why do myths about AI persist?

Sepsis kills one in five people who die worldwide and is treatable if caught early. Epic, which holds health records on over 250 million Americans, released a sepsis prediction model in 2017 that hospitals could switch on without new equipment; hundreds adopted it, the company said it reduced mortality, and its chief executive told an interviewer in 2020 that the model could identify sepsis six hours before a person could. Epic published no peer-reviewed evidence and called the model a trade secret; four years passed with no external evaluation. In June 2021 researchers at the University of Michigan Medical School, who could do the study only because their own hospital used it, found that its relative accuracy was 63 per cent rather than the 76 to 83 per cent claimed, where 50 per cent is a coin flip. Epic's adoption numbers turned out to be less impressive than they looked, since the company was paying hospitals up to a million dollars in credits for meeting conditions, one of which was using a sepsis model. In October 2022 Epic stopped selling the one-size-fits-all version and asked hospitals to train it on their own data, which removes the efficiency that was the point.

That pattern — release with fanfare, withhold the method, get adopted in consequential settings, resist scrutiny for years — is what the chapter sets out to explain, and it finds four contributors. Companies have commercial reasons to hype and none to be transparent, and some have no AI at all: an audit of one firm holding a twenty-million-dollar public safety contract with Utah found it was not using AI. Even honest companies game the metric, reporting whatever accuracy number clears ninety, because investors see the nine and stop reading.

The research community contributes for reasons of culture as much as incentive. It now depends on corporate money, with almost three-quarters of AI doctorates going to industry against a quarter two decades ago, and unlike medicine, where pharmaceutical funding produced conflict-of-interest rules, computer science treats the arrangement as normal. It also optimises benchmarks instead of understanding: accepting a Test of Time award in 2017, Ali Rahimi compared machine learning to alchemy. The same overconfidence shows in its attitude to the professions it proposes to replace. In 2016 Geoffrey Hinton said deep learning would beat radiologists within five years and added: "People should stop training radiologists now." In 2022 there was a worldwide shortage of radiologists.

Beneath this sits a measurable defect. In 2018 Odd Erik Gundersen and Sigbjørn Kjensmo reviewed four hundred papers from leading AI venues and found not one that met all their criteria for being reproducible by an independent researcher. The authors found the mechanism themselves. Civil war prediction was the one social prediction task at which machine learning appeared to excel, and when they examined the 2016 paper that started the claim, the models had been scored on data they were trained on; fixing the error left them no better than decades-old statistical methods. Leakage of that kind turned out to have affected hundreds of papers across more than a dozen disciplines, including medicine, psychiatry and genomics, one of them co-authored by Narayanan. Their online workshop on the problem, expected to draw a few dozen people, registered over seventeen hundred researchers.

The press amplifies all of this. The authors analysed fifty articles and found press releases repeated as reporting, limitations buried, and robots illustrating stories that have nothing to do with robots. A 2022 study claiming to predict crime a week ahead with 90 per cent accuracy was reported by over a dozen outlets under headlines invoking Minority Report; the model counted a prediction correct if the crime occurred on the predicted day or the day before or after. Lee Vinsel's word for criticism that ends up advertising the technology is criti-hype, which is what the 2023 pause letter did when it took a chatbot's bar exam score as evidence that jobs were about to be automated.

Where do we go from here?

The last chapter argues that the supply of snake oil is less tractable than the demand, which comes from institutions that cannot do their jobs. American newspaper advertising and circulation revenue fell from about sixty billion dollars in 2000 to just over twenty-one billion in 2022, which is why CNET published seventy-seven machine-written articles, forty-one of them carrying errors or plagiarism. Schools that cannot redesign their assessment buy AI text detectors that do not work and are biased against writers whose first language is not English. Where the problem is structural, the tools are decoration: Chicago spent nearly forty-nine million dollars on ShotSpotter before its own police review found no improvement in developing evidence in gun crimes.

The prescriptions follow the diagnosis. Argue against the purchase from inside the organisation, especially if you sit in the decision, and show up to the local processes where surveillance equipment is approved. Abandon the assumption that every allocation has a computable optimum, and where scarcity cannot be removed, use partial lotteries: everyone past a threshold enters a pool and is drawn at random. Barry Schwartz proposed this for university admissions in 2005, arguing that the pursuit of demonstrable success kills risk-taking and lets the admitted mistake luck for merit. Lotteries also make evaluation possible, as when Oregon expanded Medicaid by lottery in 2008.

On regulation they refuse the usual choice between an innovation-killing burden and a cure-all. Companies fix what damages their reputation and ignore what lands on other people, so where the harm is external, rules are the only mechanism. Three beliefs are in the way: that entirely new AI law is needed, when the United States already regulates through sector agencies and the October 2023 executive order assigned a hundred and fifty tasks to more than fifty of them; that law lags technology, when the First Amendment still governs arguments about an internet it predates; and that legislators do not understand the technology, when they have no civil engineering degrees either and we still have building codes. What they recommend is enforcing what exists and funding the enforcers in step with the industry: the Federal Trade Commission has used its deceptive-practices authority against false AI claims, ordering companies to delete not only data collected deceptively but the models built from it. Regulatory capture is the obvious risk: Sam Altman told the Senate that regulation was essential while proposing a licensing regime that would bind his competitors.

On work they are deflationary. Benchmark performance overweights the part of a job that is subject-matter knowledge, which is why predictions of imminent replacement keep failing. Of the hundreds of occupations in the 1950 American census, exactly one was eliminated by automation: elevator operator. Automation more often shrinks an occupation or changes its duties, and sometimes grows it, as automated teller machines lowered the cost of a branch and the number of tellers rose. What decides who is displaced is power inside the organisation. In 2023 the workers staffing the National Eating Disorders Association helpline voted to unionise, were dismissed four days later and replaced by a chatbot, which was caught advising users to maintain a daily calorie deficit and was withdrawn within days. Ted Chiang's line, that fears about technology are fears about capitalism, is the one they build on: the response is labour institutions, not model evaluations. The book closes with two children born around the release of ChatGPT. Fear and deference produce Kai's world, where an underfunded school sorts him by predictive model; a push to know produces Maya's, with independent researcher access, admissions by partial lottery, and copyright and labour rules that force the companies to share what they have taken. As of 2024, they write, Kai's world is the likelier.

Since 2024

The distinction the book was written to establish has largely been won. Separating predictive AI from generative AI is now standard practice in policy discussion, and "AI snake oil" has entered ordinary use. The predictive case has held up best of all, because nothing has come along to rescue it: the reasons social prediction fails were never about model capacity, and three further years of much better models have produced no visible improvement in forecasting who will keep a job, commit a crime or finish school. Epic's withdrawal of the plug-and-play sepsis model stood, and the pattern the book described, of a proprietary clinical model adopted at scale and evaluated years later by whoever happens to have access, has repeated often enough that external validation before deployment is now a named demand in health policy.

The generative-AI chapters have aged the way the authors said such chapters do. The question they left open, whether the next rung existed, got a partial answer within months: models trained to reason step by step arrived from late 2024, and the agents they described as an exciting concept stalled by poor recovery from error became products through 2025 that are useful and still fail in exactly the way they named, by compounding small mistakes over long tasks. Neither development was a threshold, which is the ladder's whole point. Their treatment of benchmarks has been vindicated from an unexpected direction: contamination of test sets by training data, leakage at industrial scale, became a mainstream concern rather than a methodological footnote.

The regulatory picture moved the way the book's own logic predicted. The executive order that the chapter treats as the organising American document was rescinded in January 2025, taking its hundred and fifty delegated tasks with it, while the European AI Act phased its obligations in on its own schedule. Content moderation went further in their direction than they predicted: Meta ended third-party fact-checking in the United States in January 2025, and trust-and-safety teams across the industry were cut rather than automated — the chapter's thesis arriving as news, since the decisions that matter were always policy decisions.

What remains contested is the strongest claim in the book. Saying that predictive AI will likely never work is a claim about the future, and critics have pointed out that the argument establishes something narrower: that individual social outcomes cannot be forecast well enough to justify the decisions made from them, while modest gains remain useful in the aggregate settings the authors themselves concede. The diagnosis they offered as almost an aside, that the appetite for AI that does not work comes from institutions too broken to do the thing properly, has proved the most portable part of the book, because it applies without modification to the generative tools the underfunded schools, newsrooms and clinics of 2026 have bought instead.