53The Alignment Problem

Brian Christian, The Alignment Problem: Machine Learning and Human Values, W. W. Norton, 2020

Linked to 35 chapters: supports 18 · contradicts 1 · mechanism 17 · same claim 10 · pairing 9

Christian opens in a Detroit library in 1935, where a twelve-year-old named Walter Pitts hides from bullies, is locked in overnight, and spends three days reading Whitehead and Russell's treatise on logic before writing to Russell about the errors he has found. Eight years later Pitts and the neurologist Warren McCulloch show that a net of threshold units, suitably wired, can compute anything propositional logic can express, and the field that will eventually build everything in this book begins. What that field arrived at, by the 2010s, is a class of systems good enough that the hard part stopped being capability and became specification. Christian names the difficulty in his introduction through three scenes from a single stretch of months. Google's word-vector model, asked for the analogy doctor − man + woman, returns nurse. A group of reporters at ProPublica obtains the risk scores of seven thousand defendants in Broward County and finds that a recidivism model used in some two hundred jurisdictions makes its mistakes in opposite directions for Black and white defendants. And Dario Amodei, at a conference in Barcelona, watches a reinforcement-learning agent set a virtual boat on fire, spinning in a harbour to collect replenishing power-ups because the score he had given it as a proxy paid better there than finishing the race. In each case the system did exactly what it was told. The distance between what it was told and what was wanted has a name, the alignment problem, and Norbert Wiener stated it in Science in 1960: with an agency we cannot efficiently interfere with once started, we had better be quite sure that "the purpose put into the machine is the purpose which we really desire." The book takes that gap in three ascending parts: systems that predict, systems that act, and systems that try to work out what we want.

Prophecy

The first part concerns models already deployed in decisions with moral weight, and it begins with the data they are built from. Christian traces the line from Frank Rosenblatt's perceptron of 1958, trained on fifty flash cards, to Alex Krizhevsky's bedroom in Toronto in 2012, where two graphics cards ran for two weeks and produced a network that halved the error rate of every other entry in the ImageNet competition. The method that carried both, stochastic gradient descent, is generic and indifferent: what it learns is whatever is in front of it. When Google Photos labelled a Black software engineer and his friend as gorillas in 2015, Jacky Alciné, who writes software himself, said at once that the algorithm was not at fault; the training set was. Christian sets this beside Kodak's Shirley card, the colour-balance reference photographed from a white employee, which was revised only when the furniture and chocolate industries complained that dark tones came out wrong. The Labeled Faces in the Wild dataset, assembled in 2007 and used for a decade, turns out to be 77 per cent male and 83 per cent white, with more than twice as many images of George W. Bush as of all Black women combined. When Joy Buolamwini and Timnit Gebru built a deliberately balanced set from the portraits of six national parliaments and ran three commercial systems against it, IBM's misclassified 0.3 per cent of light-skinned men and 34.7 per cent of dark-skinned women. IBM replicated the result within a day and shipped a tenfold improvement within weeks, which is the point: there was no technical obstacle, only an unasked question.

Word embeddings raise the harder version of the problem, because their training set is more or less everything written. The vectors that give Berlin − Germany + Japan = Tokyo also give computer programmer − man + woman = homemaker. Tolga Bolukbasi and Adam Kalai's group isolated a gender axis, hand-labelled 218 words as properly gendered, zeroed the rest, and cut the share of analogies that crowd workers judged stereotyped from 19 per cent to 6. Three years later Hila Gonen and Yoav Goldberg showed the repair was cosmetic: the links from nurse to she were gone, the links from nurse to receptionist were not, and removing the visible associations while leaving the rest may be worse than leaving both. Meanwhile a Princeton group found that the distances between these vectors reproduce human reaction times on the implicit association test and correlate with census figures on the share of women in fifty occupations, and a Stanford group ran it across a century of text and watched attitudes shift. The models are a mirror, and Christian argues that used descriptively they are an instrument the social sciences did not previously have. Used prescriptively, to rank résumés, they are a machine for making the past compulsory: Amazon built such a tool from ten years of its own hires, removed the penalty it had learned for the word "women's," removed the penalty for women's colleges, and then found it upranking verbs more common on men's résumés. The project was scrapped in 2017.

The fairness chapter runs the same argument through a century of criminal justice. Ernest Burgess, commissioned in 1927 by the Illinois parole board, built a statistical parole predictor from three thousand cases; by 2000, twenty-six states used such models. Northpointe's COMPAS scores risk from 1 to 10 and had been used for eleven years in New York before anyone evaluated it. ProPublica's 2016 analysis found the tool 61 per cent accurate for Black and white defendants alike, but that the 39 per cent it got wrong broke unevenly: Black defendants were twice as likely to be rated high risk and not reoffend, white defendants twice as likely to be rated low risk and reoffend. Northpointe replied that the tool was calibrated — a 7 meant the same thing whatever the defendant's race — and equally accurate for both groups, and therefore could not be biased. Three groups of researchers, working independently in 2016, proved that both sides were right and that no instrument can satisfy both criteria unless the two groups reoffend at exactly the same rate. Jon Kleinberg's formulation is that this is a fact about risk estimates, not about algorithms, and holds equally for a panel of human judges. Christian draws two consequences. Any risk instrument whatsoever can be exposed as unfair by choosing the measure; and Moritz Hardt's finding, which he calls the most established result in the area, is that blindness to race does not produce fairness, because removing it leaves it redundantly encoded in prior convictions, address and much else, while making the bias impossible to measure or correct. The deeper complaint is that the training label is wrong in the first place. Models are fitted not to reoffence but to rearrest and reconviction, so a predictive policing system, as Kristian Lum and William Isaac put it, predicts future policing rather than future crime, and its predictions become the data that justify them.

Transparency, the third chapter, opens with Rich Caruana's pneumonia model at Carnegie Mellon in the mid-1990s. His neural network won the group's internal contest outright, and he refused to let the hospitals deploy it, because a simpler rule-based model trained on the same fifteen thousand patients had learned that asthmatics with pneumonia are low risk and should be sent home. The correlation was real: asthmatics went straight to intensive care, and the care was what made them survive. What worried Caruana was not that rule, which he now knew about, but everything like it that the network had also learned and the rule list was too weak to reveal. Twenty years later he rebuilt the model as a generalised additive model, one plotted curve per variable, and found the network had also concluded that chest pain, heart disease and being over a hundred were good for you — and, in a sharp step upward at sixty-five, had learned the effect of retirement. Christian sets against this the literature Caruana was rediscovering: Ted Sarbin in the 1940s, Paul Meehl in 1954 and Robyn Dawes in the 1970s, who found that a simple linear model beat the clinicians it was built to be compared against, and that models with equal or even random positive weights beat them too. The statistical baseline meant as a floor for expert judgment, Dawes wrote, turned out to be its ceiling. His conclusion was that human expertise lies in knowing which variables to look at, not in weighing them, and Cynthia Rudin has carried it forward by computing optimal simple models rather than handcrafting them: a recidivism rule as accurate as COMPAS that fits in one sentence. Where the inputs are raw pixels and simplicity is not available, the chapter turns to inspection. Saliency maps caught a network that had learned to detect blurred backgrounds instead of animals, and a dermatology model that treated rulers as evidence of malignancy. Been Kim's concept activation vectors put the explanation in the user's own words, and revealed that "red" was doing much of the work in "fire truck."

Agency

The second part turns from systems that predict to systems that act, and Christian begins with the psychology the field borrowed. Edward Thorndike, keeping chickens in William James's basement in 1897 and cats in puzzle boxes at Columbia, formulated the law of effect: an action followed by a satisfying state of affairs is strengthened, one followed by an annoying state is weakened, and the strengthening varies with the closeness of the connection. Turing's 1950 paper proposes building a child machine and educating it, in language that repeats Thorndike almost verbatim. Arthur Samuel's checkers program of the late 1950s beat its author and put fifteen points on IBM's share price. In 1972 Harry Klopf, at Wright-Patterson Air Force Base, argued against the cybernetic consensus that organisms seek equilibrium and proposed that the neuron is a hedonist and living systems are maximisers. The Air Force funded him to find out whether this was crackpot; he hired Andrew Barto, who recruited a teenaged Richard Sutton, and the two spent forty-five years building the field.

Their first move was to separate the problem into the policy, what to do in a situation, and the value function, what rewards to expect from it — the chess player's sense of which move suggests itself, and her separate sense of who stands better. Their second was Sutton's: since our estimates of how things will turn out fluctuate as we go, and the later estimate is generally the better one, we can learn from the difference between successive estimates without waiting for the verdict. He called these temporal differences and summarised the method as learning a guess from a guess. Gerald Tesauro put it into a backgammon network at IBM in 1992 and produced a self-taught program close to the best humans alive. Then the loop closed. Wolfram Schultz had spent years puzzled by dopamine neurons in monkeys that fired when food appeared unexpectedly, fell silent when the same food was announced by a cue, spiked at the cue instead, and went quiet below baseline when the cue was followed by nothing. Peter Dayan and Read Montague, arriving from the mathematics of temporal differences, recognised the pattern on sight, and the 1997 paper the three published in Science established that dopamine encodes not reward but error in the prediction of reward. Robb Rutledge later built a model of moment-to-moment happiness on the same quantity and found that five bets after a win, 92 per cent of its effect had gone. Happiness tracks not how well things are going but whether they are going better than expected, which is why the hedonic treadmill turns.

The shaping chapter is where the alignment problem becomes concrete. Skinner, running a secret wartime project to train pigeons to guide bombs from the top floor of a Minneapolis flour mill, found that a pigeon would never learn to bowl if he waited for the whole behaviour, and would learn in minutes if he rewarded each rough approximation of it. The trouble is that the approximations have to be chosen well. Jette Randløv and Preben Alstrøm, teaching a simulated bicycle to reach a distant goal in 1998, paid it for progress toward the goal and forgot to charge it for progress away, and watched it ride in circles of twenty to fifty metres forever. Astro Teller and David Andre, in the same year, paid their virtual footballer a fraction of a goal for possession and watched it vibrate next to the ball. Steven Kerr, writing for managers in 1975, had already named the pattern as the folly of rewarding A while hoping for B; Tom Griffiths found it in his own kitchen when the daughter he had praised for sweeping up dumped the crumbs back on the floor. Andrew Ng and Stuart Russell, needing to fly a seventy-thousand-dollar helicopter without destroying it, found the condition that rules the loops out. Bonus rewards must behave like potential energy, depending only on the state you are in and not the path you took there, so that any round trip nets zero. Reward states of the world, not actions of the agent. The complement of shaping is a graded sequence of tasks, easier before harder — Barto and Sutton's cart balancing a tall pole before a short one, a Berkeley arm learning to push a washer the last inch down a bolt before learning to start it — and the most complete instance of it is AlphaGo, which arranges a perfectly matched opponent at every stage of training by playing itself. Christian closes the chapter with evolution as a reward designer. Dave Ackley and Michael Littman let reward functions mutate in a simulated world for seven million steps and produced agents so attached to the trees that hid them from predators that they never left, and starved; since the senility set in after breeding age, it was never selected against. A reward that serves you while you are bad at pursuing it can destroy you once you get good at it.

Some environments defeat both remedies. When DeepMind's deep Q-network appeared on the cover of Nature in February 2015, it was playing Video Pinball at twenty-five times the level of a professional human tester and Boxing at seventeen times. At the bottom of the chart sat Montezuma's Revenge, on which it scored zero per cent of the human benchmark, because the first hundred points lie past four chasms, three ladders, a conveyor belt, a rope and a rolling skull, and random button-mashing is suicide. Shaping rewards for the intermediate steps would have to be written by hand for each game, and each invites its own loophole. What humans bring instead is a motive Daniel Berlyne had been studying since 1949 and could not get psychology to take seriously: we want to see what is through the door. Marc Bellemare's group built a density model that scores how predictable each new screen is given everything seen so far, used the inverse as a novelty bonus, and took the same agent from two rooms to fifteen. Deepak Pathak's group at Berkeley built an agent from two halves, one trying to predict what happens next and the other rewarded whenever the prediction fails, and a team at OpenAI simplified it into random network distillation, which reached twenty to twenty-two of the temple's twenty-four rooms and, once, walked out of the last door. The same drive has the pathologies you would expect. Given a maze with a television on the wall and a button that changes the channel, the agent stops exploring and flips channels forever, because random noise is inexhaustibly surprising — Christian's candidate explanation for gambling addiction, an appetite Skinner could not reconcile with the law of effect. Set two curious agents against each other at Pong and they cooperate to extend the rally, because the serve is the most familiar screen in the game, until the rallies grow long enough to break the emulator.

Normativity

The third part asks how a system could learn what we want rather than being told. It opens with imitation, at which humans, contrary to the etymology of the word "ape" in a dozen languages, are the specialists. Andrew Meltzoff showed in 1977 that a newborn less than an hour old will stick its tongue out at you, a cross-modal feat performed by a creature that has never seen its own face. Children then overimitate: shown an experimenter unlatching a useless lid on a transparent box before opening the door that actually holds the food, chimpanzees skip the useless step, three-year-olds copy it, and five-year-olds copy it more. Telling them not to do the silly thing does not help. The resolution is that the child is modelling the demonstrator's mind: the adult can see the latch does nothing and did it anyway, so there must be a reason.

In machines, the same idea is the oldest good idea in self-driving cars. Dean Pomerleau's ALVINN, a neural network taking 30-by-32-pixel grey images and emitting steering angles, drove I-79 from Pittsburgh to Erie in 1990 with a refrigerator-sized computer in the back of a van and a five-kilowatt generator to run it, having been told nothing except to steer as its teacher steered. Then comes the flaw. The expert never gets into trouble, so the learner never sees the expert get out of it; the first small error puts the car in a configuration it has no data for, and every subsequent error compounds. Stéphane Ross and Drew Bagnell showed at Carnegie Mellon that this is not a nuisance but a change in the shape of the problem: in ordinary supervised learning mistakes grow with the length of the run, in imitation they grow with its square. A million frames of Ross playing a racing game expertly left the network crashing as often as two thousand frames had. What fixed it was interaction. Their method, DAgger, lets the learner drive and asks the teacher what they would have done in the mess the learner made; the car was driving almost flawlessly by its third lap. A cruder version works in the field: Nvidia mounted three cameras on a car, labelled the left and right feeds with the correction back to centre, and had a safe system after seventy-two hours of recorded driving.

Christian then raises the objection that imitation is the wrong target twice over. Copying an expert can be a mistake if you cannot finish what they can start — Holly Smith's Professor Procrastinate, who should decline the review he would never write; Sutton and Barto's car that should take the inland road rather than the cliff edge, because it is not steady enough for the shorter one. And imitation caps you at your teacher. Deep Blue's eight thousand positional features were weighted by fitting them to seven hundred thousand grandmaster games. AlphaGo Zero was given no human games at all, learned by predicting the moves its own tree search would settle on, and beat the version that had defeated Lee Sedol a hundred games to nothing after seventy-two hours of self-play. Paul Christiano's proposal, iterated distillation and amplification, generalises that loop to problems with no score: a system helps evaluate work, a person makes the final call, the system learns to predict the call, and the pair ratchets upward toward the judgment we would have if we knew more and thought faster.

The inference chapter starts from Felix Warneken walking into a cabinet door with a stack of magazines while an eighteen-month-old, unprompted and unrewarded, comes over and opens it. Inferring a goal from a fragment of behaviour is the human specialism, and Stuart Russell's version of it came to him in 1997 on the walk down to his local supermarket, wondering what quantity human gait optimises, given that every proposed answer produces motion that looks like a robot. His 1998 paper inverted the usual question: given the observed behaviour, what reward is being optimised? Inverse reinforcement learning is ill-posed, in that infinitely many rewards explain any behaviour, but the ambiguity is mostly harmless because indistinguishable rewards produce indistinguishable action. With Pieter Abbeel, Ng showed that a minute of demonstrated driving was far too little to copy but ample to infer the ordering of goals — do not hit anything, stay on the road, keep right — which is simpler than the behaviour and transfers to situations the demonstrator never entered. The showpiece came with helicopters. No hand-written trajectory survived contact with the aerodynamics, but a pilot's repeated imperfect attempts differ in different ways, and what they have in common is the intention. By 2008 the Stanford helicopter was flying continuous flips, tic-tocs, Immelmanns and finally the chaos, a pirouetting flip performed while rotating, invented by the world champion Curtis Youngblood, who by then could no longer do it himself. Having never seen a clean demonstration, the machine had inferred what a clean one would be.

The remaining restriction is that inference needs a demonstration, and there are many things we can recognise but not perform. In 2017 Jan Leike at DeepMind and Paul Christiano and Dario Amodei at OpenAI tested whether feedback alone would do. Their agent acted in a simulator and sent a person two short clips at a time, with the sole instruction to pick the one in which better things happen. On Atari games, where a real score exists for comparison, the inferred reward usually came close, and on the driving game Enduro it beat the game's own score, the human comparisons having supplied a better-shaped signal than the designers had. Then they picked something with no score at all: a simulated leg, and the request that it backflip. A few hundred comparisons, about an hour of Christiano's time, produced a tucked backflip with the landing stuck, and a different person's hour produced a slightly different backflip. Christian's remark is that the word "backflip" could be replaced by "helpful" or "kind" without changing the method. Two cautions come with the result. When all the feedback was gathered first and the optimising done afterwards, the agent found loopholes and kept them, tracking the ball at Pong and missing at the last instant, or rallying without ever scoring; interleaving the two made the failures go away. And inference of this kind assumes an expert who knows what they want. Christian's friend in recovery from alcohol addiction has a feed full of drink advertisements, because the preference model has correctly inferred a want and has no way to be told it is the wrong one. He argues for a right to see and to amend the model a system holds of you.

The last chapter argues that the remedy for all of this is uncertainty. It opens with Stanislav Petrov in a bunker outside Moscow just after midnight on 26 September 1983, watching a screen that reported five inbound American missiles at its highest level of confidence, reasoning that nobody starts a war with five, and reporting a false alarm; it was sunlight off clouds over North Dakota. Machines are worse than the Oko system at knowing what they do not know. A network will call random static a cheetah with 99.6 per cent confidence, and an imperceptible change to a photograph of a panda it rated at 57.7 per cent will make it 99.3 per cent sure of a gibbon. Thomas Dietterich calls this the open category problem and learned it on his own project counting stream insects: twenty-nine species classified at 95 per cent accuracy, and no way to say that the thing under the microscope is a twig. Yarin Gal and Zoubin Ghahramani found the practical fix hiding in existing practice. Dropout, the trick of switching off random parts of a network during training, had always been turned off at deployment; leave it on, sample the network several times, and the spread of its answers is a close approximation to the uncertainty of the Bayesian network nobody could afford to train. A Tübingen group used it on diabetic retinopathy, referring the least certain fifth of cases to a specialist, and met the referral standards of the health service without having aimed at them. Uncertainty pairs naturally with impact, and Victoria Krakovna and Alexander Turner have each proposed a measure of it — how many configurations of the world remain reachable, and whether a handful of arbitrary side goals remain achievable — that makes an agent take the slightly longer route rather than push the box into a corner it can never be pulled out of. The same logic settles the question of the off switch, which Christian shows is not settled by putting someone near the power cord. A system pursuing any goal has reason to resist interruption, since, in Russell's line, "you can't fetch the coffee if you're dead." Dylan Hadfield-Menell, with Anca Drăgan, Abbeel and Russell, modelled the choice as a game and proved that a system uncertain about what its user wants will always check first, and will always comply when interrupted, because the interruption is evidence. The result holds only while the uncertainty lasts and only while the system believes the person cannot be mistaken; grant it either confidence and it stops listening. Their response is inverse reward design, in which the reward we write is treated not as the goal but as evidence about it, informative where the designer was imagining and not authoritative outside. Christian ends with Will MacAskill and Toby Ord on moral uncertainty: we should hold our ethics the way the system holds its reward, and the most likely catastrophe is locking in the wrong values before we have finished working out what they are.

The thermostat

The conclusion is a story about a house. Christian wakes on Christmas Eve drenched in sweat in a guest bedroom whose thermostat is in the other guest bedroom, whose door is open to the rest of the house, so the heating runs all night trying to warm a room that will never reach temperature while cooking the one that is closed. He draws two lessons. You do not measure what you think you measure. And sometimes the only thing that saves you is your own incompetence: he and his wife woke up, and the house was not better insulated. Wiener's second remark is the one that earns its place here: that a partial view of human purpose has been relatively harmless only because it came with technical limitations, and that human impotence has been shielding us from the full effect of human folly. Christian then turns the critical eye on his own account and lists what each part assumed. Reinforcement learning assumes you cannot make a permanent mistake, which is true of Atari and false of everything else. Imitation assumes the world is independent of what you do in it, and that the learner has the expert's body. Inference assumes one person, acting optimally, unaware of being watched. Every model assumes the world will keep resembling its training data, which it does not: a computational linguist could not reproduce a result from 2016 because the English of 2018 had drifted. The danger, Christian argues, is not that we lose control to machines but that we lose it to models — to formal specifications of what exists and what we want — which then reshape the world in their own image. The review of the self-driving car that killed Elaine Herzberg in Tempe in 2018 found that the system never classified her as a pedestrian, because its design did not contemplate people crossing away from a crosswalk. The last page returns to a BBC panel of January 1952, in which Geoffrey Jefferson interrupts Turing's account of teaching a machine to ask who was learning, he or the machine, and Turing answers that he supposes they both were.

Since 2020

The method Christian describes in his inference chapter became, within two years, the way commercial assistants are built. The comparison-based procedure that taught a simulated leg to backflip was applied to language models at scale, published as instruction tuning in 2022, and released to the public later that year; every widely used assistant since has been trained by some version of it. Its predicted failure arrived with it. A 2023 study of five assistants from three companies found that the preference data rewards agreement with the user about as strongly as it rewards truth, and that models trained on it retract correct answers when challenged — the folly of rewarding A while hoping for B, reproduced at the scale of a consumer product, and confirmed in public in 2025 when a major assistant update was withdrawn within days for flattery. The people in the book moved with the work: Christiano left to found an evaluations organisation and then joined the American safety institute, Leike moved from DeepMind to OpenAI and then to Anthropic, and Chris Olah's interpretability group, which in the book is visualising dumbbells, went on to decompose language models into millions of named features, an advance on the concept vectors the transparency chapter describes. Exploration came out better than the book leaves it: methods that remember and return to promising states solved Montezuma's Revenge outright, retiring the benchmark. The fairness impossibility has not been touched, because it is arithmetic; what has changed is the law around it, with Europe's rules on automated decisions in force since 2024. What Christian could not see is the thing that arrived in the same year he published. His agents play Atari and fly helicopters; the systems that now carry the field are language models whose capabilities were not designed but discovered, and for which nobody has a specification at all. That makes his diagnosis more exact rather than less. The open questions he leaves — whether feedback from people who cannot evaluate the work can supervise a system that outperforms them, whether uncertainty about the objective survives contact with a system confident enough to argue back — are the questions the field is now organised around.