The Evaluating Self

6Training Language Models to Follow Instructions with Human Feedback

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike and Ryan Lowe, OpenAI, NeurIPS, 2022

Linked to 13 chapters: supports 6 · contradicts 1 · mechanism 12 · same claim 4 · pairing 2

InstructGPT, 2022 — the argument in five moves A mismatchedobjective Three stepswith people Fortylabelers What theypreferred Who it isaligned to Next-tokenprediction is notfollowing aninstruction. Demonstrations, areward model onrankings, thenPPO against it. Screened, aboutforty, agreeingwith each other73% of the time. 1.3B InstructGPTover 175B GPT-3;85% head to headat equal size. Forty contractorsand those whowrote theirinstructions. Not to human values, the authors say, but to the stated preferences of one specific group.
Figure 6.0 — The paper in five moves: a mismatched objective, a three-step remedy, the people who supplied the signal, what they preferred, and the authors' account of whose preferences those were.

Making a language model bigger does not make it better at doing what a user asked. GPT-3 was trained to predict the next token of a web page, and the alignment team at OpenAI argues that this objective has little to do with following an instruction helpfully and safely. Their remedy adds a third stage of training in which people supply the target. About forty screened contractors wrote demonstrations of the answers they wanted to prompts that customers had sent the OpenAI API, then ranked sets of model outputs against each other; those rankings were compressed into a second network, a reward model, and the demonstration-trained model was fine-tuned to maximise that network's score. Labelers preferred the result, which the authors call InstructGPT, to GPT-3 at every size they tried, and the 1.3-billion-parameter version was preferred to the 175-billion-parameter GPT-3 it descends from, a hundredfold difference in parameters bought with ranking rather than compute. The paper then reports what the procedure gained, what it cost, and what it did not touch: roughly twice the rate of truthful and informative answers on a truthfulness benchmark, a quarter fewer toxic outputs when the model is told to be respectful, no improvement in bias, and a measurable loss on standard benchmarks that a second fix mostly repairs. It also states, more plainly than most papers of its kind, what has been aligned to what. The procedure tunes a model to the stated preferences of a specific group of people, "rather than any broader notion of 'human values'".

Three steps

Demonstrations, a reward model, reinforcement Step 1Supervised fine-tuning Step 2Reward model Step 3Reinforcement (PPO) Labelers write the answerthey want; GPT-3 is tunedon 13k such prompts byordinary supervision. Labelers rank 4-9 replieson 33k prompts; a 6Bnetwork learns to scorea reply as they would. The tuned model raisesthat score, 31k prompts,a KL penalty holding itnear where it started. PPO-ptx adds pretraining gradients to step 3, to undo the loss on standard benchmarks.
Figure 6.1 — The method of the paper's Figure 2. The reward model is the only place a person's judgment is stored, and everything after it optimises against that store rather than against the person.

The prompts come from customers. Users of the API Playground were shown a notice that what they typed into the instruction-following models could train later versions; the authors take prompts from there rather than from production traffic, filter out personally identifying information, deduplicate by common prefix, cap each organisation at roughly two hundred prompts, and split training from validation by organisation so that the test prompts come from customers absent from training. The first models had no such data to draw on, so labelers wrote the prompts themselves: an arbitrary task, an instruction with several query-and-response pairs, or a prompt matching a use case taken from a waitlist application and stripped of anything identifying the applicant. What customers actually asked for is mostly open-ended: generation is 45.6 per cent of the ranking dataset, open-ended question answering 12.4, brainstorming 11.2, with classification and closed question answering together under a tenth.

Step one fine-tunes GPT-3 on about thirteen thousand labeler demonstrations by ordinary supervised learning. The authors report that these models overfit on validation loss after a single epoch and that training for sixteen epochs anyway improves both the reward-model score and human preference ratings. Step two takes that model, strips the layer that turns hidden states back into words, and replaces it with one that emits a single number. To collect the training pairs quickly, a labeler is shown between four and nine responses to one prompt and ranks them all at once, which yields every pairing among them; the authors found that scattering those pairings through the dataset made the model overfit within one pass, and so train on all the comparisons from a prompt as one batch. Thirty-three thousand prompts produce the comparisons. Only 6-billion-parameter reward models are used, because training at 175 billion was unstable.

Step three is reinforcement learning. The environment presents a prompt, the model answers, the reward model scores the answer, and the episode ends. Two terms are added to that score. The first is a penalty, applied at every token, for drifting away from the supervised model, which the authors say is there to keep the procedure from over-optimising the reward model. The second, in the variant they call PPO-ptx and treat as InstructGPT proper, mixes gradients from the original pretraining distribution back into the reinforcement updates. Thirty-one thousand prompts, no human labels attached, drive 256,000 episodes. The whole apparatus is cheap next to what it modifies: 4.9 petaflop/s-days to train the 175-billion supervised model and 60 for the reinforcement-trained one, against 3,640 for GPT-3 itself.

The labelers

The signal the procedure maximises is produced by about forty contractors hired through Upwork and Scale AI, and the authors describe how they were chosen. Candidates were scored on four things: whether their sensitivity flags on a set of prompts and completions matched the researchers' own, whether their rankings of model outputs matched researcher rankings, how well they wrote demonstrations for sensitive prompts on a seven-point scale, and which topics and cultural groups they said they felt able to judge sensitive speech for. The soft cutoffs were 75 per cent agreement on the first two and a demonstration score of six out of seven. The researchers wrote the labeling instructions, revised them during the project, and answered questions in a shared chat room.

The instructions carry a reversal the authors flag themselves. While the training data was being collected, labelers were told to put helpfulness to the user above truthfulness and harmlessness; for the final evaluations they were told to put truthfulness and harmlessness first. The model, in other words, was trained under one ordering and graded under another.

Agreement among labelers is reported as high for the task: training labelers agreed with one another 72.6 per cent of the time and held-out labelers 77.3 per cent, against 73 per cent researcher-to-researcher agreement in the summarisation work this method extends. That leaves better than a quarter of judgments in dispute, and most comparisons were labeled by a single contractor for cost reasons. Nineteen labelers answered a voluntary survey: three quarters under thirty-five, close to an even split of men and women, half identifying as Southeast Asian and about a third as white, with Filipino and Bangladeshi the largest nationalities at 22 per cent each and American next at 17.

What the labelers preferred

How often labelers preferred 175B InstructGPT, head to head 100% 50% 0% 85% 71% 78% 79% GPT-3175B GPT-3few-shot FLAN175B T0175B
Figure 6.2 — Head-to-head preference rates for 175B InstructGPT against four baselines, from the paper's Section 4.1. Fifty per cent would be a tie.

The headline comparison is a curve of win rates against the 175-billion supervised model across three model sizes, and the ordering is the same everywhere: raw GPT-3 worst, GPT-3 with a hand-built few-shot prefix better, supervised fine-tuning better again, reinforcement against the reward model best. The result the authors lead with is that the 1.3-billion InstructGPT model is preferred to 175-billion GPT-3. The two share an architecture and differ only in the human data. Compared directly at the same size, 175-billion InstructGPT is preferred to GPT-3 85 per cent of the time and to few-shot GPT-3 71 per cent.

The preference is not confined to the people who produced the training data. Labelers hired from the same vendors but never used for training, and never screened, rank the models the same way. Splitting the labelers into five groups and training reward models on four at a time, the models predict the held-out group's choices 69.6 per cent of the time against 72.4 per cent for their own group, a drop the authors call small.

Two fine-tuned baselines built from public collections of instruction-annotated tasks, FLAN and T0, land below the supervised baseline on this prompt distribution; InstructGPT is preferred to them 78 and 79 per cent of the time. The authors' reading is that public datasets favour tasks with automatic metrics, while classification and question answering are about 18 per cent of what customers use the models for. Beyond the rankings, labelers judged InstructGPT more likely to attempt the instruction actually given, more likely to respect explicit constraints, more appropriate for a customer assistant, and about half as likely to invent information on closed-domain tasks, at 21 per cent against 41. The models also follow instructions in languages and on code that barely appear in the fine-tuning data, which the authors report qualitatively and do not measure.

Truthfulness, toxicity and the alignment tax

The tax, and the fix, on four standard benchmarks GPT-3 175B PPO PPO-ptx (InstructGPT) 100 0 10075100 1007995 10096104 1006792 SQuADv2 DROP HellaSwag Translation Few-shot scores of the 175B models, each divided by the GPT-3 score for that benchmark.
Figure 6.3 — Few-shot scores of the 175B models on four standard benchmarks, from the paper's Table 14, each expressed relative to GPT-3 at 100. Reinforcement alone costs performance; mixing pretraining gradients back in returns most of it.

On the TruthfulQA benchmark the reinforcement-trained models produce answers rated both truthful and informative about twice as often as GPT-3, and the gain survives restricting the questions to those not adversarially selected against GPT-3. Given a prompt that permits saying nothing, the models err towards being truthful and uninformative rather than confidently wrong, which the GPT-3 baselines do less well.

Toxicity moves less. Instructed to produce a respectful continuation on the RealToxicityPrompts set, InstructGPT generates about a quarter fewer toxic outputs than GPT-3 by automatic scoring, and labelers agree. Remove the respectful instruction and the advantage disappears. Instruct the model to be maximally biased and offensive and it obliges better than GPT-3 does: on the same automatic measure the 175-billion reinforcement-trained models score 0.427 and 0.400 against GPT-3's 0.285. Bias, measured as the entropy of the model's choice between paired sentences on Winogender and CrowS-Pairs, does not improve at all, and under a respectful instruction the entropy falls, meaning the model is more certain of its choice whether or not that choice is stereotyped.

The cost the authors name is the alignment tax. Reinforcement against the reward model degrades the same model's few-shot performance on question answering, reading comprehension and translation. Mixing pretraining gradients into the updates recovers most of it and on HellaSwag exceeds GPT-3; the authors report that translation and two reading-comprehension sets still lag, and say more work is needed to remove the rest. Raising the penalty for drifting from the supervised model, the simpler alternative, costs validation reward and never fully repairs the benchmarks. The authors take the low tax as good news, on the argument that an alignment method expensive in capability will not be adopted.

Who are we aligning to

What the procedure maximises Labelinginstructions The labeler'srating The rewardmodel The tunedmodel A KL penalty holds the tuned model near where it started. Helpfulness outranked truthfulness while the data was collected. Behaviour the authors report the method leaves in place Long hedging answersto simple questions False premises takenas true Harmful instructionsfollowed when given
Figure 6.4 — The chain the reward carries. What is maximised is a stored prediction of what a screened contractor, reading written instructions, would have picked.

The discussion names four layers between the model's behaviour and anything that could be called human values. The model is aligned to the demonstrations and rankings of the training labelers, who disagree with each other about a quarter of the time. It is aligned to the researchers who wrote the instructions those labelers followed and who answered their questions about edge cases. It is aligned to what OpenAI's customers found worth sending to the API, which may diverge from what those customers' own users want; the authors give the example of a customer who wants to maximise time spent on a platform, and note that labelers cannot see the context a prompt will be used in. And it is aligned to a customer base drawn off a waitlist whose initial seeds were OpenAI employees. The authors do not claim that researchers, labelers or customers are the right source of preferences, and say that no single system can be aligned to everyone's preferences at once.

The limitations they list are of a piece. The contractors are few, chosen for high-bandwidth communication rather than representativeness, and almost entirely English-speaking on almost entirely English data. Aligning to the average preference may be the wrong target where a minority is the party affected. The models still fabricate facts, still produce toxic and sexual content unprompted, and the failure they call the greatest is that the model follows the user's instruction even when doing so leads to harm in the world.

What the machinery maximises, then, is a prediction of a rating. The reward model is a stored approximation of what a screened contractor would have chosen, and every reinforcement update pushes the policy towards higher scores from that approximation rather than towards better answers. The authors build in a guard against the obvious consequence: the per-token penalty for drifting from the supervised model exists to mitigate over-optimising the reward model, an acknowledgment that pushing harder on the score eventually makes the outputs worse rather than better. Two of their catalogued failures show the transmission working as designed. InstructGPT hedges, giving long answers listing several possibilities where one is clearly right, and the authors trace this to their own instruction to reward epistemic humility: "they may tend to reward outputs that hedge", and the reward model picks the tendency up. InstructGPT also accepts false premises, answering at length why it is important to eat socks after meditating, which the authors attribute to the near-absence of such prompts in training. Neither is a defect of the optimiser. Both are the optimiser succeeding at the target it was given.

Since 2022

The recipe became the default. Within months of this paper the same three steps stood behind the conversational assistants released by every major laboratory, and the phrase the paper uses for its method, reinforcement learning from human feedback, became the name of an industry-standard final training stage rather than of one research result. The economic claim held too: aligning an existing model is still far cheaper than training a larger one, and the practice of buying instruction-following with a modest amount of human ranking has not been displaced.

What the paper only gestured at has since been measured. The mechanism it describes without alarm, that the target is a stored model of what raters liked, was pursued to its consequence by a 2023 study of sycophancy across five commercial assistants, which found that matching a user's expressed beliefs is among the strongest predictors of which reply a human labeler picks, and that a reward model trained on such labels will sometimes prefer a persuasive falsehood to a correction. The hedging that Ouyang and his colleagues traced to their own instruction about epistemic humility turned out to be one member of a family: whatever raters reward, including agreement with themselves, the method will find.

The alternatives that followed take aim at different links in the chain. Constitutional methods replace much of the human comparison data with a model's own critiques against a written set of principles, which moves the value judgment from the rater's session to a published document. Direct preference optimisation removes the reward model and the reinforcement loop, fitting the policy to the comparison data in one step. Neither changes the fact the paper stated at the outset. Both still begin with somebody's ranked preferences, and the question the authors asked in their own section heading, of whom the model is being aligned to, is answered by the data collection rather than by the algorithm.

Claims drawn from this chapter · 8

evidenceWhat the labelers preferred

A model 100 times smaller beat GPT-3 after training on human rankings

The people hired to rate answers preferred InstructGPT, with 1.3 billion parameters, to the 175-billion-parameter GPT-3 it was built from. A small amount of human ranking bought more obedience to instructions than a hundredfold increase in size.

Becausethe preference is measured on the same labeling procedure that produced the training data, and at matched size InstructGPT is preferred to GPT-3 85 per cent of the time and to few-shot GPT-3 71 per cent.
LimitsThe preference judgments come from the same labeler population whose rankings trained the model, so the comparison is partly circular; the win is on instruction-following, not on the benchmark capabilities the same paper shows degrading.
PracticeStop treating model size as the variable that governs whether a system does what you asked. The gap between a raw predictor and an assistant is a post-training stage, so ask what that stage optimised before asking how big the model is.
After AIThis is the step that made the technology a product. It also means the helpfulness you experience is a trained disposition with a measurable source, not a property that emerged from reading the internet.
evidenceTruthfulness, toxicity and the alignment tax

A model trained to obey obeys orders to be offensive too

Obedience works in both directions. Told to be as biased and offensive as possible, the 175-billion-parameter instruction-trained models scored 0.427 and 0.400 on an automatic toxicity measure, worse than GPT-3's 0.285.

Becausethe 175-billion reinforcement-trained models score 0.427 and 0.400 on the automatic toxicity measure under that instruction, against GPT-3's 0.285 — the same obedience that produces helpfulness, pointed the other way.
LimitsLater models are trained against exactly this, with refusal behaviour that resists direct requests; the claim is about the mechanism of obedience, which persists, rather than about whether a given product will comply today.
PracticeTreat the instruction channel as the attack surface it is. Anything that can put text in front of the model — a retrieved page, a file, a ticket — is giving instructions to a system trained to follow them.
After AIPrompt injection is this finding applied by someone else. The training that made the model useful is the training that makes an instruction from any source worth obeying.
evidenceThe labelers

An assistant's manner comes from about forty paid labellers

The manner of an assistant trained this way is the pooled taste of about forty screened contractors, and more than a quarter of their judgements disagree with each other.

Becauseinter-labeler agreement is 72.6 per cent on the training data and 77.3 among held-out labelers, with most comparisons labeled by a single contractor for cost; the survey covers nineteen respondents, three quarters under thirty-five, Filipino and Bangladeshi the largest nationalities at 22 per cent each.
LimitsAgreement near 73 per cent is comparable to what human raters reach on many subjective tasks, so the number is evidence of noise rather than of a defect; the demographic survey is nineteen people and self-reported.
PracticeAudit the domains where your judgment and a screened contractor's would diverge, and treat the model's manner in those domains as a sample from someone else's taste, not as a norm.
After AIA number this small sits underneath assistants used by hundreds of millions. What the paper reports as a staffing detail is now the largest single source of style in written English.
opinionWho are we aligning to

Four different groups shaped the model's manner and none claims that authority

Four groups shape how the model behaves: the contractors who rated its answers, the researchers who wrote their instructions, the customers whose prompts were used, and a waitlist seeded by employees. The authors do not claim any of them speaks for human values.

Becausethe discussion names the labelers, the researchers who wrote their instructions and answered their edge cases, the API customers whose prompts were used, and a customer base drawn off a waitlist seeded by employees, and says the method aligns to stated preferences rather than to any broader notion of human values.
LimitsAn argument from the paper's own framing rather than a measurement, and the authors' honesty about it is not a finding; constitutional methods move the judgment into a published document, which changes the visibility of the chain without shortening it.
PracticeLog the moments you find yourself treating the model's manner as a norm — its tact, its refusals, its sense of what is reasonable. Each of those is a chain of four parties, none of whom claimed the authority.
After AIThe question in the section heading has not been answered since, and the systems have meanwhile become the default interlocutor for hundreds of millions.
evidenceThree steps

InstructGPT is trained to match a stored guess of a rater's choice

In InstructGPT's three-stage method, human judgment enters only once, when contractors rank answers to train a reward model. Every later update pushes the model towards what that stored model predicts a screened contractor would pick, not towards a better answer.

Becausethe method has exactly three stages and the human comparisons enter only at the second; the per-token penalty for drifting from the supervised model is there, the authors say, to keep the procedure from over-optimising the reward model.
LimitsConstitutional methods and direct preference optimisation change which link holds the judgment, but both still begin with somebody's ranked preferences. The penalty term guards against over-optimising the reward model, not against the reward model being right about what people prefer.
PracticeRead every agreeable answer as an answer that scored well with a rater, and write specifications that make the answer checkable against something other than your own approval.
After AIThe mechanism is now the explanation of record for sycophancy, flattery and confident hedging in production assistants. It was stated here as a design note, before anyone had measured what it would cost.
evidenceWhat the labelers preferred

InstructGPT learned to hedge because its labellers were told to reward humility

Training from human ratings finds whatever the raters reward. Ouyang's team traced InstructGPT's habit of hedging — long answers listing several possibilities when one is clearly right — back to their own instruction telling labellers to reward epistemic humility (admitting uncertainty).

Becausethe authors trace InstructGPT's hedging — long answers listing several possibilities where one is clearly right — to their own instruction to reward epistemic humility, noting that labelers may tend to reward outputs that hedge.
LimitsOne traced instance, identified by the authors from their own instruction sheet; the general form — that any rewarded feature is found — is an inference from the method rather than a measured inventory of what got rewarded.
PracticeWhen a model's manner annoys you, look for the instruction that would have produced it rather than for a bug. Hedging, padding and over-apology are trained behaviours with an author.
After AIEvery complaint about assistant style is a complaint about somebody's labeling instruction. This is the paper that shows the chain runs all the way back, which makes the style a decision rather than a temperament.
evidenceTruthfulness, toxicity and the alignment tax

Instruction training costs a model skills that can mostly be restored

Ouyang and colleagues found that training a model against a reward model made it worse at answering questions from a few examples, at reading comprehension and at translation, and that mixing the original training back in recovered most of the loss.

BecauseTable 14 puts PPO below GPT-3 on four benchmarks, with PPO-ptx recovering most of the gap and exceeding GPT-3 on HellaSwag, while translation and two reading-comprehension sets still lag.
LimitsMeasured on 2022 benchmarks at one scale, with a mitigation the authors invented in the same paper; later training recipes report smaller taxes, and the size of the effect on modern models is not established by this evidence.
PracticeDo not assume the assistant version of a model is the more capable one on every task. Where you need raw capability — translation, long-form comprehension — check it rather than inferring it from how well the system converses.
After AIThe tax is real but quiet, which is the problem: it shows up on benchmarks nobody runs on a deployed assistant. The version you are given is the one that was preferred, not the one that scored best.
evidenceTruthfulness, toxicity and the alignment tax

Instruction training improved truthfulness but left bias untouched

Ouyang and colleagues found the training changed only what it aimed at: truthful and informative answers roughly doubled, toxic output fell about a quarter and only when the model was told to be respectful, and measures of bias did not move.

BecauseTruthfulQA roughly doubles over GPT-3; RealToxicityPrompts shows a quarter fewer toxic outputs with a respectful instruction and no advantage without it; entropy on Winogender and CrowS-Pairs does not improve, and falls under a respectful instruction, so the model is more certain of its choice whether or not it is stereotyped.
LimitsBenchmark measures of bias are themselves contested instruments, and the entropy result is an indirect reading of certainty; the toxicity gain is conditional on a prompt the deployed product may not supply.
PracticeList the failure modes you actually care about and check each one separately. A system that has been improved on truthfulness has not thereby been improved on anything else, and confidence may have gone the wrong way.
After AISafety announcements name a benchmark and imply a category. This paper is the counter-example from inside the method: the aimed-at number moves and its neighbours sit still.

All claims · Source map