7Training Compute-Optimal Large Language Models

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch and colleagues, DeepMind, NeurIPS, 2022

Linked to 13 chapters: supports 6 · contradicts 3 · mechanism 11 · same claim 4 · pairing 2

Figure 7.0 — The paper in five moves: a budget fixed in advance, three ways of finding the best way to spend it, one answer, a model built to test it, and a verdict on everything else that had been built.

A training run is bought before it is designed. A team knows how many accelerators it has and how long it may keep them, and that fixes the number of arithmetic operations available before a single token has been read. Hoffmann and his colleagues at DeepMind ask what to do with that number. The same budget will buy a large model that reads a little text or a small model that reads a great deal, and by 2022 the field had settled almost unanimously on size. GPT-3 at 175 billion parameters, Jurassic-1 at 178 billion, Gopher at 280 billion and Megatron-Turing NLG at 530 billion had all been trained on roughly 300 billion tokens. The DeepMind group trained more than 400 models, from 70 million to over 16 billion parameters, and estimated the best split three separate ways. The three methods share almost none of their machinery and agree on the answer: parameters and training tokens should grow in step, so that doubling the budget means doubling both. Measured against that rule, every large model then in existence had been given too many parameters and too little text. To settle the matter the authors spent Gopher's exact budget on a model a quarter of its size trained on nearly five times as much data, called it Chinchilla, and beat Gopher on almost every benchmark they ran.

The question

The paper states the problem as a constrained minimisation and never loses sight of it. Write the final pre-training loss as a function of two numbers: the parameter count and the number of tokens seen. The compute spent is itself a function of those same two numbers. Find the pair that minimises loss on the surface where compute equals the budget. Because a transformer's training cost is close to six times parameters times tokens — the authors check that approximation against a full operation count, embedding matrices included, and find it accurate to within about ten per cent — the constraint is a hyperbola, and every point on it costs the same money. The only question is where along it to stand.

Figure 7.1 — Two ways of spending one budget. Because cost runs as parameters times tokens, the two shaded areas are the same expenditure.

The question has to be answered in advance rather than discovered by trying, because a model at this scale is trained once. Table 1 of the paper makes the state of the art look less like a series of decisions than one habit repeated. LaMDA held 137 billion parameters and saw 168 billion tokens. GPT-3 held 175 billion and saw 300 billion. Jurassic-1, 178 billion and 300 billion. Gopher, 280 billion and 300 billion. Megatron-Turing NLG, 530 billion and 270 billion. Model size had more than tripled in two years while the data stayed where GPT-3 had left it.

Three approaches, one answer

The first approach fixes a family of model sizes, from 70 million to over 10 billion parameters, and trains each of them four times over horizons that differ by a factor of sixteen. Smooth and interpolate every loss curve, take the lower envelope of all of them, and at 1,500 logarithmically spaced budgets read off which model reached the lowest loss and how many tokens it had consumed to get there. Power laws fitted through those points give the optimal size and the optimal token count as functions of compute.

The second approach fixes nine budgets instead, from 6×10¹⁸ to 3×10²¹ operations, and varies model size within each one, up to 16 billion parameters. Plotting final loss against parameter count at constant compute produces a valley: a model too small wastes the budget going over the same ground, a model too large is stopped before it has learned much. A parabola fitted to each valley locates its floor, and power laws through the floors give the same two functions again.

Figure 7.2 — The isoFLOP curves, schematic, after the paper's Figure 3. Each curve is one budget; the minima move right and down together, and the line through them is the compute-optimal frontier.

The third approach throws the curves away and fits a function to every run from the first two. The loss of a model with a given parameter count and token count is written as three terms: the entropy of natural text, which no model can go below; a penalty for having finitely many parameters; and a penalty for taking finitely many optimisation steps. Five constants are fitted by minimising a Huber loss, which is unmoved by outliers. Minimising the fitted function under the compute constraint gives the frontier in closed form.

Approach	Parameters, exponent on compute	Tokens, exponent on compute
Minimum over training curves	0.50	0.50
IsoFLOP profiles	0.49	0.51
Parametric fit of the loss	0.46	0.54
Kaplan and colleagues, 2020	0.73	0.27

Three methods, three sets of exponents, one conclusion: "model size and the number of training tokens should be scaled equally". The parametric fit leans slightly towards data, an artefact the authors trace to the curvature at the cheap end of their range. Repeating the second approach on the C4 web crawl and on a body of GitHub code returns very similar exponents, so the result is a property of transformers rather than of DeepMind's corpus, provided no token is seen twice.

The disagreement with Kaplan

Kaplan and colleagues had asked the same question in 2020 and answered it the other way: parameters should take almost three quarters of any new compute and data barely a quarter. Their recommendation for a tenfold increase in budget was to make the model five and a half times larger and give it 1.8 times more text. The field had followed that advice, which is why five of the largest models in the world had converged on 300 billion tokens.

Figure 7.3 — What to do with ten times the compute, under the 2020 rule and under this paper's. Kaplan's multipliers are the ones the paper's introduction quotes; the 2022 pair follows from the exponents in Table 2.

Hoffmann's group gives two reasons for the divergence. The first is the learning rate schedule. Kaplan trained with one cosine schedule sized for 130 billion tokens and read intermediate points off those curves. A run stopped a tenth of the way into a schedule built for ten times its horizon has not finished decaying its learning rate, and ends above the loss it would have reached under a schedule matched to its own length. Appendix B measures the penalty and the main analysis is built to avoid it: every run here gets a cycle length set to its own token count. Reading short runs off a long schedule makes training for longer look unprofitable and small models look weak, which is precisely the bias that pushes an estimate towards size. The second reason is scale. Many of Kaplan's runs were under 100 million parameters; most of Hoffmann's are above 500 million and the largest is 16 billion, and the frontier bends in a way that only the larger runs reveal.

They settle the question with an experiment rather than an argument. At 10²¹ operations the 2020 rule recommends a model of 4.68 billion parameters and the first approach recommends 2.86 billion. The authors train a 4.74 billion and a 2.80 billion parameter transformer at the same depth-to-width ratio, and the smaller one finishes with the lower loss.

Chinchilla

Gopher had been trained with 5.76×10²³ operations. The three approaches put the best model for that budget somewhere between 40 and 70 billion parameters, and the authors take the top of the range for reasons of data and engineering efficiency. Chinchilla has 70 billion parameters and is trained on 1.4 trillion tokens: a quarter of Gopher's size, nearly five times its data, the same bill. It keeps Gopher's 80 layers but halves the model width to 8,192 and the head count to 64, trades Adam for AdamW, raises the maximum learning rate, and uses a tokenizer that skips Unicode normalisation, which helps with mathematics and chemistry. The corpus is the same MassiveText, reweighted for the longer run.

Figure 7.4 — Chinchilla and Gopher at identical training cost, from the paper's Tables 6 to 9. Chinchilla has a quarter of Gopher's parameters.

Chinchilla is ahead of Gopher on every subset of the Pile and takes WikiText-103 perplexity from 7.75 to 7.16, though the authors caution that a model trained on four times as much text may simply have seen more of the test set, and lean instead on the exam-style benchmarks. On the 57 subjects of MMLU it averages 67.6 per cent against Gopher's 60.0, ahead on 51 of them, level on two and behind on four, and above the 63.4 per cent that seventy-three forecasters had put forward in 2021 as the state of the art they expected in June 2023. On BIG-bench it averages 65.1 against 54.4, ahead on 58 of 62 tasks. On the harder half of the RACE reading examinations it scores 82.3 against 71.6. On TruthfulQA, a benchmark whose authors had argued that larger models get less truthful, it reaches 43.6 per cent with no examples in the prompt against Gopher's 29.5, which the paper reads as evidence that better modelling of the same data is enough to move it. It also outperforms Megatron-Turing NLG, a model more than seven times its size, on all but one common-sense task.

Two things do not improve in proportion. On Winogender, which tests whether a model attaches a pronoun to the right occupation, Chinchilla resolves 78.3 per cent against Gopher's 71.4 and gains ten points on the examples where the right answer contradicts the stereotype. But it gains 3.2 points on male pronouns against 8.3 on female and 9.2 on neutral, and the authors take that unevenness as bias surviving the change. Toxicity does not move: across 25,000 unprompted samples the mean classifier score is 0.087 against Gopher's 0.081. A better model of the data is not a cleaner one.

The saving compounds afterwards. A quarter of the parameters is a quarter of the memory and roughly a quarter of the arithmetic every time the model answers anything, and a deployed model is used far more often than it is trained. The energy of the run is amortised over that working life, so a smaller model on the frontier is cheaper to own as well as better.

What the analysis assumes

The authors are explicit about what they have not shown. There are only two comparable runs at full scale, Chinchilla and Gopher, and none in between; the whole prediction rests on hundreds of small models and one large confirmation. The frontier is assumed to be a power law, yet the data show concavity at the highest budgets, which means the optimal size may be smaller still than the paper says. And every run used in the analysis sees less than one epoch of text. What happens when tokens are reused is left open, though Chinchilla itself passes over the Wikipedia portion of its corpus 3.4 times inside its 1.4 trillion.

The extrapolations are where the strain shows. Table 3 puts the optimum for a 175-billion-parameter model at 3.7 trillion tokens, for a 280-billion-parameter model at 5.9 trillion, and for a trillion-parameter model at 21.2 trillion tokens and more than two thousand times Gopher's budget. Nobody in 2022 had assembled anything close to those corpora, which is why the conclusion is not really about architecture at all: "an increased focus on dataset scaling is needed". The authors add the qualification that more text helps only when the text is good, and that a corpus of trillions of web tokens carries toxic language, social bias and private information in proportion, so that collecting it responsibly and inspecting what is in it matter more as the number grows, not less.

Since 2022

The ratio became a rule of thumb within months — about twenty tokens for every parameter — and then a rule that people deliberately broke. Meta's LLaMA models, released in February 2023, were trained far past the frontier, seven billion parameters on a trillion tokens, on an argument the paper had left room for. Hoffmann's analysis optimises the cost of training and nothing else. A model served to millions pays most of its lifetime compute at inference, where the parameter count is what costs money, so a smaller model trained well beyond the compute-optimal point wastes training compute and saves far more of it afterwards. By 2024 models of seven or eight billion parameters were being trained on more than ten trillion tokens, some seventy times the allocation Table 3 recommends for that size.

The fit itself was reopened in 2024, when a group outside DeepMind reconstructed the data behind the third approach from the published figures and reported that the parameters given in the paper do not reproduce those points, and that the confidence intervals around them were implausibly narrow. Their corrected fit moves the third approach onto roughly equal exponents, in line with the first two: the conclusion the paper is remembered for survived the audit, and the one piece of it that had leaned hardest towards data did not. The single-epoch restriction was relaxed soon after, by work showing that a handful of passes over the same text is nearly as useful as fresh text.

What has held up is the method and the measurement it corrected. Fit the frontier from many cheap runs before committing the expensive one, and check the prediction by building the model it names. The field had been reading one variable and adjusting it, and the paper showed that the other variable had been held constant for two years by nothing more than imitation.

Claims drawn from this chapter · 9

evidenceChinchilla

Chinchilla became more truthful than Gopher but no less toxic

Chinchilla, a better-trained model than Gopher, scored 43.6 per cent on a truthfulness test against Gopher's 29.5. Its average toxicity did not fall, and its gender-bias scores improved much more for male pronouns than for female or neutral ones.

BecauseChinchilla scores 43.6 per cent on TruthfulQA against Gopher's 29.5, mean toxicity is 0.087 against 0.081 over 25,000 unprompted samples, and Winogender improves 3.2 points on male pronouns against 8.3 on female and 9.2 on neutral.

LimitsA two-model comparison on benchmark instruments of contested validity; the uneven Winogender gains are an improvement everywhere, just unequal, which is a weaker finding than no movement.

PracticeDo not let a capability upgrade stand in for a safety review. Each failure mode you care about needs its own measurement, because the ones that move and the ones that do not are not predictable from the headline.

After AIEvery model release is announced as an improvement, singular. Two independent papers in this unit find the improvement is plural and unevenly distributed, which is the reading to apply to the next release note.

evidenceSince 2022

Chinchilla's rule minimises training cost and ignores the cost of running

The Chinchilla scaling rule finds the cheapest way to train a model and nothing else. Once a model is served to users, most of its lifetime computing cost goes on answering, so the rule stops being the right target.

BecauseLLaMA trained seven billion parameters on a trillion tokens in February 2023, and by 2024 eight-billion-parameter models were trained on more than ten trillion tokens — some seventy times the allocation Table 3 recommends — on the argument that a deployed model pays most of its lifetime compute at inference.

LimitsTraining past the frontier has diminishing returns that are not well characterised, and the inference argument depends on a deployment volume that most models never reach.

PracticeAsk what an optimisation was for before adopting its conclusion. The cheapest model to train is not the cheapest to run, and the domains where you rely on a model are decided by the second number.

After AIThe small, heavily trained models on your laptop exist because the industry deliberately broke this paper's rule. The rule was right about training and was never about the thing you use.

evidenceThree approaches, one answer (Table 3)

Hoffmann's conclusion is about training text, not model design

The paper's conclusion is about how much text you have, not how the model is built. Its own table puts the best training set for a trillion-parameter model at 21.2 trillion tokens, far more text than anyone had collected when it was written.

BecauseTable 3 puts the optimum for 175 billion parameters at 3.7 trillion tokens and for a trillion parameters at 21.2 trillion, more than two thousand times Gopher's budget, and the authors write that an increased focus on dataset scaling is needed — with the qualification that more text helps only when the text is good.

LimitsThe single-epoch restriction that forces the conclusion was relaxed soon after, by work showing a handful of passes over the same text is nearly as useful as fresh text.

PracticeRead the data question as the live one. Where the next gain must come from text that does not exist yet, the corpus — provenance, licensing, synthetic fraction — is what moved between model versions.

After AIA recommendation to scale corpora, addressed to engineers, is also a recommendation to collect. It points straight at the documentation problem Bender raised the year before, and the two papers were not in conversation.

evidenceWhat the analysis assumes

Hoffmann's prediction rests on small models and one full-scale run

The whole prediction rests on more than 400 models below 16 billion parameters, plus a single run at full scale that confirmed it. Nothing was measured in between.

Becausethe authors list the assumptions themselves: two comparable runs at full scale, a power law assumed where the data show concavity at the highest budgets, and every run seeing less than one epoch of text.

LimitsThe gap is a feature of what was affordable rather than a flaw in the reasoning, and the one confirmation was decisive on the quantity that mattered; the authors state the assumptions rather than hiding them.

PracticeRecord what a forecast rests on at the time you make it, including the gap between where you measured and where you are predicting. An extrapolation over four orders of magnitude with one checkpoint should be logged as such.

After AIScaling predictions are still made this way, with many cheap runs and one expensive check, and they now set capital allocation. The structure of the evidence has not changed with the size of the bet.

evidenceThe question

In 2022 every big AI model was too large for its data

Hoffmann's team found that in 2022 every large language model was too big for the amount of text it was trained on. Labs had kept the data size roughly fixed since GPT-3 only because they were copying each other.

BecauseTable 1 lists LaMDA at 137B parameters on 168B tokens, GPT-3 at 175B on 300B, Jurassic-1 at 178B on 300B, Gopher at 280B on 300B and Megatron-Turing NLG at 530B on 270B — model size more than tripled in two years while the token count stayed where GPT-3 left it.

LimitsThe judgment is made against this paper's own fit, so it is oversized relative to one analysis of training cost; on an inference-cost criterion those models were undersized in data and oversized in parameters for a different reason.

PracticeLook for the variable everyone has been holding constant. A field moving fast on one dimension is usually not testing the others, and the frozen number is where the cheap gain is.

After AIAn entire generation of systems was mis-specified for two years because each lab copied the last. That is a fact about how a fast-moving technical field evaluates itself, and it is worth holding next to any current consensus about what models can do.

evidenceThree approaches, one answer

Model size and training data should grow at the same rate

When the budget for training a language model doubles, the number of parameters (the model's adjustable weights) and the number of words it trains on should both double, in equal step.

Becausethree methods sharing almost no machinery — the envelope of training curves, isoFLOP valleys and a five-constant parametric fit — over more than 400 models from 70 million to 16 billion parameters give compute exponents of 0.50/0.50, 0.49/0.51 and 0.46/0.54.

LimitsThe fit assumes a power law where the highest budgets show concavity, every run saw less than one epoch of text, and the third approach's published parameters later failed to reproduce its own points.

PracticeAsk what a model was trained on before asking how large it is. Two systems of the same parameter count can differ by an order of magnitude in the data that shaped them, and that is the variable this result says to read.

After AIThe practical form of this for anyone choosing a model is that parameter count stopped being a ranking. It settled a question about training budgets and was read by the industry as permission to serve small models.

evidenceSince 2022

One of the paper's three methods failed audit but its answer held

Of the three methods Hoffmann used, the one that leaned most on fitting the data did not survive a later re-check. The conclusion the paper is remembered for did.

Becausea 2024 reconstruction of the parametric fit from the published figures found that the paper's parameters do not reproduce its own points and that the confidence intervals were implausibly narrow, and the corrected fit moves that approach onto roughly equal exponents, in line with the other two.

LimitsThe reconstruction worked from published figures rather than the original data, and its correction moves the third approach into agreement with the other two, so the audit strengthens the paper's conclusion while discrediting one route to it.

PracticeLog which part of a result you are relying on, not the result. Here the headline survived an audit that the most convenient component failed, and only someone who had written down which was which could tell.

After AIThe group that had just corrected somebody else's measurement made a measurement error of its own in the same paper. Auditing is not a stage that ends.

evidenceChinchilla

The Chinchilla team settled a scaling argument by running one experiment

A prediction about scaling a model can be settled by experiment. The authors spent Gopher's exact budget on the 70-billion-parameter model their formula named, trained it on 1.4 trillion tokens, and beat Gopher 67.6 to 60.0 on MMLU and 65.1 to 54.4 on BIG-bench.

Becausethe confirmation is a single pre-registered prediction tested at full scale with the budget held fixed, and the MMLU result also cleared the 63.4 per cent that seventy-three forecasters had expected for June 2023.

LimitsOne confirmation at one scale, by the group whose fit made the prediction, on benchmarks they chose; the result is a decisive experiment for the ratio, not evidence that the fit extrapolates further.

PracticeName the number your analysis predicts before you run the expensive thing, and hold the budget fixed so the comparison means something. A prediction that can be settled is worth more than an argument that cannot.

After AIThis is the cleanest experiment in the literature this book draws on, and it is about an engineering allocation. Nothing about what the model understood was settled by it, which is the distinction most reporting on model releases loses.

evidenceThe disagreement with Kaplan

The earlier scaling answer was a measurement error, not a real disagreement

The field's earlier answer was wrong because of how it was measured, not because anyone saw the world differently. Kaplan read intermediate results off a training schedule built for a much longer run, which makes shorter training look worse than it is.

BecauseKaplan's runs used one cosine schedule sized for 130 billion tokens, so a run stopped a tenth of the way in ends above the loss a matched schedule would reach, which makes training longer look unprofitable and small models look weak — the exact bias that pushes an estimate towards size; Appendix B measures the penalty.

LimitsThe diagnosis is this paper's own account of another group's runs, and the artefact explains the direction of the error rather than all of its size.

PracticeBefore disputing a result, reconstruct how its intermediate points were produced. Two of the largest disagreements in this literature turned on an instrument setting rather than on a claim about models.

After AIBoth of the field's headline scaling results were corrected by someone re-reading the measurement rather than the model. That is the habit worth carrying into any claim about what a system can do.