9Are Emergent Abilities of Large Language Models a Mirage?

Rylan Schaeffer, Brando Miranda and Sanmi Koyejo, NeurIPS, 2023

Linked to 16 chapters: supports 11 · contradicts 2 · mechanism 8 · same claim 7 · pairing 2

Figure 9.0 — The claim, the mechanism proposed against it, and three tests: rescored arithmetic, a survey of one benchmark's metrics, the effect manufactured to order.

By 2022 one shape of plot had become the standard picture of what scale does to a language model: performance sits flat near chance across two or three orders of magnitude of parameters, then at some size lifts off. Wei 2022 defined the abilities behind those plots as "abilities that are not present in smaller-scale models but are present in large-scale models", with two properties — sharpness, the passage from absent to present with nothing in between, and unpredictability, that passage arriving at a scale nobody could have named in advance. Schaeffer, Miranda and Koyejo do not dispute the plots; they dispute what produced them. Holding the outputs fixed, they argue the cliff is manufactured by the choice of scoring rule: a metric demanding a whole sequence of tokens be exactly right turns a smoothly improving per-token error rate into a step, and a few hundred test items are too coarse to show the small models were ever above chance. Score the same outputs by a rule that moves in proportion to the error rate and the step flattens into a ramp. The paper then runs the demonstration backwards, choosing metrics that make image autoencoders and character classifiers appear to acquire an ability nobody had claimed for them.

Sharpness and unpredictability

Emergence reaches machine learning from physics, by way of P. W. Anderson's 1972 essay More Is Different: as a system grows more complex, properties may appear that could not be predicted from an exact account of its parts. The idea arrived in language models with a warning attached. If capabilities appear abruptly at sizes nobody can forecast, a larger model might one day acquire a dangerous one without notice.

Schaeffer, Miranda and Koyejo begin from where the reported cliffs are found: under particular metrics. Multiple Choice Grade scores one when the model puts the highest probability mass on the correct option and zero otherwise; Exact String Match scores one when the output matches the target exactly and zero otherwise. Both are step functions applied to a quantity that moves smoothly. In a hand-annotated catalogue of 137 claimed emergent abilities, more than 92 per cent appear under one of those two.

An alternative explanation

Suppose, as the literature on neural scaling laws reports, that a model family's per-token cross-entropy falls as a power law in the number of parameters; the authors need only that it falls smoothly. Exponentiating the negative of it gives the per-token probability that the model picks the correct token, which climbs smoothly towards one.

Now suppose the task demands a run of tokens all correct: the five digits of a sum, with no additions, deletions or substitutions. If the per-token errors are independent — an assumption the authors mark as false and serviceable — the chance of scoring one is that probability raised to the power of the target length, and the exponent is where the cliff comes from. A per-token probability of 0.6 scores 0.08 across five tokens; 0.9 scores 0.59. The underlying quantity improved by half and the score sevenfold. Against a logarithmic parameter axis that is the familiar flat-then-rising curve.

Figure 9.1 — Schematic. A per-token probability rising smoothly with scale gives a step under exact match across five tokens and a ramp under token edit distance.

Swap the scoring rule and the shape changes without a single output changing. Token edit distance — the additions, deletions and substitutions needed to turn the output into the target — falls in proportion to the per-token error rate rather than geometrically, and draws a straight improvement. The same exchange works on the discontinuous side: Multiple Choice Grade produces a step, the Brier score a ramp.

A second factor supplies the flat stretch on the left. A test set sets a floor on what can be measured: with a few hundred items the smallest non-zero accuracy observable is a few thousandths. If the small models' true accuracy on five-digit targets is one in ten thousand, every one of them scores zero, and zero reads as the absence of an ability rather than as a number below the instrument's resolution. A third factor sharpens the picture: model sizes are sampled sparsely at the large end, so the rise is seen in one jump.

Arithmetic in the GPT family

These factors yield three predictions, tested on the one family the authors could query: four models of 350 million, 1.3 billion, 6.7 billion and 175 billion parameters, reachable through a public interface in March 2023. The tasks are the ones the emergence papers had used: two-shot multiplication of two-digit integers and two-shot addition of four-digit integers.

The first prediction is that changing the metric while holding the outputs fixed removes the emergence. Scored by accuracy, both tasks show the published pattern whenever the target runs to four or five digits: flat near zero, then a rise at the largest model. Scored by token edit distance, the same outputs improve smoothly across the whole range.

The second prediction is that the small models are not at zero but beneath the resolution of the test set. With additional test data generated, every model in the family, the 350-million-parameter one included, scored above chance on both tasks, improving smoothly with scale even under accuracy.

Figure 9.2 — Schematic, after Fig. 4 of the paper. With a few hundred test items the three smaller models read as zero; with enough items they read as small numbers on a smooth line.

The third prediction is what makes the account more than a re-description. Whatever the metric, lengthening the target should degrade performance in the way the metric's algebra says: accuracy approximately geometrically in the target length, token edit distance approximately linearly. Both held across targets of one to five digits. The metric has exactly the effect one should predict, which is the difference between an explanation and an objection.

Metrics, not task-model pairs

The other families said to show emergence were closed, so the authors turned to the published results on BIG-Bench, a benchmark of more than two hundred tasks scored under many metrics at once. The prediction concerns where emergence lives: if an ability belongs to a model family on a task it should appear under any reasonable way of scoring that task, and if it belongs to the metric it should appear under the nonlinear and discontinuous rules alone.

Using the emergence score defined by the benchmark's own authors — the span between a family's best and worst performance across scales, signed by direction and divided by the typical step between neighbouring scales — the authors asked which of BIG-Bench's 39 preferred metrics carry any emergence at all. At most five do; most carry none. Against the hand-annotated list the count falls to four, and two of those, Multiple Choice Grade and Exact String Match, account for more than 92 per cent of the claimed abilities. Because the benchmark often scores one task under several metrics at once, the absence under the others is a comparison drawn within the same outputs.

Figure 9.3 — Counts from Fig. 5 of the paper. At most five of 39 metrics carry any emergence; two carry more than 92 per cent of the annotated claims.

Then the direct test. LaMDA's outputs are available through the benchmark, so the authors took the tasks on which that family shows emergence under Multiple Choice Grade and rescored them under the Brier score, already among the benchmark's own metrics. The emergence disappeared.

Emergence induced in vision networks

The third test runs the argument in reverse. Vision was chosen because abrupt jumps in capability have not, to the authors' knowledge, been reported there. Shallow nonlinear autoencoders trained on CIFAR-100 images show mean squared reconstruction error falling smoothly as the bottleneck widens. The authors define a new metric: the fraction of test images whose squared reconstruction error falls below a threshold. At thresholds of 0.003 and 0.004 the same networks acquire a sharp, seemingly unpredictable ability to reconstruct images. Autoregressive transformers classifying Omniglot handwritten characters improve smoothly with scale when scored one image at a time; scored on whether all three images in a sequence are right, they acquire an emergent ability. A LeNet network on MNIST does the same when accuracy is redefined as classifying six, eight or ten test images correctly, and sharper still when model sizes are sampled sparsely.

Figure 9.4 — Schematic, after Figs. 7, 8 and 10. Smooth curves become cliffs once the score requires several outputs to be right at once.

What the paper does not claim

The authors are explicit about the boundary. Nothing in the paper argues that a language model cannot display an emergent ability; the argument is that the abilities claimed in the papers under examination are likely artefacts of the analysis. The scope is narrower still: one task, one model family, one fixed set of outputs. Within that frame the researcher can choose a metric that creates an emergent ability or one that removes it, so that "emergent abilities may be creations of the researcher's choices".

Four things follow for practice. A task and a metric are separate decisions, and a benchmark is the pair. Whoever picks a metric should work out its effect on the per-token error rate and size the test set to match. Claims of this kind face multiple comparisons on a scale rarely acknowledged: BIG-Bench alone offers around 220 tasks, 40 metrics per task and 10 model families, a million combinations among which some will look emergent by chance. And the investigation was possible only where outputs could be obtained.

Their reading of the prior literature is generous. The BIG-Bench authors had already noticed that accuracy can look sharp where cross-entropy does not, and had suggested the metric might be partly responsible; this paper turns that suggestion into predictions. Two rival accounts are named rather than dismissed: one explains emergence as a real break between two power laws, the other as real under particular assumptions about the training data.

Since 2023

The paper was given an outstanding-paper award at NeurIPS in December 2023, and the objection it raised has been absorbed rather than settled.

The reply from the other side does not contest the mathematics. It grants that accuracy is discontinuous and per-token loss smooth, and holds that this is beside the point: what a person gets from a model is the whole answer, right or wrong, so a metric that scores the whole answer measures the thing that matters. On that reading the sharpness is real in the only units anyone uses. The counter-reply is that an explanation deriving the jump from a smooth quantity has already removed the unpredictability, and unpredictability was half the original definition.

The empirical work since has gone after resolution. If accuracy on small models is too small to see with a few hundred examples, the answer is to see it with far more: sample each model thousands of times on each item and estimate the probability of a correct answer several decimal places further down. Measured that way, most tasks turn out to be predictable from smaller models across orders of magnitude, as the mirage account requires, while a few survive with an acceleration the extrapolation misses, as the rival account requires. Other work attributes the residue to in-context learning rather than to any new capability, and the authors have since argued that what makes benchmark scores hard to forecast is the step from a continuous per-token measure to the discrete score built on it.

Practice has moved further than vocabulary. Scaling reports now publish a continuous measure beside the headline accuracy, and a claim that an ability appeared at a particular scale is expected to survive a change of metric before it is believed. The word emergence is still in use. It now carries a question about the ruler.

Claims drawn from this chapter · 8

evidenceArithmetic in the GPT family

A model scoring zero may only mean the test was small

The flat stretch on the left of an emergence curve shows the limit of what the test can detect, not the absence of an ability. With a few hundred test items, the smallest score above zero is already a few thousandths.

Becausea few hundred test items put the smallest observable non-zero accuracy at a few thousandths, and with additional test data generated every model in the family, including the 350-million-parameter one, scored above chance on both tasks and improved smoothly even under accuracy.

LimitsDemonstrated on two arithmetic tasks in one open model family; the larger models whose curves are actually in dispute could not be rescored, because the outputs were not available.

PracticeSize your test set to the effect you are trying to see. A zero on a small sample is a statement about the sample, and the same applies to your own record of how often a thing goes wrong.

After AIA model that gets your task wrong every time may be improving steadily and invisibly. Whether you would notice depends on how many times you have tried, which is usually not many.

contestedSince 2023

A model's new ability still arrives suddenly for the person using it

The answer to the claim that sudden new abilities are an artefact of scoring is that a person using a model gets a whole answer, right or wrong. On that measure the change is still abrupt.

Becausethe counter-argument grants the mathematics and disputes the relevance, and the authors' reply — that deriving the jump from a smooth quantity has already removed the unpredictability, which was half the original definition — concedes that the experienced sharpness is not in dispute.

LimitsContested on both sides: the sharpness is real in the units of use, and the unpredictability that made emergence interesting is gone once the jump is derived from something smooth.

PracticeAudit the domains where you need whole answers rather than good tendencies. Where a task is scored all-or-nothing by the world, per-token improvement underneath buys you nothing until it crosses.

After AIThis is the practical residue of the whole dispute. A model either does your job or does not, and the continuous improvement that explains the transition is not something you can act on.

evidenceMetrics, not task-model pairs

Schaeffer counts a million ways to find a surprising benchmark result

Schaeffer and colleagues point out that around 220 tasks, 40 scoring measures and 10 model families give about a million combinations to look at. With that many, some will show a sudden jump in ability by chance alone.

Becausethe arithmetic is BIG-Bench's own and is stated in the paper's implications for practice, alongside the recommendation to size the test set to the metric's effect on the per-token error rate.

LimitsAn argument about the search space rather than a demonstration that a particular claimed ability is a false positive, and most such claims are reported without any correction for it either way.

PracticeCount the comparisons that were available before treating a striking result as a finding, in benchmarks and in your own logs. A surprise drawn from a million cells is not a surprise.

After AIModel evaluation now runs hundreds of benchmarks against every release, and the striking numbers are selected for publication. The multiplicity is larger than it was when this was written.

evidenceArithmetic in the GPT family

Schaeffer's explanation predicted how accuracy falls as the sums get longer

Schaeffer's account of sudden jumps in ability predicted how scores would fall as arithmetic problems got longer: accuracy geometrically, edit distance linearly. Making a prediction that then held is what makes it an explanation rather than only an objection.

Becauselengthening the target degrades accuracy approximately geometrically and token edit distance approximately linearly, exactly as each metric's algebra requires, and both held across targets of one to five digits.

LimitsThe prediction concerns the behaviour of metrics rather than of models, so it confirms the algebra rather than the claim that the observed instances of emergence were artefacts.

PracticePrefer the criticism that makes a prediction. An objection that says only that a result could be an artefact is worth less than one that says what else should follow if it is, and then checks.

After AIThis is what a well-formed challenge to a capability claim looks like, and it is the shape to imitate when a vendor's benchmark result does not match what you see in use.

contestedWhat the paper does not claim

Schaeffer's paper attacks particular examples not the idea of emergence

The paper argues that the reported cases of abilities appearing suddenly are artefacts of how they were measured. It does not argue that such jumps are impossible, and it covers one task, one model family and one fixed set of outputs.

Becausethe authors state the boundary explicitly and name two rival accounts rather than dismissing them — a real break between two power laws, and a real effect under particular assumptions about the training data.

LimitsContested because the paper's careful scope and its popular reading have come apart: it is widely cited as having refuted emergence, which is a claim the authors decline to make.

PracticeState the scope of your own objection when you raise one, and say what would still be true if you are right. It is the difference between a correction that gets absorbed and one that gets fought.

After AILater work sampling models thousands of times per item finds most tasks predictable from smaller models, as this account requires, and a few that survive with an acceleration, as the rival account requires. Both halves held.

evidenceEmergence induced in vision networks

Schaeffer's team produced sudden new abilities on demand in image networks

Schaeffer and colleagues made sharp, sudden ability appear in image networks simply by choosing a harsh scoring rule, in a field where nobody had claimed such abilities. The jump comes from the measure, not from the model.

Becauseshallow autoencoders on CIFAR-100 acquire a sharp ability once the score is the fraction of images below a reconstruction threshold of 0.003 or 0.004, Omniglot transformers show it when scored on all three images at once, and a LeNet on MNIST shows it when accuracy is redefined as six, eight or ten images correct — sharper still under sparse sampling of model sizes.

LimitsInducing an artefact shows that the artefact is available, not that a particular reported case is one; the constructive demonstration is deliberately the strongest form of the authors' weakest claim.

PracticeTreat a sharp curve as a claim about measurement until someone shows it survives a change of metric. If the effect can be induced on demand in a network nobody thinks is intelligent, its appearance proves nothing on its own.

After AINothing about the mechanism requires a language model, which is the cleanest way to see that the argument is about rulers. It travels to any pass-or-fail measure of a system that is improving continuously.

evidenceAn alternative explanation

Scoring rules create the sudden jumps in AI ability, not the models

Schaeffer and colleagues show that the sudden jump in ability is made by the scoring rule. A score that demands a whole answer be exactly right turns a steadily falling per-word error rate into a sudden step.

Becausethe algebra is explicit — a per-token probability of 0.6 scores 0.08 across five tokens and 0.9 scores 0.59, so the underlying quantity improves by half while the score improves sevenfold, with the target length as the exponent.

LimitsThe mechanism explains how a step can be manufactured, not that every reported step was; the authors say so, and rival accounts — a genuine break between two power laws, an effect driven by the training data — remain live.

PracticeCheck the shape of a scoring rule before reading a curve drawn with it. Any all-or-nothing measure over several steps will make gradual improvement look like an event, in a model or in a person.

After AIAnyone grading an assistant's output pass-or-fail is running this metric. The system improving steadily underneath a flat success rate is the normal case, and the flat rate is what you will feel.

evidenceMetrics, not task-model pairs

Sudden abilities come from the scoring rule, not the task or model

Sudden jumps in ability track the way performance is scored rather than which task or model is involved. At most 5 of BIG-Bench's 39 preferred scoring rules show any such jumps, and two of them account for almost all.

Becausethe benchmark's own emergence score applied across its metrics puts Multiple Choice Grade and Exact String Match behind more than 92 per cent of 137 hand-annotated claimed abilities, and rescoring LaMDA's emergent tasks under the Brier score makes the emergence disappear.

LimitsThe rescoring was possible only where output probabilities were published, so the evidence is strongest exactly where the models are open and weakest on the frontier systems the claims are made about.

PracticeAsk which metric a claimed capability appeared under, and whether it survives a change of ruler. Two scoring rules out of thirty-nine carrying a phenomenon is a fact about the rules.

After AIBenchmark scores are now the public vocabulary for what a model can do. This result says the vocabulary has two words in it that behave differently from the rest, and they are the two most used.