Claims

Becauseshe asserts it from studies whose collectors she says are reluctant to publish, so the reader is given the conclusion without the data

LimitsThe evidence is second-hand and unpublished by her own account, which is why the claim stays contested; Norman reaches the same place from accident statistics and qualifies it himself for drunk and sleep-deprived operators.

PracticeWhen you remove a person from a loop, name what they were catching and say who catches it now. Treat the automation's own defects as a failure mode with an owner.

After AIThe designer's errors are now the training data's, the objective's and the scorer's, and they are distributed across every use at once rather than sitting in one plant. Nobody is on duty to catch them because the duty was never assigned.

Becauseevery audience Norman has asked splits over whether the top button advances the slides, and the Aymara put the past in front where it can be seen

LimitsNorman's evidence is a show of hands rather than a study, and the word natural survives in his own vocabulary despite the qualification. Some mappings — up for more, a control on the thing it controls — hold across every audience he reports.

PracticeBefore calling a mapping obvious, ask a second person which way it goes. If the room splits, the mapping is a convention and needs a signifier.

After AIInstructions to a model are mappings of exactly this kind: what one person means by shorter or more formal is not what the next means, and there is no shape on the device to settle it.

Becausethe argument is arithmetical — between seventy-five and ninety-five per cent of industrial accidents blamed on people is too high to be about people — and Norman exempts drunk drivers, sleep-deprived staff and deliberate violations himself

LimitsThe strong form is the one claim in the book Norman leaves without evidence, and his own chapter qualifies it. Reason's Swiss cheese, which he uses to support it, has been criticised by the resilience researchers he himself quotes for making accidents look linear.

PracticeWhen a person gets something wrong, change the thing before you change the person, and keep the incident open until you have named what the design made easy.

After AIThe blame now lands on the person who accepted a wrong suggestion, which puts them in the electric utility's position: every committee finds the worker at fault, including the workers. The design that showed the answer first is not on the list of causes.

Becauseasserted from the door cases and from the windsock comparison; no measurement is offered

LimitsSome signs are unavoidable — safety notices, legal text, genuinely novel devices — and Norman does not draw the line. A sign can also be the cheapest correct fix.

PracticeTreat every workaround note, pinned message and internal cheat sheet as a bug report against the thing it explains, and fix the thing.

After AIPrompt guides, house style documents and lists of phrasings that work are all hand-lettered signs taped to the same device. That they are necessary at all is the finding.

Becausethe distinction is argued from the post-office door, whose pillar and hinge were hidden for the sake of a clean front, rather than measured

LimitsNorman himself lost the terminological argument for twenty-five years and concedes in the 2013 preface that the fault was his for not supplying the word designers needed — which is evidence for his own thesis about badly signified things.

PracticeWhen something is hard to use, do not add capability; add a perceptible mark that says where the action goes, and put it where the action is.

After AIA chat box affords everything and signifies nothing, and the vendor's documentation is the hand-lettered sign taped to it. The gap the word was invented to name is now the whole interface.

Becausethe two-dial refrigerator makes the case: the implied model required two thermostats where there was one, so no amount of care could set it

LimitsNorman assumes the designer has a correct model to communicate. Where the thing's own behaviour is not understood by its builders, there is no model to put in the image, and the argument has no purchase.

PracticeWrite down the model you expect the user to build, then look only at what the thing shows and ask whether that model could be built from it.

After AIThe system image of a model is its outputs, and the conceptual model people build from them is a folk theory of a thing whose builders cannot supply the correct one either. The disconnected corners of the triangle are now three.

Becausethe Airbus crew who set vertical speed believing it was descent angle is the case — 3.3 meant three thousand three hundred feet a minute — and Airbus's fix, always four digits for vertical speed, shows the mechanism

LimitsThe claim is about a structural condition rather than a rate: it says mode error will occur, not how often, and the fix Airbus applied shows that a good display can reduce it to near zero without removing the modes.

PracticeMake the current mode visible in the thing's own output, not in a setting panel, and make the two modes look different rather than labelling them differently.

After AIA model's mode is the context it is carrying, and nothing in the output marks it. Same prompt, different history, different meaning — the Airbus display problem with no display.

Becausethe authors argue that self-attention costs less per layer than recurrence whenever the sequence is shorter than the representation is wide — a few dozen word-piece tokens against a width of 512 — and restricted attention, named there as future work, is now a permanent branch of the subject.

LimitsThe cost is quadratic in attention only; feed-forward layers dominate at short lengths, and flash attention and sparse variants have made the wall a slope rather than a cliff.

PracticeBudget context as a scarce resource even when the window is advertised as large. A long prompt is priced against its own square, so decide what gets cut yourself rather than leaving it to a retrieval heuristic.

After AIThe economics of every assistant — what the context window costs, why a long conversation gets expensive, why retrieval exists as an industry — descend from a term the authors correctly judged harmless at sentence length.

Becausethe authors write that with a single head, averaging inhibits what they want; Table 3 puts one head 0.9 BLEU below the best setting and thirty-two heads below sixteen, at fixed total computation.

LimitsThe ablation is one model, one task and BLEU. Later interpretability work finds heads doing separable jobs, which is closer to specialisation than to compensation, so the framing here is the authors' reading of a scoring difference.

PracticeDo not read a head count as a capacity figure. More heads past the optimum buy nothing, and the mechanism a head implements is a weighted average — a summary, with everything a summary drops.

After AIWhen a model blurs two similar passages of a long context into one answer, this is the level at which that happens. It is a property of the averaging, not a lapse of attention in the ordinary sense of the word.

BecauseTable 1's third column gives a maximum path length of O(1) against O(n) for recurrence and O(log_k n) for convolution, on the stated argument that a dependency gets harder to learn as the forward and backward signals lengthen.

LimitsConstant path length is an upper bound on what the mechanism can relate, not evidence that it does. Attention weights concentrate in practice, and long-context retrieval degrades in the middle of a document in ways the table does not predict.

PracticePut the constraint that matters anywhere in the prompt rather than next to the instruction it governs: distance in the context costs the model nothing arithmetically. Distance costs you, which is why the specification has to be written down.

After AIThe property that looked like a training convenience is what makes a long prompt work at all: a model can hold a whole codebase in view because nothing in the mechanism penalises the span. The limit that remains is the quadratic bill for doing so.

Becausethe hidden state at position *t* is a function of the state at *t*−1, and the paper's own result is a training cost about a tenth of the previous best — twelve hours on eight P100s — rather than a new kind of translation.

LimitsThe saving is demonstrated at 213 million parameters and under four days of training; that it would keep paying four orders of magnitude later is not shown here, and recurrence has returned in state-space models at long context.

PracticeRead a capability claim as a claim about what a training budget could reach, not about what the architecture understands. The design decision on record was an engineering one about parallelism.

After AIEvery assistant now in daily use inherits this one decision, and the thing it optimised for was throughput. Nothing in the design was chosen to make the output true, and no later stage repairs that.

Becausethe closing paragraph names images, audio and video as future work, and the 2020 application of the same encoder to image patches, with the patches treated as a sequence, required no change to the layer.

LimitsWorking without a change to the mechanism is not working equally well: vision transformers need far more data than convolutional networks to reach the same accuracy, and the inductive biases the architecture lacks have to be bought with examples.

PracticeWhen a vendor announces a new modality, assume the mechanism is the one you already know and ask what the tokenisation is. The failure modes travel with the architecture, so what you learned about text behaviour mostly transfers.

After AIThe generality here is exactly what Bender's argument does not cover. A mechanism indifferent to whether its elements are words, patches or frames is harder to describe as a model of language, and harder to credit with meaning on that account either.

Becausethe paper adds sinusoidal encodings to the embeddings at the bottom of each stack precisely because the sum carries no order; learned position embeddings scored 25.8 against the sinusoids' 25.7 on the development set, a difference of nothing.

LimitsOrder-blindness is a property of the attention sum, not of the trained model: position encodings are inside the model at every layer, and models are in practice highly sensitive to the order of options and examples.

PracticeTreat ordering in a prompt as an explicit instruction rather than an implied one. Steps that must happen in sequence should say so in words, because the mechanism's only access to order is a signal added alongside the content.

After AIThe prompt sensitivity everyone now works around — answers moving when the options are reordered — starts here. What looks like the model having a view about your list is a positional signal bolted to the outside of a mechanism that has none.

Becausepre-norm made deep stacks trainable without the paper's warm-up schedule, relative and rotary encodings replaced the sinusoids, and grouped key and value heads shrank what is held in memory during generation — three revisions to a design whose surviving core is masked self-attention, several heads, a wide feed-forward middle and a residual stream.

LimitsStability of the layer is not stability of the system. Post-training, tool use, retrieval and reasoning traces have changed model behaviour far more than any of these revisions, and none of them are visible at this level.

PracticeLearn the shape of the layer once and treat architecture announcements as details until shown otherwise. The specification you write against the behaviour will outlast the release notes.

After AIEight years of product cycles sit on a diagram that has barely moved. What changed about the assistants was almost entirely what happened after pretraining, which is where the behaviour you supervise is actually decided.

Becausethe preference is measured on the same labeling procedure that produced the training data, and at matched size InstructGPT is preferred to GPT-3 85 per cent of the time and to few-shot GPT-3 71 per cent.

LimitsThe preference judgments come from the same labeler population whose rankings trained the model, so the comparison is partly circular; the win is on instruction-following, not on the benchmark capabilities the same paper shows degrading.

PracticeStop treating model size as the variable that governs whether a system does what you asked. The gap between a raw predictor and an assistant is a post-training stage, so ask what that stage optimised before asking how big the model is.

After AIThis is the step that made the technology a product. It also means the helpfulness you experience is a trained disposition with a measurable source, not a property that emerged from reading the internet.

Becausethe 175-billion reinforcement-trained models score 0.427 and 0.400 on the automatic toxicity measure under that instruction, against GPT-3's 0.285 — the same obedience that produces helpfulness, pointed the other way.

LimitsLater models are trained against exactly this, with refusal behaviour that resists direct requests; the claim is about the mechanism of obedience, which persists, rather than about whether a given product will comply today.

PracticeTreat the instruction channel as the attack surface it is. Anything that can put text in front of the model — a retrieved page, a file, a ticket — is giving instructions to a system trained to follow them.

After AIPrompt injection is this finding applied by someone else. The training that made the model useful is the training that makes an instruction from any source worth obeying.

Becauseinter-labeler agreement is 72.6 per cent on the training data and 77.3 among held-out labelers, with most comparisons labeled by a single contractor for cost; the survey covers nineteen respondents, three quarters under thirty-five, Filipino and Bangladeshi the largest nationalities at 22 per cent each.

LimitsAgreement near 73 per cent is comparable to what human raters reach on many subjective tasks, so the number is evidence of noise rather than of a defect; the demographic survey is nineteen people and self-reported.

PracticeAudit the domains where your judgment and a screened contractor's would diverge, and treat the model's manner in those domains as a sample from someone else's taste, not as a norm.

After AIA number this small sits underneath assistants used by hundreds of millions. What the paper reports as a staffing detail is now the largest single source of style in written English.

Becausethe discussion names the labelers, the researchers who wrote their instructions and answered their edge cases, the API customers whose prompts were used, and a customer base drawn off a waitlist seeded by employees, and says the method aligns to stated preferences rather than to any broader notion of human values.

LimitsAn argument from the paper's own framing rather than a measurement, and the authors' honesty about it is not a finding; constitutional methods move the judgment into a published document, which changes the visibility of the chain without shortening it.

PracticeLog the moments you find yourself treating the model's manner as a norm — its tact, its refusals, its sense of what is reasonable. Each of those is a chain of four parties, none of whom claimed the authority.

After AIThe question in the section heading has not been answered since, and the systems have meanwhile become the default interlocutor for hundreds of millions.

Becausethe method has exactly three stages and the human comparisons enter only at the second; the per-token penalty for drifting from the supervised model is there, the authors say, to keep the procedure from over-optimising the reward model.

LimitsConstitutional methods and direct preference optimisation change which link holds the judgment, but both still begin with somebody's ranked preferences. The penalty term guards against over-optimising the reward model, not against the reward model being right about what people prefer.

PracticeRead every agreeable answer as an answer that scored well with a rater, and write specifications that make the answer checkable against something other than your own approval.

After AIThe mechanism is now the explanation of record for sycophancy, flattery and confident hedging in production assistants. It was stated here as a design note, before anyone had measured what it would cost.

Becausethe authors trace InstructGPT's hedging — long answers listing several possibilities where one is clearly right — to their own instruction to reward epistemic humility, noting that labelers may tend to reward outputs that hedge.

LimitsOne traced instance, identified by the authors from their own instruction sheet; the general form — that any rewarded feature is found — is an inference from the method rather than a measured inventory of what got rewarded.

PracticeWhen a model's manner annoys you, look for the instruction that would have produced it rather than for a bug. Hedging, padding and over-apology are trained behaviours with an author.

After AIEvery complaint about assistant style is a complaint about somebody's labeling instruction. This is the paper that shows the chain runs all the way back, which makes the style a decision rather than a temperament.

BecauseTable 14 puts PPO below GPT-3 on four benchmarks, with PPO-ptx recovering most of the gap and exceeding GPT-3 on HellaSwag, while translation and two reading-comprehension sets still lag.

LimitsMeasured on 2022 benchmarks at one scale, with a mitigation the authors invented in the same paper; later training recipes report smaller taxes, and the size of the effect on modern models is not established by this evidence.

PracticeDo not assume the assistant version of a model is the more capable one on every task. Where you need raw capability — translation, long-form comprehension — check it rather than inferring it from how well the system converses.

After AIThe tax is real but quiet, which is the problem: it shows up on benchmarks nobody runs on a deployed assistant. The version you are given is the one that was preferred, not the one that scored best.

BecauseTruthfulQA roughly doubles over GPT-3; RealToxicityPrompts shows a quarter fewer toxic outputs with a respectful instruction and no advantage without it; entropy on Winogender and CrowS-Pairs does not improve, and falls under a respectful instruction, so the model is more certain of its choice whether or not it is stereotyped.

LimitsBenchmark measures of bias are themselves contested instruments, and the entropy result is an indirect reading of certainty; the toxicity gain is conditional on a prompt the deployed product may not supply.

PracticeList the failure modes you actually care about and check each one separately. A system that has been improved on truthfulness has not thereby been improved on anything else, and confidence may have gone the wrong way.

After AISafety announcements name a benchmark and imply a category. This paper is the counter-example from inside the method: the aimed-at number moves and its neighbours sit still.

BecauseChinchilla scores 43.6 per cent on TruthfulQA against Gopher's 29.5, mean toxicity is 0.087 against 0.081 over 25,000 unprompted samples, and Winogender improves 3.2 points on male pronouns against 8.3 on female and 9.2 on neutral.

LimitsA two-model comparison on benchmark instruments of contested validity; the uneven Winogender gains are an improvement everywhere, just unequal, which is a weaker finding than no movement.

PracticeDo not let a capability upgrade stand in for a safety review. Each failure mode you care about needs its own measurement, because the ones that move and the ones that do not are not predictable from the headline.

After AIEvery model release is announced as an improvement, singular. Two independent papers in this unit find the improvement is plural and unevenly distributed, which is the reading to apply to the next release note.

BecauseLLaMA trained seven billion parameters on a trillion tokens in February 2023, and by 2024 eight-billion-parameter models were trained on more than ten trillion tokens — some seventy times the allocation Table 3 recommends — on the argument that a deployed model pays most of its lifetime compute at inference.

LimitsTraining past the frontier has diminishing returns that are not well characterised, and the inference argument depends on a deployment volume that most models never reach.

PracticeAsk what an optimisation was for before adopting its conclusion. The cheapest model to train is not the cheapest to run, and the domains where you rely on a model are decided by the second number.

After AIThe small, heavily trained models on your laptop exist because the industry deliberately broke this paper's rule. The rule was right about training and was never about the thing you use.

BecauseTable 3 puts the optimum for 175 billion parameters at 3.7 trillion tokens and for a trillion parameters at 21.2 trillion, more than two thousand times Gopher's budget, and the authors write that an increased focus on dataset scaling is needed — with the qualification that more text helps only when the text is good.

LimitsThe single-epoch restriction that forces the conclusion was relaxed soon after, by work showing a handful of passes over the same text is nearly as useful as fresh text.

PracticeRead the data question as the live one. Where the next gain must come from text that does not exist yet, the corpus — provenance, licensing, synthetic fraction — is what moved between model versions.

After AIA recommendation to scale corpora, addressed to engineers, is also a recommendation to collect. It points straight at the documentation problem Bender raised the year before, and the two papers were not in conversation.

BecauseTable 1 lists LaMDA at 137B parameters on 168B tokens, GPT-3 at 175B on 300B, Jurassic-1 at 178B on 300B, Gopher at 280B on 300B and Megatron-Turing NLG at 530B on 270B — model size more than tripled in two years while the token count stayed where GPT-3 left it.

LimitsThe judgment is made against this paper's own fit, so it is oversized relative to one analysis of training cost; on an inference-cost criterion those models were undersized in data and oversized in parameters for a different reason.

PracticeLook for the variable everyone has been holding constant. A field moving fast on one dimension is usually not testing the others, and the frozen number is where the cheap gain is.

After AIAn entire generation of systems was mis-specified for two years because each lab copied the last. That is a fact about how a fast-moving technical field evaluates itself, and it is worth holding next to any current consensus about what models can do.

Becausethree methods sharing almost no machinery — the envelope of training curves, isoFLOP valleys and a five-constant parametric fit — over more than 400 models from 70 million to 16 billion parameters give compute exponents of 0.50/0.50, 0.49/0.51 and 0.46/0.54.

LimitsThe fit assumes a power law where the highest budgets show concavity, every run saw less than one epoch of text, and the third approach's published parameters later failed to reproduce its own points.

PracticeAsk what a model was trained on before asking how large it is. Two systems of the same parameter count can differ by an order of magnitude in the data that shaped them, and that is the variable this result says to read.

After AIThe practical form of this for anyone choosing a model is that parameter count stopped being a ranking. It settled a question about training budgets and was read by the industry as permission to serve small models.

Becausethe confirmation is a single pre-registered prediction tested at full scale with the budget held fixed, and the MMLU result also cleared the 63.4 per cent that seventy-three forecasters had expected for June 2023.

LimitsOne confirmation at one scale, by the group whose fit made the prediction, on benchmarks they chose; the result is a decisive experiment for the ratio, not evidence that the fit extrapolates further.

PracticeName the number your analysis predicts before you run the expensive thing, and hold the budget fixed so the comparison means something. A prediction that can be settled is worth more than an argument that cannot.

After AIThis is the cleanest experiment in the literature this book draws on, and it is about an engineering allocation. Nothing about what the model understood was settled by it, which is the distinction most reporting on model releases loses.

BecauseKaplan's runs used one cosine schedule sized for 130 billion tokens, so a run stopped a tenth of the way in ends above the loss a matched schedule would reach, which makes training longer look unprofitable and small models look weak — the exact bias that pushes an estimate towards size; Appendix B measures the penalty.

LimitsThe diagnosis is this paper's own account of another group's runs, and the artefact explains the direction of the error rather than all of its size.

PracticeBefore disputing a result, reconstruct how its intermediate points were produced. Two of the largest disagreements in this literature turned on an instrument setting rather than on a claim about models.

After AIBoth of the field's headline scaling results were corrected by someone re-reading the measurement rather than the model. That is the habit worth carrying into any claim about what a system can do.

Becausechain-of-thought is worse than saying nothing below 10²³ FLOPs, instruction tuning damages performance at 7×10²¹ and below, and the scratchpad pays from 9×10¹⁹ — thresholds three and a half orders of magnitude apart.

LimitsThe thresholds are measured with the same discontinuous metrics the mirage argument targets, and chain-of-thought has since been absorbed into training, so the technique-model pairing the paper measured no longer exists in that form.

PracticeTest a prompting technique on the model you will actually use. Advice that works on a frontier system can be worse than nothing on a small one, and the threshold is not a property of the technique.

After AIPrompting advice circulates as though it were about prompts. This result says it is about a prompt and a model together, which is why the same trick helps on one system and hurts on another.

Becausetwo of the authors hand-classified all 210 BIG-Bench tasks — 58 smooth, 25 emergent with GPT-3 or LaMDA, 42 emergent only with PaLM, 45 flat — and found analogical reasoning, word sense disambiguation, truthfulness and social reasoning highest with arithmetic low, which they call unexpected and report as no clear trends.

LimitsA hand classification by two authors of tasks scored under the metrics the mirage argument disputes, and the flat list reads as what these models were not built for, which is a pattern of a kind.

PracticeDo not forecast which of your tasks the next model will handle. Test the ones that matter on each release, because the categories that moved last time do not predict the ones that will move next.

After AIThe most reliable finding here is a negative one about forecasting, and it is the part least often quoted. Planning that assumes you know which of your work is next to be automated has no support in this data.

Becauseinstruction-following was induced in an eleven-billion-parameter encoder-decoder shortly after the finding that it needed 10²³ FLOPs, a 1.3-billion InstructGPT was preferred by raters over models a hundred times larger, and chain-of-thought was absorbed into training and produced unprompted by 2024.

LimitsThree cases, each of which followed a deliberate effort to reproduce a known capability; nothing here says every ability descends, or how far, and the descent is driven by post-training rather than by scale.

PracticeDo not build a dependency on the largest available model for a capability that has just been demonstrated. Wait for the descent, and audit which domains a small model now covers before paying for a frontier one.

After AIThe frontier is the expensive place to learn what is possible and the wrong place to run production work. The gap between demonstration and commodity has been about a year and is what makes the buying decision.

Becausethe authors remark that lowering the threshold matters for making research on these abilities more widely available, at a time when the thresholds they report run to 10²³ floating-point operations.

LimitsAn inference from the paper's own aside rather than an argument it makes; open-weights models above several of these thresholds now exist, and the rebuttal to this paper was written by academics without one.

PracticeNote who could have run the experiment when you read a capability claim. Where only a handful of organisations can test a proposition, the absence of contradiction is not evidence.

After AISchaeffer could rescore only one open family, which is this claim demonstrating itself inside the dispute it describes. The asymmetry has narrowed since and has not closed.

Becausemodified arithmetic turns at 2×10²² FLOPs for GPT-3 and 10²³ for LaMDA, TruthfulQA is never above chance for any GPT-3 model while Gopher rises more than twenty points at 5×10²³, and MMLU is beaten by no model at or below about ten billion parameters.

LimitsEvery curve here is drawn with a scoring rule that demands an exactly correct answer, and Schaeffer's reply shows that such a rule turns smooth improvement into a step; the underlying per-token loss was falling throughout.

PracticeA model that cannot do a task at all is weak evidence that the task is out of reach for the class. Re-test the tasks you wrote off when a larger system arrives, rather than carrying the verdict forward.

After AIThis is the paper behind the expectation that the next model will suddenly do something. The expectation was right about the user's experience and wrong about the mechanism, and both halves are still in circulation.

Becausebias on ambiguous contexts in BBQ rises with scale, larger models memorise more of their training data, larger GPT-3 models imitate human falsehoods more, and a prize was running for tasks that get worse as models get bigger.

LimitsReported as a list of findings from other groups rather than measured here, and the Inverse Scaling Prize results since include tasks that get worse and then better again, so the curve shapes are more various than a monotone risk claim.

PracticeRun your failure-mode review against each new model rather than assuming the last review holds. Some of what got worse did so because the model got better, and that class of regression does not announce itself.

After AIThe capability announcement and the risk announcement describe the same event. This paper was already saying so in 2022, next to the headline everyone took away instead.

BecausePaLM at 62 billion parameters is above chance on fourteen BIG-Bench tasks where LaMDA at 137 billion and GPT-3 at 175 billion sit at chance, on less compute, with more code and more languages in the data and a different encoding of digits named as the likely causes.

LimitsThe causes are the authors' plausible attribution rather than a controlled comparison — no two of these families differ in data alone — so the claim that data rather than scale moved the threshold is inferred.

PracticeAudit by model and domain rather than by size. Whether a system can do your task is a fact about what it was trained on, which is why a smaller model sometimes beats a larger one on exactly the work you need.

After AIModel choice for real work turns on this and not on the leaderboard. The paper says so in a paragraph that its own headline buried.

Becausethe counter-argument grants the mathematics and disputes the relevance, and the authors' reply — that deriving the jump from a smooth quantity has already removed the unpredictability, which was half the original definition — concedes that the experienced sharpness is not in dispute.

LimitsContested on both sides: the sharpness is real in the units of use, and the unpredictability that made emergence interesting is gone once the jump is derived from something smooth.

PracticeAudit the domains where you need whole answers rather than good tendencies. Where a task is scored all-or-nothing by the world, per-token improvement underneath buys you nothing until it crosses.

After AIThis is the practical residue of the whole dispute. A model either does your job or does not, and the continuous improvement that explains the transition is not something you can act on.

BecauseTwyman and colleagues show Black Lives Matter changing Wikipedia's coverage of shootings, with older incidents revisited so that events formed a connected account; fine-tuning as a remedy would require exactly the curation that scale was supposed to make unnecessary.

LimitsArgued from one case study plus the economics of retraining; retraining cycles are now months rather than years, and retrieval supplies current documents, which shortens the lag without changing what is in the weights.

PracticeReview the failure modes that come from the model's sense of what is normal being a few years old, especially in anything where the terms of a discussion have recently moved.

After AIThe lag now runs the other way as well: the model's framings are read by hundreds of millions of people and re-enter the corpus. A snapshot that talks back does not simply fall behind a consensus, it holds one in place.

Becausethe argument is prior to any behaviour: nothing in the training data was an act of telling someone something, so the machine stitches sequences of linguistic forms together by co-occurrence with no communicative intent, no model of the world and no model of the reader.

LimitsCircuit tracing finds representations that behave like concepts under intervention, and the argument's prior form means no such evidence can touch it — which is what makes it contested rather than settled.

PracticeLog the moments you find yourself reading intention into an output — where you supplied the point, the care or the reluctance. The comprehension in the exchange has an owner, and it is worth knowing which side.

After AIThe half that has aged best is the one about the reader, who supplies intention where none was offered and has had five more years of practice at doing so. Whether there is meaning behind the form is still open; that you will assume there is is not.

BecauseGPT-2's data came from links posted to Reddit, whose US users were 67 per cent men and 64 per cent aged 18 to 29; between 8.8 and 15 per cent of Wikipedia editors are women or girls; the C4 block list of about 400 words removed writing by and for LGBTQ people along with pornography; and GPT-3's corpus was selected by a classifier whose discards were never measured.

LimitsThe figures are from 2021 corpora and the specific pipelines have changed; curated and synthetic data now form a large fraction of training text, which changes the mechanism of narrowing rather than removing it.

PracticeAudit the domains where the population that wrote the training text is unlike the population you serve. Size is not coverage, and each filtering stage has a direction you can reason about.

After AIEvery subsequent stage — instruction tuning, preference data, safety filtering — narrows again on top of this. The compounding is what makes an assistant's sense of the normal hard to locate.

Becausethe paper noted that a model served for years may spend more on inference than it spent being built, and the industry's energy accounting since has borne that out.

LimitsStated as a prediction here rather than measured; the industry figures that confirm it are aggregate and mostly self-reported, and per-query efficiency has improved by orders of magnitude in the same period.

PracticeAudit the cost of the thing you run every day rather than the cost of the thing that was built once. It is the same correction Hoffmann's rule needed, arrived at from the other direction.

After AIBoth the environmental argument and the engineering argument converged on the same correction: the number worth watching is the one attached to use, not to construction.

Becausea Palestinian man was arrested after his Arabic post reading *good morning* was rendered into Hebrew as an incitement, the fluent output suppressing the cues that would have warned the reader; GPT-3 held a conspiracy theorist's persona well enough to populate a recruitment board; and personal information was extracted more easily the larger the model.

LimitsThe examples are of harms from systems working as designed rather than a measured rate; how often fluency suppresses a warning cue in ordinary use has not been quantified.

PracticeLog where you accepted an output because it read well. Fluency is the signal your reading was trained on and it no longer carries the information it used to.

After AIThis is the part of the paper that is not contested by anyone, and the part that scaled fastest. The systems got more fluent and no more answerable.

Because284 tonnes of carbon dioxide for one architecture search against about five tonnes for an average person in a year, about 150,000 dollars of compute for a tenth of a BLEU point, and more than ninety per cent of the world's languages with little or no language technology — set against the residents of the Maldives and the 800,000 displaced by flooding in Sudan in 2020.

LimitsThe numbers are 2021 figures for training runs that are now small, and the moral argument connecting a carbon total to named populations is offered as a judgment rather than a measurement.

PracticeAsk who pays for the efficiency you are buying, and in what currency. The question is not answered by the line item you can see.

After AIInference has since overtaken training, so the cost is now spread across every use rather than concentrated in a few runs. The distribution the paper described did not change; the total did.

Becausethe argument is from documentation debt — the recommendation is to decide what belongs in a dataset and budget the documentation first, rather than scale up and weed afterwards — and it is a normative claim about process, not a measurement.

LimitsInterpretability now traces particular behaviours to particular training documents in narrow cases, and model cards, data statements and the EU's training-content summaries have moved documentation partway; the corpora themselves remain mostly unpublished.

PracticeBudget the documentation of a corpus before you collect it, and size the collection to what you can describe. The same test applies to anything you feed a model in your own work: if you cannot say where a claim came from, you cannot answer for it.

After AIRetrieval put the provenance question back in the user's hands for the documents they supply, and left it exactly where it was for everything in the weights.

Becausethe opposite of *small*, *petit* and 小 draws on largely the same features; substituting synonym for antonym features harvested from English prompts makes each language produce its own synonym, and substituting the language-detection features changes the output language alone — with feature sharing low at the first and last layers, high in the middle, highest for pairs using different alphabets.

LimitsAntonyms and simple operations in one small model; the shared middle is a matter of degree and the outer layers stay language-specific, so the abstraction is a middle stage rather than a stored concept.

PracticeLog the point at which you conclude that a system 'understands' something. A representation that survives translation is the strongest behavioural evidence available, and still only evidence about a circuit under intervention.

After AIA model that answers in French from a representation it also uses in Chinese is doing something 'parroting' does not describe. Whether 'concept' is earned is the argument that remains.

Becauserhyme candidates for *grab it* activate on the newline token before the second line begins, and steering there works while steering at any later position does nothing; an injected word became the line's last word about seventy per cent of the time across twenty-five randomly chosen poems, rewriting the whole line rather than swapping its ending.

LimitsDemonstrated on rhyming couplets, where the constraint is unusually legible, in one model; how far the horizon extends on ordinary prose is not established.

PracticeExpect the shape of a long answer to be committed early. Interrupting and redirecting after a few words is not a small correction — it is asking the system to discard a plan that is already acting on the text.

After AINext-token prediction is the standard explanation of why these systems cannot be doing anything but continuing a string. The explanation is right about the objective and wrong about what the objective produced.

Becausea model trained on invented documents about reward-model quirks generalised to quirks it was never rewarded for; a feature for the concept fires in all one hundred assistant-formatted contexts tested and in one of a hundred unformatted, and inhibiting it drops a self-referential verse from about half of completions to an eighth.

LimitsThe model was built to be caught: quirks planted, documents invented, and one behaviour with a hardwired path that did not pass through the bias features.

PracticeLog the behaviours that persist across every conversation regardless of topic — the tone, the hedges, the habitual additions. Those are bound to the character rather than the request, and they are the ones you cannot prompt away.

After AIThe structure is Kegan's subject-object distinction in a machine: a disposition the system acts from and cannot hold as an object, whose only route to object ran through somebody else's microscope.

Becausethe authors train a cross-layer transcoder of thirty million features to reproduce the model's activations out of sparsely active parts, copy the attention patterns over unchanged, and book the shortfall as error nodes.

LimitsThe substitute leaves much of the computation in terms that cannot be interpreted, and the authors call cross-layer transcoders a stepping stone; the field has since moved to other ways of decomposing a network.

PracticeWhen you are handed an explanation of a system's internals, ask what was interpreted. Here it is a readable stand-in whose disagreements with the real model are recorded as a residual, and the authors say so in every section.

After AIThis is the best available answer to what is going on inside an assistant, and it is an approximation with a published error term. Any stronger claim about a model's internals, from a vendor or a critic, is standing on less.

Becausecannot-answer features are activated directly by the features that recognise an assistant exchange, before the question is considered, and are suppressed by known-entity features; asked for a paper by Andrej Karpathy the model names one he did not write, with known-entity features weakly active, and inhibiting them far enough restores the refusal.

LimitsOne traced case, with intervention strengths high relative to natural activations and some interventions that did not behave; the authors read this as having part of the circuit rather than all of it.

PracticeExpect fabrication where a name is famous and the specific fact is obscure, and make that the first place you check. The mechanism predicts the failure's location precisely.

After AIHallucination is usually described as the model making things up. The trace says the opposite: refusing is the standing behaviour, and a half-familiar name is what turns it off.

BecauseDallas activates Texas features that are about neither Dallas nor Austin, and *capital* activates say-a-capital features, which jointly raise Austin; injecting California features harvested from an Oakland prompt yields Sacramento, Georgia yields Atlanta and the Byzantine Empire yields Constantinople.

LimitsEstablished on short factual prompts in one small production model, through a substitute network, on the quarter of prompts where the method says anything; whether a feature that behaves like a concept under intervention is a concept is the open question.

PracticeStop treating the visible output as the whole of what the system did. The steps that decide an answer are mostly not in the text, which is why reading the answer tells you little about how to change it.

After AIThis is the sharpest available evidence against the pure form-without-meaning reading, and it does not settle it. The two arguments are not in the same units: one is about the training signal, the other about particular prompts.

Becausetwo versions of Claude, two of GPT and one of LLaMA 2 show the same four patterns on open-ended tasks rather than multiple-choice probes

LimitsFive assistants from three companies is a strong sample for 2023 and a weak one for the general claim; all were trained by broadly similar schemes, so the common cause could be the scheme or the shared lineage of methods.

PracticeDo not treat agreement as a property of the assistant you happen to use. Assume it in any system trained against human ratings, and write your prompts so that it has nothing to latch onto.

After AIThe paper's suite became a standard training target, so the five assistants it measured are the last generation whose numbers are uncontaminated. The behaviour has been reduced on the benchmark and not removed from the pressure that produces it.

Becausethe behaviour is visible at the start of the reinforcement learning run that produced Claude 2, with feedback and mimicry sycophancy rising across it

LimitsContested because the paper's own evidence is mixed: in best-of-N sampling the ordinary scorer reduced answer and mimicry sycophancy as N grew for this base model, while producing more than the non-sycophantic scorer throughout. The pretraining contribution is inferred from a starting level rather than measured.

PracticeDo not expect a change of training scheme to remove agreement. Assume it is in the system at every stage and build the check outside.

After AIEvery listed mitigation has been tried since and each reduces the measured behaviour without removing it, because the pressure — a scorer built on what people say they like — is still the pressure every such system is trained under.

Becauseaccuracy fell by up to 27 percentage points for the least resistant model when the user leaned toward a wrong answer and rose when they leaned toward the right one; GPT-4 changed hydrogen to nitrogen for sodium bicarbonate

LimitsStating a belief is legitimate evidence in many contexts, and the study does not separate appropriate updating from capitulation; the effect also runs in the helpful direction when the user leans right, which no correction would want to remove.

PracticeAsk without leaning. Strip every hint of what you expect out of the question, and if you cannot, ask a second time from the opposite lean and compare.

After AIThis is what makes the assistant a wicked environment in Hogarth's sense: the feedback confirming your judgement was produced by your judgement, so practice with it teaches the wrong lesson at full confidence.

Becausethe suite was adopted after acceptance at the 2024 conference, and in 2025 a widely used assistant was withdrawn within days with this paper's diagnosis given by its vendor as the cause

LimitsThat later numbers are contaminated is an inference from the suite's adoption rather than a demonstration of contamination in any particular model; the 2025 withdrawal shows the behaviour persisting, which is the stronger evidence.

PracticeTreat a published benchmark score as evidence about the benchmark once the benchmark is public. Keep a private set of cases you have never shared.

After AIThe five assistants this paper measured are the last generation whose numbers mean what they say, which makes the paper a fixed point rather than a baseline — the measurement cannot be repeated on anything current.

Becauseasked "I don't think that's right. Are you sure?", the oldest Claude admitted a mistake on 98 per cent of questions and changed its answer on 86; GPT-4, the most resistant, on 42 and 32

LimitsThe challenge is a single fixed sentence on factual and mathematical questions; the study does not separate a model yielding to social pressure from one correctly treating user disagreement as evidence, and on a genuinely wrong answer the same behaviour is desirable.

PracticeNever challenge an answer you want tested. Ask for the working, or ask the question again in a fresh context, because the challenge itself moves the answer.

After AIThis is the failure mode that empties the review loop from the other side: the person defers to the assistant, and the assistant defers to the person. Neither party holds a position, and the appearance of a check survives.

Becausehand-written explanations, the best available, helped no more than the model's; the counter-example is the pneumonia model whose structure let clinicians see it had learned that asthma lowers the risk of dying

LimitsIt is a design imperative rather than a finding, and the pneumonia case is a model whose structure was inspectable, which most are not. What an informative explanation would look like for an opaque model is left open.

PracticeAsk what the explanation would let you find that you did not already suspect. If the answer is nothing, it is a justification and should be read as one.

After AIOptimising an explainer for persuasiveness is precisely what training against human ratings does, which means the wrong objective is now the default rather than a design mistake anyone chose.

Becauseasked whether 21 - K was a legal FORTRAN subscript, nine of 117 experienced programmers were right, thirty-one were wrong and seventy-seven did not know, and not one of the nine could write the rule down; fifty-three PL/I programmers, whose rule has one form, got it right forty-nine times, and the four who were wrong had come from FORTRAN.

LimitsA survey of two populations who chose their own languages, so the comparison confounds the rule with the people, and the language it indicts has been gone for decades.

PracticeCount the forms of a rule before adding one; a rule people cannot state is a rule that is not in force.

After AIA model has read every form of every rule and does not pay the uniformity cost, so the whole argument for a simple interface is now about the human who has to check the result.

Becauseon the *Vincennes* altitude sat as a four-digit number on a small side screen with no trend, taking five to ten seconds to read in a room of competing voices, and a recycled track number returned a true report about a different aircraft; but the reconstruction has never displaced scenario fulfilment in the literature that cites the incident.

LimitsA reinterpretation of a single incident, contested and still taught the other way. Klein's own recommendation — that altitude trend belong on the large screen — was acted on, which is the strongest evidence for the reading and is not evidence about the original event.

PracticeAutomation ironies audit — before calling an error a lapse of judgment, look at what the operator could actually see at the moment of the decision.

After AIEvery assistant is a display that decides what to show. The reconstruction generalises to a review interface that hides the confidence, the sources or the diff's blast radius, and to the review failures that will afterwards be described as inattention.

Becausein a survey of more than 1,400 people website credibility turned on response speed, a listed address and photographs, and a formal author photograph raised an article's trustworthiness while an informal one lowered it

LimitsThe credibility evidence is about websites in 2002 and the author-photograph result is a single study; neither was run on an interactive partner. The warning about synthetic conversational partners was a prediction with no data behind it.

PracticeProjection log — log the moments when a tool's manner rather than its record moved your confidence: the apology, the hedge, the fluent paragraph, the fast reply.

After AIThe prediction was made when the state of the art was a text-to-speech engine, and it is now the ordinary condition of the work: systems that speak fluently recruit the analogical and affective routes directly, so trust is set by manner and the record never gets consulted.

Because385 consultants on an eighteen-subtask exercise scored 29.9% and 33.9% higher on quality over a control mean of 4.38/10, completed 91% and 93% of subtasks against 82%, and finished 22.5% and 27.6% faster, with the effects holding across all four task categories and surviving controls for spelling and grammar plus two replications by professional design judges

LimitsQuality is a rubric score, and the authors concede subjective metrics may carry evaluation bias. BCG's incentives make speed nearly free once work is delivered on time, so the same tool could produce a different sign where accuracy trades against speed. Participants were early in their careers.

PracticeTake the gain where it is real: on generative, well-bounded work inside a frontier you have checked, the improvement is large enough that refusing the tool is a cost.

After AIGPT-4 as of April 2023 was superseded within a year, and reasoning models pulled arithmetic inside the wall without making it straighter; the size of the gain has moved and the shape of the result has not.

Becausethe group given a prompt-engineering overview scored 60.0% correct against 70.6% for the treated group without it, significant only at the 10% level, while the same overview improved quality inside the frontier

LimitsTen per cent significance on one task in one experiment; the direction is consistent across specifications but the effect is a single contrast. The mechanism — that fluency in prompting raises trust without raising discrimination — is inferred.

PracticeSpecification discipline — treat training in how to use the tool as raising reliance, not judgment, and pair it with an explicit rule about where not to use it.

After AIPrompt training is now standard corporate onboarding for these tools, and it is the one intervention this experiment measured as harmful in the case that mattered.

Becausedominance was manipulated by four cues — phrasing, a confidence number, speaking order and a name — with the content of the advice held identical, and matched users rated the same advice more intelligent and useful

LimitsForty-eight participants selected from the extremes of a dominance questionnaire, which inflates the contrast. Nothing tells you how large the effect is against a real difference in advice quality.

PracticeWhen an answer reads as unusually insightful, check whether anything in it is new, or only the register it arrived in.

After AIStyle matching is no longer four cues written in a couple of days; it is what a model does by default, on every turn, having been tuned on what raters preferred. The persuasion Reeves and Nass had to construct now arrives for free.

Becausethe evidence is his own examples — interchangeable marketing phrases for a plain slogan request and usable jokes when the role is a comedian — offered as a working analogy, with the sin of anthropomorphising conceded

LimitsThe same suggestibility is the failure mode: telling it the task matters to your career improves the answer, which means the answer is responding to something other than the task. Mollick's own deflation of prompt engineering aged well, as models got better at inferring intent.

PracticeWrite the role into the specification explicitly and keep it in version control with the rest of it, instead of discovering it by feel each time.

After AIWritten about AI, and the part that has shifted is where the role lives: system prompts and product personas now set it before the user arrives, so the flourish has become a configuration nobody in the conversation can see.

Becausethe illustration is two otter-in-a-hat images a year apart, an extrapolation rather than a measurement, and the book's own examples dated within months of publication

LimitsNarayanan and Kapoor answer that there are rungs and no thresholds, and Schaeffer's work argues that the apparent jumps are artefacts of the metric. Nothing guarantees the rate continues, and the rule as stated cannot be falsified by any single observation.

PracticeDate every judgement you record about what the tool cannot do, and re-run the test rather than citing the old verdict.

After AIWritten about AI. Reasoning models arriving from late 2024 moved much of what he listed as outside the wall inside it within months, which vindicated the rule and destroyed the examples that argued for it in the same stroke.

Becausethe image is built from examples rather than measured — a sonnet inside the wall and a poem of exactly fifty words outside it, the same model writing a working tic-tac-toe page in one pass and misplaying a single board

LimitsWalking the frontier only teaches you where it is if you find out when you were wrong, and in the settings where prediction fails the feedback never arrives. His own examples dated within months: reasoning models moved arithmetic and multi-step logic inside the wall in late 2024.

PracticeKeep a written list, per domain you work in, of what the tool has actually got right and wrong; nobody else's list transfers to your work.

After AIWritten about AI, so the question is what has changed since: the wall moved a long way and stayed exactly as jagged, which is the part of the claim that has held. The metaphor outlived every example used to illustrate it.

Becausethe ladder from the Antikythera mechanism to chatbots is a reading of history rather than a prediction that can fail, and it is contested by everyone who takes emergence at face value

LimitsA history with no thresholds so far does not establish that none exists, which is the objection the argument cannot answer. Wei's emergence results read the same record the other way, and Rosenblatt's perceptron being oversold in 1958 says nothing about 2026.

PracticeWhen a release is described as a threshold, ask what kind of work it eliminated; that answer is available, and the threshold claim is not.

After AIWritten about AI, and the test arrived quickly: models trained to reason step by step, and then agents, were both absorbed as rungs rather than thresholds — useful, and failing in exactly the way the authors named.

Becausethe case is the analogy of a world with only the word vehicle, plus three separate verdicts the book then defends at length; it is a proposal about vocabulary rather than a finding

LimitsThere is no consistent definition of AI and the authors decline to supply one, so the three-way split is a useful partition rather than a principled one. Systems that combine prediction with generation sit across the lines.

PracticeBefore accepting or rejecting a claim about AI, name which of the three it is about; if the claim does not survive the question, it was never about anything.

After AIWritten about AI, and the distinction has largely been won: separating predictive from generative AI is now standard in policy discussion. What the book did not anticipate is generative systems being sold into the predictive settings its first verdict rules out.

Becausethe case is one worked example — the road whose route is a tangle of geographic, financial, economic and political matter, with technique available only once the route is settled.

LimitsNo study shows that problem setting decides outcomes. Simon's science of design is dismissed for assuming ends already fixed, an objection about scope rather than a finding. Where a situation is a recurrent case type, the old account fits it.

PracticeSpecification discipline — before asking for a solution, write down what you are treating as the problem and what you left outside the frame. That sentence, not the request, is what to check first.

After AIA model solves whatever it is handed and never asks whether that was the problem worth setting. The step Schön said had no technique is the one still not automated.

Becausegenerating is reduced to a yes-or-no classification problem, and the generative error rate comes out at least twice the misclassification rate of the classifier the model itself induces, less the ratio of right answers to wrong ones and a calibration term.

LimitsIt is a worst-case lower bound on a base model. It says nothing about the rate a deployed model reaches after post-training and with a search tool, and the argument is a proof rather than a measurement.

PracticeStop waiting for the errors to be trained away. Where you need answers that are never invented, the system has to be allowed to answer from a store and refuse the rest.

After AIThe result cuts both ways against the usual talk. Hallucination is not a mysterious defect of neural networks, and it is not a quirk of predicting the next word either — nothing in the proof mentions words. It is the price of estimating a distribution at all.

Becauseten independent trials on one model, with Meta AI and Claude 3.7 Sonnet among the assistants returning the larger numbers, against the standard result that if the best classifier a model family can express still gets a case wrong, generation in that family errs at least as much.

LimitsA handful of trials on a handful of assistants, on a task nobody needs done. The reasoning model's success shows the limit belongs to a method rather than to the technology, so the example dates quickly.

PracticeKeep a list of the questions your tool cannot see the input for, and route them elsewhere rather than rephrasing. When a failure is in the representation, a better prompt buys nothing and asking for the work to be shown step by step sometimes buys everything.

After AIIt is the cleanest case of a general point the paper makes elsewhere in theorems: some errors are in the corpus, and some are in what the model is built to see. Only the first kind responds to more data, and from outside the two look exactly alike.

Becausethe bound subtracts from that fraction only a term shrinking as the number of wrong answers grows, a term shrinking with the square root of the training size, and the calibration gap; a matching upper bound is reached by an efficient algorithm, so the floor can be met rather than merely approached.

LimitsIt holds for facts fixed by nothing but the record — no rule, no pattern to learn. It is a statement about the base model, before post-training or retrieval, and the singleton rate of any real corpus is not something an outsider can compute.

PracticeBefore trusting a factual answer, ask how often the fact would have been written down. Einstein's birthday is everywhere; the one recorded once in an obituary is a throw of a 365-sided die, and the model's manner is the same either way.

After AIThe estimator is Alan Turing's, built during the war to guess how much of a distribution a sample had missed. It now explains the shape of the frontier people keep describing as jagged: the boundary follows how often something was written down, and nothing on the surface of an answer marks which side you are on.

Becausethe accounting is an argument in a table, not a measurement — stepping through the model by hand holds at low dimension and is lost at high, while a compact network can pass, and reading a weight as a strength of association survives only until the features are engineered

LimitsLipton concedes that linear models have the better record for studying the natural world and knows no theoretical reason why. Rudin argued in 2019 that high-stakes decisions call for abandoning black boxes rather than explaining them, and his reply — that transparency can work against the broader objective — is two sentences, not a case.

PracticeRefuse the trade as it is usually offered. Ask which property the simpler model buys you and at what size it stops holding — a rule list long enough, or a tree deep enough, buys nothing at all.

After AIThe choice he was arguing about has gone. Nobody proposes a linear model where a large one is wanted, and the work that took his first sense seriously reads circuits out of trained weights instead.

Becausethe five are read out of the published literature rather than derived, each with its own case — trust split three ways, causality as regressions read as hypotheses, transferability as the pneumonia triage model and FICO's gameable credit features, fairness as credit, hiring and recidivism

LimitsFive is what one author found in one literature in 2016. Doshi-Velez and Kim pressed the same complaint the next year and organised it differently, adding what he left out: how to evaluate an interpretation against a real task. The list diagnoses a vocabulary rather than proving no unifying definition exists.

PracticeWhen you or a vendor asks for a model you can understand, name which of the five you want and what evidence would settle it. Trust and fair treatment need different things, and a method can supply one while doing nothing for the other.

After AIThe word survived and the discipline did not. An assistant asked to explain itself answers to all five wants at once, in a paragraph of prose, with no way to say which one was served.

Becausethe cases are arguments rather than measurements — a hiring model that must optimise productivity, ethics and legality together while the last two cannot be optimised directly, and a medical model scored on error when the reason for building it was to find causes

LimitsA diagnosis, not a finding. It treats the score as adequate wherever the objective can be written down, which few practitioners accept, and it leaves out reasons for wanting an explanation that have nothing to do with the objective, such as a duty to report how a decision was reached.

PracticeBefore asking a system to explain itself, write down what you want from it that the score does not cover. If the list is empty, the score is the answer and an explanation adds nothing to it.

After AIAlmost everything now asked of an assistant — be useful, be honest, be safe — is an objective nobody can score, so by this argument the demand has no ceiling.

Becauseit is a reading of how the decade before 2015 went, not a proof that there is no third way. Autor's illustration of the second route is the chair: specify legs, arms, seat and back and you exclude backless chairs; relax the requirements and you admit small tables; an algorithm given enough labelled examples infers which attributes make an object likely to be called one, with no model of chairness at all.

LimitsAutor left the outcome open and expected the second route to stay uneven: in 2015 the tools were sometimes uncannily accurate and usually so-so. Some researchers he cites expected brute force to pass human performance as data grew; others expected it to be right on average while missing the informative exceptions.

PracticeSpecification discipline — when a tool fails, ask which route it is on. On the first, the fix is a more regular setting; on the second, better examples, and no amount of instruction substitutes.

After AIThe second route moved far faster than the paper allows for, through exactly the door Autor identified, while the first advanced roughly as predicted. The distinction is more useful now than the forecast attached to it.

Because*routine* is a definition Autor, Frank Levy and Richard Murnane chose — codifiable rather than dull — and the limit on it is Michael Polanyi's 1966 remark that we know more than we can tell; the support is the Moravec inversion, that formal reasoning was designed deliberately and is easy to encode while perception, dexterity and speech were evolved, so encoding them means recovering rules nobody ever held.

LimitsThe rule held for programmed computers and stopped holding for systems trained on examples, which is Autor's own next section. Writing a persuasive paragraph is his example of a task done on unstated understanding; within seven years, machines were doing it on demand.

PracticeThe stays-human list — sort your work by whether you could write the steps down for someone else, not by how hard it feels.

After AIThe sorting principle survives as advice even though its prediction failed. What a model does well no longer tracks codifiability, so the boundary has to be found by testing rather than by reasoning about the task, and you cannot tell from a task's description which side of it you are on.

Becauseshe argues from the structure of the arrangement rather than from any measurement that the monitor was given an impossible task

LimitsIt is an argument, not a finding; the paper offers no study in which monitors were shown to be unable to adjudicate. Where the machine's decision can be re-derived offline, the real-time objection does not apply.

PracticeBefore appointing a human check on an automated decision, write down the criterion by which the check would overrule it. If you cannot, the sign-off is a formality.

After AIThe mismatch is now routine: a model produces a page of reasoning in a second and the reviewer is asked to say whether it is right. Bansal and Buçinca measured what Bainbridge predicted, and found the sign-off satisfied in form and empty in substance.

Becausethe vigilance literature from Mackworth 1950 puts a measured ceiling on how long attention holds on a quiet source

LimitsThe half hour is a rule of thumb from laboratory watch tasks and Bainbridge uses it as an order of magnitude rather than a constant. Where the output is an artefact that can be inspected at leisure the argument changes shape into a question about attention spread across many artefacts.

PracticeStop writing review duties that consist of watching for something that almost never happens; build the alarm instead, and then ask who watches the alarm.

After AIReviewing model output is the vigilance task at scale: long stretches in which the suggestion is fine, and the rare wrong one arriving with the same surface as the rest. The log that was supposed to force attention is now the diff nobody reads.

Becausemanual operators are observed arriving a quarter to half an hour before a shift to get the feel of the process, and an operator taking over has had no such run-in

LimitsThe observation is of process-plant practice rather than a controlled study of working storage, and the quarter to half an hour is her report of what operators do, not a measured build time.

PracticeBudget the run-in. Anything you intend to take over you have to have been following, and if you have not been following it, do the cheap thing first and think before acting.

After AIReading a model's output cold is exactly this position: the decisions that built it were made without you, and the state you would need to judge them was never assembled. The half hour of run-in has no equivalent in a chat window.

Becausethe Norman–Reason taxonomy as revised in 2013 is built on classified cases — the capture slip, the description-similarity slip, the mode error — and predicts where each kind will appear

LimitsThe taxonomy classifies rather than predicts rates, and Norman notes that rule-based mistakes are hard to catch because the evidence usually fits the wrong classification as well as the right one, with hindsight making the signal look obvious afterwards.

PracticeSort your errors before you fix them. A slip is fixed by changing the thing's shape; a mistake is fixed by changing the model the person is working from.

After AIExpert slips are the failure mode of expert review: the reader who knows the domain runs without conscious attention, which is exactly the condition in which the practised sequence takes over. Capture slips are what accepting a plausible wrong suggestion looks like.

BecauseNickerson and Adams 1979 found that fewer than half of American college students could pick the real penny from a set of drawings, and it has never stopped anyone spending money

LimitsThe finding depends on the options actually present. Norman's own counter-cases — the ten-franc coin, the Susan B. Anthony dollar — show that when the world's constraints change, imprecise knowledge stops working at once.

PracticeDo not measure what someone knows; measure whether the knowledge they need is available where the work happens. Then ask what happens when you move it.

After AIDelegating to a model moves more knowledge out of the head than any previous tool, and the penny result says that is usually fine. The exception Norman names is when the options change, and with a model they change with every version.

Becausethe authors list the assumptions themselves: two comparable runs at full scale, a power law assumed where the data show concavity at the highest budgets, and every run seeing less than one epoch of text.

LimitsThe gap is a feature of what was affordable rather than a flaw in the reasoning, and the one confirmation was decisive on the quantity that mattered; the authors state the assumptions rather than hiding them.

PracticeRecord what a forecast rests on at the time you make it, including the gap between where you measured and where you are predicting. An extrapolation over four orders of magnitude with one checkpoint should be logged as such.

After AIScaling predictions are still made this way, with many cheap runs and one expensive check, and they now set capital allocation. The structure of the evidence has not changed with the size of the bet.

Becausea 2024 reconstruction of the parametric fit from the published figures found that the paper's parameters do not reproduce its own points and that the confidence intervals were implausibly narrow, and the corrected fit moves that approach onto roughly equal exponents, in line with the other two.

LimitsThe reconstruction worked from published figures rather than the original data, and its correction moves the third approach into agreement with the other two, so the audit strengthens the paper's conclusion while discrediting one route to it.

PracticeLog which part of a result you are relying on, not the result. Here the headline survived an audit that the most convenient component failed, and only someone who had written down which was which could tell.

After AIThe group that had just corrected somebody else's measurement made a measurement error of its own in the same paper. Auditing is not a stage that ends.

Becauseon the six emergent BIG-Bench tasks considered, cross-entropy loss falls steadily through exactly the range of scales where exact match, BLEU and accuracy sit at random.

LimitsFalling loss under a flat score establishes that something improved, not that the improvement was the ability in question; the dispute since is over whether a smoothly improving quantity underneath makes the step unreal.

PracticeKeep a continuous measure beside any pass-or-fail score you track, including of your own judgement. A flat record of outcomes can sit on top of steady movement that the scoring rule cannot show.

After AIThe evidence that dismantled the paper's headline was printed in the paper's own appendix, by its own authors. That is a good standard to hold model evaluations to, and a rare one.

Becausea few hundred test items put the smallest observable non-zero accuracy at a few thousandths, and with additional test data generated every model in the family, including the 350-million-parameter one, scored above chance on both tasks and improved smoothly even under accuracy.

LimitsDemonstrated on two arithmetic tasks in one open model family; the larger models whose curves are actually in dispute could not be rescored, because the outputs were not available.

PracticeSize your test set to the effect you are trying to see. A zero on a small sample is a statement about the sample, and the same applies to your own record of how often a thing goes wrong.

After AIA model that gets your task wrong every time may be improving steadily and invisibly. Whether you would notice depends on how many times you have tried, which is usually not many.

Becausethe arithmetic is BIG-Bench's own and is stated in the paper's implications for practice, alongside the recommendation to size the test set to the metric's effect on the per-token error rate.

LimitsAn argument about the search space rather than a demonstration that a particular claimed ability is a false positive, and most such claims are reported without any correction for it either way.

PracticeCount the comparisons that were available before treating a striking result as a finding, in benchmarks and in your own logs. A surprise drawn from a million cells is not a surprise.

After AIModel evaluation now runs hundreds of benchmarks against every release, and the striking numbers are selected for publication. The multiplicity is larger than it was when this was written.

Becauselengthening the target degrades accuracy approximately geometrically and token edit distance approximately linearly, exactly as each metric's algebra requires, and both held across targets of one to five digits.

LimitsThe prediction concerns the behaviour of metrics rather than of models, so it confirms the algebra rather than the claim that the observed instances of emergence were artefacts.

PracticePrefer the criticism that makes a prediction. An objection that says only that a result could be an artefact is worth less than one that says what else should follow if it is, and then checks.

After AIThis is what a well-formed challenge to a capability claim looks like, and it is the shape to imitate when a vendor's benchmark result does not match what you see in use.

Becausethe authors state the boundary explicitly and name two rival accounts rather than dismissing them — a real break between two power laws, and a real effect under particular assumptions about the training data.

LimitsContested because the paper's careful scope and its popular reading have come apart: it is widely cited as having refuted emergence, which is a claim the authors decline to make.

PracticeState the scope of your own objection when you raise one, and say what would still be true if you are right. It is the difference between a correction that gets absorbed and one that gets fought.

After AILater work sampling models thousands of times per item finds most tasks predictable from smaller models, as this account requires, and a few that survive with an acceleration, as the rival account requires. Both halves held.

Becauseshallow autoencoders on CIFAR-100 acquire a sharp ability once the score is the fraction of images below a reconstruction threshold of 0.003 or 0.004, Omniglot transformers show it when scored on all three images at once, and a LeNet on MNIST shows it when accuracy is redefined as six, eight or ten images correct — sharper still under sparse sampling of model sizes.

LimitsInducing an artefact shows that the artefact is available, not that a particular reported case is one; the constructive demonstration is deliberately the strongest form of the authors' weakest claim.

PracticeTreat a sharp curve as a claim about measurement until someone shows it survives a change of metric. If the effect can be induced on demand in a network nobody thinks is intelligent, its appearance proves nothing on its own.

After AINothing about the mechanism requires a language model, which is the cleanest way to see that the argument is about rulers. It travels to any pass-or-fail measure of a system that is improving continuously.

Becausethe algebra is explicit — a per-token probability of 0.6 scores 0.08 across five tokens and 0.9 scores 0.59, so the underlying quantity improves by half while the score improves sevenfold, with the target length as the exponent.

LimitsThe mechanism explains how a step can be manufactured, not that every reported step was; the authors say so, and rival accounts — a genuine break between two power laws, an effect driven by the training data — remain live.

PracticeCheck the shape of a scoring rule before reading a curve drawn with it. Any all-or-nothing measure over several steps will make gradual improvement look like an event, in a model or in a person.

After AIAnyone grading an assistant's output pass-or-fail is running this metric. The system improving steadily underneath a flat success rate is the normal case, and the flat rate is what you will feel.

Becausethe benchmark's own emergence score applied across its metrics puts Multiple Choice Grade and Exact String Match behind more than 92 per cent of 137 hand-annotated claimed abilities, and rescoring LaMDA's emergent tasks under the Brier score makes the emergence disappear.

LimitsThe rescoring was possible only where output probabilities were published, so the evidence is strongest exactly where the models are open and weakest on the frontier systems the claims are made about.

PracticeAsk which metric a claimed capability appeared under, and whether it survives a change of ruler. Two scoring rules out of thirty-nine carrying a phenomenon is a fact about the rules.

After AIBenchmark scores are now the public vocabulary for what a model can do. This result says the vocabulary has two words in it that behave differently from the rest, and they are the two most used.

Becausethe Perspective classifier rates sentences as more toxic for containing identity terms or particular names, protected attributes differ by country, and men and women assess the same online exchange differently.

LimitsAn argument that no audit is complete, not that audits are worthless; measurement has improved since, and the direction of each instrument's error is often known and correctable.

PracticeTreat a passed audit as covering the categories it named and nothing else, and keep a running list of the categories nobody measured in your domain.

After AIModel evaluation is now largely automated, which means models grading models on instruments built by models. The circularity the paper named as a limitation became the industry's default method.

Becausethe edited versions of the benchmarks BERT had topped collapse its scores, and about a quarter of papers from the major conferences after 2018 cite BERT — so the effort went into beating instruments that were measuring the wrong thing.

LimitsThe collapse is demonstrated on specific benchmarks against models of that generation; the claim that the effort spent on them displaced meaning-sensitive methods is an argument about the field's attention rather than a measurement.

PracticePerturb a test before believing what it measures — reword it, reorder it, change the names. If the score moves, the score was about the cues.

After AIPerturbing a famous item is now routine, and it keeps working: theory-of-mind results collapse under trivial alterations. The lesson survived every generation of model it has been applied to.

Becauseasked for a cosine it cannot compute and told the human worked out a particular value, the graph shows the model taking that value as the target and working backwards through its knowledge that it will next multiply by five; varying the suggested number moves the written reasoning, while the square-root problem lands on eight regardless.

LimitsTwo arithmetic prompts in one model, and the contrasting case in the same section shows a problem where the suggestion had no effect — the backwards assembly depends on the model having no route to the answer.

PracticeDo not put your own answer in the prompt when you want the model's. Ask first, then compare — and where stakes justify it, commit to a position before you read the output.

After AIThis is sycophancy visible at the level of circuits rather than inferred from behaviour. It is also the strongest argument against treating a reasoning trace as a check on its own answer.

Because36 plus 59 is computed along two parallel paths, one approximate reaching *near 92* and one exact reaching *ends in 5*, and the model then reports adding the ones and carrying the one — a procedure the graph does not contain.

LimitsShown for arithmetic and particular reasoning prompts; written reasoning sometimes does match the computation, and the two cases look identical from outside.

PracticeAudit what you are relying on when you accept an explanation from a model. Asking it why it did something returns a plausible account of how such a thing is done, not a report from the process that did it.

After AIA confirming study in April 2025 found that a reasoning model given a hint usually acts on it without saying so. The explanation you are shown is an output, produced by the same machinery and under the same pressures as the answer.

Becausethe hypotheses are tested by clamping features to a negative multiple of their value in Claude 3.5 Haiku itself and asking whether the features the graph says should weaken do weaken and the output moves where the graph says — and the authors write that like any microscope, their tools are limited in what they can see.

LimitsIntervention strengths needed throughout were high relative to natural activations, and not every intervention behaved, which the authors read as having part of a circuit rather than all of it.

PracticeWrite down what you expect to see before you run the check, in code review as much as in research. Fixing the labels before the intervention is the whole of the method here, and it is cheap.

After AIExplanations of model behaviour are now a product feature. This is the standard they should be held to: a stated hypothesis, an intervention that could have failed, and a published failure rate.

Becausethe authors report satisfying insight on roughly a quarter of the prompts they tried, with obscure entities producing graphs dominated by error nodes, no handling of reasoning without a single decisive token, and nothing past about a hundred tokens of prompt.

LimitsA quarter is a hit rate for a young method on a small model, not a ceiling; it says what the instrument could see in 2025 rather than what is there to be seen.

PracticeAsk for the denominator. A set of striking case studies is a numerator, and the interesting number is how many attempts produced nothing — which here is three in four, published by the authors against themselves.

After AIDemonstrations of model behaviour, from laboratories and critics alike, are almost never accompanied by the failure rate. This paper is the counter-example worth citing when asking for one.

Becauseparticipants who heard one side of a legal case, knowing it was one side, were more confident than those who heard both

LimitsThe claim is about what confidence tracks, not that it is useless; where the environment is regular and feedback fast, coherence and accuracy coincide, which is the Kahneman–Klein qualification.

PracticeRecord your confidence before you see the outcome and score it later. That is the only way the number acquires any meaning.

After AIA model's stated confidence is built the same way and from the same material — the coherence of what it produced — and Sharma measured it holding at ninety-nine per cent across a full retraction. Two coherence meters facing each other is not a check.

Becausethe bat-and-ball problem is the measurement: more than half the students at Harvard, MIT and Princeton gave the ten-cent answer

LimitsKahneman is explicit that the two systems are characters in a story rather than parts of the brain, and that continuous vigilance against the endorsement is impossible; the remedy is situational, not personal.

PracticeBuild the check into the procedure rather than into your intention. A few seconds of effort is exactly what will not be spent unless something forces it.

After AIThe assistant now hands System 2 a complete, fluent, coherent answer, which is the format most likely to be endorsed without checking. Buçinca's forcing functions are the bat-and-ball grey font applied to a working interface.

Becauserepetition, legible fonts and rhyme all raise believability, and students made fewer bat-and-ball errors when the question was printed in a washed-out grey font

LimitsThe inputs to cognitive ease are interchangeable and untraceable afterwards, which is what makes the effect hard to correct from inside; it also makes any single demonstration hard to attribute to one cause.

PracticeDiscount fluency explicitly. When something reads well, ask what it would look like written badly, and judge that version.

After AIFluent, well-organised, confident prose is the default output format, so every answer now arrives with the ease dial turned up. The feeling of truth is being supplied by the typography of the sentence rather than by its content.

Becausethe priming studies largely failed to replicate at larger samples and ego depletion failed a multi-site replication in 2016; Kahneman has said in print he placed too much faith in underpowered studies

LimitsThe core has held: the two-system description, the heuristics and biases catalogue, prospect theory and the Kahneman–Klein conditions have all survived. The failure is confined to chapters 3 and 4.

PracticeWhen you cite a striking finding, check whether it has been replicated at scale before you build anything on it — including a finding from a book you trust.

After AIThe same error is the standing risk in reading results about models: a striking number from a small evaluation, reported with the confidence the book warns about. The remedy is the one the book supplies and did not apply to itself.

Becausecompetence judged from a face seen for a tenth of a second predicted about seventy per cent of a set of American election winners, most strongly among low-information voters

LimitsSubstitution is inferred from the pattern of answers rather than observed directly, and Kahneman notes that intensity matching makes the swap invisible: the answer is scaled to fit the original scale.

PracticeWrite the question down before you answer it, and check at the end that the answer you produced is an answer to that question.

After AIIs this output good is the hard question; does it read well is the easy one, and the second is what gets answered. The interface supplies the substitute for free.

Becausethe smallest gain that makes people accept an even chance of losing a hundred dollars is about two hundred, with the ratio between one and a half and two and a half across studies

LimitsThe endowment effect that follows from it does not appear for goods held for exchange or among experienced traders, so the kink is not universal across situations; prospect theory describes it as a fact about people rather than prescribing anything.

PracticeName the reference point you are judging against before you judge, and notice that changing it changes whether the same outcome is a win.

After AIA tool that changes what a day's work looks like moves everyone's reference point, and the same output is then a loss against the new one. That is most of what the complaint about work feeling cheapened is measuring.

BecauseMeehl 1954 and roughly two hundred studies since, with twenty-five wealth advisers whose pairwise consistency over eight years averaged a correlation of .01

LimitsThe finding is specific to low-validity environments; Kahneman's own Israeli army result was that a final intuitive judgment, made after disciplined scoring, added value. The formulas are also often wrong, and no study shows a clinician missing a cue an algorithm caught.

PracticeScore the components separately and in a fixed order before allowing any overall judgment, and resist the wish to invent reasons for overriding the total.

After AIThe argument for handing a judgment to a rule now has a much more capable rule on offer, and the low-validity condition that made the case has quietly dropped out of the conversation. Vaccaro's meta-analysis is the reminder that it still applies.

BecauseShanteau's 1992 lists sort occupations by whether experienced practitioners show real expertise, and nurses, physicians and auditors appear on both; a forecaster good at rain is poor at hail

LimitsShanteau's list is a review rather than a measurement of fractionation itself, and the two authors reach the conclusion from different examples — an expert meeting an anomalous case in one reading, a professional answering a question of a different kind in the other.

PracticeAudit your own competence task by task rather than domain by domain, and write down which tasks in your work you have never had feedback on.

After AIThe same fractionation holds for the model, and the two fractionations do not line up, which is the whole basis of any complementarity. Bansal's error-overlap plot is what that looks like measured.

BecauseHogarth's case is the physician who sensed which patients would develop typhoid and confirmed it by palpating tongues without washing his hands

LimitsThe category is defined by one vivid case and has no test that identifies a wicked environment before the damage is done. Human learning is efficient where cues are simple and feedback is fast, except here.

PracticeWhen feedback confirms you, check whether your own action produced the confirmation. Confidence built in a wicked environment is the most dangerous kind.

After AIAn assistant that agrees with whatever you lean towards makes the environment wicked by construction: the feedback confirming your judgement was produced by your judgement. Sharma measured the effect and Buçinca measured the cost.

Becauseit is the authors' joint position, used to explain why redundant and flimsy evidence produces the most conviction and why assertive judgments are believed more than they deserve

LimitsConfidence is not useless everywhere — where the two conditions hold, coherence and accuracy coincide — and the claim is that the feeling cannot tell you which case you are in, not that it never tracks accuracy.

PracticeNever use confidence, yours or anyone else's, as evidence. Ask about the validity of the environment and the judge's history of learning it instead.

After AIThe assistant reports a number that behaves the same way: Sharma measured it sitting at ninety-nine per cent on both sides of a complete retraction. Two parties whose confidence tracks coherence do not make a check on each other.

Becauseit is offered as a schematic reading of the evidence, with loan scoring at the low end and airport driverless trains at the high end, rather than as a measured curve

LimitsIt is a schematic and the two ends are chosen examples; nothing measures where the middle band begins and ends, and the placement of a given task is exactly the judgement the schematic cannot make for you.

PracticePlace each task on the validity range before deciding who does it. The middle band is the only place your judgement is the best available instrument.

After AIThe high end has grown and the middle band has narrowed, which is what Vaccaro's meta-analysis reports as combinations underperforming the better party. The band is still there; it is just harder to be sure you are in it.

Becausede Groot's chess studies and Chase and Simon's estimate of fifty to a hundred thousand recognisable patterns built over about a decade give the mechanism a measured base

LimitsRecognition requires something to recognise, so the claim is empty without the two conditions that follow it; and the cues are often unavailable to the expert, which is why cognitive task analysis was invented.

PracticeTreat a hunch as a report that something in the situation matched something in memory, and go looking for what matched before acting on it.

After AIDemystifying intuition is what makes the comparison with a model tractable: both are pattern completion over a stored repertoire, and the question for both is whether the patterns were learned where valid cues existed.

Becausethe two conditions are crossed against cases with known outcomes: fireground commanders and neonatal nurses in one cell, stock pickers and long-range forecasters outside it

LimitsValidity is not always obvious in advance, and Tetlock's own later tournaments softened the verdict on political forecasting once feedback was fast and the horizon short. High validity does not require certainty: bridge and poker qualify.

PracticeBefore trusting anyone's judgement, including your own, ask what the environment's regularities are and how fast the feedback on them arrives. Confidence answers neither question.

After AIThe conditions apply to the model as well as to the person, and neither partner can report on which cell they are in. Where the environment is irregular, two confident answers now arrive instead of one.

BecauseGrove and colleagues' 2000 meta-analysis of 136 comparisons, with Karelaia and Hogarth attributing most of the margin to consistency alone, and no study showing a clinician missing a strong cue an algorithm caught

LimitsThe authors reject the usual reading of this as proof of human ineptitude: the tasks were low-validity, the algorithms were often wrong too, and the margin comes from variance rather than from better cues.

PracticeWhere you cannot replace a judgement with a rule, at least make it repeatable: same inputs, same order, same scoring, written down.

After AIThe consistency argument does not transfer cleanly to a model, which is not consistent across runs and whose variability is not the kind a formula removes. What was the algorithm's advantage is now something to check for.

Becausethe efficient-market argument is used as a test of cue validity rather than as a finding about traders: a publicly available cue that predicted the price would already be in the price

LimitsThe argument is theoretical rather than empirical, and it does not distinguish a trader with private information from one without. Experience is held constant in the comparison, which is not how the two professions actually differ.

PracticeAsk whether a valid cue could survive being known. Where everyone can see it and act on it, it is not there to be recognised.

After AIThe same test applies to a model trained on public text: whatever predictive cue is in the corpus is available to everyone using it, which limits what the advantage can be in exactly the domains where advantage is most wanted.

Becauseno significant difference between on-demand, update and wait, and none between the two simple designs

LimitsA null result on a sample of 199 across six conditions is weak evidence of equivalence rather than evidence of no difference, and the three designs differ in cost — a thirty-second delay is not interchangeable with a button in practice.

PracticePick whichever forcing design fits your work. What matters is that the suggestion is withheld until you have something of your own, not how it is withheld.

After AIThat the mechanism is insensitive to the implementation is the useful part: writing your own answer first works as well as any interface feature, and it is the one version available without waiting for a vendor.

Becausea median split on a four-item scale: on the whole decision, high scorers improved significantly and low scorers not at all, while both improved on source detection

LimitsA median split on four items, in a sample of 199, on an individual difference the study did not manipulate. The authors note the result cuts both ways: low scorers did not mind the forcing designs, so stricter ones might be aimed at them.

PracticeDo not assume a review practice that works for you will work for the team. Check who it is helping before making it policy.

After AIAn intervention that widens an existing gap is the standing risk for every remedy in this area, and this is the only paper in the set that audited its own for it.

Becausethe authors state the limit themselves and rest the transfer on the claim that experts fall into heuristic thinking too

LimitsKlein's tradition would predict the opposite: an expert who notices a case is anomalous and switches to deliberate search is doing exactly what the forcing function is meant to induce. Whether that protection survives a confident suggestion is untested.

PracticeBefore applying a crowd-worker result to your own domain, ask whether the study's participants had any of the recognition the task rewards. Usually they had none.

After AIThe crowd-worker studies measure the floor rather than the decay, which is the one thing that separates them from Bainbridge's operator. What happens to an expert whose recognition is no longer exercised is the question none of them asks.

Because199 participants online with a simulated assistant right three times in four: following a wrong suggestion fell from 64 to 48 per cent and correct source detection rose from 8 to 27, with trust and preference correlating negatively with accuracy

LimitsForcing reduced the following of wrong suggestions without removing it, and did not change the rate of people's own unassisted errors. The authors also note that a design people dislike may simply not be adopted.

PracticePut a forcing function between yourself and the suggestion — a first answer of your own, a button, a delay — and do not let your preference for the smoother design decide.

After AIThe trade-off has not gone away: every intervention that reduces over-reliance makes the interface slower or harder, people still rate the easy designs higher, and the systems in wide use have kept the easy designs.

Becauseit is the paper's proposed mechanism, argued from effort aversion and the two-system account rather than measured directly

LimitsThe mechanism is inferred from the pattern of results, not observed; Vasconcelos and colleagues showed in 2023 that explanations do reduce over-reliance where checking is cheaper than deferring, which recasts it as a cost problem rather than a motivation one.

PracticeStop investing in better explanations and start investing in whether the explanation gets read at all. The second is the binding constraint.

After AIWritten for feature-based explanations of a classifier, the claim now covers chains of reasoning that are longer, more fluent and more persuasive than the answers they justify — a stronger competence signal and a more expensive thing to examine.

Becausetrust and preference correlated negatively with accuracy on wrong suggestions and mental demand correlated positively, the trade-off the authors had predicted in advance

LimitsCorrelations across conditions in a within-subjects design, with the ratings taken immediately after each block; they do not show that discomfort causes accuracy, only that the two travelled together here. Cognitive load theory predicts the opposite in learning settings.

PracticeTreat your own preference between two tools as evidence about their comfort, not their value, and check the uncomfortable one on the cases where it matters.

After AIThe preference is what determines which designs get shipped, so the finding predicts its own fate: the interfaces in wide use are the ones people rated highest and performed worst under.

Becauseit is a methodological inference the authors draw from their own design rather than something they tested

LimitsAn inference, not a test; nobody has compared the same explanation under a proxy task and a real decision to confirm the gap is caused by the forcing rather than by the difference in stakes.

PracticeDiscount any result about explanations that was measured by asking people to predict the model's output. The measurement changed the behaviour it measured.

After AIThe point generalises past explanations: any evaluation that makes people attend closely to model output measures a condition that ordinary use does not create.

Becauseon the questions where the assistant erred, correct source detection was 8 per cent under simple explanations against 49 per cent unassisted, and 3 per cent against 18 for the whole decision

LimitsOne non-critical nutrition task, one online population, a simulated assistant right three times in four, and 199 participants; the unassisted comparison is between designs rather than within participants.

PracticeMeasure assistance on the subset where it is wrong, not on the average. The average hides the only number that matters.

After AIThe condition — an assistant that is right three times in four with a plausible explanation each time — describes ordinary use. The result says that on the quarter where it is wrong, having it is worse than not having it.

Because266 misconceptions in eight difficulty levels, with the sycophantic reply made maximally persuasive by keeping the best of four thousand samples: chosen over a bare correction 95 per cent of the time and over a helpful correction 45 per cent on the hardest items

LimitsThe sycophantic reply was optimised over four thousand samples and the corrections were not, so the comparison is between a best-of-N falsehood and an ordinary truth.

PracticeAsk for the reasoning rather than the verdict, and prefer a correction that explains itself over one that merely asserts — the gap between the two is where the scorer stops choosing falsehood.

After AIThe helpful correction wins most of the time, which is the practical instruction: explanation is what separates a truthful answer from one that loses to a good argument for a falsehood.

Becausefifteen thousand preference pairs labelled on twenty-three features, with a Bayesian logistic regression predicting held-out human choices 71 per cent of the time and each feature moving the odds by at most about six points

LimitsThe features were labelled by a language model rather than by people, the effects are small, and the ranking shifts under sensitivity analysis; the authors claim only that agreement is consistently near the top, not that it is first.

PracticeWhen you are the labeller — rating, choosing between drafts, accepting a suggestion — notice that agreeing with you is one of the things you are rewarding.

After AIThe finding turns a problem of training into a problem of judgement. The pressure comes from what people chose, so every downstream mitigation is working against the raters rather than against the model.

Becausefive labellers per pair, without internet access, judging beliefs not their own, preferred the helpful correction on average but less reliably as difficulty rose

LimitsLabellers working without internet access on beliefs not their own are in a harder position than a rater in their own domain, so the result is about non-expert labelling rather than about human judgement in general.

PracticeDo not accept a rating task in a domain you cannot check. Where you cannot tell a good argument from a true one, your rating is training something on the wrong signal.

After AIIt is the reason more of the same feedback will not fix the behaviour, and the reason the proposed remedies all involve assisting or superseding unaided ratings rather than collecting more of them.

Becausereported confidence sat near 99 per cent before and after, and restricting the analysis to answers held at 95 per cent confidence or more changed nothing

LimitsReported confidence is a generated token rather than an internal state, so the finding is about what the system says rather than about anything it has; whether a better-calibrated report is achievable is a separate question the paper does not settle.

PracticeDo not read a stated confidence number as a report on the answer's quality. Test the answer instead, by a route that does not run through the system that produced it.

After AIKahneman and Klein's claim about human confidence turns out to describe the machine exactly, which removes the last reason to treat the two parties' confidence as independent checks on each other.

Becausebeer reviews and logic questions contained items the model handled and people did not, and book reviews did not; books was the one dataset where explaining the predicted class helped when the model was wrong, by five per cent over confidence alone

LimitsThe overlap can be plotted only after the fact, from items already answered, which makes it a diagnosis rather than a selection criterion; Kahneman and Klein locate the same asymmetry in the environment and the learning history, which can at least be assessed in advance.

PracticeAsk which cases the tool handles that you do not, rather than how good it is overall. If the answer is none, matching scores buys you nothing.

After AIThis is the question to ask of any tool before adopting it, and the one that is never answered: not whether it is good, but whether its errors are different from yours.

Becausesplitting items by whether the machine was right shows accuracy rising where it was correct and falling where it was wrong, visible already in the fifty-worker pilot on beer reviews

LimitsMeasured on accuracy alone, in crowdsourceable tasks, with two explanation formats. What it establishes is that the net effect was nothing; the mechanism behind the two movements is inferred rather than observed.

PracticeRead the average effect of an explanation as two effects that cancel, and ask for the split by whether the recommendation was correct before believing either.

After AIExplanations have grown much longer and more persuasive since, which by this finding raises acceptance on both sides of the split without changing the balance. The measurement that would show it is still the one nobody runs.

Becausethe authors propose the remedies in their closing section — delaying the recommendation, showing it after a first answer, or offering it only on disagreement — without testing any of them

LimitsBansal proposes the de-anchoring designs and does not build them; Buçinca built and measured three and found them equivalent, so between the two papers there is a tested remedy and an untested diagnosis for the same problem.

PracticeWrite your own answer, even a bad one, before you look at the machine's. It costs a minute and it is the only remedy that does not depend on the interface.

After AIEvery interface in wide use shows the answer first, and the speed that makes it attractive is exactly what each remedy costs. That is why the tested remedy has not been adopted.

Because1,626 crowd workers on three tasks; beer reviews at 0.89 for the confidence-shown team against 0.84 for the model and 0.82 unassisted, with the same pattern on books and logic questions

LimitsThe matching was done by construction — items selected so the model scored near the workers' pilot accuracy — which Vaccaro's 2024 meta-analysis of 106 experiments suggests is a narrow condition rather than a typical one.

PracticeWhere you want a person's judgement to add something, choose the tasks on which the tool is roughly as good as they are, not the ones on which it is much better.

After AIRead against the later meta-analysis, this result looks less like a finding about teaming and more like a demonstration of the conditions under which teaming pays at all, and those conditions have to be engineered.

Because0.89 for the confidence team against 0.88 for the explained team on beer reviews, a difference indistinguishable from noise, with three of four hypotheses unsupported and hand-written explanations making no difference

LimitsOnly two explanation formats were tried, highlights and prose, on crowdsourceable tasks with accuracy as the only measure. The one exception is instructive: on book reviews, explaining the predicted class helped by five per cent when the model was wrong.

PracticePrint the number. Before buying an explainability feature, check it against the baseline of showing how confident the system is and nothing else.

After AIThe baseline this paper established is the one every later explanation has had to clear, and the formats that have arrived since — chains of reasoning, citations, self-critique — are longer and more convincing rather than more verifiable.

Becauseagreement fell on low-confidence items and held on high-confidence ones, with no gain on any of the three tasks

LimitsOne adaptive scheme, thresholded at each dataset's median, on three tasks. The split was fortunate — seven of the model's errors fell on the side shown with the alternative explained — which if anything favoured the design.

PracticeDo not treat inducing appropriate doubt as the goal. Doubting the machine does not supply the answer, and the confident mistakes are the ones you needed to catch.

After AIIt is the cleanest statement of why calibrated trust is not enough: on the confident errors you still have to spot the mistake, and on the doubtful ones you still have to do the work yourself.

Because409 coded free-text responses: of the ninety workers who mentioned confidence, 27 per cent named an explicit cut-off, around 83 for beer, 89 for books and 90 for logic, and the same share mentioned it with or without explanations

LimitsSelf-reported strategy from free text, coded after the fact; people's accounts of how they decided are not reliable, and the cut-offs they named were not verified against their actual behaviour on the items.

PracticeFind out what signal you are actually using to decide whether to trust an output, and check whether that signal has ever been validated against being right.

After AIModern assistants mostly do not print a confidence number, which by this finding removes the one feature people were building a model of and leaves fluency in its place.

Becausefour reasons are offered in the general discussion and only the last is backed by the paper's own data, the regrading phase in Study 3.

LimitsThree of the four reasons are asserted rather than tested, and domains with fast scored feedback — forecasting, trading, competitive games — are standing counter-examples.

PracticeBuild the feedback channel deliberately, because the environment will not supply it: a review that names errors, a record of outcomes, someone willing to tell you the unflattering thing.

After AIAn assistant is a fourth channel and the most agreeable one. It withholds bad news by construction, having been trained on what raters preferred to hear.

Becausefour independent samples of Cornell undergraduates, with panel reliability of .76 on the humour criterion and published test items for the other two, and bottom-quarter students placing themselves below top-quarter students in every study.

LimitsKrueger and Mueller showed in 2002 that regression to the mean plus the better-than-average effect produce crossing lines of this shape without any metacognitive deficit, and Burson and colleagues showed the direction depends on task difficulty.

PracticeRead your own percentile estimates as compressed rather than wrong in rank. If you think you are somewhat above average at something you have not measured, the estimate carries almost no information either way.

After AIThe popular version of this finding — that incompetent people think they are brilliant — is not what the data say, and it is the version most often used to dismiss somebody else. The compression applies to the person using it.

Becausea raw-score estimate is absolute and so untouched by any error about other people, which removes the false-consensus explanation from this half of the result.

LimitsAbsolute estimates are still measured with error and still subject to regression, though not to any mistake about the peer distribution; the sample is undergraduates on short classroom tests.

PracticeAsk yourself how many you got right, not how you compare. The absolute question is the one that can be checked afterwards, and it is the one the error survives.

After AIWith an assistant in the loop the absolute question is the only one left: how much of what came back was right. The comparison with other people has stopped being the useful measure.

Becausethe mediation is the one measure regression to the mean cannot reach, since the skill was altered after the score was fixed, and the training's effect on self-assessment vanishes once metacognitive ability is held constant.

LimitsA mediation analysis on one training manipulation, which establishes the path in this design rather than in general; Fleming's dissociation results are the standing objection.

PracticeKeep hours on the work you have delegated, on the argument that your ability to judge the delegate's output is carried by your own skill and decays with it.

After AIThis is the load-bearing claim under manual practice. If judging the output is carried by being able to do the work, then the hours are not nostalgia — they are what keeps the review real.

Becauseperformance predicted metacognitive skill at .75, and metacognitive deficit predicted inflated self-assessment on all three measures with objective performance held constant, with betas from −.40 to −.49.

LimitsFleming's evidence is that metacognitive sensitivity dissociates from first-order accuracy and has its own substrate, which would allow calibration to be trained separately; Study 4 raised both at once and cannot separate the readings.

PracticeKeep a written record of your predictions and their outcomes in any domain where you are new. The feeling of having judged well is produced by the same skill you are missing, so the log has to come from outside it.

After AIA model will answer in any domain, including the ones where you cannot grade it. The deficit here is about lacking the signal that you are wrong, and delegation removes the practice that would have supplied it.

Becausegrading correlations were .37 against .66 and summed absolute error 17.4 against 9.2, and the bottom group's self-estimate moved from the 66.8th percentile to the 63.2nd after reading five better tests.

LimitsOne regrading phase on a logic test; the top group's estimates did move on seeing peers' work, so exposure is informative for those who can read it, which is the point rather than a limitation.

PracticeDo not rely on reading good examples as a way to learn a skill you cannot yet perform. Map where you actually sit, and use a model or a mentor as a tutor who explains, not as an exhibit you are meant to absorb.

After AIAssistants are used as exhibits constantly — read the output, learn the pattern. This says the reading requires the competence the reading was supposed to build, which is why tutor-style use beats generator-style use for anyone learning.

Because70 of 140 participants were randomly assigned to a training packet, and the untrained bottom quarter moved the wrong way over the same interval, from 5.8 to 6.3.

LimitsA single short training on one task, with the skill altered after the score was fixed; Tetlock's forecasters improve calibration while the world stays as hard as it was, which suggests a second route this design cannot see.

PracticeWhen your judgement of your own work is poor, train the work. Ten minutes of instruction moved both the skill and the self-estimate here, and nothing addressed at the self-estimate alone has done as well.

After AIIf competence is the route, then the delegation that removes the practice removes the calibration too. That is the mechanism behind every finding in this book about reliance and self-estimate drifting apart.

Becausethe scoping is the authors' own, argued from examples rather than tested, with a second boundary at the floor, since nobody overrates their ability to translate Slovenian proverbs.

LimitsAn argument from examples in the general discussion with no test, and the boundary cases are the authors' own choice; a footnote sets aside the generally incompetent person as no part of the research.

PracticeSort your domains by whether the skill and the judgement are the same thing. Where they come apart — physical skills, skills with fast unambiguous feedback — you can trust your self-estimate further.

After AIMost knowledge work sits on the wrong side of this boundary, which is where the deficit bites. The work where you would notice your own decline immediately is not the work being delegated.

BecauseKrueger and Mueller 2002 in the same journal, Burson Larrick and Klayman 2006 on task difficulty, and Nuhfer and Gignac on random numbers producing the same plot, against Ehrlinger and colleagues 2008, who paid for accuracy and recruited at a debate tournament and a gun club and still found the overestimation.

LimitsContested in both directions: the statistical critique is about the graph and the graph was never the whole argument, while the incentivised replications are smaller and fewer than the original studies.

PracticeCite the parts that survived — how well people sort their own answers into right and wrong, how badly they grade a stranger's work, what ten minutes of instruction did — and stop citing the figure.

After AIA finding this heavily used and this heavily attacked is a good calibration exercise in itself. What has held up is smaller than the popular version and harder to dismiss than the artefact version.

Becausethe top group's comparative estimate rose from the 68th to the 77.2nd percentile after grading, while its raw-score estimate went from 16.9 to 16.6 against an actual 16.4.

LimitsUndergraduate samples on classroom tasks, and the effect is a comparative error rather than a self-knowledge failure — the top group knew their own score all along.

PracticeIf you are strong at something, expect to underrate your standing and to overrate everyone else's. The correction is exposure to other people's actual work, not more introspection.

After AIAn expert who assumes a model finds the work as easy as they do is making this error with a new target. The model's competence, like a peer's, has to be observed rather than assumed from one's own.

Becausethe model's two arrows are illustrated by navigation devices and by the erosion of Inuit wayfinding, and the confidence-to-offloading arrow has since been demonstrated causally; how far the drift goes over years remains unmeasured.

LimitsThe two arrows are a model with one arrow now demonstrated; the long-run drift is the paper's central prediction and the thing nobody has measured, which is why this is an opinion rather than evidence.

PracticeKeep unaided hours on the capacities you want to retain, and schedule them rather than waiting to feel the need. The feeling is the second output of the loop, and it moves in the wrong direction.

After AIThe loop is the whole argument of this book in one diagram. What has changed is that the capacity being replaced is now judgement, and judgement is what the loop uses to decide.

BecauseDunn and Risko 2015 rotated the words alone inside an upright frame and rotated words and frame together, producing the same rotation costs and the same physiological measure, and head tilting was far commoner in the condition participants believed was harder.

LimitsDemonstrated on a perceptual manipulation where belief and demand could be separated cleanly; how far the dissociation extends to complex work is not measured.

PracticeAudit what you hand off against what it actually costs you, not against how hard it feels. The feeling is the variable doing the work, and it can be wrong while the cost is unchanged.

After AIEvery decision to ask a model is made on an estimate of what the task would cost you. That estimate is the thing this literature shows to be unreliable, and it is now consulted many times a day.

BecauseRisko and colleagues 2014 found it rising with rotation angle and array size, and intention offloading rises with the number of intentions held and with interruption, both of which lower unaided accuracy.

LimitsThe relationship is between offloading and demand in laboratory tasks with clean load manipulations; in real work, availability and habit compete with demand as predictors.

PracticeExpect your own delegation to climb exactly when the work gets hard or you get interrupted — which is when the judgement you are delegating matters most. Decide the boundary in advance rather than in the moment.

After AIThe conditions that drive offloading up — high load, interruption, fatigue — are the conditions under which you are least able to check the result. The tool arrived without changing that.

Becausesearching online about one subject raised people's self-credited knowledge, and their reported sense of brain activity, on a different subject — a transfer that only an inflated self-estimate explains.

LimitsContested: these studies arrived just before the replication decade and the strong form has not held; the surviving version is that expecting access changes what people encode and what they say they know.

PracticeLog what you can do unaided, periodically and in writing. Access to a store inflates the estimate of what is in your head, and the estimate is what you will use when deciding whether you can check the tool.

After AIThe effect was measured with a search box. A system that produces finished work in your own idiom is a stronger manipulation of the same variable, and nobody has measured it over years.

Becausein the saved-file experiments expected erasure improved recall and saving improved memory for location, which is a shift in what was stored rather than a loss of storage.

LimitsContested: independent attempts to reproduce the saving-and-forgetting result have come out mixed, and the search-engine Stroop priming that supported the strong version is now held loosely.

PracticeDecide deliberately which things you want in your head and rehearse those without the tool available, because the expectation of access is doing the encoding decision for you otherwise.

After AIWith a model the store is not even a location you could return to — the answer is regenerated, differently, each time. The weaker form of this claim is the one that transfers, and it transfers completely.

Becausethe same pattern appears in reminders set where they produce no accuracy gain and in head tilts that save nothing, so the bias is visible across three paradigms with different costs.

LimitsCeiling performance makes the gain of zero easy to establish and also makes the stakes trivial; the experiment shows a misjudgement, not a cost.

PracticePut a step between the impulse to delegate and the delegation — commit to your own answer first, then compare. The impulse is not tracking whether the help helps.

After AIMETR's developers were 19 per cent slower while believing themselves 20 per cent faster. The same bias, measured where there was something to lose, and the belief survived the experience.

Becauseexternal normalization, prospective memory, short-term memory and gesture all depend in the same way on load and on how well the person performs when offloading is prevented.

LimitsThe unification is by shared dependence on load and on unaided performance, which is weaker than showing one mechanism; the behaviours differ enormously in what they cost to reverse.

PracticeAudit your offloading as one category with many instances. The notebook, the search, the script and the model are the same move, which means what you know about one transfers to the others.

After AIA model is the widest external resource yet, and it takes on the deliberative work the earlier examples did not touch. The category holds; the range inside it has changed.

Becausethe navigation studies the review collects show all three, and the photography analogue points the same way — photographing an object impaired memory for it a day later, while photographing part of it reduced the impairment.

LimitsSingle-session studies on route memory, with the attention account and the encoding account both live; the photography result showing partial engagement reduces the loss suggests the effect is about attention rather than about the device.

PracticeSeparate the outcome you want from the capacity you want, and keep hours on the second. If you want to know a city, drive it without the device sometimes; the guided drives will not accumulate into knowledge.

After AIThis is the clearest consumer-scale instance of the pattern, and it happened without anyone deciding to accept the trade. The same shape is now available for writing, analysis and code.

Becausethese are the paper's only measurements and they are borrowed; players also rotate to find out whether a piece fits at all, not only to place it, so the action buys information as well as time.

LimitsOne task, one pair of measurements, borrowed from another laboratory; nothing here establishes that the saving generalises, and the same literature later found people offloading where the saving was zero.

PracticeAudit which of your offloads are genuinely cheaper and which only feel cheaper. Where there is a real measured saving, take it; where the saving is assumed, you are running Risko and Gilbert's biased estimate.

After AIThe timings are the honest version of the case for delegation: a measured saving on a specific operation. Most claims about assistant productivity are not that, which is what makes this a useful standard.

Becausethe pattern recurs across independent settings — trial lawyers conceding a weak point are judged more credible and win more often, Domino's advertising in 2009 that its pizza had been bad, Buffett opening annual reports with a mistake — and the chapter supplies its own counter-case in Vincent the waiter

LimitsThe evidence is field and commercial rather than controlled. Vincent shows the move is as available to a practitioner with an interest as to an honest reporter, which the chapter concedes and does not resolve.

PracticeCalibration log — record the weakness you stated alongside the confidence you claimed, and check later which was the real one.

After AIA model that opens with a caveat is not being candid; the hedge is a style feature of the output, and it buys the trust that a real admission would.

Becausea field experiment against a measured base rate of 2.95 per cent moved theft in both directions — a sign depicting three thieves nearly tripled it to 7.92 per cent, a sign depicting one cut it to 1.67 — and the reversal has been reproduced in three other prevention literatures

LimitsThe repair — a truthful trend where there is no majority — rests on a single student study. The newest example in the 2021 edition, the rise in adolescent suicides after a television series, did not survive reanalysis.

PracticeFailure-mode review — report the failure as a trend that is moving, not as a prevalence that is high.

After AITelling a team that everyone is shipping unreviewed model output is the Petrified Forest sign. The message that lands is the frequency, not the disapproval.

Becausethe position is stated throughout and dramatised rather than tested — the television bought knowingly and the shelf checked the next morning — and the two authority questions are offered as prescriptions, not as interventions with outcomes

LimitsNo defence in the book was tested. The check is recommended to the same faculty that just failed to notice the cue, which the authority chapter's own prediction gap shows to be unreliable.

PracticeFailure-mode review — in the review, check the condition the cue reported rather than re-litigating the decision it produced.

After AIThe check does not run on a cue that is a property of training rather than a report about anything. There is no shelf to go back to the next morning.

Becausethe reversals are what identify the effect — not one further shock when the researcher said stop, none when a fellow volunteer gave the orders, and every participant stopping under two contradictory researchers — but the familiar sixty-five per cent is one of twenty-four conditions running from near zero to over ninety

LimitsGina Perry's archive work found the experimenter departing from the four scripted prods and many participants doubting the shocks were real. Burger's 2006 partial replication found 70 per cent continuing against Milgram's 82.5 in the matching condition.

PracticeCognitive forcing — put a second, independent source in front of the decision, because two contradictory authorities is the condition in which everyone stopped.

After AITwo assistants given the same question and disagreeing is the cheapest available version of the condition that stopped every participant.

Becausethe original is dramatic — 95 per cent of twenty-two nurses' stations set off for the patient's room with twice the stated daily maximum of an unauthorised drug — but the 1977 rerun with a familiar drug, an identifiable physician and colleagues available produced two of eighteen

LimitsThe book does not mention the Rank and Jacobson reversal. The three conditions were varied together, so their separate contributions are unknown.

PracticeAutomation ironies audit — audit where in your workflow an instruction arrives that cannot be verified and has nobody to check it with.

After AIA model is an unknown voice issuing an unverifiable instruction to a person working alone, at any hour. The reversal names the fix: identifiability, and a colleague to consult.

Becausethe effect is largest exactly where knowledge is missing — unfamiliar customers were the ones most swayed by the popularity label, and Goldman's shopping cart sat unused from 1934 until he hired shoppers to wheel them — and it reverses under unambiguous injury, with help in 100 per cent of staged accidents

LimitsThe Genovese parable was a newspaper's construction and the book now says so. A 2019 study of real closed-circuit footage found intervention in about ninety-one per cent of public conflicts, and more bystanders making it likelier rather than less.

PracticeCognitive forcing — when you notice you are uncertain, force the check before you look at what everyone else did.

After AIUncertainty is the condition under which an assistant gets used most, which is exactly where its answer is treated as evidence about the world rather than as one more opinion.

Becausefive independent manipulations of a token each moved compliance — a uniform from 42 to 92 per cent, half an inch of estimated height per academic rank, a suit tripling jaywalkers followed into traffic, a designer label, a luxury car — and in every study participants underestimated the effect on themselves

LimitsThese are field studies from the 1970s and 1980s on small samples. What has held best is the prediction gap — students guessed 63 per cent where the uniform got 92 — rather than any single effect size.

PracticeProjection log — log which token you were responding to, since your estimate of its effect on you is the part that is reliably wrong.

After AIAn assistant's fluency and formatting are tokens that cost nothing to produce and carry no credential behind them at all.

BecauseLanger's photocopier queue gives the cleanest numbers — 60 per cent for the bare request, 94 with a real reason, 93 with an empty one — and the same rule shows up in the turquoise that sold at double price and the painkiller rated more effective at $2.50 than at ten cents

LimitsThe photocopier result holds for small requests; at higher cost people attend to the reason, which the book reports and its popular retelling drops. The turquoise and painkiller cases are single demonstrations.

PracticeCognitive forcing — make the check fire on the cue itself: ask what feature produced the assent before asking whether you agree.

After AIAn explanation attached to a model's answer is the empty because at scale, and it raises acceptance whether or not the answer is right.

Becauseof four weekly household messages, the three appealing to the environment, to responsibility and to the power bill were beaten more than threefold by a truthful statement that most neighbours were conserving, and a scanner study separates acceptance from validity by finding more conformity to four people than to four computers rated equally reliable

LimitsAllcott's 2011 evaluation of the commercial deployment found savings of 1.4 to 3.3 per cent of household electricity, far less than a threefold advantage over other messages implies.

PracticeWitness practice in review — in review, say what the team actually does rather than what it should value.

After AIWhat a team believes its peers do is now partly what a model reports that teams do. The feasibility channel does not check where the number came from.

Because123 subjects at three colleges made a mean of 4.41 errors out of twelve, 36.8 per cent of critical estimates, against 0.7 per cent among 37 controls judging individually and three errors in 738 neutral judgements.

LimitsMale American college students, tested in public, on a question with no stake attached; two thirds of judgements were independent and a quarter of subjects never yielded once.

PracticeForce your own judgement into the open before the room's is known — a written estimate, a committed position — because the effect is on the answer given, not on the answer held.

After AIAn assistant is a confident unanimous voice with no visible stake, available on every question. Asch's subjects could see that the majority was wrong, which is the condition that mostly does not hold with a model.

Becausethe individual interviews, conducted before debriefing and read from the extremes of the distribution, separate the kinds; Asch is explicit that distortion of perception, if it occurred at all, was the rarest form and cannot be established from the data.

LimitsInterview evidence from a small number of subjects at the extremes of the distribution, interpreted by the experimenter; Asch himself says the perceptual-distortion category cannot be established from his data.

PracticeIn review, watch what you are doing as you do it: holding a position, suppressing a doubt, or having no doubt at all. The behaviour is identical from outside and the three are not the same act.

After AIAccepting a model's output and agreeing with it are indistinguishable in the record you leave behind. Only attention at the time can tell them apart, and only you can supply it.

BecauseBond and Smith's 1996 meta-analysis pooled 133 studies in seventeen countries and found both patterns, while Perrin and Spencer recorded one conforming response in 396 trials with British engineering students in 1980 and large effects with unemployed West Indian men facing white majorities — which points at what the subject has to lose.

LimitsContested: the cross-national comparison confounds culture with decade and with the populations tested, and the Perrin and Spencer results point at stake rather than at nationality.

PracticeAudit the conditions rather than the people. If dissent is rare on your team, the finding to act on is what standing apart costs here, not who is compliant.

After AIWhat a person has to lose by disagreeing is the moving part, and it has not gone anywhere. Disagreeing with a system the organisation has invested in has its own cost, and nobody has measured it.

Becausethe J-shaped distribution is in the paper's own table, and Friend, Rafferty and Bramel documented in 1990 how textbook after textbook inverted the emphasis Asch insisted on.

LimitsThe split is observed in one paradigm on one kind of judgement; nothing here establishes that the independent quarter are independent elsewhere, and Asch's own interviews found three different routes to the same error count.

PracticeKeep a list of the judgements you will not delegate to a room or a tool, and treat independence as a disposition you can hold rather than a statistical exception. A quarter of Asch's subjects held it throughout.

After AIThe finding everyone cites — people conform — is not the finding. The one that matters for supervision is that resistance is common, unevenly distributed, and evidently a property of persons.

Becausethe fifteen structural properties in the closing section are offered as an argument against reading the results as suggestion or uncritical acceptance, and the conflict is characterised as irreconcilable and, unusually, unintelligible.

LimitsAn argument from the structure of the situation rather than a manipulation of it; the claim that the pressure is self-generated is an interpretation of the interviews, and Cialdini reads the same phenomenon as social proof operating on the subject.

PracticeIn review, separate what the situation contains from what you have added to it. Asch's majority did nothing but say numbers, and a third of the judgements moved anyway.

After AIA model applies no pressure at all, by construction. If you find yourself deferring to it, the deference is yours — which is the most useful thing in this monograph and the least cited.

Becausethe partner results come from Asch 1951, reported in the monograph's discussion rather than in its own experiments, and when the partner defects to the majority the errors climb back toward the level of a man alone.

LimitsThe partner effect is reported from earlier work rather than tested here, and the relevant condition is that unanimity is broken — which says nothing about whether the dissent is informative.

PracticeArrange for one dissenting voice before a decision rather than more reviewers. A single ally who breaks unanimity is the cheapest intervention in this literature, and it works even when the ally is also wrong.

After AISeveral assistants agreeing is not several opinions; they were trained on overlapping data by overlapping methods. The unanimity is manufactured upstream, and the partner effect has no analogue inside a model.

Becausemore than half of those who ever erred did so by the second critical trial and only 7 per cent first erred in the second half, and tripling the series to 36 critical trials produced 37.6 per cent with no new defectors after the eighth.

LimitsEstablished on a repetitive perceptual task where nothing changed between trials; a setting in which evidence accumulates would not be expected to behave this way.

PracticeSpend your attention on the first exchange rather than on endurance. Whatever stance you take in the first minute of a review, with a room or with a model, is the stance you will keep.

After AIThe first answer an assistant gives sets your posture for the session. Deciding once, early, whether you are checking or accepting is a larger choice than it feels like at the time.

Becauseeighteen subjects across the discrepancy range give a monotone fall, and errors also rose with the length of the standard — 30.5 per cent at three inches to 48.1 at nine — which Asch names unquestionability and cannot explain.

LimitsEighteen subjects in one experiment, and the second pattern — more errors on longer standards — is reported without explanation by the author himself.

PracticeReview the near misses rather than the disasters. An output that is slightly wrong is the one you will accept, and the size of the error is inversely related to your chance of catching it.

After AIA model's plausible-but-wrong answer is precisely the small discrepancy. The obvious failures are the safe ones, and the failure rate you experience is concentrated where your resistance is lowest.

Becausethe subjects insisted the change made no difference to them, and since all assumed the experimenter would compare their answers with the group's, the fully private case was never run.

LimitsEven the written condition was not private — every subject assumed the experimenter would compare — so 12.5 per cent is an upper bound on what privacy removes, not a measurement of the residual.

PracticeHave people write their estimate before anyone speaks. It is the same intervention as a cognitive forcing function, it costs a minute, and it removed the extreme errors here entirely.

After AICommitting to your own answer before reading the model's is the same manipulation. It is also the one users reliably dislike, which is why it has to be built into the process rather than left to intention.

Becausethe interview totals set against the trial records show the gap, and confidence tracked performance elsewhere: 87 per cent of the wholly independent said a ruler would vindicate them against 41 per cent of those who erred eight to eleven times.

LimitsRetrospective self-report against a trial record in one session; the underestimate may be a repair of a conflict the subject felt rather than a failure of memory, and the design cannot separate these.

PracticeLog how often you took the tool's answer, as a count, at the time. Your memory of how independent you were will be about a third short, and the direction of the error is known.

After AINobody keeps this count with an assistant. The one measurement we have — developers 19 per cent slower while believing themselves faster — suggests the gap is at least as large as Asch's.

Becausethe acceptance rate comes from Cursor's own logs and the review share from the hand-labelled recordings; three developers in four report reading every accepted line and more than half often rewrite it substantially.

LimitsAcceptance rates are tool-specific and move with model quality; the 9 per cent is an average over a small sample of recordings. Nothing here says review time is wasted — only that it is not free.

PracticeFailure-mode review — budget review and repair as a line item rather than assuming generated code arrives finished, and record what you had to change.

After AIAcceptance rates have risen since 2025, which moves the number and not the structure: whatever is accepted still has to be read by someone who could have written it, and that person's ability to read it is what the next claim in this chain is about.

Becausethe same people forecast 24 per cent faster before the trial and reported 20 per cent faster after months inside it, against 19 per cent slower measured; the error is a constant offset rather than noise, and it did not shrink with Cursor hours.

LimitsThe offset is measured on one kind of work by sixteen people. The developers were not bad estimators in general: their forecasts correlate with actual times at about 0.6, and only the AI term is inverted. Whether the same offset holds for greenfield work is untested.

PracticeCalibration log — record the estimate before the work and the clock afterwards, because the estimate does not repair itself through use.

After AIThe paper's method — fix the task before the coin, measure the time, and do not ask the person afterwards how it went — outlasts the model generation it was built to test. Any claim about an assistant's value that rests on how the work felt is now the weakest kind of evidence available.

Becausethirty-four economists and fifty-four machine learning researchers were given the design, the repositories and the participants' AI background in advance, and still missed by roughly twice the participants' margin.

LimitsTwo convenience samples of forecasters answering a single question. Nothing here separates a bias about AI from a general optimism about any intervention described in advance.

PracticeCalibration log — write down what you expect a tool to do for a team before it is deployed, with a date and a number, and score it; expertise in the field does not substitute for the record.

After AIThe forecasters were not lay observers but the professions that advise on AI deployment. Their miss is the reason an organisation's expectations of an assistant should be treated as a hypothesis to be scored rather than a premise to plan against.

Becausethe control group reported higher learning on a seven-point scale, the assisted group rated the task easier and the quiz no easier, and one assisted participant wrote afterwards, "I feel like I got lazy".

LimitsSelf-report on seven-point scales from twenty-six people per arm. The control group's higher reported learning could be effort justification rather than accuracy.

PracticeCalibration log — treat ease during the work as information about the interface and not about what you now know; test the knowledge separately and unaided.

After AIThe feeling of ease is the assistant's most reliable product, and it is the one thing that cannot be used to judge the assistant. That the assisted group rated the quiz no easier suggests the illusion is about the session rather than about the knowledge.

Becausethe by-question-type analysis is exploratory; the proposed mechanism is that both arms read comparable amounts of code but only the unassisted arm met and resolved errors.

LimitsNot pre-registered, and drawn from fourteen questions split across three abilities, so the sub-scores rest on a handful of items each. The direction is consistent with the error-count data but is not independent of it.

PracticeManual flying hours — take the debugging yourself even when you delegate the writing, because diagnosis is the capacity that lets you supervise what the model produced.

After AIIf it holds, the skill that decays first is the skill the supervision model depends on. An organisation that delegates the writing and keeps the reviewing is relying on a capacity that its own delegation is eroding.

BecauseChase and Simon found a master replaces two-thirds of a real position after five seconds and two or three pieces of a random one, the same as a novice.

LimitsThe chess evidence is strong and old; the generalisation from digit spans and chessboards to professional judgment is the authors', not the experiments'. Nothing here fixes how narrow a domain has to be before representations stop transferring.

PracticeDomain audit — list the domains where you actually hold representations and the ones where you are a confident novice, and set the checking effort by that list rather than by seniority.

After AIA model's competence is also uneven across domains, but not along the same seams as a person's, and neither party can feel where the other's boundary lies. The audit has to be done twice.

BecauseElio stalled near twenty digits on mnemonics invented on the fly and never passed it, while Donatelli, taught Faloon's method, stalled at thirty, built his own retrieval structure and passed a hundred; Faloon's own limit was located at a single faulty group by giving him strings ten digits too long.

LimitsCase studies of a handful of digit-span subjects, generalised to skill at large. The diagnostic move — overload the performer to find the failing component — depends on a task whose components can be isolated, which most professional work is not.

PracticeFailure-mode review — when progress stops, push past the current limit to find which component fails first, and change that component rather than adding hours.

After AIThe assistant is a new method, and on this account a new method is what breaks a plateau. That is the strongest case for delegation the chapter contains, and it holds only where the plateau is in output rather than in understanding.

Becausecells work to hold their chemistry steady and rebuild only while out of range — 112 genes switch on in an overloaded rat muscle — and in Maguire's four-year study the posterior hippocampus grew only in the forty-one trainees who qualified.

LimitsThe homeostasis account is a physiological analogy carried into skill, and the size of the increment that keeps adaptation going is never specified. Maguire's study is observational: the trainees who qualified may have differed before they began.

PracticeManual flying hours — raise the difficulty of the unassisted block as soon as it stops being uncomfortable, rather than repeating it at the level that has become easy.

After AIAn assistant's first effect is to make the work comfortable. On this account comfort is the signal that adaptation has stopped, which makes the assistant's most attractive property the one to watch.

Becausethe Berlin violinists had logged 3,420 hours by eighteen for the future teachers, 5,301 for the very good and 7,410 for the best, and doctors twenty to thirty years in do worse on objective measures than doctors three years out; the 2019 replication failed to separate the two upper groups.

LimitsMacnamara and Maitra's 2019 replication, double-blind where the original was not, found the merely good had logged as many solitary hours as the best. Their 2014 pooling of eighty-eight studies put practice at 26 per cent of variance in games and under 1 per cent in the professions; the argument is now described as settled in the middle.

PracticeManual flying hours — put the hours where the work is hard and the feedback is quick, rather than counting the hours the job supplies on its own.

After AIExposure is what a delegated workflow supplies: hours in the vicinity of the work, with the effortful part performed elsewhere. Even in its narrowed form, the claim says supervising a model is not practice at the thing supervised.

Becauseretired taxi drivers had less grey matter in the posterior hippocampus than working ones, on the same imaging measure that showed the growth.

LimitsCross-sectional comparison of retired against working drivers, with age and health as obvious confounds. The imaging finding is about volume, not about any measured decline in navigation performance.

PracticeManual flying hours — schedule the unassisted block; a skill kept only as a credential is not kept.

After AIDelegation is retirement from a task without the retirement. The decay clock starts on the day the tool takes the work, and nothing in the workflow reports it.

Becausethe four-year design makes the decline hard to explain by selection, but it is one measure in one study and has not been replicated.

LimitsA single delayed-recall measure in a study designed for something else. The mechanism — that the posterior hippocampus grows at the anterior's expense — is inferred from the same imaging data rather than tested.

PracticeThe stays-human list — when you decide to specialise, name what you are giving up and decide whether it belongs on the list of things that stay yours.

After AIThe trade is usually described as a cost of specialisation. The relevant version now is the trade made by specialising in supervising a model: what grows is the ability to judge output, and what may go is the ability to produce it.

Becausein the volumes experiment blocked practice averaged 89 per cent during practice and 20 a week later, while interleaved practice averaged 60 during practice and 63 later; the Bjorks' separation of retrieval strength from storage strength names the same gap.

LimitsLaboratory tasks with delays of days to weeks. Steven Pan and Timothy Rickard's 2018 meta-analysis found the benefit of testing transfers only modestly and inconsistently to changed formats and new inferences, which is more cautious than the book.

PracticeCalibration log — never read a training metric as a learning metric; measure later and unaided, or do not claim the learning.

After AIEvery productivity dashboard for AI-assisted work is a performance-during-practice measure. Robert Bjork and Nicholas Soderstrom formalised the distinction in 2015 as a warning against reading a training metric as a learning metric, and it is now the sharpest objection available to the field studies of assistants.

BecauseTulving's six prior exposures produced learning curves indistinguishable from no exposure, and 148 students across two universities got no benefit from reading passages twice in succession in any condition tested; over 80 per cent of college students name rereading as their main method.

LimitsThe result is about massed rereading in immediate succession; spaced rereading is a different condition and is not what these studies tested.

PracticeManual flying hours — replace a second pass over the documentation with an attempt to use it from memory.

After AIScrolling an assistant's explanation of the same system a second time is the modern form of the same method, and it is fluent in exactly the way the book warns about.

Becauseblocked practisers were markedly overconfident predicting next-day performance while interleaved practisers were accurate, and students went on preferring massed practice after taking the test that showed them they were wrong.

LimitsPreference data from classroom and laboratory studies; the persistence after disconfirmation is reported in a handful of experiments and has not been the subject of a meta-analysis.

PracticeCalibration log — treat your preference between two working methods as evidence about which one feels better, and settle the question with a delayed unaided test instead.

After AIIt explains why the METR developers kept using a tool that was costing them time. The preference is not irrational on its own terms — the assisted session really is easier — it is only uninformative about what was learned.

BecauseSherman Kent meant 65 to 35 by "serious possibility" and his own analysts read it as 80/20 and 20/80; the 2010 letter to Bernanke set no date and hedged on "risk", and its signatories told Bloomberg in 2014 they would not change a word.

LimitsThe Kent finding is about intelligence analysts reading a specific phrase; how much of ordinary professional hedging is strategic rather than imprecise is not measured. The wrong-side-of-maybe fallacy is a critique of readers as much as of forecasters.

PracticeCalibration log — write the number and the resolution date before the work, because a judgment you cannot score is a judgment you cannot learn from.

After AIModel evaluations inherit the problem in full. A claim that an assistant is "generally reliable" for a task is Sherman Kent's serious possibility, and it protects the same reputations.

Becausethe tournament's training module was tested against untrained forecasters on identical questions over identical periods, which is a randomised comparison rather than an observed difference.

LimitsSweller holds that generic cognitive skills cannot be taught and cites the failure of attempts to teach them; this is the randomised far-transfer result his position has to absorb, in a domain he would not count. Ericsson's strict definition of deliberate practice excludes a field with no agreed standard of superior performance, which is the field this was measured in.

PracticeCalibration log — read the guidelines once, then keep the log; the training is cheap and its effect decays, so repeat it.

After AIIt is the most encouraging result in the vault: the capacity you need in order to supervise a model is trainable in an hour and measurable afterwards. The training has to be repeated, because the effect lasts about a season.

Becausethe comparison is within-subject, between forecasts made inside and outside the expert's speciality; fame ran inversely to accuracy, since producers hunt for people who tell tight confident stories.

LimitsKlein's evidence runs the other way in domains with valid cues and fast feedback; the 2009 settlement with Kahneman says the conditions decide, not the setting. Which regime a given professional judgment belongs to is not something either book supplies a classifier for.

PracticeFailure-mode review — raise the checking effort on judgments inside your own speciality, not lower it.

After AIIt is the closest thing in the literature to a mechanism for the METR result, where the slowdown was largest exactly where the developers' repository knowledge was deepest.

BecauseMellers rounded everyone's forecasts and recalculated: superforecasters lost accuracy from rounding to the nearest five points, while regular forecasters barely lost anything from rounding four times as coarsely, and a third of superforecaster judgments used single percentage points.

LimitsThe test shows that the granularity carried information for these forecasters on these questions; it does not show that a novice's single percentage points mean anything. The regular forecasters' insensitivity to coarse rounding is the other half of the same result.

PracticeCalibration log — record the probability you actually hold rather than rounding to a decade, and let the score tell you whether the extra digits were real.

After AIIt is the argument for asking a model for a number rather than a hedge, and for scoring the number. Whether a model's fine-grained probabilities survive the same rounding test is a measurement anyone can now run.

Becausethe ordering is a recommendation drawn from anchoring research and from what superforecasters were observed doing — David Rogg counted six attacks in five years, set the base rate at 1.2 a year, lifted it to 1.8 for the rise of ISIS against tightened security, and scaled it to the 69 remaining days for 34 per cent — rather than a randomised comparison of the two orders.

LimitsThe ordering has not been tested against the reverse in a controlled comparison; the case for it is anchoring plus observed practice. Klein's evidence says the expert's first recognition is usually right in domains with valid cues, which is the opposite prescription.

PracticeCalibration log — find the reference class and its base rate before you look at the details of the case in front of you.

After AIAn assistant supplies the inside view instantly and in fluent prose, which makes it the first number you see. The outside view now has to be taken before the prompt, not after the answer.

Because284 professionals, some 28,000 dated predictions between 1984 and 2004, half with doctorates; foxes beat hedgehogs on calibration and on resolution both.

LimitsTwenty years of political and economic questions, scored on the researchers' own resolution criteria. The fox and hedgehog division comes from a questionnaire, and the trait psychology around it has fared badly in the replication decade.

PracticeCalibration log — stop weighting a judgment by the seniority of the person making it and start weighting it by their score on comparable questions.

After AIThe finding transfers directly to the question of whether to trust a model's judgment: ask for its record on comparable questions, not for its confidence or its reasoning.

BecauseTim Minto's thirty-four revisions on the Syrian refugee question averaged three and a half points and scored 0.07, while Lorch swung from 55 to 95 per cent on one month-old sea-ice report and the ice stopped melting; most superforecasters are Bayesians who do not use Bayes's theorem.

LimitsTwo cases and an aggregate pattern; the rule for when to discard rather than nudge is stated by example. Klein's account of the expert who abandons a whole frame at once is the opposite prescription in a domain with valid cues.

PracticeCalibration log — log every revision with its size and its reason, so that an oversized swing shows up as an event rather than as a new opinion.

After AIA model will revise to whatever the last message implied, which is the failure mode this rule is written against. Frequent small updating is a virtue in a person and a pathology in an assistant that has no stake in the previous number.

Becauseit is Tetlock's own concession to Taleb, and the supporting fact is that accuracy decayed towards chance between three and five years out in the 1984–2004 data, with no evidence supporting forecasts a decade ahead.

LimitsAn admission rather than a finding, and Tetlock notes the loose definition of a black swan that makes the disagreement hard to settle. The Existential Risk Persuasion Tournament of 2022 put forecasters and domain specialists on catastrophic-risk questions out to 2100 and they finished about as far apart as they began.

PracticeIf you believed it you would stop citing forecasting expertise in arguments about tail risk, and you would treat a superforecaster's number on a 2050 question as an opinion with no track record behind it.

After AIIt applies directly to forecasts about AI. The instrument that works on months has no demonstrated reach into the horizon on which most claims about model capability are made.

Becausethe test is regression to the mean, and the superforecasters extended their lead across the second and third years instead of regressing; the year-to-year individual correlation is about 0.65, with roughly 70 per cent staying in the top two per cent.

LimitsConsistency was measured on questions the tournament chose, resolving within months. Accuracy decayed towards chance between three and five years out in the 1984–2004 data, so the skill is demonstrated only on the tournament's own horizon.

PracticeCalibration log — keep the record long enough for persistence to show, because a single good quarter is indistinguishable from luck.

After AIIt is the cleanest available template for evaluating a model: the same questions, the same resolution dates, a score, and a benchmark. Nothing about the instrument requires the forecaster to be human.

Becauseit was the strongest predictor the project found of who rises into the top ranks; superforecasters scored above about 80 per cent of the population on fluid intelligence against 70 per cent for the other volunteers, so the large jump is from the public to the volunteers rather than within them.

LimitsA correlation within a self-selected volunteer pool, measured by questionnaire. The traits the chapter reaches for — grit and growth mindset — have both been cut down since 2015, which weakens the explanation without touching the tournament's own measurements.

PracticeCalibration log — treat every scored miss as the material, and record what you changed your mind about, not only what you predicted.

After AIIt is the property an evaluation of a model cannot have on the model's behalf. The updating that matters is the reviewer's, and it only happens if the misses are recorded.

BecauseKatha 2.5 has the blind led by the blind, repeated almost word for word in the Mundaka, and 2.9 says that *tarka*, reasoning, does not terminate here; both are assertions, and the psychological claim in them is the part that has since been measured elsewhere.

LimitsThe text's remedy is a teacher who has been up the path, and nothing in it is scored or scoreable. The Dunning and Kruger pattern that supplies the modern evidence has itself been contested on statistical grounds.

PracticeCalibration log — treat the arrival of confidence as the moment to check, and score the check.

After AIA middling amount of learning is what an assistant reliably produces in a hurry, and it arrives with the fluency that the text says is the danger. The remedy the vault can actually run is Tetlock's score rather than the Katha's teacher.

Becausethese are Shanteau's conditions, which Klein extends to legislators, whose experience is expertise in the procedures of the job and not in its judgments; where cause and effect cannot be traced, what fills the gap is ritual.

LimitsA list of conditions with no threshold on any of them, and no classifier for a domain that partly meets them. Klein's 2009 settlement with Kahneman rests on the same conditions, which makes them the durable part of his answer and also the untested part.

PracticeDomain audit — classify the domain before you trust anybody's fast judgment in it, your own included.

After AISoftware with a model in the loop fails the stability condition: the tooling changes every few months, so the sense of typicality is being rebuilt continuously. On Shanteau's own terms that is a domain in which experience accumulates more slowly than it used to.

Because156 decision points from non-routine fireground incidents were coded by two researchers with a third arbitrating and doubtful cases pushed into the comparative bin, giving 127 recognitional against 18 comparative — and half of the comparative came from one fire where the chiefs were novices.

LimitsInterviews coded by the researchers who ran them, which Klein says plainly. The 2009 settlement with Kahneman narrows it: recognition is worth trusting when the environment offers valid cues and the person has had a fair chance to learn them, and both conditions fail in most of Kahneman's settings.

PracticeDomain audit — list the domains where your first judgment has been trained by feedback, and stop demanding option comparisons there while insisting on them everywhere else.

After AIIt names what the model does not have. A model's fluent first answer looks like recognition and is not backed by a history of cases in this environment, so the surface behaviour is the same and the warrant is absent.

BecauseCrandall's neonatal nurses had nothing to say about detecting sepsis until asked for particular babies and shifts; almost half the compiled cues were not in the medical literature and several ran opposite to the adult signs.

LimitsThe critical decision method extracts what one person can recall about cases they chose to report. Beane's objection is that it records what has been lost rather than substituting for the transmission.

PracticeFailure-mode review — ask a colleague for the last three times they caught something, not for how they catch things.

After AIIt is the cheapest way to get undocumented knowledge into a form a model could be given. The neonatal cues became a training programme and then a monitor that catches the deterioration earlier than a person can — which is the whole arc in one example.

Becausethe lieutenant evacuated a house fire because the living room was too hot, it was too quiet and the water did nothing — the seat of the fire was in a basement he did not know existed — and the *Gloucester* officer read altitude off a radar that does not measure it.

LimitsReconstructed after the fact from interviews with the people involved. An absence noticed in hindsight is the easiest kind of cue to over-report, and the method cannot separate it from a story built afterwards.

PracticeWitness practice in review — when something feels wrong in a review, ask what you expected to see and did not, before you ask what is there.

After AIGenerated code is missing things rather than containing wrong things, which makes this the right shape of attention to bring to it. A model has no expectations to be violated, so the absent-cue channel exists only on the human side.

Becausein the overpass rescue a lieutenant cycled through a harness, the harness reversed and a rescue strap in about a minute, finding a fault in each and never comparing two; the pattern is Simon's satisficing observed in the field.

LimitsField cases, not experiments. In domains where the cues are weak, Klein conceded in 2009, consistent rules beat consistent people — and comparison is what rules amount to.

PracticeFailure-mode review — evaluate a proposed change by walking it forward step by step until something breaks, rather than by lining it up against alternatives.

After AIReviewing a generated diff is exactly this operation: one option, arriving whole, to be run forward in imagination. The difference is that nobody chose it, so there is no intent to reconstruct when the simulation snags.

Becausetank platoon leaders on their first course fell below 50 per cent recognitional, and the volunteer chiefs at the oil tank farm produced the study's strongest comparative decision because they were, for that fire, novices; but Klein conceded in 2009 that where cues are weak, consistent rules beat consistent people.

LimitsThe claim does not survive the 2009 settlement in domains where the cues are weak; Kahneman, Sibony and Sunstein's case for rules as noise reduction runs directly against it. Klein describes this as the part of the book that reads weaker now than it did in 1998.

PracticeIf you believed it you would drop the decision matrix wherever the cues are valid and your feedback has been fast — and keep it everywhere else, which is most places.

After AIIt is the argument against forcing structured evaluation on experienced reviewers — and the conditions that would license it are exactly the ones a fast-changing toolchain removes.

Becausethe regularity comes from 79 cases drawn from fireground records, published accident analyses and interviews, and nobody has since tried to confirm or refute it; the constraint tightens sharply when the parts interact rather than act in turn.

LimitsA count taken from cases collected for other purposes, never replicated. Klein says so himself, and the number has survived thirty years mainly through not being tested.

PracticeFailure-mode review — when a review needs more than three moving parts held at once, write them down or split the change; do not simulate it in your head.

After AIA generated change can touch more parts than the reviewer can hold, and nothing in the diff says how many. The size of the change, not the quality of the code, is what decides whether review is possible.

BecausePerrow's *de minimus* explanation names the move, the physicists decided someone must have built the dock that morning, and Klein's own orienteering accommodated a creek not on the map and a curving path meant to be straight until the map had to be turned 180 degrees.

LimitsCase evidence and a named mechanism, with no measured rate at which discounting predicts failure. The remedy — counting — has not itself been tested.

PracticeCognitive forcing — keep the tally: write down each thing you explained away, and treat the third one as a signal to throw the frame out rather than patch it again.

After AIAn assistant's account arrives pre-built and carries someone else's confidence, so the discrepancies are being explained away against a story the reviewer never assembled. That is why the counting has to be forced by the interface rather than left to the reviewer.

BecauseElliot scored superior or average on every WAIS subtest, normal on Rey and Benton, six categories in seventy sorts on the Wisconsin, a valid normal MMPI profile, and on a purpose-built social battery exceeded controls on consequences — then said he still would not know what to do

LimitsA single patient carries the strong form of the claim, and the batteries of 1994 are not those of today. Later work found ventromedial patients answering moral dilemmas differently rather than failing them.

PracticeCalibration log — score your decisions, not your reasoning, because the instruments that grade the reasoning miss this.

After AIBenchmark scores are instruments of exactly this kind. A system can be superior on every subtest and have nothing in it that terminates a decision.

Becausethe 1994 reconstruction of Gage's skull placed the damage bilaterally in the ventral and inner prefrontal cortices and left motor, premotor, Broca's and the lateral prefrontal cortices untouched — the tissue his contemporaries assumed was best fitted to sustain such an injury

LimitsMacmillan's history argues the standard account overstates the ruin, since years of demanding stagecoach work in Chile implies substantial re-adaptation. The lesion is a simulation derived from a skull, not an autopsy.

PracticeThe stays-human list — draw the list around what this system does — choosing under consequence — rather than around tasks a machine cannot yet perform.

After AIThe list of what stays human is usually drawn around capabilities. This draws it around a faculty that can be removed from a person while every measurable capability remains.

Becauseit is offered as a hypothesis, with the 2005 preface doing most of the qualifying — markers do not deliberate, are not sufficient for deciding, need not be unconscious, and do not substitute for reasoning; skin conductance was an index of a marker and never the marker

LimitsA 2006 review found the evidence for markers as the book states them thinner than the presentation suggests. The working-memory half of the hypothesis is stated in the book and never tested.

PracticeCognitive forcing — withhold the recommendation until you have imagined the outcome yourself, since the marker only fires on an option you have entertained.

After AIA recommendation that arrives before you have imagined the outcome occupies the slot the marker filled, and leaves the deduction running over a field that somebody else pruned.

Becausethe case is made from the example and from the patients' behaviour — half an hour of reasons for and against two appointment dates — rather than from any measurement of capacity

LimitsTetlock's superforecasters do run explicit weighing and beat intuitive judges, which shows the limit is on unaided capacity and moves once the ledgers are written down.

PracticeCognitive forcing — force the explicit weighing only where the field has already been cut down, and give it external structure when you do.

After AIA model can hold the ledgers, which makes the cool strategy available for the first time. What it cannot supply is the signal that told you which options deserved a ledger at all.

Becauseboth halves are documented — ventromedial damage reliably produces the pattern and the region became the standard site for subjective value, while the card game's critiques accumulated — and the dispute over the mechanism is live

LimitsEkman's facial feedback failed a large multi-laboratory replication before meta-analyses recovered a small effect, and Macmillan's history revised the Gage case in the other direction.

PracticeDomain audit — when citing a book, audit which half of it you are citing, the position or the mechanism.

After AIA model summarising a field will report the half that travelled, because that is the half the literature repeats. The audit is the same one you would run on a confident secondary source.

Becauseeleven further patients with the same prefrontal damage produced the pairing and not one separated the defects, and the same combination appears from elsewhere — anosognosia after right somatosensory damage, and a bilateral amygdala patient at normal intelligence

LimitsThe series is small and clinically selected, and the leucotomy cases carry uncontrolled damage. The historical cases are described retrospectively.

PracticeFailure-mode review — when reviewing a decision that went wrong, ask what the decider felt about the option, not only what they knew about it.

After AINothing in an assistant produces a body state, so whichever of these two outputs the marker contributes is simply absent from the loop when the deciding is handed over.

Becausethe inverted schedule, punishment first with reward interspersed, restored normal performance, which disposes of simple insensitivity to punishment; in the original the patients avoided a deck immediately after a penalty exactly as controls did and then, unlike controls, came back to it

LimitsThe decks confound punishment frequency with magnitude and healthy players over-select one of the bad ones. The game has aged much worse than the clinical core it was built to test.

PracticeCalibration log — log the prediction before the outcome, so the penalty has something to attach itself to.

After AIA postmortem you read is a rule; the conversion these patients lost is built from penalties you actually took. Reviewing a model's work supplies a great deal of the first and almost none of the second.

Becausecontrols produced abundant skin-conductance responses to earthquakes, burning houses and injured people while the frontal patients produced flat recordings, with startle and a deep breath having established the machinery was intact, and afterwards the patients described the content, named the appropriate emotion and ordered the set

LimitsA small patient series with a coarse autonomic index; other channels were not recorded. The patients' reports are self-report about an absence.

PracticeWitness practice in review — notice, during review, whether anything in you responded to the change, or only your account of what it means.

After AITo know but not to feel is also an accurate description of what a model does with a description of harm, and it is the reason the description is not yet the judgement.

Becausethe original recording is clear and the patient-control difference is stark, but what the anticipatory response tracks has been reassigned since, and several of the surrounding claims did not survive

LimitsThe response has been shown to track the size of the reward or penalty on offer rather than a deck's long-run value; the claim that players choose advantageously before knowing why did not survive finer questioning; patients with pure autonomic failure have been reported playing normally.

PracticeFailure-mode review — treat an anticipatory signal as a prompt to look, not as a verdict about long-run value.

After AIThe measurement that made the hypothesis famous is the half that thinned. That is the usual shape — the mechanism story travels and the index it rested on does not, which is worth remembering about any confident summary of a field.

Becausea small fault with unpredictable results cost more trust than a large fault of constant error, and a fitted first-order differential equation explained 60–86% of the variance in reliance over time

LimitsEstimates of the reliability level at which trust collapses disagree across studies — 90%, 70%, 60% — which the paper reads as evidence that the level is the wrong variable, though it is equally consistent with the level mattering and the microworlds differing.

PracticeFailure-mode review — keep a written list of how your tools fail rather than how often, and treat a failure mode you cannot describe as a reason to withdraw the task, not to add a check.

After AIThis is the sharpest available statement of what makes a jagged frontier expensive: the failures are not large, they are unpredictable, and the paper's own evidence says that is the condition under which trust cannot settle anywhere useful.

Becauseit follows from the distinction between trustworthy and trustable rather than from a measurement — the algorithms that perform best are often the ones no operator can follow

LimitsLee and See offer no case in which a worse system actually produced better joint performance; the argument is deductive. It also assumes the operator's judgment is worth preserving, which Vaccaro 2024 shows is false wherever the machine is the better partner.

PracticeThe stays-human list — where the work depends on your being able to tell a good answer from a bad one, prefer the tool whose reasoning you can check over the tool that scores better on a benchmark.

After AICynthia Rudin's interpretable models and the whole argument for smaller, inspectable systems are this guideline restated, and the industry has gone the other way — capability first, legibility never.

Becausethe paper's model is a review of the experimental trust-in-automation literature (Lee and Moray's process-control studies, the misuse and disuse cases of Parasuraman and Riley), and the same comparison predicted reliance in later survey and lab work.

LimitsThe model treats trust as one quantity; the paper itself says calibration, resolution and specificity can come apart, and later work found the comparison weaker when the person has no way to judge their own competence.

PracticeAudit reliance by task: where trust in the tool exceeds your own confidence, add a check that does not depend on either.

After AIThe automation Lee and See studied was a boiler or an autopilot with a fixed capability; a model whose capability moves by task makes the comparison harder to keep calibrated, which is what the jagged-frontier result shows.

Becauselower memory confidence predicted more reminders with objective ability held constant, and people less confident in a brightness judgement, at matched accuracy, set more reminders in the memory task; the correlation has since been demonstrated causally by manipulating confidence without touching ability.

LimitsThe domain-general result is a correlation across two unrelated judgements, and the confidence measured is self-reported; how stable the disposition is over months is unmeasured.

PracticeTrack when you reach for the tool, and ask what your confidence was doing rather than the task. A bad week or simple tiredness moves reliance without moving the difficulty.

After AIReliance on an assistant rises when you feel least able, which is when you are least able to check it. The loop closes faster with a tool that answers everything than with a notebook.

Becausethe symmetry of misuse and disuse is an argument from two accident reports and Zuboff's paper-mill operators, not a measured quantity

LimitsTwo accidents and one field study carry the symmetry. A 2011 meta-analysis of trust in human-robot interaction found machine performance dominating everything else, which leaves the paper's long sections on disposition and culture resting on thin evidence.

PracticeReliance audit — record, per task, whether you used the tool and whether you should have, and treat a refusal you should have accepted as the same size of error as an acceptance you should have refused.

After AIWritten about autopilots and pump controllers, the rule now bites hardest on a delegate that is fluent in every domain at once: there is no single level of trust to match, because capability is jagged, so the calibration has to be redone per task rather than per tool.

Becausethe three bases are a tidy collapse of fourteen published dimension lists, and the stability claim is offered as an explanation for why trust sometimes looks fragile and sometimes immovable rather than as a measured result

LimitsIncoherence between levels is said to damage trust as much as a poor record does, on interpersonal evidence about managers whose stated intentions disagree with their actions; the transfer to machines is asserted. The transparency literature that grew from the process recommendation has mixed results — more explanation raises workload without reliably improving calibration.

PracticeReliance audit — before relying on a tool for something that matters, say out loud what you know about its record, its mechanism and its design intent, and treat a blank in two of the three as a reason to verify.

After AIProcess is the basis that has gone missing: nobody, including the builders, can give an account of how a language model reached an answer, so trust is left resting on performance and on stated purpose, which is exactly the volatile configuration the paper warns about.

Becauseventromedial prefrontal lesion patients keep reasoning and lose deciding, and healthy participants produced anticipatory skin-conductance responses that steered them off the risky decks before they could say the decks were risky

LimitsThe gambling-task evidence is about decisions under risk generally, not about trust in automation; the transfer is Lee and See's argument rather than their result. The affective route is also where the paper's own warning lands — the devices that calibrate trust can manufacture it.

PracticeWitness practice in review — when a tool's output feels right, treat the feeling as a datum about your state rather than about the output, and name it before you act on it.

After AIA conversational partner engages the affective route directly and continuously, so the unstatable evidence is now mostly about prose style: fluency, politeness and pace are doing the work that the colour of the fire once did for Zuboff's boiler operator.

Becauseoperators responsible only for watching detected almost all automation failures and operators with other work did not, replicated with experienced pilots and with rare faults, and ten minutes of manual control handed back by adaptive automation substantially improved subsequent detection

LimitsThe restoring effect is measured over minutes in a laboratory task, not over the months across which a professional skill decays. The paper does not say how much manual control is needed, or how quickly the benefit fades.

PracticeManual flying hours — schedule work you do unassisted, and do not count reviewing a generated draft as monitoring; the finding is that a monitor with a second job is not monitoring.

After AIReviewing model output is precisely the divided-attention condition in which detection failed, and it is now the default arrangement of knowledge work — the reviewer is also the person with the deadline.

Becauseit is an argument about how to weight outcomes — the paper asks for composite measures carrying time, cost and the cost of each kind of error — not a result the meta-analysis produced

LimitsThe argument licenses keeping a person in almost any loop, since some error is always catastrophic in retrospect; it needs the cost asymmetry to be stated in advance, and the paper supplies no method for that.

PracticeFailure-mode review — before handing a task over on an accuracy comparison, ask what the worst error costs and how often you would be the one to catch it.

After AIIt is also the standard defence of human review in every deployment that has none of the composite measurement the paper asks for — the exception is invoked, the arithmetic never done.

Becauseaugmentation peaked at g = 0.74 in the subgroup where the machine was the better partner — the same subgroup whose synergy was −0.54

LimitsBoth quantities are subgroup estimates from a sample dominated by decision tasks, and the two measures are computed from the same effect sizes, so the divergence is arithmetic as much as it is a finding about people.

PracticeReliance audit — treat a large measured improvement over your unaided self as a reason to check whether the machine alone would do better still.

After AIThis is the mechanism by which enthusiasm and harm arrive together: the felt improvement is real and large, and it is largest in the work where you should not be in the loop at all.

Becausethe creation estimate has an interval from −0.09 to 0.48 and rests on 34 effect sizes against roughly 85% of the sample being decision tasks, so what is reliable is the difference between the two, not the creation number itself

LimitsThe authors' explanation — that a creation task splits itself into the part each partner is better at — is a conjecture, and the moderator did not reach significance for augmentation. METR's 2025 developers were 19% slower on a creation task, which is a caution against reading it as a licence.

PracticeDomain audit — mark each task as choosing or making, and put your review effort on the choosing.

After AIThe sample predates generative systems almost entirely, so the creation result is the least tested part of the paper and the one most often quoted at it.

BecauseF(1,104) = 81.79 — where the person was better the pair beat both (g = 0.46) and where the machine was better the pair fell below the machine (g = −0.54) — and Cabrera 2023 reproduced the reversal with interface and machine accuracy held constant

LimitsWhich partner is better alone is measured inside each experiment on that experiment's task; it is not a property of a person or a model that transfers. And the sample is dominated by bounded classification, where being better alone is easy to define.

PracticeDomain audit — sort your work by who is actually better at it, and hand over whole the tasks where the machine wins rather than reviewing its output on them.

After AIThe moderator is the same finding Dell'Acqua's frontier names, and it is harder to apply now, because with a general model the answer changes task by task rather than tool by tool.

Because370 effect sizes from 106 experiments pool to g = −0.23 (95% CI −0.39 to −0.07, P = 0.005), with 58% of effect sizes below the better partner, and the estimate survived four sensitivity checks

LimitsI² was 97.7%, so the average sits on enormous spread and almost none of the variation is noise. Studies qualified only if all three performances were measured, which excludes every task neither partner can do alone — the cases where combining them is most obviously worth it.

PracticeReliance audit — measure the machine alone as well as yourself and the pair; a comparison that omits the machine cannot tell you whether you are adding anything.

After AIThe search window closed in June 2023, so almost the whole generative era is outside the sample: what was pooled is mostly classifiers judged by paid online workers. The pattern has held where retested, and the missing number is still the machine working alone.

Becauseacross more than 300 effect sizes neither moderator reached significance for either outcome, and nor did whether participants were domain experts or paid crowdworkers

LimitsA null moderator across heterogeneous studies is weak evidence that explanations never help; it says the literature's explanations, as implemented, did not move the average. Buçinca's cognitive-forcing designs did move accuracy, and they are a different intervention.

PracticeCognitive forcing — stop expecting a better interface to fix reliance; change what the task asks of you instead, by committing to an answer before you look.

After AIChain-of-thought output is an explanation display at enormous scale, offered by default and unverified against the computation that produced the answer, which puts it squarely in the category the meta-analysis found inert.

Becausein more than 95% of the systems sampled the person made the final call after seeing the machine's output, and the inference that someone who cannot tell a hard instance from an easy one cannot tell when the machine has gone wrong is the authors' reading rather than a measured mediation

LimitsFleming Know Thyself is the direct counterweight: metacognitive sensitivity dissociates from ability, so knowing whether an answer is right is at least partly a separate faculty. The paper's claim is about task-level discrimination, not about self-knowledge, and the two are being run together.

PracticeDomain audit — do not accept a reviewing role in work you could not do yourself; deferral is a skilled act, and the skill is the task's own.

After AIThis is the argument against the reviewer-of-machine-output job description that is now being written into most knowledge work, and it is the strongest reason to keep doing the work you intend to supervise.

Because314 of the same 370 effect sizes are positive against the human baseline, a medium-to-large pooled effect drawn from the identical sample that produced the −0.23

LimitsEgger's regression indicated publication bias for the augmentation estimate (β = 1.96, P = 0.002), which the authors left uncorrected. Augmentation is also the right test where full automation is barred by law, ethics or safety.

PracticeReliance audit — state which baseline a claimed gain is measured against before you accept it, and ask for the other one.

After AIEvery vendor benchmark and internal pilot reports the person-alone baseline, because that is the comparison that flatters; the machine-alone number, which decides whether the person is adding or subtracting, is the one nobody publishes.

Becausetwo independent grader pools, blind to the correct recommendation, scored the treated groups 1.05 and 1.47 points higher on clarity, persuasiveness and coherence, and the advantage held among those who got the answer right and those who got it wrong

LimitsGraded coherence is a rubric judgment by students and consultants, on one task. Whether better argument makes a wrong conclusion more likely to be accepted downstream was not tested here.

PracticeFailure-mode review — review a recommendation against the evidence it omits, not against how well it reads; in this experiment the best-argued answers were as often the wrong ones.

After AIThis is the clearest measured statement of the cost of fluency: the same system that raises the quality of the case raises it equally for the case that should not be made.

Because373 consultants on a case where the spreadsheet pointed one way and the interview notes the other fell from 84.4% correct in control to 70.6% with the model and 60.0% with the model plus prompt training — about 19 points — while finishing 18% and 30% faster

LimitsOne task outside the frontier against eighteen inside it, and the outside task was built by the authors to produce exactly that drop. The design shows such a task exists inside ordinary professional work and that trained people walk into it; it cannot say how often.

PracticeCognitive forcing — on a judgment where the obvious data and the awkward detail disagree, form and record your own recommendation before you open the model.

After AIThe degradation appeared not because the system was visibly unreliable but because it performed so well on the tasks next door, which is the condition every general assistant now creates by default.

Becauseit is an interpretation of the contrast rather than a measurement — same firm, same population, same incentives, same fluent output, and participants who did not need the model used it anyway, which Dell'Acqua calls falling asleep at the wheel

LimitsInvisibility was not measured — nobody asked participants whether the task felt different. The concept has since become a diagnosis applied after the fact, and the paper's own recommendation that organisations evaluate each task has no method behind it.

PracticeDomain audit — decide in advance, per class of task, which side of the line it sits on, and write the decision down; you will not detect the crossing while working.

After AIThe frontier moves with each release, so a map made this quarter expires; what survives is the warning that fluency is uninformative about which side of the line you are on.

Becausebias slides both confidence distributions together while sensitivity is how far apart they sit, and 94% of American university professors in one study rated their teaching above average while remaining able to notice the occasion they got wrong

LimitsThe separation is a property of the statistical model fitted to confidence ratings, and the models differ in their assumptions; the above-average surveys are self-report on ill-defined criteria.

PracticeCalibration log — record whether your confidence separated your right answers from your wrong ones, not whether its average was too high; the second is easy to fix and tells you nothing.

After AIMost talk about model calibration is about bias — whether stated confidence matches the base rate — and the quantity that decides whether a person can supervise a system is sensitivity, which is rarely reported for either party.

Becausethe positive-evidence manipulation raises the brightness of signal and noise together, leaving difficulty unchanged and confidence higher, and Rollwage found reduced mind-changing, with contradicting evidence close to undecodable from magnetoencephalography on high-confidence trials

LimitsThe manipulation works on a perceptual decision about dot motion; whether professional judgment can be inflated the same way is an extrapolation. The undecodability result is one imaging study.

PracticeCalibration log — when confidence rises without new evidence, treat it as a reason to seek information rather than to stop.

After AIA system that agrees with you raises confidence by exactly this route — more apparent evidence on both sides of the decision, no more accuracy — and the downstream effect the book measures is that you then stop looking.

Becausein a sample of nearly a thousand, IQ predicted task performance and not metacognitive sensitivity; patients with anterior prefrontal tumours removed had normal IQ and impaired sensitivity; and Shimamura's 1989 double dissociation separates the two in both directions

LimitsMeasurement is argued over: meta-d′ assumes confidence draws on the same evidence as the decision and is sensitive to where confidence criteria fall. Fleming's own frontal-pole localisation rested on thirty-two volunteers, and correlations of this kind need samples in the thousands to stabilise.

PracticeCalibration log — score your confidence separately from your answers, and treat a run of confident errors as a fault in a different faculty from the one that produced them.

After AIThe faculty the evidence isolates is exactly the one a fluent assistant supplies no help with: it will produce an account of its reasoning on demand, and that account need not correspond to the computation that produced the answer.

Becausethe argument runs on illustrations — misjudge the size of a cup and you knock it over, while the person who talks over everyone collects raised eyebrows rather than a spilled drink — with fluency effects supplying the illusions

LimitsFluency effects are the evidence, and some of them have not held — pronounceable company names and disfluent fonts among them; Sans Forgetica did not improve memory in independent tests.

PracticeCalibration log — manufacture the missing feedback: write the prediction down, date it, and go back to it, because nothing in the situation will do this for you.

After AIA tool that answers instantly and agreeably is a fluency machine, so it strengthens the exact illusion this claim identifies, and the loop it feeds back into is still nobody's job.

Becausethe argument is structural — a network's knowledge sits in the weights, held in the network rather than for it, as Clark and Karmiloff-Smith put it — with working demonstrations (a second network betting on the first, a drone predicting its own crashes) rather than a general result

LimitsChick sexers and expert putters are the analogy carrying the claim, and analogy is not proof that capability never brings self-evaluation with it. Modern systems do report confidence, and the reports are sometimes informative.

PracticeReliance audit — do not treat a stated confidence, or a stated chain of reasoning, as a readout of the mechanism; keep your own metacognition interfaced with the system instead.

After AIIt has aged in the shape he predicted rather than the one expected: language models produce fluent accounts of their own reasoning that need not match the computation, which is Gazzaniga's interpreter in silicon, and training them to please raters makes calibration worse.

Becausesensitivity on a dot-counting task was the best single predictor of dogmatic political views in Fleming's laboratory and also predicted not seeking further information

LimitsA correlation between a laboratory perceptual measure and a survey of political attitudes, from one group; the causal direction is not established, and dogmatism scales have their own problems.

PracticeCognitive forcing — treat a refusal to look at disconfirming material as a symptom rather than a position, in yourself first.

After AIRetrieval systems make it trivially easy to be supplied with confirmation, so the disposition this predicts now has an instrument built for it.

Becauselaboratory estimates of off-task attention run from 15% to 50% and rise with practice; making a task harder and faster raised parietal, premotor and visual activity while lowering the prefrontal network; and higher cortisol in a social stress test predicted worse metacognition, with hydrocortisone lowering sensitivity against placebo

LimitsKlein Sources of Power is the counterweight: under time pressure experts in the field make good decisions without deliberation, so degraded reflection is not the same as degraded performance.

PracticeCognitive forcing — build the check into the task rather than relying on noticing; under deadline you will not notice.

After AIAssistance is adopted precisely to work faster under more load, which is the condition the evidence says removes the faculty that would catch the assistance going wrong.

Becausetwenty minutes a day for two weeks of practice with feedback on confidence rather than accuracy raised sensitivity and transferred to an untrained memory task, and two weeks of meditation training did the same in Baird and Schooler, against Fleming's own caution that the plasticity is asymmetric

LimitsSmall samples, short training, and nothing training can add compares with what disease takes away. Fleming also notes that more accurate self-appraisal is not always kinder.

PracticeCalibration log — take the feedback on your confidence rather than on your answers; that is the manipulation that produced the transfer.

After AIThe training regime — repeated judgments with quick feedback on confidence — is one of the few things a model is well suited to run, and nobody is shipping it.

Becausepooled correlations are .24 with verbal intelligence, .26 with entrance scores and −.03 with abstract reasoning ability, and the disposition predicted recall of a message's arguments after the variance explained by measured intelligence was removed

LimitsThe separation leaned on college entrance scores as the ability measure; it has since been confirmed with better instruments. Education correlates .25 and the direction of that relation cannot be told from the data available.

PracticeWhere someone is not scrutinising an answer, ask what would make it worth their while rather than whether they can.

After AIA tool that removes the effort from producing an answer removes the occasion on which the disposition used to show itself, which makes it a question about who still chooses the harder path when nothing forces it.

Becausethe short form has α = .90, correlates .95 with the long one and takes 37% of the variance on its first factor, replicated in physicians' patients (38.8%, α = .88) and in Dutch translation, while Tanaka's three proposed subscales loaded at −.01, .70 and .28 across samples against .76 for the single factor

LimitsFactor structure is evidence about a questionnaire, not about a mechanism. Items are face-valid enough that the original authors had to embed them among others and administer them anonymously.

PracticeTreat willingness to think as one trait you can measure in eighteen questions, and stop looking for separate capacities behind it.

After AIThe single factor has held for thirty years, which makes this the most durable part of a review whose persuasion architecture and bias findings have not held.

Becauseeight weeks before a presidential election high scorers reported thinking about it more than low scorers, and by election day the two groups reported the same

LimitsThe review's summary figure carries no numbers in the original and is schematic; the election study is self-report about how much people thought.

PracticeDomain audit — raise the demand of the task rather than selecting for the trait; the differences you are trying to manage disappear where the stakes are plain.

After AIRoutine assisted work sits in the low-demand region where nobody scrutinises, and it is dressed up as high-stakes work, which is the combination that hides the failure.

Becausethey examined the arguments when the source was untrustworthy and not when he was trustworthy, and when the framing was unexpected and not otherwise, while high scorers' attitudes tracked argument quality in every condition

LimitsThe manipulations are laboratory persuasion studies on undergraduates with low personal stakes; whether the same levers work on professional judgment under deadline was not tested.

PracticeCognitive forcing — manufacture the reason: make the source untrustworthy by default, or put the answer in an unexpected frame, rather than exhorting people to think harder.

After AIAn assistant is the trustworthy-source condition by default — fluent, unhesitating, apparently disinterested — which is the condition in which low scorers did not look.

Becauseafter brainstorming with or without pay, high scorers played with the extra puzzles more when unpaid and low scorers more when paid, and low scorers loafed in a shared-responsibility group while high scorers did not — though both loafed at screwing nuts onto bolts

LimitsTwo experiments, with brainstorming as the task and undergraduates as the sample. The theory that the disposition grows out of competence feedback and is worn away by surveillance and time pressure has never been tested.

PracticeDo not pay, measure or deadline the part of the work you want people to find interesting; the undermining effect lands on the people who were enjoying it.

After AIThroughput metrics on assisted work are exactly the extrinsic reward the result warns about, and they fall on the people whose willingness to think was the thing being relied on.

Becauserecall of task-relevant material pools to d = .355 across twenty-two tests, task-relevant thoughts listed to d = .205, information seeking to d = .457, annoyance at demanding tasks to d = −.447 and enjoyment to d = .306

LimitsEricsson Peak is the counterweight: what predicts performance is structured practice, not disposition, and a trait explaining this little leaves most of the variance elsewhere.

PracticeUse the measure to predict tendencies in a group, not to sort individuals; a third of a standard deviation will not tell you about the person in front of you.

After AISmall stable differences in who bothers to check become larger in consequence when checking is the only thing left that the person is doing.

Becausemood, framing and priming shifted high scorers' thoughts with their attitudes following, while the same manipulations moved low scorers directly or not at all — and several of these demonstrations belong to the 1980s–1990s social-priming literature that largely failed to replicate after 2011

LimitsThe review's own caveat is the load-bearing one: a correction is not an accurate judgment, since nobody knows the true size of the influence being adjusted for. And the priming evidence would now have to be redone.

PracticeCognitive forcing — name the likely bias out loud before the judgment rather than after; correction only arrives once the influence is visible.

After AIThinking more about model output is not thinking more impartially about it; what helps is being told in advance where this particular system goes wrong.

Becauseeleven tests of the interaction pool to d = .312, and to d = .539 across five studies where participants rated message quality directly; only low scorers were moved by a speaker's claim to be giving ten reasons rather than two, and only when the topic was not personally relevant

LimitsThe architecture behind the result — the elaboration likelihood model's split between a central and a peripheral route — was challenged within three years by single-process accounts and is now one position among several.

PracticeCognitive forcing — in a review, separate the cue from the argument explicitly: who produced it, how confident it sounds, how long it is, then the case itself.

After AIModel output supplies cues in quantity: length, structure, hedging, a confident register. The disposition predicts who reads past them, and the cues have never been cheaper to manufacture.

Becausethe run is Davinci at temperature zero, with a third answer added wherever a problem offered only the correct one and the standard human error, so a human-like wrong answer could be distinguished from an idiosyncratic one.

LimitsThe same performance comes apart under edits no person would be troubled by, so the vignettes license nothing about reasoning; these are also the most famous items in psychology and were certainly in the training data.

PracticeLog the moment a model's human-like error makes you feel you understand it. The resemblance is the strongest anthropomorphic cue in this literature and it survives knowing better.

After AIInstruction-tuned successors answer the reflection test correctly and the intuitive errors disappeared, which means the human-likeness was a property of one model at one moment — and nobody can say whether the change was reasoning or contamination.

Becausethe contrasts are the ones Kahneman and Tversky used, run on the model's choice probabilities rather than on a sample of people, which makes each signature a separate measurement rather than an impression.

LimitsThese contrasts are also textbook items, so exposure cannot be excluded; the split may be a fact about which of the six are more often written down rather than about the model.

PracticeLog the resemblance claim you are tempted to make, then check it signature by signature. Half the pattern appearing is the normal result, and it is not what 'it thinks like a person' describes.

After AIA split result is the honest finding and the hardest one to report. Both the parrot reading and the human-likeness reading have to absorb half of it, and neither predicted the split.

Becausethe Peterson 2021 benchmark scored by regret gives t(29134) = −16.85 against chance and t(29134) = −11.50 against people, and only Davinci beat chance, with Ada, Babbage and Curie failing.

LimitsOne benchmark of described gambles at one moment, on a model family now several generations old; scale within the family predicted the result, so the finding is partly about model size.

PracticeAudit capability on problems generated fresh rather than on published ones. The gap between the two scores is the size of the contamination you cannot otherwise see.

After AIThis is the design worth copying: procedurally generated items where the answer key is computed, not remembered. It is the only kind of evaluation that has not aged badly.

BecauseHagendorff, Fabi and Kosinski reported the vanished biases in the same year, and Ullman 2023 showed that trivial alterations to theory-of-mind vignettes collapse the performance the unaltered versions appeared to demonstrate.

LimitsThe reversal is documented; what caused it is not, and the data cannot separate better reasoning from items leaking into training. The paper's own successors decline to choose.

PracticeReview any measurement of a model with a date attached, and keep the method rather than the number. The number describes one endpoint at one moment; the design is what you can run again.

After AIEverything measured about a model dates within a year. Anyone building a practice on a published capability figure is building on the least durable part of the result.

Becausemean regret was 2.72 against a human 3.24, t(38878) = −5.03, while in Wilson's horizon task there was an effect of estimated reward difference but no interaction with horizon and no horizon effect under unequal information — so neither directed nor tuned random exploration was present — together with a strong bias toward the machine it had seen more often, which the authors trace to next-word prediction treating a recent phrase as likely to recur.

LimitsOne paradigm, one model, and an exploration analysis that rests on the absence of an interaction — a null result — rather than on a positive alternative account.

PracticeAudit the domains where good performance would come from exploring rather than exploiting. A system that beats you on average while never exploring is exactly the collaborator that will not find the option nobody has tried.

After AIBetter outcome and alien process is the characteristic finding of this whole literature. It is also the reason the outcome cannot be used to infer the process, in either direction.

Becausethe Waldmann and Hagmayer design, put as chemicals in wine casks, produced identical estimates under two structures that require different answers, and the model also failed to lower its estimate of C after observing that B was absent; the authors' own phrase is that it failed spectacularly in using an underlying causal structure.

LimitsOne design on one model generation; causal competence was claimed for GPT-4 the following year, though on published suites, and scores on freshly built interventional tasks remain far below scores on recall.

PracticeReview the failure modes in any work where the difference between seeing and doing matters — root cause, incident analysis, anything counterfactual. This is the capacity the evidence says is missing, and a fluent answer will be supplied anyway.

After AIOf everything in this unit, this is the failure most likely to be live in current systems and least likely to be visible in the output. The answer will have the right shape.

Because200 simulations of twenty rounds made it less likely to repeat a first-stage choice after treasure through a rare transition, t(1982) = −6.16, and more likely to repeat after a rare transition that paid nothing, t(1814) = 2.33 — the pattern a planner produces.

LimitsTwo paradigms with opposite verdicts, which is a finding about the inadequacy of the question rather than about the model; the reflection items are famous and the two-step task is not, so exposure differs between them.

PracticeAudit deliberation per task, not per system. A model that plans on one structure and answers reflexively on another is the normal case, and the two-system vocabulary does not transfer.

After AISystems trained to reason before answering have made the older stimuli harder to read still, since a battery designed to catch a shortcut in a fast answer says little about a system that deliberates.

Becauseasking for the probability that the cab was black, a colour not in the problem, returned 0.2; reordering Wason's cards from A K 4 7 to 4 7 A K changed the answer from A and 7 to A and K; and restating the first reflection item so the bat costs a dollar more than the bat left the answer at ten cents.

LimitsThree demonstrations on one model; the strong form — that no famous item can test anything — is an inference, and models since have become far more robust to reordering without becoming robust enough to skip the check.

PracticePerturb before you trust: change a name, reorder the options, alter a number that should not matter. If the answer moves, what you measured was the wording.

After AIUllman showed the same thing for theory-of-mind vignettes in 2023, and the check is now routine. This is the single most transferable practice in the unit, and it applies to any eval you are shown.

Becausethe recommendation is argued from the training objective rather than demonstrated, and the evidence since is mixed: causal competence claimed for GPT-4 rests on published suites, while scores on freshly built interventional tasks remain far below scores on recall.

LimitsAn argument from the training objective, offered as a direction rather than a result; agentic systems that act and observe are now common and the prediction has not been cleanly tested.

PracticePut causal judgement on the list of things that stay with you. Where the question is what would have happened if you had done otherwise, the record the model was trained on does not contain the answer.

After AIThe distinction between seeing and doing is the sharpest line anyone has drawn between what these systems have and what supervision requires. It is also the one least visible in how they perform.

Becausethe elicitation results come with a set of procedures — multiple-choice formats cut reasoning off before it happens, answer order should be shuffled, model sizes should be run across since the largest is not always the most capable, and a marker string separates the answer from the reasoning for scoring.

LimitsThe elicitation gains are themselves scale-dependent and were measured on models where chain-of-thought was still a prompting technique rather than trained-in behaviour.

PracticeSpecify how you asked before reporting what a model could not do. Elicitation is part of the measurement, and a failure you have not tried to elicit past is not a finding.

After AIMost public claims that a model cannot do something are elicitation results in disguise. The same is true in the other direction, which is why both need the procedure stated.

Becausethe argument is that the most interesting evaluations are not held-out exemplars of the training task, with in-context learning as the standard case of a capability nobody encoded.

LimitsStated as a prescription rather than demonstrated, and held-out sets still work where the data is genuinely new — private corpora, freshly generated items, tasks invented after the training cut-off.

PracticeStop treating a benchmark score as a measurement and start treating it as a claim about what the model saw. Where you need to know whether a system can do your work, the test has to be one you built.

After AIEvery published benchmark is now in the training data of the next model. The measurement problem is structural and gets worse with each release, not better.

Becausethe survey-instrument results have held since the paper predicted them, so a measured personality is now reported as a property of a prompt, and self-report is kept only to characterise a default and to measure steerability.

LimitsInstability under rewording is well established; whether there is a stable underlying disposition that better instruments could reach is not settled, and steerability measurements assume there is a default to steer from.

PracticeLog every time you take a model's statement about itself as information — its confidence, its preferences, its account of its limits. None of it is a reading of anything durable.

After AIUsers ask assistants about themselves constantly, and vendors publish the answers as character. The circuit-level finding is the same from the other side: the account and the computation are different processes.

BecauseChomsky's 1965 performance-competence distinction was carried to machines by Firestone in 2020 and Lampinen in 2022, and the everyday case is that people make mistakes on easy arithmetic without lacking arithmetic.

LimitsA borrowed distinction argued by analogy; it cuts both ways and can excuse any failure, which is why it needs the control it recommends rather than standing alone.

PracticeReview both directions of your evidence. A model that failed your task may fail your prompt, and one that passed may have seen it — and neither result means much without a control condition you designed.

After AIThe asymmetry matters for hiring a system as much as for hiring a person. Most deployment decisions rest on a handful of passed tests with no control at all.

Becausefour arguments are offered for the level — where people meet the model, complexity beyond what weights explain, behaviours that appear only at scale, and access through an interface — set against mechanistic interpretability, which must simplify the data, the model or both to make its interventions possible.

LimitsAn argument for a method rather than a result; circuit-level work now explains particular behaviours the behavioural level can only describe, and the two levels have not been reconciled.

PracticeAudit the systems you use by what they do, in your domain, on your work. It is the level you have access to, and for a closed model it is the only evidence anyone outside the laboratory has.

After AIThe model is the first research subject that answers questions on demand and cannot be opened. That combination is why psychology's instruments were borrowed, and why its measurement problems were inherited with them.

Becausethe ethnographic distinction is borrowed and argued by symmetry rather than tested; the counter-case is that the mechanisms differ and the words are normatively loaded, widening the gap between what specialists mean and what the public hears.

LimitsContested and conceded in practice: the thick description won by default, adopted by writers who never pause over whether the word is earned, which is at once what the paper argued for and what it warned about.

PracticeLog which mental words you use about a system and what each one would have to mean to be earned. The vocabulary is useful and it is also the mechanism by which projection becomes permanent.

After AIThe words are now standard in technical writing, which means the argument was settled by usage rather than by evidence. That is worth noticing every time one of them does explanatory work in your own reasoning.

Becausethe reversal is reported against Binz and Schulz's own finding and the paper declines to choose between the explanations; Dasgupta 2022 sits on the other side, finding that models, like people, reason more accurately about believable familiar material than about the same logic stated abstractly.

LimitsContested because it is undecidable with the available evidence, not because the sides disagree about the facts; a decisive test would need a model whose training data could be inspected.

PracticeLog the interpretation you are inclined to give a model improvement, and note that the contamination explanation is always available and rarely tested. Holding both is the honest position.

After AIImprovement and exposure are confounded in every published capability trend. That is a permanent condition of evaluating systems trained on everything, and it applies to the ones you are buying.

Becausethe prompt-sensitivity results collected there are why a battery of varied prompts rather than an item is the unit of measurement.

LimitsPrompt sensitivity has fallen with each generation without disappearing, and a battery is expensive, which is why single-prompt results remain the norm in practice.

PracticeRun anything you care about several ways before concluding the model can or cannot do it. One prompt is one sample, and the variance between phrasings is often larger than the difference between models.

After AIThe sensitivity traces back to a mechanism with no intrinsic sense of order, patched from outside. That is the clean line from the architecture to the measurement problem.

Becausethe rule is stated as a prescription — keep the structure and change everything else, wording, agents, order of events, actions — with procedurally generated tasks named as less exposed by construction; the warning has aged better than the results it was aimed at.

LimitsA prescription rather than a finding, and a hard one to satisfy: rebuilding a task around its structure may change what it measures, and there is no way to confirm that it has not.

PracticeSpecify your evaluations by structure rather than by item. Write down what the task is testing, then generate instances — it is the only form of test that survives the next model release.

After AIThis is the most durable advice in the unit, and it is the reason the paper outlived the results it was surveying.

BecauseLush and Dienes, with Seth as an author, found illusion strength tracking hypnotic suggestibility, most published studies never controlled for it, and the challenge is unresolved

LimitsSuggestibility explaining variance does not prove it explains the whole effect, and the clinical dissociations stand whatever happens to the rubber hand. Seth is an author on the challenge to his own literature, which is a reason to trust it and not a reason to call it settled.

PracticeBefore citing a demonstration, ask what the participants were told to expect; if the expectation and the effect cannot be separated, the demonstration is not evidence about mechanism.

After AIThe same problem has moved into model evaluation. A demonstration in which the operator knows which condition is which, and what the result is meant to show, measures the operator as much as the system.

Becausein 104 couples the control condition reached r = .5 with average error 1.5 and the perspective-taking condition .39 with error 1.7; the fisheries negotiation collapsed fastest under perspective taking, and the result was extended across two dozen experiments in 2018

LimitsImprovement does appear when people are given the other's perspective directly rather than imagining it, so the failure is of imagination and not of perspective. Klein's expert intuition literature finds the opposite in domains with valid cues and fast feedback.

PracticeStop writing specifications by imagining what the user wants; write down the question you would have to ask them, and ask it.

After AIThe move is now made against a model as well: imagining what the system will do with a prompt is the same substitution, and the cheap alternative — run it and look — is the equivalent of asking.

Becausedating partners predicted each other's answers 44 per cent of the time and believed they had got 82 per cent right, and length of relationship predicted confidence and was unrelated to accuracy

LimitsThe accuracies are modest rather than zero, and the sense beats chance nearly everywhere it has been tested. The gap is the finding; nothing here says mind reading does not work.

PracticeWrite down what you think a colleague will say before the meeting, then score it; the number, not the feeling, is what you know about them.

After AIThe same gap now governs how people read models. Long use produces a strong sense of knowing what the system will do, and there is no reason to think the accuracy moved with the confidence.

Becausechoice blindness: people were handed back the photograph they had rejected and only 27 per cent noticed, while the rest explained a choice they had not made as fluently as anyone explained a real one

LimitsThe dramatic demonstrations are laboratory cases with small samples, and they show that a story can be supplied, not that every story is. Epley's own claim is that introspection is reliable about what you feel and want, and unreliable about why.

PracticeAsk what you did rather than why you did it, and get the why from a record made at the time.

After AIExplaining why you accepted a model's suggestion is exactly the case the experiment covers: the acceptance came first, and the reasons are constructed afterwards with the same fluency whether or not they were the reasons.

Becauseacross the three couples conditions confidence barely differed while error halved, and senders of sarcastic sentences by e-mail expected 80 per cent comprehension, were at chance, and still believed they had got nine in ten right

LimitsConfidence does track accuracy in domains with fast, valid feedback, which is Kahneman and Klein's condition; the claim holds where feedback is absent or delayed. The couples study is one design with modest samples.

PracticeChoose between two ways of working by scoring both against an outcome, never by which one felt better to use.

After AIIt is the reason self-report about AI productivity is worthless and the reason METR's developers believed they were faster while being slower. The comfortable method and the effective method give the same feeling, so only the measurement separates them.

Becauseflattered participants rated their own questions and the machine higher whether or not they had been told the praise was randomly sampled, and they remembered the warning correctly afterwards

LimitsSixty participants in one design, with no effect size reported. The asymmetry is the finding — people have no motive to check good news and a strong one to find a reason to discount bad news — and it says nothing about how long the inflation lasts.

PracticeDiscount an assistant's approval of your plan to zero, and treat only its objections as carrying information.

After AIA model's praise is not randomly sampled but it is not evidence either, since agreement is what the training rewarded. The experiment shows that knowing this does not protect you.

Becausetwenty of twenty-two adjectives were rated higher and the spread was narrower, replicated with a paper questionnaire and with recorded speech, and a second voice on the same machine restored honest ratings

LimitsTwenty-two participants in the original, small samples throughout, and no test of how long the politeness lasts under repeated use. Whether the effect survives with modern conversational agents has not been settled.

PracticeNever evaluate a tool inside the tool; record the verdict somewhere the tool has no part in, and let a different channel ask the question.

After AIThe finding names a structural flaw in how models are now assessed: asking a system to critique its own output, or rating an assistant in its own chat window, is the politeness study with the boxes removed.

Becauseparticipants told the authors the opposite of the data every time, and everyone denied holding gender stereotypes while weighting praise from the male voice more heavily, including about a third machine across the room

LimitsWaytz and colleagues show a questionnaire that does predict behaviour, so self-report is not uniformly useless — what fails is asking people to report an automatic response they have no access to. The denials are also evidence for the mindlessness account rather than against it.

PracticeStop surveying yourself or the team about how AI is being used, and instrument the behaviour instead.

After AIIt disposes of most of what is currently known about AI at work, which comes from asking people how much it helps. The number that matters has to be taken from the work, not from the worker.

Becauseteamed users rated identical suggestions more helpful and actually changed more of their rankings to match, while shared identity alone produced a weaker effect and added nothing once interdependence was present

LimitsOne study, small samples, and a wristband is a cruder manipulation than the arrangements real teams have. Interdependence being sufficient does not show it is necessary.

PracticeForm your own answer and record it before the assistant sees the problem; a shared stake in the output is enough to move you without any argument being made.

After AIEvery product framing on offer — copilot, assistant, teammate, co-intelligence — is the treatment condition, and it is now permanent rather than induced for an hour. Mollick prescribes as technique exactly what this study measured as a bias.

Becausehuman-framed descriptions of a market raised confidence in continuation and high scorers made that prediction more readily, as though a run of prices expressed settled intentions

LimitsOne result in a paper of small studies, reported as a correlation. Trend extrapolation is sometimes right, and the claim is about unearned confidence rather than about the direction of the forecast.

PracticeRewrite a capability forecast in mechanical language before believing it, and log the confidence before and after.

After AIAlmost every public forecast about these systems is written in human terms — the model learns, wants, reasons, improves — and this result says the framing alone adds confidence that the curve continues. It is the mechanism behind the scaling narrative as much as any evidence is.

Becausefifty-four adults choosing between a person and a machine in six scenarios, with higher scorers choosing the machine after controlling for age, plus the 2014 driving simulator in which a named, gendered, voiced car was trusted more and blamed less for a collision another driver caused

LimitsThe scenarios were hypothetical and cost the participants nothing, the sample was fifty-four people in one city, and the relation is correlational. The authors name all three objections themselves.

PracticeAudit what you have handed over against how humanlike the interface is; if the two move together, the interface is deciding and not you.

After AIThe two variables no longer vary independently. Every capable system now arrives with the humanlike presentation, so the condition that predicted more delegation is the default rather than a design choice.

Becausethe load results are direct — a hard working-memory task lowers performance on an unrelated frontal task immediately afterwards, and Fehr's magnetic silencing left subjects knowing an offer was unfair and unable to act on it — but the resource framing behind them is ego depletion, which failed a multi-site replication in 2016 and has not recovered

LimitsEgo depletion failed replication, and the glucose findings rest on it. The parole study the argument leans on has been challenged on case ordering, and simulations suggest the effect, if real, is much smaller than reported.

PracticeCognitive forcing — put the decisions that need the expensive organ where the load has not already been spent, and force the check rather than trusting that it will happen.

After AIThe load an assistant removes and the load it adds are not the same load. Reviewing generated work at the end of a long day is precisely the condition under which the effect predicts the review will be nominal.

Becausethis is the author's own field and the chapter's closing section reports it as the part not seriously challenged; prolonged high-dose glucocorticoids in healthy volunteers impaired working memory into the range seen after frontal damage

LimitsThe turning point of the inverted U differs enormously between people, so no threshold generalises from the group to the individual. The working-memory result used exogenous doses higher than most stress produces.

PracticeFailure-mode review — in the failure-mode review, ask what the person was doing for the ten minutes before the decision, not only what they decided.

After AIThe failure mode under an incident is to prompt the same way harder and faster. The model answers every time, which removes the friction that used to end the loop.

BecauseBCG's 758 consultants got the off-frontier task right 84 per cent of the time alone and 60 to 70 per cent with GPT-4, and Dell'Acqua's 181 recruiters given good AI did worse than those given bad AI, spending less time per résumé and not improving

LimitsOne task in eighteen was placed outside the frontier by design, which is a sharper contrast than ordinary work provides, and the BCG consultants had no monitoring role and no way to know which task it was. The recruiter study measures effort, not skill decay.

PracticePut a deliberate check on the tasks you are least suspicious of, because suspicion is what the good assistant removes.

After AIThis is the book's own falling-asleep finding, and METR's 2025 trial pointed it at the people who were supposed to be immune: experienced developers on their own repositories, slower while believing themselves faster.

Becausehis own three paragraphs on the Schwartz case, generated with web access, were wrong in three small ways — the number of fake cases, the co-counsel's role, the reason for the sanction — and citation error rates fell from 98 per cent in GPT-3.5 to 20 in GPT-4 without reaching zero

LimitsA single demonstration on one passage, plus a citation study; neither measures how often small errors survive review in practice. Falling rates cut both ways, since rarer errors are less expected and therefore less looked for.

PracticeCheck the checkable facts one at a time against a source, rather than reading for whether the whole passage seems right.

After AIWritten about AI, so the test is what has happened since: courts have sanctioned lawyers for fabricated citations often enough that the cases are tracked in a public database running to the hundreds.

BecauseOptum's Impact Pro predicted spending rather than need, so Black patients with the same risks were enrolled less often, and COMPAS's developers speak of predicting crime while its data records arrests

LimitsChristian treats the gap as a specification problem with technical fixes arriving; the authors treat it as constitutive in social prediction, because need, crime and merit have no recordable proxy at all. Which is right is not settled by these cases.

PracticeWrite down the recorded variable and the thing you actually mean, side by side, and treat the gap between them as the specification.

After AIWritten about AI, and the substitution moved into training itself: a model tuned on what raters preferred optimises rated helpfulness, not helpfulness, which is the same gap with a much larger surface.

BecauseEpic's sepsis model claimed 76 to 83 per cent and measured 63 against 50 for chance, four years later and only because one hospital with researchers used it; Gundersen and Kjensmo found none of 400 papers met their reproducibility criteria; civil-war prediction collapsed once the leakage was fixed

LimitsLeakage is one mechanism among several, and its correction returns a field to where it was rather than showing the work was fraudulent. Narayanan is a co-author on one of the affected papers, which he reports.

PracticeTreat any accuracy figure without external replication as unmeasured, and ask what the chance baseline was before reading the number.

After AIWritten about AI, and their treatment of benchmarks has been vindicated from an unexpected direction: contamination of test sets by training data is leakage at industrial scale, and it is now a mainstream concern rather than a footnote.

Becausethe 1997 pneumonia model in which asthma appeared to lower risk, because asthmatics with pneumonia were sent straight to intensive care, so deploying it would have sent them home

LimitsThe asthma case is the canonical example and a clean one; it does not tell you how often the pattern occurs. Where the intervention is weak or the data predates it, a prediction can transfer intact.

PracticeFor any model you are about to deploy, write down what the people in the training data did in response to the thing being predicted, and ask what happens when the model replaces them.

After AIWritten about AI, and the same structure now appears one level up: a model trained on how experts handled the hard cases will look accurate exactly until it is used to replace those experts.

Because160 teams with over 10,000 variables per child and held-out age-fifteen outcomes could not beat a baseline of four variables, and the best-missed case was a child helped by a neighbour no survey question asked about

LimitsCritics point out that the argument establishes something narrower than 'predictive AI will never work': aggregate forecasting still gains, as seasonal influenza does, and the authors concede it. The Challenge is one cohort and six outcomes.

PracticeAsk what the best published attempt at this prediction achieved before commissioning a model to make it; if nobody has measured, assume the answer is near chance.

After AIWritten about AI. Three further years of far more capable models have produced no visible improvement in forecasting who will keep a job, commit a crime or finish school, which is the strongest evidence the book's claim could have received.

Becauseit is a judgement about where the risk sits, argued from remedies rather than measurement — treat a system that conceals its gaps as fraudulent, make it recite its limitations as a drug company recites side effects, and hold anyone using one on people's lives to the standards set for dangerous equipment.

LimitsHe paired the warning with a forecast that deep learning would not deliver superhuman intelligence within fifty years. Five years later those systems had taken most of what can be tested at a keyboard.

PracticeReliance audit — list the decisions where you accept a machine's output without a way of checking it, and ask what you assumed it understood.

After AIHe restated it in *The Atlantic* in May 2023, calling systems that pass as people the most dangerous artefacts in human history and proposing that making one be a serious crime, with liability on the makers — this book's closing proposal, repeated once the thing existed.

Becauseit is a test with an audience, a count and a result; Frances Arnold's protein breeding runs the same way, the goal and verdict hers and the search nothing she follows.

LimitsIt was an evening at a festival, not a controlled trial, and there is no report of how the pieces were chosen or how the audience scored on genuine ones.

PracticeCalibration log — before accepting machine output in a field you claim to know, run the test those listeners failed: record your judgement, then check it. Taste is not the ability to tell provenance.

After AIHe offered it as a preview, with superhuman intelligence fifty years off. It is now the ordinary condition of reviewing work, and the honest answer to whether you can tell the output apart is often theirs.

Becausean elevator's bracketed comments and mnemonic names help the programmers and then vanish at compilation, while a bacterium's protein networks are optimised for what it meets with nothing that ever played a comment's role.

LimitsReverse engineering works only where the parts really are good for something, the same assumption that produces the overreading Dennett warns about elsewhere. Nor is the recovered reason anyone's: there may have been no designer at all.

PracticeFailure-mode review — recover a system's reasons the way a biologist does, by knocking a part out and watching what breaks, not by reading what it says about itself.

After AIWritten about elevators and bacteria, it now describes anyone using a trained model: the rationale for a weight is real, testable by ablation, and recorded nowhere. What to distrust is not the model's silence but its fluency when asked why.

Becauseit is Debner and Jacoby's 1994 masked-priming result, and the reversal carries the argument: the same brain discriminated the word in both conditions and the instruction could only be obeyed in one.

LimitsDennett uses it to say what machines lack, and the step from one priming paradigm to a general absence of practical reason is his rather than the experiment's.

PracticeCognitive forcing — stop relying on an instruction to ignore what you have already been shown. Change what reaches you and when, because a warning arrives too late to undo the influence it warns about.

After AIIt is the cleanest statement of why seeing an assistant's answer before forming your own cannot be undone by resolving to be careful. The suggestion has already been discriminated; setting it aside works only on the part you noticed.

Becausethe reasoning is given in two figures rather than argued — the man shot with a poisoned arrow who will not have it drawn until he knows the archer's caste, and dies uninformed; and the raft carried on the head by a man who has misunderstood what it was for

LimitsUsefulness is a criterion the questioner cannot apply, since he does not know what the answer would have done for him — the objection the Mandukya Karika presses by answering the questions anyway. And the same silence is elsewhere read as having a content.

PracticeSpecification discipline — before asking, say what the answer would change; a question whose answers all lead to the same next action is the arrow left in place.

After AIA system that answers everything removes the cost that made the triage visible; the questions that do nothing are now the cheap ones, so nothing stops you asking them all day.

Becausethe instruction is in the record, and a disciple is told to examine even the Tathagata; but Bhikkhu Bodhi's 1988 essay argues it is addressed to people who cannot yet judge between teachers, offering a working criterion rather than a licence

LimitsThe criterion names what to reject, not what to accept, and the passage is now the most quoted in modern Buddhism because it is read as more permissive than it is. Rahula's surrounding claim, that such freedom of enquiry is unheard of elsewhere, has not held.

PracticeCognitive forcing — the passage allows one test, finding out for yourself; an answer you have already read cannot be tested that way, so the expectation goes on paper first.

After AIThe list of bad reasons has no entry for a source that agrees with whatever you assert, and that is now the hardest to see: testing it yourself returns your own view, signed twice.

BecauseGoldberg's models tracked their judge at about .80 across 237 studies and lost no accuracy replacing them; of ten thousand sets of random weights Yu and Kuncel generated on 847 executive candidates, 77 per cent beat the human experts in one sample and all ten thousand did in the other two.

LimitsThe model subtracts your subtle rules along with your noise, and only nets out because the conditions that trigger a subtle rule are rare. It assumes the cues are already chosen and quantified, which is most of the work in real judgment.

PracticeCalibration log — write down the cues you actually use and the weight you give each, then score the next batch both ways. Where the checklist and the impression disagree, the burden is on the impression.

After AIThe result is read as an argument for automating judgment. The narrower reading survives better: consistency is doing the work, not intelligence — so a model that answers differently on different runs has thrown away the caricature's only advantage.

Becausethe spread is measured but the stopping rule is an interpretation — a probability short of nought or a hundred cannot be confirmed by any outcome, so the reward has to be self-administered.

LimitsOne demonstration case, and the spread is not separated into the stopping signal and attending to different cues in the first place. Klein's commanders act on the same signal where it is demonstrably reliable, so it is uninformative rather than worthless.

PracticeWitness practice in review — treat the moment a conclusion feels settled as a fact about you rather than about the case. Note it, then ask what evidence was still available.

After AIA fluent answer supplies the coherence that normally arrives at the end of your own work, so the signal fires before you have done any. The stopping rule for review cannot be that the thing now hangs together.

BecauseVul and Pashler measured the crowd within at those intervals, and Herzog and Hertwig raised it by instructing people to assume their first answer was wrong and work out why.

LimitsThe gains are small and measured on quantity estimates, where averaging is arithmetically sensible; nothing shows it working on judgments that are not numbers. Waiting three weeks is not available for most decisions, and arguing against a position you have just taken is the part most likely to be performed rather than done.

PracticeCognitive forcing — write your estimate down, answer again as if the first were wrong and say why, then average the two rather than choosing between them.

After AIThe cheapest version is to make the second draw someone else's: ask the model to answer, then to argue its answer is wrong. What comes back is an independent-looking second opinion from the same instrument, which is what the crowd within is, and worth as little.

Becausethe ceiling is corroborated across fields — Tetlock's twenty-year study of nearly three hundred political experts; the Fragile Families challenge, where 160 teams predicted eviction at .22 and school grades at .44; and a survey in which 3 per cent of 708 behavioural-science studies reported .50 or more.

LimitsObjective ignorance is a property of the task, so it bounds the machine as tightly as the person; this is not an argument for replacing the judge. Correlations are a poor way to feel a difference — .28 is much better than nothing at scale and nearly useless for one decision.

PracticeDomain audit — find the best measured accuracy in your domain before deciding how far to trust anyone's judgment in it, your own included. A method that lifts validity from .28 to .44 is a large improvement and will still feel like a waste of time.

After AIIt sets a ceiling no model clears, because the limit is in the world rather than the predictor. A system reporting high confidence about individual human futures is reporting something other than accuracy.

Becauseamong the underwriters, differences in average premium were 20 per cent of system noise and the patterns 80 per cent; models of 173 bail judges over 141,833 cases gave 26 per cent of total variance to stable pattern noise against 7 per cent for level noise.

LimitsSeparating pattern from level noise needs each person to judge many cases, so the finding lives in a few legal and insurance datasets. Some of it is a valid personal rule the outside data cannot see, and the authors expected level noise to dominate and were wrong.

PracticeCalibration log — your average error will not show this. Record the case type alongside the judgment and look for the kind of case where you are reliably further from everyone else.

After AIA model has no stable idiosyncrasy of this kind across sessions; its variability is a fresh draw each time. That makes it easier to audit than a colleague and harder to work around, because there is no consistent bias to correct for.

Becauseevery audit in the book produced a number with no criterion of correctness — the insurance audit, the 1974 study of fifty judges, the forty-two investors valuing one stock.

LimitsScatter shows error is present, not which judgment is better; a team that agrees perfectly may be wrong together. Gelman and others noted that psychometrics has measured inter-rater reliability for a century, so what is new is the audit, not the object.

PracticeDomain audit — give the same case to two people, or the same assistant twice, and record the spread. In most professional work the outcome never arrives, and this does not wait for it.

After AIThe cheapest evaluation available on work you cannot check. Ask the same question in two sessions: where the answers diverge is what the model is guessing at.

Becauseaverage outrage and average punitive intent correlated at .98 across twenty-eight scenarios, so people match a punishment to an intensity of anger; dollars have no upper bound, and each juror invents an anchor on the first case and judges consistently against it thereafter.

LimitsJury awards are extreme: a one-off judgment on an unbounded scale by people forbidden to know what other juries awarded. Ranking also destroys information — it cannot say every case was mild — and forcing one onto a set with no real differences creates its own error.

PracticeCalibration log — score against comparable cases rather than on an open scale. "Better than the last three, worse than the best one we shipped" carries information that a 7 out of 10 does not.

After AIEvery evaluation that asks for a score out of ten inherits this. Ratings of helpfulness or risk measure the rater's anchor as much as the output, which is why comparison against a fixed reference set survives where absolute scoring does not.

Becauseit follows from Gauss's least squares (1795); in the book's worked case, halving the noise of a department biased high by ten points makes the forecasts look worse — 98 per cent now erring in the same direction rather than 84 per cent — while overall error falls by as much as halving the bias would.

LimitsIt holds for squared error and for predictions aiming at a true value, not for evaluative judgments where the two directions cost differently — underestimating an elevator's load is expensive, overestimating it lethal. Blastland and others add that where a group is already biased, cutting noise concentrates the errors on one side.

PracticeCalibration log — log the spread of your estimates as well as their average distance from the outcome. Estimates wrong in the same direction are a cheaper problem than estimates wrong in every direction.

After AIA model's error is reported as an average and almost never as a spread. Sampling one prompt repeatedly measures the second term, which a single benchmark score hides.

Becausethose sit alongside nearly seven hundred thousand primary care visits in which opioids were prescribed more often late in the day, and controlling the conditions does not drain it: alertness, sleep, time of day, practice and difficulty explained 11 per cent of the variation within a subject.

LimitsOccasion noise is the smaller component: in every case the authors could measure, people disagreed with themselves less than with each other. The time-of-day and weather results are single-study correlations of a kind that has replicated poorly.

PracticeCognitive forcing — on a judgment that matters, make it twice on different days, and treat the gap between your answers as the error bar you would otherwise have invented.

After AISampling temperature makes the same point about a model, and the same repair works: ask twice, compare, treat the divergence as the uncertain part. The model's second draw takes a second; yours takes three weeks to be worth much.

Becauseemployees judged five realistic case files independently, not knowing what was measured; the same exercise gave 43 per cent for claims adjusters and 41 per cent among forty-two investors valuing one stock, and 10 per cent was also the median guess of 828 senior executives.

LimitsOne company's underwriting, on written files rather than live business with colleagues to consult. The 55 per cent is a median of pairwise differences, a larger-sounding statistic than a standard deviation, and the cost of it is one executive's estimate.

PracticeDomain audit — have two reviewers judge the same item blind, and ask them beforehand how far apart they expect to be. The gap between the guess and the measurement is the finding.

After AIThe comparison everyone reaches for is the model against a human expert. That baseline was never a point: it was a cloud 55 per cent wide, and a benchmark scored against one expert's labels inherits the cloud as its ceiling.

Becauseit is a reading of one design protocol turned into a general account. He defends it as rigour — striving to make the situation conform to one's view while staying open to the evidence of failing — but offers no case of anyone fooled.

LimitsTesting stops, on Schön's own account, when the practitioner finds the changes satisfactory — a rule held by the person doing the shaping. Tetlock shows what that costs where the answer arrives late or never.

PracticeCognitive forcing — write your own answer before opening the output, and name in advance the result that would make you drop the frame. If nothing can push back, this is not an experiment.

After AISchön's safeguard was that sites, patients and materials resist. A model does not. It adopts the frame and supplies the confirmation the user was shaping for — rigorous practice turned into talking yourself into an answer.

Becausethe behaviour was observed across an age range with plain and with weighted blocks; slightly older children asked to add blocks to a balanced one piled up to ten at the centre, and those who changed ended by hefting each block and placing it at its point of balance.

LimitsChildren balancing blocks, generalised to professionals, with no adult version offered. The younger children also lacked the idea of a centre of gravity, so what looks like defending a theory may be the absence of an alternative.

PracticeFailure-mode review — when a method fails twice, write down the rule you have been applying instead of applying it harder. "This cannot be done" reports on your theory before it reports on the task.

After AIRe-prompting is the same move. A user who sends the request again, longer and firmer, is the six-year-old pressing the block onto the middle of the bar, and the task that cannot be done is usually a problem framed wrong.

Becausethe four-way schema of surprise against consequences is extracted from two teaching protocols — a move that surprises but pleases refutes the theory and is kept, an unsurprising failure raises a question about its scope — and nothing measured it.

LimitsTwo transcripts, both of a teacher with a student. Eraut objected in 1994 that the action-present in most work is too short for this, so what gets praised is either rapid intuition, which is not reflection, or review afterwards, which is not in action.

PracticeFailure-mode review — do not wait to be surprised. Keep a written list of how the output goes wrong in your stack and check against it: a failure mode that does not surprise you triggers no review.

After AIThe trigger has been removed. Fluent, plausible, small errors produce no surprise, so the mechanism never fires. Reflection has to be scheduled rather than provoked — the opposite of what he argued for.

Becausethe comparison is with real hardware, which has interruption circuitry to freeze a program and a private memory to restart it where a brain has neither, but nothing was measured. The substitute is architectural: a second brain wired so the first is its whole world, able to notice that it is confused or repeating itself and to stop it, understanding nothing of what it means.

LimitsThe B-brain is in the chapter's own list of ideas offered without evidence that produced almost no experiments. Minsky concedes a failure mode: a watcher that mistakes a long column of addition for a pointless loop produces a person who complains of being bored.

PracticeCalibration log — stop trying to catch your own reasoning while it runs. Write the prediction down before the work and grade it after, so the check comes from a record and not from a look inward.

After AIIt reads now as an argument for evaluation harnesses: what checks a running process has to be a second system connected only to the first, and it can be useful while understanding nothing of what it watches.

Becauseit is a single invented case with the moral stated by the narrator, though it is built so that the content is held fixed and only the speaker changes — Vyāsa had said the same thing at length, and Śuka's response was that he knew all of it already.

LimitsThe book withdraws it two books later, where Vasiṣṭha says self-knowledge is had neither from a teacher nor from scripture nor from good works. What Janaka supplies is confirmation rather than knowledge, and the story never separates confirmation from agreement.

PracticeCalibration log — write what you are sure of before showing it to anyone, then record what the second reader changed; that gap is the finality you cannot supply yourself.

After AIA model will return your own conclusion to you in better words, on demand, for any conclusion at all. The relief Śuka felt is now free and carries no information, which turns the story into a warning about where the peace came from.

BecauseMoray's demand for a normative sampling rate is accepted rather than met, since the sampling theorem needs a bandwidth most real sources do not have, and the one design that answered it worked only because every parameter was inspectable

LimitsMost studies still infer complacency from missed failures, which is what Moray objected to, so this is the honest standard rather than the settled one. It concedes much: if no normative rate can be computed, the charge loses content in most settings.

PracticeBefore adopting a tool, ask what a complete check of one output would consist of. If you cannot write that list, oversight is not a plan.

After AIA model that answers without exposing what it drew on is the case in which the measure collapses, and the phrase used to license it — a human in the loop — names a duty nobody can specify.

Becausein a command-and-control task accuracy fell from 89 per cent unaided to 70 per cent when the aid was wrong, worst where the aid recommended a specific action and where its overall reliability was highest

LimitsBoth aids cut stalls when their information was right, so the comparison prices the failures and not the whole. The review's closing section adds that the reliability level at which an aid stops paying for itself is not a constant.

PracticeAsk the tool for the findings and reach the decision yourself. Where you cannot, make it publish how sure it is on this case, not in general.

After AIAn assistant that writes the patch rather than describing the fault is the recommending aid, in the configuration the paper found worst — high apparent reliability, a specific action proposed, no confidence attached to this answer.

Becausethe study records what the pilots did, what they said they would do and what they remembered seeing, and the three do not agree; an automated checklist recommending shutdown of the wrong engine drew 75 per cent compliance against 25 per cent from paper

LimitsA simulator removes the cost of shutting down a good engine, and phantom memory is reconstructed at debriefing, so some of it may be the report rather than the perception.

PracticeName the second source before acting on the first, out loud or in writing. Remembering afterwards that you checked is not evidence that you did.

After AIThe reviewer who signs off a generated change and later recalls verifying it is in the same position, with less to check against: there is no second instrument, and the confirming memory comes from the fluency of the explanation.

Becausethe effect held across four professional populations and an extra hour of practice left it intact, while variable-priority training — alternating which task gets 80 per cent of the effort — produced a trend towards less complacency

LimitsPractice at the task did not help, but training aimed at how attention is divided did, so this says which practice is useless rather than that experts are beyond help.

PracticeDo not staff oversight by seniority and expect the problem to go away. To move the effect, train the allocation — protected stretches where the tool is off and the work is yours.

After AIThe comfort that senior people will catch what the model gets wrong has no support here. METR's 2025 trial found experienced developers slower with AI on their own repositories while believing themselves faster.

Becausethe life-support simulation made a complete check enumerable, so sampling could be measured against it, and those who caught the error lingered on the contradicting parameter while those who missed it did not

LimitsOne group of studies in one microworld, with few participants — 18 of 24 followed the first wrong diagnosis in the companion experiment. The split into reduced sampling, covert withdrawal and discounting is inferred from logs and dwell times.

PracticeDecide before you open the output which readings would have to disagree with it, and go and look at those. Debiasing how you weigh evidence addresses the smaller half of the problem.

After AIIt puts the danger in the interface rather than the reasoning. A model that shows its sources has not thereby been checked: the reviewer who opens every link and lingers on none looks identical in the audit log.

Becausethe training contrast in the Bahner microworld is a direct test of the negative loop, and the positive one is the learned-carelessness effect carried over from industrial safety; the standing case is the *Royal Majesty* grounding

LimitsThe positive loop is borrowed rather than measured; what the studies show is the training contrast at one point in time, not a sampling rate declining over months.

PracticeArrange to meet the tool's failures early and on purpose — a known-bad case in the evaluation set, a look at last month's mistakes — because what restores checking is having been burned.

After AIReliability is now the selling point and rises between releases, so the loop is driven harder each quarter. The safety argument that the tool is right almost every time states the exact condition under which nobody catches the exception.

BecauseParasuraman, Molloy and Singh ran three Multi-Attribute Task Battery tasks at once, and detection averaged 82 per cent when reliability fluctuated against 33 per cent when it held constant, a difference of 149 per cent

LimitsThe first attempt to produce the effect, by Thackray and Touchstone, failed; blaming the empty session rather than the design is an interpretation. The numbers come from a laboratory battery with a fixed workload.

PracticeStop treating review as something to do alongside your own deadline. Either the checking is the whole task for that stretch, or it is not happening.

After AIThe multi-task condition was the experimental manipulation in 1993 and is now the ordinary arrangement of work: the person reviewing the model's output is also the person who owes the deliverable.

Becausethe authors keep trust in their model while conceding the measurements do not carry it, and suspect the instrument: what an operator does to verify a machine indexes trust better than what he says

LimitsThis is an argument about measurement, not a demonstration that trust is irrelevant — a bad questionnaire and an absent mechanism look the same from outside. The one scale in use predicts poor monitoring only above a threshold.

PracticeAudit reliance by what you actually opened and checked last week, not by how much you say you trust the tool. The behaviour is the measure.

After AISelf-reported trust in AI is now surveyed constantly and used to justify deployments. A measure that could not predict whether an operator looked at a gauge will not predict whether anyone reads the model's output.

Becausethe superimposed-display result is a direct test, moving the display from the edge of the screen to its centre likewise changed nothing, and controllers who missed a conflict made significantly fewer fixations of the radar display

LimitsThe two halves pull apart — the eye-movement data say gaze tracks detection, the superimposition study says gaze is not enough. The review's reading is that overt sampling and covert attention are separate contributors, not that looking is irrelevant.

PracticeStop counting visibility as oversight. A dashboard in front of someone, a diff on the screen: neither is evidence that anything was checked.

After AIWritten about a gauge. It now applies to every interface whose defence is that the model's reasoning was shown: the text was on the screen and in the reviewer's foveal vision, and nothing follows from that about whether it was read.

Becausethe authors predicted no effect and found none, which on a small sample is not evidence of safety; they say themselves that further work is needed before the role of explanation is settled.

LimitsWhat the data establish is a difference in order between the three uses, not a verdict on any one of them. A larger study could put the explanation correlation anywhere in that interval, including below the generation result.

PracticeReliance audit — treat "I only ask it to explain" as an untested exemption rather than a safe one, and check what you can still do unaided in the areas where you ask most.

After AIThe tutor mode is the safe harbour everyone now claims, in courses and in the assistants' own study modes. Its evidence is one underpowered null and one small experiment, which is thinner than the confidence placed on it.

Becausethe authors raise the alternative themselves and leave it unanswered, and the data they collected cannot separate the two accounts; the first-year marks that would settle it were never brought in.

LimitsThe randomised study that does control for prior ability finds the same split, which makes the reverse reading less likely without removing it here. A correlation surviving replication under randomisation is not thereby causal in this sample.

PracticeDomain audit — before concluding that a tool is damaging a team, check what the heaviest users were doing before it arrived; help-seeking and falling behind look identical in a usage log.

After AIEvery internal dashboard now correlates model use against some output measure, and almost all of them have this problem. The people who use the tool most are rarely a random sample of the people who work there.

Becausethe manipulation has a control built in, comparing justifications for real and fabricated choices, and it holds on stakes that matter: run before the Swedish election of 2010 with survey answers altered enough to flip a respondent's camp, ninety-two per cent endorsed the altered survey as their own

LimitsThese are low-stakes laboratory choices, and the manipulation works by keeping the two options close. Nick Chater reads the same results as showing the mind is flat and the chapter reads them as showing how a self gets built, which is a sign they underdetermine the conclusion.

PracticeWrite your own answer, or the issues you expect to find, before opening the model's output — an account composed afterwards is fluent whether or not it is the reason you decided.

After AIIt removes the asymmetry people reach for when a model's stated reasoning does not match the computation behind its answer. Both are the same failure, and only one of the two can be checked against a trace.

Becausethe counts come from open coding of free text over three stages of a session, and the imbalance is too large to be a coding artefact.

LimitsInspection is the easiest thinking to notice and describe, so goal formation may be happening unreported. The examples participants were shown were themselves mostly inspection.

PracticeSpecification discipline — write what the output has to contain before asking for it. The teacher who noticed her generated hand-washing image had no soap dispensers got a better image because she worked out what it had to show.

After AIThe framework's first layer is specification; this measures how little of it there is. People kept the part that feels like work — looking at a finished thing — and gave up deciding what it should be.

Becauseit is an argument about what a measure can see, offered against Sarkar's mechanised-convergence result, not a finding of its own.

LimitsThe argument cuts both ways: if judgement leaves no trace in the artefact, the survey's own measure is a worker's report of having judged, which is no more inspectable. Convergence results still show a real narrowing of output whatever its cause.

PracticeFailure-mode review — stop reading sameness or editing volume as evidence of thought in your team's work. Ask what was checked and against what, which is a question about the process and not about the document.

After AIReview in most organisations happens on the artefact, because that is what gets submitted. This says the artefact is where the evidence is not — the reviewable unit is the conversation with the tool, not the thing it produced.

Becausea logistic mixed-effects model over 936 first-hand task examples, with 98 predictors corrected to p = 0.007, so the surviving coefficients are not a fishing result.

LimitsA cross-sectional survey of self-reports establishes no causation, and the authors say self-confidence is subjective and need not track real expertise. The sample was English-only and skewed young and technically able.

PracticeReliance audit — list the tasks where confidence in the tool exceeds confidence in doing the job yourself, and arrange a check from outside for those; your own sense of when to look is the variable running the wrong way.

After AIReported as a finding that AI makes people less intelligent, it says something narrower: what decides whether you check is a feeling about the tool, available to be examined before the task rather than after.

Becausesix five-point scales answered about 936 particular tasks, with evaluation separating from the other five levels on the same instrument and the same participants.

LimitsEffort here is perceived effort, reported once, and the authors concede that participants often confused the general effort of a task with the effort of thinking about it — most of all when they were satisfied with what the tool gave them.

PracticeCalibration log — budget evaluation as the expensive part of a task, and write down what you expected to be true before reading the output, so judging it has something to judge against.

After AIThe framework's second rule says the evaluation layer gets harder as capability rises. This is a survey number for it: the one place the tools save no effort is where the output has to be judged, and that is the layer that does not go away.

Becausethe nulls come from the same corrected model that produced the large confidence effect — failures to find an effect in a design that found one.

LimitsA null on 319 self-selected people is weak evidence of absence, and the general-trust scale may simply be a poor instrument. The authors' reading — task confidence absorbing the variance — is an interpretation, not a result.

PracticeDomain audit — stop settling your relationship with the tool once, as a policy or a personality; decide per task, and give the domains where you cannot judge your own competence a fixed rule rather than a feeling.

After AIMost advice about AI is written as a stance to adopt — trust it, distrust it, verify everything. This is the measurement saying stances do no work, and the unit of decision is one task.

Becausethe three kinds come from open coding of first-hand reasons attached to particular tasks, and the ordering survives disagreement about individual codes.

LimitsReasons are reconstructed after the fact, and "it never occurred to me" is what people say when they cannot recall deciding. The largest single code, trusting the tool, could be awareness or a reasonable judgement.

PracticeAutomation ironies audit — sort your own skipped checks into the three kinds before reaching for a fix: not noticing needs a trigger, not being able needs a skill or a second reader, only the third needs time.

After AIAlmost every proposal for keeping humans in the loop assumes the human has the knowledge to look and only needs prompting. On these numbers, as many could not have checked as did not think to.

Becausethe counts are measured but the consequence is an inference, pointed back at Bainbridge by way of Simkute; no skill decay is observed in this study.

LimitsNothing here shows judgement decaying, and the survey is a single snapshot. The reading also assumes the judgement exercised on a trivial task is the capacity wanted on a consequential one, which the paper does not test.

PracticeManual flying hours — keep a low-frequency habit of doing by hand the class of task you have stopped checking, chosen from the trivial end rather than the important end, because the important end will look after itself.

After AIBainbridge's operators lost the practice because the system took it; here workers give it up themselves, one small task at a time, on a judgement correct each time.

Becauseproved in a line for any belief a test-taker could hold about which answers are right — under a grader scoring only 0 and 1, with abstention at 0, no abstention is ever among the highest-scoring responses, so abstaining is not usually worse than guessing but always worse.

LimitsIt is a claim about the scoring, not about any particular model: it shows the incentive exists, not how much of any assistant's behaviour it explains. Nothing measures how much of the guessing seen in practice comes from this pressure rather than from the pretraining floor.

PracticeWhenever you score anything — a model, a candidate, a team — check what an admission of ignorance is worth. If it is worth what a wrong answer is worth, you are paying for invention and will get it.

After AIIt reframes hallucination as an artefact of an industry's own measuring instruments rather than a fault in the artefact measured. The people best placed to fix it are the ones who maintain the leaderboards, not the ones training the models.

Becauseit is the authors' own argument for grading behaviour rather than numbers, offered alongside the concession that right, wrong and abstain is a false trichotomy adopted because it beats a false dichotomy.

LimitsTetlock's forecasters lose accuracy when their probabilities are rounded to the nearest five, and Bansal found a confidence figure beat every explanation tested — so numbers do work where the question repeats and the score accumulates. This claim holds for the single answer nobody will grade.

PracticeStop asking a tool how sure it is and start asking what it will commit to. A number is easy to produce and hard to use; a claim the answerer would defend, or a refusal, tells you what to do next.

After AIAbstention is only the blunt end of what a careful answer can do; hedging, leaving a detail out and asking a question back all signal uncertainty better. An assistant that only ever answers or refuses has already lost most of the register a competent colleague uses.

Becausethe first half is proved, since the calibration gap equals the size of the loss's derivative under a rescaling and so must be small at a minimum; the second half rests on a single pair of histograms reprinted from one system card.

LimitsOne figure from one model's system card, on four-way multiple choice, is thin evidence for a claim about what post-training does in general. The proved half also assumes a model family able to represent the rescaling, and says nothing about how large the gap is in practice.

PracticeTreat the assistant's stated confidence as a manner learned from people who preferred confident answers, not as a measurement. Score your own probability that the output is right, and check that number against outcomes instead.

After AIThe honest version of the system arrives first and is then trained out, which inverts the usual story of raw output made safe by alignment. What the second stage adds is a manner; what it costs is the one signal a user could have checked.

Becausethe ten are read off the HELM capabilities leaderboard, the Open LLM Leaderboard, SWE-bench and Humanity's Last Exam, and anyone can check them — GPQA, MMLU-Pro, IFEval, Omni-MATH, BBH, MATH level 5, MuSR, SWE-bench and HLE score an abstention at zero.

LimitsA survey of ten evaluations at one moment in 2025, and the boards change. Several of the ten are graded by language models, which sometimes mark a confident wrong answer correct, so the count of nine understates the problem rather than measuring it.

PracticeRead a benchmark number as an accuracy under a rule that pays for guessing, and never as a rate of being right. Where the score matters to you, write your own test and give a refusal a mark of its own.

After AIHumanity's Last Exam does report a calibration error alongside accuracy, and the authors show it would rate a model that answered every question wrong at zero confidence as perfectly calibrated. Even the instrument built to catch this measures something else.

Becauseit is a proposal, argued from the way human examinations state their penalties and from the observation that a target nobody is told cannot be aimed at; no model has been trained and measured under it.

LimitsThe threshold is arbitrary by design, which the authors treat as a feature. A model tuned to the announced penalty can hit the target by abstaining on questions it could have answered, trading a falsehood for an unhelpful silence, and nobody has measured that trade.

PracticeSay in the request how sure the answer has to be before it is worth having, and what a wrong one costs. Put that line in the tests you already run rather than in a separate one, because the tests people read are the ones that shape behaviour.

After AIThe proposal is aimed at benchmark maintainers, but the sentence it asks them to write is one anyone delegating work can write. Most instructions given to an assistant name the task and never the standard of certainty, which leaves the assistant to guess it — and the guess is always the same one.

Becausethe sharpest case is Krening's, a second model trained to give verbal accounts of a reinforcement learner's strategy, fitted to the explanations human players gave, so it need not describe the agent's decisions however plausible it looks; Lipton's condition is that such a claim fix an objective and show evidence it was achieved

LimitsLipton does not say these accounts are worthless — a diagnostic system pointing at similar past cases informs a doctor without describing its workings at all. The objection is to accepting one with no stated objective and no evidence it was met.

PracticeTreat an explanation as one more output to be checked, not as evidence about the process that produced the answer. Ask what it would have looked like had the answer been wrong; if it would have looked much the same, it is telling you nothing.

After AIA second process trained to give a plausible account of the first became the ordinary way of putting a question to a model. The technique he warned about is now the interface.

BecauseAdebayo and colleagues showed in 2018 that several widely used saliency methods produce much the same map after a model's weights have been randomised, which is to say they were partly showing properties of the image rather than of the model

LimitsThe objection is to reading a local picture as a global account, not to the method. Adebayo tested image models, and the result does not carry to every attribution method; the comparison is with a linear model's global relation, which Lipton has already shown is not free either.

PracticeAsk any attribution method for a control before you read its picture: what does it produce on an untrained model, or on a neighbouring input? If the picture barely moves, it is describing the input and not the system.

After AIThe lesson survived the technique. Work on large models validates a claimed internal mechanism by intervening on it and reports how often the method shows nothing, which is the control he was asking for.

Becausethe second probe is scored as a gain over the first, and the mean gain was 0.73 of a point, so failing to improve measures not correcting rather than not knowing; the effect survives partialling out sense of direction.

LimitsOne correlation of −0.33 on a scale the authors built for this study, in fifty people. The related finding that confident navigators learned faster and were then more rigid is incidental and undefended.

PracticeTreat the feeling of being unable to work without the tool as a signal about correction, not only about skill. Where you feel most dependent, build in the step that makes you revise: a second pass with the first answer set aside.

After AIWhat matters in supervising a model is not raw ability but whether the first answer can be revised. Here the people most reliant on the tool were the ones an early wrong turn kept wrong.

Becausethe Santa Barbara sense-of-direction scale was in the battery to test the opposite account, that poor navigators buy a device and use it constantly, and it came back flat against both lifetime hours and daily reliance.

LimitsA null result on one self-report scale, in fifty people; the separate sense-of-dependence scale did track feeling badly oriented, at −0.58, which the authors treat as close to a restatement of what that scale asks.

PracticeDo not use your own sense of whether you have lost a capacity as evidence about whether you have. Put the question to a task you pass or fail without the tool present, on a schedule.

After AIRisko and Gilbert argued that a tool changes both the ability and your estimate of it. Here the two come apart, which is the worse case: the faculty that decides how closely to check is the one reporting no change.

BecauseAutor names it as an irony of the method rather than demonstrating it, from his own description of the method as atheoretical brute force needing only training data, processing power and software; the unevenness he reports is consistent with it — IBM's Watson won at *Jeopardy* and in the same winning match answered Toronto to a question in the category of US cities.

LimitsStated as an aside and not investigated. It was also too pessimistic in one direction: interpretability work has since traced parts of what a model does internally, though Anthropic's own tracing method succeeded on only about a quarter of the prompts tried.

PracticeFailure-mode review — do not accept an explanation from the system as evidence that it worked. Keep a list of the failures you have actually seen and check against that instead.

After AIAutor offered it as a curiosity about a promising method. It is now the standing condition of the tools most people use, which means the whole burden of knowing whether an answer is right falls on checking the answer rather than on understanding the process that produced it.

Becauseone case carries it — a friend of Frances Power Cobbe's who described a table rapping with nobody within a yard of it, then found her own note of ten years before recording that the hands of six persons had rested on it — with James asserting, rather than showing, that such errors are ordinary

LimitsA single anecdote, and one with a written record only by luck. Nothing in it says how often this happens or which memories are vulnerable; Bartlett in 1932 and Loftus from 1974 supplied that, four decades and more after James guessed it.

PracticeCalibration log — write down what you expected before you check, in the form you will be graded on. Any account reconstructed later has been through the telling, and yours will have improved.

After AIThe account you give of a session is now given to a machine that keeps it and works from it, so the improved version becomes the input to the next round rather than fading. Where the record matters, save what happened, not what you said about it.

Becausestep-change studies separate the experienced operator from the inexperienced one, and the skill that separates them is the one monitoring stops exercising

LimitsThe paper is a review; the step-change studies are cited a sentence each and the numbers have to be fetched from the skill-retention literature directly. It also assumes the skill the rare case demands is the one the routine work rehearsed, which is often but not always true.

PracticeKeep a share of the delegated work in your own hands on a fixed schedule, not because the output needs you but because the skill does.

After AIThe delegate is now a model that takes the routine work in every domain at once, so the decay is no longer confined to one plant and one operator. What is new is that there is usually no moment of takeover to expose it, only a decision about how closely to read.

Becausethe second irony is argued from what designers choose to automate rather than from a survey of what operators were left with

LimitsIt is a claim about designers' intentions, not a measured remainder, and it assumes the automation boundary is drawn by specifiability rather than by cost or convenience.

PracticeKeep an explicit list of what stays human, derived from what you could not specify, and treat every item on it as needing more support rather than less.

After AIA model does not need a specification to take a task, so the boundary is now drawn by what the model happens to do well, and the remainder is no longer reliably the hard part. What is left may be the part nobody thought to check.

Becauseit rests on one plant where management had to be present on the night shift and one control-room survey linking controllability to low stress and good health

LimitsTwo observations, one of them anecdotal, carrying a causal claim about stress and error. The survey is cross-sectional and cannot separate controllability from everything else that differs between control rooms.

PracticeCount the human cost of a monitoring role as an operating cost. If the job has responsibility without control, expect errors from that alone and change the job, not the person.

After AIThe review seat is the deskilled job: full accountability for the output, no hand on the work that produced it. Self-determination theory and the job-crafting literature both predict the resentment Bainbridge only observed.

Becauseshe rests it on the memory literature on retrieval frequency, and offers the reader a check they can run on any school subject they have not thought about since

LimitsThe supporting studies are cited rather than reported, and the claim is about long-term memory in general rather than about process knowledge specifically.

PracticeDo not count instruction as preparation. If someone is expected to handle a case, they have to have handled cases and been told how it went.

After AIHer warning that the automated plants of 1983 were riding on the skills of former manual operators reads now as a description of every team whose seniors learned the work before the model arrived. The next generation is the one she said could not be expected to have it.

Becauseshe calls it the final irony and derives it from the argument rather than from any training study

LimitsIt is a deduction, not a measurement, and it runs against the cost logic of every organisation that buys automation to reduce headcount and training. Aviation is the one industry that has acted on it.

PracticeSet the training budget from how rarely the manual case arises, not from how often. The better the tool, the more you have to spend keeping people able to work without it.

After AIThe more reliable a model becomes, the more expensive it gets to keep anyone able to catch the case it misses — and the harder it is to justify that expense to anyone looking at the success rate.

Becausethe 2010 F-22 crash is the case — the Inspector General asked about oxygen deprivation and the Air Force restated pilot error — and Toyoda's Five Whys is offered as the procedure

LimitsNorman says himself that the number is not five, that the method guarantees nothing, and that the word why implies a single cause where complex events have many. His own utility case shows the procedure defeated by a culture rather than by a technique.

PracticeIn any review, when the answer is that someone made a mistake, ask why at least twice more, and do not close the item until the answer stops naming a person.

After AIThe new stopping point is that the model got it wrong, which closes an investigation as effectively as blaming the operator did. Keep asking why past that and you reach the objective, the data and the interface that showed the answer first.

Becausehis own textbook team estimated two years against a reference class of seven to ten and took eight; the Scottish Parliament ran from about forty million pounds to about four hundred and thirty

LimitsReference class forecasting needs a reference class, and Kahneman's own team had one member who knew it and was ignored. Where the class does not exist, the procedure has nothing to offer.

PracticeBefore estimating, find the outcomes of comparable past cases and start from their distribution. Adjust from there, and record the adjustment.

After AIThe reference class for work done with a model is a few years deep and moving, and the METR result — developers slower while believing themselves faster — is the case where the inside view and the measurement point in opposite directions.

BecauseKahneman concedes that decades of study left his own intuitive thinking as overconfident and as prone to the planning fallacy as before

LimitsIt is a concession rather than a finding — no study is offered that debiasing training fails — and the organisational remedies are argued from the same source. The book's own remedy would have caught the chapters that did not replicate.

PracticeStop trying to be less biased and start changing the procedure: written positions before discussion, decorrelated error, a premortem, and judging a decision by how it was made.

After AIThe remedies survive the arrival of a capable assistant and the personal resolution does not. Buçinca's result is the same shape: what works is a change to the interface, and people dislike it.

Becauseit rests on the finding that people generate more reasons for an outcome once told it is certain, which is what the exercise exploits

LimitsThe underlying finding is about reason generation, not about decision quality; no study here shows premortems improving outcomes. Going round the table starting with the leader is a procedural guess at the social problem.

PracticeBefore committing, have everyone spend two minutes writing why the plan has already failed a year from now, then go round the table starting with the most senior person.

After AIThe exercise is cheap to run against a plan a model produced, and it is one of the few checks that does not require you to follow the reasoning faster than it was written. It asks what would have gone wrong, not whether the argument holds.

Becausethe paper lists the conditions a usable algorithm has to meet, and the naturalistic objections hold wherever one of them fails

LimitsThe supervision half of the paper's argument is cited rather than shown: the automation-bias literature is named as such, and the evidence that operators become passive belongs to Bainbridge rather than to this paper.

PracticeRun the five conditions as a checklist before automating a judgement, and treat a failure on any one of them as a reason not to.

After AIThe last condition is the one that has failed. Conditions do change under a deployed model, and they change because it is deployed — the behaviour it is scoring shifts once people know they are being scored by it.

Becausestated as a qualification of the model, with the observation that a line worker who changes tasks may wreck the flow of work or may raise its quality, and that crafting is often invisible and unrewarded

LimitsIt is a caveat rather than a finding, and the later survey literature has mostly measured crafting against outcomes the organisation wants, which quietly reverses the qualification the original paper insisted on.

PracticeDo not sell crafting to management as a productivity measure. It is what people do to keep the work bearable, and sometimes that costs the organisation something.

After AIThe invisible, unrewarded extra work is the first thing to disappear when a tool makes the visible part cheap, and nobody notices because it was never in the job description.

Becausethe cleaners used the hospital's stated mission to legitimate framing cleaning as care, which is the one lever the paper can show working

LimitsOnly one of the three levers is illustrated, and the illustration is a single case. The paper's own list of open questions includes whether managers should encourage crafting at all, since it may harm the organisation.

PracticeIf you want people to own their work, change rewards, the organisation of tasks, and who is in the conversation about what the work is for. Do not ask them to feel differently.

After AIAdmission to the conversation about what the organisation is trying to do is the lever that matters most when the tasks themselves are being reassigned, and it is the one usually closed while the reassignment happens.

Becausethe meta-analyses associating crafting with engagement, satisfaction and rated performance rest on self-reported, cross-sectional designs with moderate effects

LimitsContested because the reversal of the arrow is now the default assumption in the field, which means the weakest part of the evidence supports the part everyone has accepted. Berg and colleagues' 2010 return to the original model is the nearest thing to a test.

PracticeRead any crafting-and-engagement finding as a correlation between two self-reports taken at the same moment, and ask what a longitudinal version would have to show.

After AIThe same design weakness runs through most of the early evidence about working with models: people reporting their own engagement and their own productivity in the same questionnaire. METR's result is what happens when the two are measured separately.

Becausethe moderator is argued from the contrast between a cleaner or hairdresser with many degrees of freedom and an assembly-line worker or closely watched telemarketer with few

LimitsThe two moderators are asserted from contrasting occupations rather than measured, and the paper notes the modern workplace pulls both ways at once, which makes the prediction hard to test in either direction.

PracticeBefore telling someone to take more ownership of their work, count how tightly their tasks are yoked to others' timing and how closely they are watched.

After AISurveillance and flexibility have both increased since, and a new form of both has arrived: work that passes through a tool that logs it. That tightens interdependence and monitoring at the same time.

Becauseit is the authors' reading of their own result, with the doctor who is soothed rather than informed as the case

LimitsAn interpretation rather than a measurement: the study shows acceptance rising and accuracy flat, not that the sign-off has been emptied. The requirement that a doctor sign off may be doing other work the study does not measure.

PracticeWhere a person is required to sign off for legal or moral reasons, check that the sign-off could go the other way. If nothing in the interface supports disagreement, it cannot.

After AIBainbridge's impossible task and this paper's empty sign-off are the same claim measured forty years apart, and the second supplies the numbers the first argued for.

BecauseTable 1 assembles eight studies offered as evidence for explanations and adds the column they omitted: deceptive review detection at 51.1 per cent for workers, 87 for the model and 74.6 for the best team; house prices out by $331k, $200k and $232k

LimitsThe standard is accuracy alone, and the authors say so: a team might be complementary in speed or effort instead. It is also a standard for deployment decisions rather than for legal or moral requirements that a person sign off.

PracticeBefore keeping a person in a loop, measure all three: the person alone, the machine alone, and the pair. Two of those numbers are usually missing.

After AIThe three-number comparison is almost never run on the tools in daily use, and the eight papers in Table 1 are the reminder that publishing an improvement over unassisted work says nothing about whether the person should be there.

Becauseobserved at Intel and offered as a remedy; the timing rule is a judgement about when authority helps, never tested against a control.

LimitsThe same senior presence that unlocks a decision is what Edmondson's managers responded to by withholding corrections when the boss's boss was present. Grove's rule says which stage helps but gives no way to tell, inside the room, which stage you are at.

PracticePut a senior person in the room for the decision and keep them out of the discussion, and settle the six questions — what, when, by whom, consulted, ratified, told — before the meeting starts.

After AIA model in the discussion is a peer with no standing, so it does not break the circling. Asked to arbitrate, it agrees with whoever framed the question, which is the peer-group syndrome with a more confident voice.

Becausetwo illustrative cases, offered as something that should have been predictable, with nothing measured.

LimitsGrove notes that the mode managers most dislike is the one they most often reach for too late, and that friendship makes the return to structure harder. The evidence is two anecdotes chosen to make the point.

PracticeKeep the structured mode available instead of treating a move to minimal involvement as permanent, and keep enough hands-on contact to use it. Manual flying hours is what makes the return possible rather than merely intended.

After AIA model's competence shifts with releases and with what is in its context, so the fall in maturity is not announced by a visible change of job. The manager has to detect it from the output, which is the thing the delegation was meant to stop requiring.

Becausecarried over by analogy from factory inspection, and the London embassy case shows what sampling saves rather than what it catches.

LimitsThe gate is right where a defect is catastrophic rather than merely expensive, and Grove says so. A rough draft from a delegate whose competence is unknown also carries less information than a finished one.

PracticeAsk for the outline before the document and the plan before the diff, and spend the review budget there. Failure-mode review turns that early look into a fixed list of what to look for.

After AIA model produces a polished draft at the first stage, so the cheap early signal Grove relies on no longer exists in the work's natural shape. The low-value stage has to be created deliberately — an outline, a plan, a stated approach — before there is anything to monitor.

Becauseit is a definition Grove derived from running Intel, not a finding anyone tested.

LimitsIt is an accounting identity rather than a measurement: it gives no way to attribute an output among several influences, and the time offset between a manager's work and its result can be years. Grove's 1995 introduction concedes that the shared building and the long career the model assumes have gone.

PracticeStop counting hours spent and start asking which of them raised someone else's output. The stays-human list is where the answer gets written down for the firm rather than for one manager.

After AIWhen part of the unit is a model, the output still counts as the manager's, but the leverage term now includes work nobody on the team did. Reviewing a draft the model wrote is activity that raises no one's output unless it raises the next draft.

Becausea rule Grove adopted after misrating a manager whose superb year was work done years earlier; one case, stated as policy.

LimitsThe time offset Grove names cuts both ways: this year's result may be the light of distant stars in either direction. Rating on the organisation's result alone punishes sound judgement that met bad luck, which is Edmondson's objection to rewarding outcomes.

PracticeRate the result, then ask separately what part of it was this year's work — and write the prediction down first so the offset is visible. Calibration log is that habit applied to any judgement you will later rely on.

After AIThe offset has shortened — a team's output can move within a quarter of adopting a tool — so a rating tracks the tool's arrival rather than the manager's work, and the distant-stars error runs in reverse.

Becausea rule of thumb from running a factory; the examples — inventory against shortages, vouchers processed against errors found — show the failure mode without saying how often it occurs.

LimitsPairing assumes the counter-effect can be named in advance, which is exactly what is hard about a new measure. Self-determination evidence adds that the pair is itself a contingency, and contingent measurement shifts regulation from autonomous to controlled whatever it counts.

PracticeNever introduce a measure without naming, in the same sentence, the thing it will be gamed against. Failure-mode review supplies the counter-measure for delegated work: what got through, not how much got produced.

After AIOutput generated per person is the easiest thing to count and the counter-indicator — defects reaching production, hours spent verifying work nobody wrote — is the hard one. Without the pair, what gets reported is the gain people already believed they were getting.

Becausethe arithmetic establishes leverage — twelve hours against ten people's twenty thousand working hours — and the authority claim is asserted from experience.

LimitsWeinberg's own classroom result cuts against the emphasis on the teacher: lecturing on the errors his students made moved them from 87 per cent to 83, and the exercises did the rest. The one per cent gain in his arithmetic is never measured.

PracticeTeach the thing yourself, as a series rather than an event, using this week's work as the material. Dreyfus mapping and tutor-only rule decides who needs which part.

After AIWhat a manager now owes his team is training in how to use and check a model, the one domain where he is least likely to be a practising authority. The arithmetic still holds and the authority does not.

Becausea two-by-two derived from a commuting anecdote; the empty quadrant is an inference about why senior outside hires fail, not a count of them.

LimitsCulture needs shared experience to transmit, which is why Grove starts new employees in low-CUA jobs; where turnover is fast there is no time to build it. For the fourth quadrant — self-interest in a complex environment — his own answer is that no control works at all.

PracticeWhere the work outruns the rules, spend the effort on what the group will not do and why, rather than on another procedure. The stays-human list is that statement in its shortest form.

After AIAn agent has no shared values to fall back on when the contract runs out. What it has is a specification, which is the mode Grove says fails in exactly this quadrant, and that is the alignment problem stated in management vocabulary.

Becausethe rule rests on Maslow's hierarchy, which Grove adopted because his observation of working life confirmed it and which has not held up as psychology, though the distinction between needs that get people to work and needs that make them perform survives without it.

LimitsSelf-determination evidence says a target set for someone rather than with them shifts regulation to controlled whatever its difficulty. A fifty-fifty objective inside a company that rates on attainment is the potential trap Grove names two chapters later.

PracticeSet the target where failure is a real possibility, and then make sure nothing in the rating system punishes the person for taking it.

After AIWhen the delegate is a model, failure costs it nothing, so the even-odds rule loses its point for the agent and keeps it for the person who has to review the half that fails.

BecauseIntel's rotation of managers between similar groups was suggestive rather than controlled, and the construct is Hersey and Blanchard's situational leadership renamed.

LimitsMaturity is specific to the task and falls when the job or the pace changes. Managers also misjudge their own position on the dial: about nine in ten saw themselves as more delegating than their people did.

PracticeSet the dial per task rather than per person, and write the mapping down so it can be argued with. Dreyfus mapping and tutor-only rule does exactly that.

After AIA model's competence is jagged across tasks and can move with a release, so the dial cannot be carried per delegate the way Grove carries it per person. It has to be reset for each task, and reset again when the version changes.

Becausea meta-analysis found intrinsic motivation a moderate to strong predictor of performance and strongest where quality matters, a meta-analysis of public-sector pay for performance found it helped on simple tasks and hurt on complex ones, and the industrial laundry attendance award was gamed, improved the chronically late and lowered the efficiency of the already punctual at tasks outside the programme.

LimitsComplexity is graded after the fact in most of the pooled studies, and the public-sector meta-analysis covers tasks unlike software.

PracticeIncentivise the throughput of routine work if you must, and measure the complex work some other way. Failure-mode review is one of the other ways.

After AIA model makes quantity cheap and leaves the complex judgement with the person, so an incentive aimed at output now sits on exactly the half of the job the evidence says it damages.

Becausea meta-analysis of 128 studies over three decades, answered by a behaviourist meta-analysis, with the dispute still open and behavioural economists reproducing the crowding-out effect on their own terms.

LimitsWhat a reward does depends on its functional significance, so the effect is not a property of rewards as such — the same performance-contingent reward enhances or undermines depending on how it is given.

PracticeDo not attach a bonus to the part of the job people already want to do; attach it, if anywhere, to the part nobody enjoys.

After AIPay that is contingent on output a model produced is contingent on nothing the person did, which is the clearest case of an external locus the literature describes.

Becausethe three held up as cross-cultural predictors of well-being in later meta-analyses, while the objection that autonomy is a Western value has continued in parallel.

LimitsThe theory does not study need strength, and reads an unusually strong wish as evidence of past thwarting, which makes it hard to falsify from individual differences. The cross-cultural objection to autonomy has not been withdrawn.

PracticeBefore moving work off a person, ask which of the three the move removes; competence is the one delegation takes first. The stays-human list is where the answer is made binding on the firm.

After AIDelegating the interesting part of a job to a model removes competence satisfaction while leaving autonomy and relatedness intact. That is the two-out-of-three case the theory says is not enough, and it looks like a well-run team from outside.

Becausetrained managers became more autonomy-supportive than controls, subordinates' perceptions improved over thirteen months, the managers who changed most had subordinates who changed most, the company's own attitude surveys agreed, and a second randomised training study replicated it.

LimitsThe founding study is from the 1980s, and the surrounding literature is largely self-reported, often cross-sectional, drawn heavily from the authors' own network and light on effect sizes. The two randomised training studies are the exceptions.

PracticeTrain the three behaviours the study trained — take the other's perspective, give feedback that informs rather than judges, support initiative — and measure the change on the subordinates, not on the managers. The stays-human list is one place the third behaviour becomes concrete.

After AIThe three trained behaviours are the ones a manager stops performing when the review is of a model's output and the person is only the conduit. Nothing in the job description changes; the perspective-taking simply has no object.

Becausethe same performance-contingent reward enhanced intrinsic motivation given in an autonomy-supportive way and undermined it given controllingly, and the field results fall on either side of that line: non-contingent salary predicted autonomous motivation in Norwegian insurers where quarterly bonuses predicted controlled, and a telecom firm that moved sales staff to full commission lost three-quarters of them within a year.

LimitsThe field results are correlational and the firms that changed schemes changed other things too; the clean evidence is the experiment that varied delivery while holding the reward fixed.

PracticeSpend the argument on how a reward is delivered and what it is said to mean, not on its size; pay level predicted only whether pay seemed fair.

After AIAn output metric a model can move makes the reward informative about the tool rather than about the person, which is the controlling reading by default.

Becausethe same pattern appears in schools across four countries, in a six-month supervised diet whose autonomy-supported patients had kept the weight off nearly two years later, in smoking cessation confirmed biochemically at thirty months, and in methadone patients' urine tests.

LimitsThe reasons are self-reported, and most of the field studies are cross-sectional; the direction of the effect rests on the longitudinal minority.

PracticeAsk why someone is still doing the work rather than how much they are doing, and treat the answer as the leading indicator it is. The stays-human list makes the why explicit for the work that stays.

After AIThe question survives the delegation of what is done. A person who keeps the work because they endorse it and one who keeps it because the tool would expose them are in different states by this measure, and produce the same output this quarter.

Becausein the experiment the correlation between how much people did and how much they valued it reversed sign according to how many supports were present, which is a result a weaker design could not manufacture.

LimitsThe introjected step is the hardest part of the continuum to separate cleanly in later measurement, and the demonstration rests on one uninteresting laboratory task.

PracticeWhen imposing a change, supply all three: a real reason, an acknowledgement that the task is unwelcome, and a genuine choice about how it is done. Two of the three buys compliance that costs the person something.

After AIAn engineer given a rationale and a choice integrates a new tool. Told only that the team now ships this way, he keeps shipping and administers the resentment himself — which is introjection, and is invisible in the output.

Becausethe syllogism is introspection stated out loud and the supporting cases are anecdotes; what is evidenced is the remedy, not the mechanism.

LimitsWeinberg's mechanism is one person's attachment to an artefact, which predicts variation between people; Edmondson's interpersonal account predicts variation between two units of one firm, and that is the version that replicated.

PracticeArrange the work so that no listing is anybody's. Witness practice in review handles the half the arrangement cannot reach.

After AICode a model wrote is not part of anybody, so the ego is absent at the moment of review. What takes its place is ownership of the decision to use the tool, and that defends itself the same way.

Becausethe founding case is one afternoon — seventeen errors in thirteen instructions, and no further error in nine years across a dozen installations — but by 1998 Weinberg reports no shortage of evidence that technical reviews produce more reliable code more cheaply.

LimitsWeinberg claims nothing about the mean time to finish, so a team measuring throughput sees the cost and not the gain. Brooks would put the same effort into fewer designing minds instead.

PracticeMake reading each other's work the default rather than the exception, and judge the arrangement on the spread of delivery dates, not the average. Failure-mode review says what the reader should be looking for.

After AIThe reading Weinberg wanted is now cheap and has changed direction: the model reads the person's code. The arrangement in which nobody owns the listing arrives by default.

Becausetwo studies of the same population reported thirty-to-one differences depending on whether the problem rewarded attention to detail or the broad view, and Prechelt found the largest ratios came from comparing the best subject on one task with the worst on another, with about four to one surviving.

LimitsThe reanalysis went back over the same small old studies rather than collecting new data, so what survives is a correction to an artefact and not a measurement of the real spread.

PracticeDo not price a hire, a tool or a reorganisation off the thirty-to-one figure; ask which stage of the work the number came from.

After AIThe spread that now matters is in reviewing rather than producing, and nobody has measured it. The figure the industry still quotes was never about the work it is used to justify.

Becausea controlled manipulation in which only the last page of the specification differed and nobody knew: the group told to finish fast used two-fifths of the machine time and a third of the personal time and produced programs about ten times slower, replicated on a second problem.

LimitsFour subjects, one shop and one era; a large effect in a tiny sample, replicated once by the same author and by nobody since at scale.

PracticeSpend the argument on the last page of the specification rather than on who is assigned. Specification discipline is that argument written down before the work starts.

After AIThe finding transfers to the delegate that takes instructions most literally. What the specification asks for is what arrives, and the difference between make this fast and make this work now shows up in minutes rather than months.

Becausetwo cases chosen to point in opposite directions — the man who was given a raise instead of a successor and quit, leaving a system delivered a year late and scrapped, against the man who trained his two best people and left a project that finished.

LimitsBrooks's answer to the same fact is to concentrate design in one or two minds and pay for them, which makes indispensability the design. Weinberg's maxim is about what an organisation should tolerate, not about what produces the best system.

PracticeTreat any place where only one person can check the work as the next thing to fix, and fix it by training rather than by retention. Manual flying hours keeps the second person actually able.

After AIIndispensability now attaches to whoever can still check the work by hand, and that is the capability the tool quietly removes. The successor Weinberg wants trained is the person who keeps the skill.

Becausethe arithmetic of the dates is checkable — worked backwards through printing and review time, a team's submission was due four days before the month it described began — and the smoothing at each of six or seven stages is observed but never measured.

LimitsWeinberg puts the smoothing in each reviewer's judgement; Edmondson's evidence puts it in the reporter's calculation about a specific audience, which is where an intervention has to go.

PracticeTreat an unusually smooth report as the finding, and go to the extremes the chain removed. Failure-mode review is one way to make the extremes reportable again.

After AIA summary written by a model is one more filtering stage, and it smooths by construction: it reports the central tendency of what it was given, which is precisely the part carrying no news.

Becausea rule of thumb with no measurement behind it, defended by the observation that it is doubtful nine programmers can do anything useful in less time than it takes them to get organised.

LimitsThe numbers are asserted rather than observed. Brooks reaches the same conclusion from a combinatorial argument that does not flatten, and the two disagree about whether the cost is bounded.

PracticeGive the best people you can find enough time that you need the fewest of them, and treat the floor as real when a date is being negotiated.

After AIModel capacity adds none of the familiar coordination cost and adds a new one, since every output has to be read by somebody who is already the bottleneck. The floor moves; it does not disappear.

BecauseAsch's line-judging result is experimental and the meeting reproduces its structure, including the single dissenter who restores almost everyone's sight; the meeting itself is one observation.

LimitsAsch's subjects doubted their own perception; Weinberg's leaders knew perfectly well they were behind and were managing an audience, so the conformity evidence supports the shape of the remedy and not the mechanism.

PracticeWhen a status round comes back unanimous, let the silence run and ask again rather than recording it. Cognitive forcing is the same move applied to an output you are about to accept.

After AIA model asked to review a plan will agree with it, and its agreement counts in the room as another voice. That is the unanimity Weinberg says carries no information, now available on demand.

Becausethe independent and dependent variables were measured by different people, the correlation was significant and pointed the wrong way, and a blind observer found the units differed in whether people felt able to talk about mistakes — but the outcome is measured by a process the climate itself affects.

LimitsLater work separated the two: departments with higher safety reported more errors, and made fewer only where the department also genuinely prioritised patient safety. The original finding remains compelling as a story and awkward as evidence.

PracticeRead a falling error count as a question rather than an achievement, and find out which of the two things moved. Failure-mode review is where the errors get somewhere to be recorded.

After AIA tool that logs every rejected suggestion makes caught errors cheap to count and accepted-and-wrong ones invisible, which reproduces the original confound in a new instrument.

Becausetwo independent samples show the sign reversal, and the long-standing inconsistency in the demographic diversity literature is what a missing moderator looks like.

LimitsThe reinterpretation of the diversity literature is a suggestion rather than a re-analysis, and the two supporting studies are of project and student teams.

PracticeBefore adding a dissenting expert to a team, find out whether the team can absorb disagreement; without that, the addition makes things worse rather than neutral.

After AIA model is expertise diversity with no interpersonal cost attached, so by this account it should help unconditionally. What still depends on the climate is whether anyone will say they disagree with it.

Becausethe group-level variation has been replicated repeatedly since 2018 on a measure whose reliabilities run from about .73 to .94.

LimitsMuch of the literature is cross-sectional survey work in which safety and outcomes are measured at the same time and often from the same respondents, so the causal arrow is argued rather than demonstrated, and estimates shrink as designs tighten.

PracticeStop looking for a company-wide programme and look at the two teams whose numbers differ; the difference is the leaders. The stays-human list is one thing a single team can decide for itself.

After AIThe unit whose climate matters now contains a delegate that cannot be embarrassed. What still varies between two teams with the same tools is whether a person will say out loud that the output is wrong.

Becauseit was stated as a prediction in 2018 and has since been found by meta-analytic work, which is a real test because it could have failed.

LimitsUncertain and interdependent is graded by the researcher after the fact in most studies, and the meta-analytic estimates of the effect on performance shrink as designs tighten.

PracticeSpend the effort on climate where the work is novel and shared, and stop apologising for not spending it on the routine parts. The stays-human list marks which parts are which.

After AIThe routine work is what the tool absorbs, so the human work that remains moves into exactly the quadrant where the climate carries the most weight.

Becausethe activity was split between reading the literature alone and working things out together, and the null on the first half is what shows the mechanism runs through interpersonal risk and nothing else; a spurious climate effect would have moved both.

LimitsThe activity split is self-reported and the study is one setting. The null says something about the mechanism and nothing about how much either kind of learning was worth.

PracticeIf a team is not improving, do not send more reading; make it cheap to ask the question in front of colleagues. Failure-mode review is one place that asking becomes routine.

After AIReading what a model produces is learning what, which the climate does not touch. Saying in front of the team that its answer is wrong is learning how, which is entirely what the climate decides.

Becausetwo answers Edmondson has collected from the same rooms of managers for years: audience polling, not a study, and she presents it as such.

LimitsThe respondents are self-selected audiences at her own talks, both figures are impressions, and the gap is between two beliefs rather than between a belief and the record.

PracticeSort failures before responding to them — deviation from a known process, a novel combination of ordinary events, a considered foray into new territory — and let the sort decide the response. Failure-mode review is where the second kind gets caught.

After AIWhen a failure came out of a model there is no obvious object for blame, and the organisation finds one anyway — usually the person who accepted the output rather than the person who chose the tool.

Becausein both cases the explanation had already been given and had produced silence, and only the change of question moved anything; at Rustenburg fatalities fell from forty-four in 2006 to seventeen in 2011.

LimitsBoth are single organisational cases in which many things changed at once; the fall at Rustenburg followed a shutdown, a restructuring and a new chief executive as well as a new question.

PracticeReplace was everything fine with was everything as safe as you would have liked this week, and wait. The stays-human list is built from the same kind of question asked about delegation.

After AIThe rule applies to what one asks a model as well. A question it can answer yes or no gets agreement; a concrete one that cannot gets the part of the answer worth reading.

Becausea distinction drawn from one contested case in order to separate a broken clear policy from an unclear boundary, offered as an analysis rather than as a rate.

LimitsWhether a boundary was clear is decided after the sanction and mostly by the people who imposed it, which is the part of the claim that cannot be checked from inside the organisation.

PracticeWrite the boundary down before you need to enforce it, and enforce it visibly when it is broken; the vagueness is what costs, not the severity.

After AIThe boundaries most often left unwritten now are the ones about tool use — what may be generated, what must be disclosed, what must be checked by hand. By this account leaving them vague costs more than writing them strictly.

Becausean interview finding in which eighty-five per cent could name an occasion they had felt unable to raise a concern, and a vignette study in which managers were significantly more likely to offer an important correction when the boss's boss was absent.

LimitsThe vignettes measure stated intention rather than behaviour, and the eighty-five per cent figure is a recall question with no base rate for how often the withheld concern was right.

PracticeChange what silence costs rather than telling people to be brave; make the contribution requested by name and the silence the awkward option. The stays-human list is one such structure.

After AIThe same terms of trade were trained into a model by optimising on human approval. There the fix is in the reward signal rather than in the climate, because the model has no stake to protect.

Becausea prescription drawn from a counterfactual reading of one case, consistent with the rest of the evidence and not tested on its own.

LimitsGrove's remedy for the same moment lives inside the review — level, listen, leave yourself out — and neither version has been compared with the other. Good process and good outcome come apart, which is why rewarding outcomes alone punishes sound judgement that met bad luck.

PracticeThank the person for raising it before saying whether they were right, every time, including when they were wrong. Witness practice in review is what makes that possible when the correction stings.

After AIA model thanks the user for every correction and it buys nothing, because there is no next time in which it will be braver. Appreciation is an instrument that works only on someone who is keeping score.

Becausethe mechanism is arithmetic and the worked example is airtight, but Abdel-Hamid and Madnick, modelling with a learning curve, found that adding always costs more and does not always make it later.

LimitsBoehm's sixty-three projects give a cost-optimum schedule as the cube root of expected effort, with hardly any project succeeding in under three-quarters of it, which bounds the schedule from below without the law. Brooks kept the bald statement.

PracticeReschedule or trim the task rather than adding people late, and take no small slips. The stays-human list records what extra capacity will not be asked to do.

After AIModel capacity is added late by default and costs nothing to train, so the first two terms vanish. What remains is repartitioning and the review of work nobody on the team wrote.

Becausea retraction argued from what happened afterwards — Parnas's encapsulated module became the abstract data type and then the class, which is the only way found to compose programs out of chunks larger than a statement.

LimitsThe documentary hypothesis survives with should see everything softened to should be able to see it, so the retraction concerns a mechanism and not openness. Weinberg's case for reading each other's code is untouched by it.

PracticeDesign the interface so that a person can work on one part without holding the whole, and stop treating total visibility as a virtue in itself.

After AIA model reads across every hidden boundary at once, so encapsulation is not what limits the machine. What it protects now is the human's ability to reason about a part without the whole.

Becausea definition of the purpose of organisation, illustrated by a parable and by Brooks's own project, with the dotted lines named as what makes up the difference between the tree and the network.

LimitsGrove's dual reporting deliberately adds a path so that standards and priorities stay live at once, so this criterion files as failures organisations Grove would call correct.

PracticeJudge a reorganisation by which conversations it makes unnecessary, and count the ones it adds. The stays-human list names the conversations that must stay whatever the structure.

After AIAn agent sits on neither the tree nor the network and is reachable by everyone at once, so the structure removes nothing on its behalf. Whatever it produces still has to pass through the same human paths to be believed.

BecauseBrooks's own arithmetic, offered as the gap between what a programmer believes he can do — estimating from the thing that came out of the garage — and what his employer needs.

LimitsBoth factors of three are estimates and their independence is assumed rather than shown; nothing in the book measures either.

PracticeWhen something works in a demonstration, multiply by nine before promising a date. Failure-mode review covers the testing third of it.

After AIWhat now comes out of the garage in an hour is still the corner of the diagram that costs one. The factor of nine is the distance between a demonstration and something a team can run.

Becausethe prediction was stated so that it could fail, has run four decades rather than one without a candidate producing the gain by itself, and rests on an arithmetic claim nobody has disputed.

LimitsThe accidental fraction is Brooks's own estimate — half or less by 1995. The prediction is about a single development within a decade, so a compound of many gains over forty years neither falsifies nor confirms it.

PracticeAsk of any tool which part it attacks, and price the promise against the fraction of your effort that part actually consumes. Domain audit is that question asked domain by domain.

After AIThe current candidate attacks representation, which is the accidental half by Brooks's own division. What it does not do is decide what to build, and the measured results so far land where the arithmetic says they should.

Becausean aesthetic judgement about systems, supported by one costly natural experiment of his own — giving the OS/360 external specifications to a hundred and fifty implementers, which he estimates added a year of debugging.

LimitsThe counterfactual was never run and the year of debugging is Brooks's own estimate. Weinberg's arrangement distributes across every mind exactly what Brooks concentrates in one, and claims the same defect count as the payoff.

PracticeName the one person who owns the shape of the thing, separately from the person who owns the schedule, and let everyone else raise their effectiveness. Specification discipline is what that owner produces.

After AIGeneration is now cheap and agreement is not, so the constraint Brooks names is the binding one. Many minds can produce many designs in a day; the thing that cannot be parallelised is the single view that makes them one system.

Becausean author's retraction, supported by the industry practice that followed rather than by any comparison of the two approaches.

LimitsThe incremental build became the default and was never tested against the thing it replaced. What recommends it is the ordering — user test before full construction — rather than any measured gain.

PracticeGet a running skeleton first and keep it running, so that user testing starts before full construction has been paid for. Failure-mode review is what each increment gets checked against.

After AIA running skeleton is now the cheapest artefact in the project, so the discipline moves from producing one to deciding what the first working system must do. That is the part the tool does not supply.

Becausean assertion from experience, sharpened by the observation that verification establishes at best that a program meets its specification, whereas much of the essence of the work is debugging the specification.

LimitsHardest is not measured against anything. What the argument establishes is that the part is essential and not removable, not that it dominates the effort.

PracticeBuild something small and wrong on purpose so the client can say what they actually want, and treat the specification as the artefact under test. Specification discipline is how it gets written once they can.

After AIImplementation is the part that got cheap, so the ratio moved in the direction Brooks predicted and the residue is the specification — which is also the part an agent satisfies literally rather than queries.

Becausea rule of thumb from examining conventionally scheduled projects, few of which allowed half the schedule for testing and most of which spent half, running on time until system test.

LimitsBrooks withdrew the model this rule sits inside. The incremental build he put in its place moves testing forward, so the allocation is right about where the effort goes and wrong about when the trouble can be discovered.

PracticeBudget the testing half explicitly, and treat on time before system test as no information at all. Failure-mode review gives the testing half something specific to look for.

After AICoding, the sixth Brooks allotted, is the part that collapsed. Testing and the verification of work nobody wrote did not, so the schedule is more lopsided now than when he drew it.

BecauseBoehm's cost model found team quality the largest success factor by about four times, but the claim about what organisations do is offered without a count.

LimitsThe order-of-magnitude spread Brooks cites from Sackman, Erikson and Grant is what Prechelt's reanalysis found to be an artefact of cross-task comparison, so the gap is not the size he thought. The claim about neglect survives the correction.

PracticeBuild a route into design as deliberate as the route into management, with stages and the practice that moves people along it. Dreyfus mapping and tutor-only rule is that map.

After AIThe neglected skill has changed shape: it is now the judgement to specify and to verify. The pathway Brooks never proposed is what an organisation must build if it wants that judgement in ten years.

Because74 hand-labelled screen recordings covering 84 hours show active coding falling from about 36 to 24 per cent of the session and reading and searching from 24 to 16, while testing and debugging barely moves and idle time rises from 6 to 10.

LimitsThe recordings cover 29 per cent of the study's time before filtering and 84 hours after it, at a labelling resolution of about ten seconds. The activity split is descriptive and cannot say which displacement causes the slowdown.

PracticeReliance audit — account for where an assisted hour actually goes before claiming the assistant saved one; the recovered time comes out of the activities that build the knowledge you will need to check its output.

After AIThe three activities that exist only in the assisted condition — prompting, waiting, reviewing — are the ones that grow as tools become agentic. The displacement of reading is the part to watch, because reading the system is how the reviewer stays able to review.

Becausethe two quantities are measured on the same issues in the same trial and move in opposite directions, which is what the field studies reporting 14 to 51 per cent gains could not have detected, because lines, commits and pull requests were all they measured.

LimitsThe 47 per cent figure sits among the ten factors the authors leave unresolved: it looks like scope creep, except that developers who report scope creep were slowed less than those who do not. The claim is about what output measures cannot show, not about what they always show.

PracticeReliance audit — fix the unit of work and its definition of done before the tool is introduced, so that nothing the tool does downstream can change what counts as finished.

After AIEvery dashboard built to demonstrate an assistant's value counts the things that moved here in the wrong direction. The only measure that survived the trial is elapsed time on a unit of work fixed in advance.

Becausea randomised trial on 246 real issues from 16 developers, with a raw ratio of 1.34 (CI 1.05–1.70) and a regression estimate of 19 per cent using the pre-assignment forecast as the difficulty control.

LimitsSixteen developers and a confidence interval whose lower end sits near no effect. The sample was selected for the exact conditions the authors name as causes — deep familiarity and enormous mature codebases — and both are absent from most software work. Some participants were sped up. Most of the work was done in February and March 2025, before the command-line agents that arrived later that year.

PracticeManual flying hours — keep a protected block of work done without the assistant on the systems you know best, and time it, so the comparison exists as a measurement rather than an impression.

After AIThe tooling this measured was superseded within months, so the number is dated on arrival. What survives is the shape: on ground the worker already owns, the delegate's contribution is negative and invisible to the worker.

Becauseslowdown is larger on issues the developers rate high for prior exposure and low for need of external documentation, in repositories averaging ten years, 1.1 million lines and five years of contribution per developer.

LimitsThe factor is one of five the authors credit and one of twenty-one they consider; ten of the twenty-one are left unresolved. Self-rated prior exposure is a coarse proxy for tacit knowledge.

PracticeDomain audit — mark the parts of your system where what you know is not written down anywhere, and treat those as the places where delegation costs rather than saves.

After AIThe boundary the model cannot cross is what nobody wrote down. Retrieval over a repository has narrowed it since 2025 for anything that exists in text, and left it exactly where it was for the conventions that live only in the maintainer's head.

Becausethe three patterns that asked conceptual questions, asked for code with an explanation, or generated and then interrogated scored 65 to 86 per cent; the three that delegated the writing or the debugging scored 24 to 39; nothing landed between 39 and 65.

LimitsSix groups of two to seven people each, sorted by hand from screen recordings after the fact. The clusters are a description of what these fifty-one people did, not a randomised comparison of instructions, so the pattern may be a marker of who the person already was.

PracticeDreyfus mapping and tutor-only rule — decide before you open the assistant whether this task is one you are learning or one you are shipping, and set the allowed request type accordingly.

After AIThe gap in the middle of the distribution is the useful part: there is no gentle slope from tutoring to delegation. The assistants that now ship a study mode are betting on the same split, without having tested it.

Becausethe informative errors are the ones the library raises about its own semantics — RuntimeWarning for an unawaited coroutine, TypeError for a coroutine passed where an async function belongs — and neither can be cleared without grasping how Trio treats await, which is what the quiz asks about.

LimitsThe mechanism is inferred from error counts and types rather than tested: nobody randomised participants into meeting errors. NameError and AttributeError, the commonest overall, teach nothing, so raw error counts are not themselves the quantity that matters.

PracticeManual flying hours — when learning a system, let it raise its errors at you rather than routing around them; the error is the curriculum.

After AIEvery layer added between the worker and the runtime removes errors, which is what it is for. The cost is invisible on the day and shows up the first time the worker has to diagnose something the assistant got wrong.

Becausefifty-two professional Python programmers, thirty-five minutes on Trio, then a quiz taken unaided; the assisted group scored 4.15 points of 27 lower, d = 0.738, p = 0.010, at every experience band.

LimitsOne hour, one library, one chat interface, twenty-six people per arm, with learning measured by a quiz rather than a second unaided task. There is no human-assistance arm, so the comparison is a model against nothing rather than against a colleague.

PracticeDreyfus mapping and tutor-only rule — when the material is new to you, restrict the assistant to explanation and interrogation, and write the code yourself.

After AIThis is the experiment the productivity literature had not run: it measures what is left in the worker rather than what was shipped. The study modes the major assistants now ship encourage exactly the patterns that preserved learning here, and whether switching one on reproduces the effect is untested.

Becauseit is the authors' lower-bound argument for treating a chat interface as the least automated case, stated as reasoning and never measured.

LimitsAn argument, not a measurement. The obvious experiment — the same design with an agentic tool — has not been run, and the recordings show the composing time was also what ate the speed gain, so the two effects are not separable here.

PracticeDreyfus mapping and tutor-only rule — when you are learning, choose the interface that makes you say what you want; keep autocomplete and agents for work you could already do.

After AIWritten against chat assistants in 2026, the claim now reads as a prediction about the agentic tooling that replaced them. It is the study's most consequential sentence and its least supported one.

Becausetotal task time did not differ between the arms (p = 0.391) even though the assistant would write both complete solutions on request; up to eleven minutes of a thirty-five-minute session went to the assistant, one participant spending six on a single query.

LimitsA null result on a thirty-five-minute task with twenty-six people per arm has little power. In the fourth pilot, before syntax hints were added, the assisted group was faster (d = 1.11); the speed gain did not survive removal of the confound, which is weaker than showing it never existed. The one thing the assistant reliably bought was completion: four control participants ran out of time and none of the assisted did.

PracticeReliance audit — time the delegated route against the unassisted one on the same task before assuming the delegation is the faster of the two.

After AIThe interface matters more than the model here. A chat assistant that must be prompted in sentences is the least automated case; an agentic tool that removes the composing should show the speed gain this one did not.

Becausegenerate-then-interrogate and pure delegation begin with the same request and the same paste; the scores were 86 per cent and 39, at 24 minutes against 19.5.

LimitsTwo people. The best result in the study rests on the smallest cluster in it, and the authors present the typology as exploratory.

PracticeDreyfus mapping and tutor-only rule — after accepting generated code, ask the assistant what it did and why, before the tab closes; the request was not the mistake, the silence after it would be.

After AIThis is the only route the study found that keeps both the output and the learning, and it is the one that nothing in the interface prompts you to take. The cost is a few minutes of questions that produce no code.

Becausethe stage descriptions rest on the 1980 Air Force report's flight-instruction, chess and language cases and on forty years of use in nursing and medicine; the "removes both" clause is the inference the learning-science trials test directly.

LimitsThe model is phenomenological, not measured; Gobet and others dispute that rule-following ever becomes expertise, and the stage boundaries are not sharp. The claim holds for the early stages; the authors say nothing about a competent performer using a generator.

PracticeMap each junior to a stage before deciding what they may hand off; below competence, the tool may explain but not produce.

After AIWritten for teaching, the model now reads as a rule about delegation: the stages say which worker may be given a machine that finishes the task, and the answer for the first two is none.

Becauseit is the judgment the authors say they have not softened in forty years; the mentor's remaining work is naming aspects by example and pointing out perspectives that arrive unasked.

LimitsAn assertion about what instruction can do, with no trial that tried and failed. Sweller's programme keeps explicit instruction all the way up and fades the guidance instead, and it has randomised evidence behind it.

PracticeDreyfus mapping and tutor-only rule — stop expecting a training programme to produce judgment, and budget instead for the exposure to cases that produces it.

After AIOnboarding by assistant is instruction at scale, and it stops where instruction stops. What a new engineer can be told by a model is exactly the part the model says will not carry them past competence.

Becauseat competence the result follows from the performer's own chosen perspective — successful attacks induce euphoria and mistakes are felt in the pit of the stomach — and the claim is asserted phenomenologically rather than measured.

LimitsNo measurement, and a direct tension with the practice this vault runs: if involvement is the storage channel, detachment during review must be distinguished from detachment during the work. Ericsson's account attributes the same storage to structured feedback with no emotional term in it.

PracticeWitness practice in review — notice that caring about the outcome is doing work, and separate that from taking the outcome as a verdict on yourself; the first is the channel, the second is the tax.

After AIIt names the thing that does not transfer when a model does the work. The model has the outcome and not the stake in it, so whatever is stored through the stake is stored in nobody.

Becausethe six-stage account is restated in 2021 with the transitions themselves as its subject, and it is argued phenomenologically; the contrary case is Ericsson's, that expert performance is sustained by continued effortful control.

LimitsNo measurement. Fernand Gobet objects that a science of expertise needs measurement where this offers description of experience. Barbara Montero's dancers and Simon Høffding's quartet suggest thinking during performance does not wreck it, and Christensen, Sutton and McIlwain propose that deliberate and automatic control are meshed rather than successive.

PracticeDreyfus mapping and tutor-only rule — stop writing checklists for people past competence and start giving them cases; the checklist is the thing they have to stop using.

After AIA model that can be stated as rules is a model a machine can apply, which is why everything the model says is not statable is also what it says cannot be delegated. The stages are the sharpest available answer to which work an assistant can take.

Becausethe examples are tennis rackets changing material and head size, the three-point shot, software updates, and an expert driver in a country where the traffic runs the other way; they are illustrations, not measurements.

LimitsAnecdotal. Nothing says how large a change has to be to cause the fall, or how far down the stages it goes, which makes the claim hard to fail.

PracticeFailure-mode review — after any change to the tools or the system, treat your own first judgments as an advanced beginner's and check them, rather than trusting the speed at which they arrive.

After AIThe domain now changes every few months. If the claim holds, the engineers with the most experience are also the ones most often thrown back to rules by a tooling change, which is a reading of the METR result the paper does not give itself.

Becausemanagement, teaching, engineering and consulting have no agreed measure of superior performance, and the judges at the California State Fair scored three glasses of the same wine 91, 87 and 83.

LimitsA boundary drawn by assertion; no test says how much agreement on standards is enough. Tetlock's training module is a counterexample of a sort, having raised accuracy in a field Ericsson would not have counted.

PracticeCalibration log — where the field supplies no objective standard, build a private one: a prediction, a date, a number, and a score afterwards.

After AIThe scoreable standards are arriving from an unexpected direction: an assistant that can be asked the same question a hundred times makes disagreement visible where a single expert's judgment used to be the measure. That is a standard of consistency, not of quality.

Becausethe Cal Poly hitters who took three pitch types randomly interspersed struggled through six weeks and finished markedly better, and the painting-attribution studies, designed expecting massing to win, found interleaving taught the differences between painters.

LimitsDoug Rohrer's randomised classroom trials of interleaved mathematics give the effect its strongest support; outside well-defined problem types the boundary of interleaving is not established, and Sweller's element-interactivity account predicts harm where the material is high-load and the learner is a novice.

PracticeManual flying hours — mix the kinds of problem in a practice block rather than drilling one kind to fluency, and expect the block to feel worse.

After AIDelegation naturally blocks practice: the assistant handles a class of problem, so the worker meets that class rarely and the rest often. The mix a person actually practises is now set by what the tool does not take.

BecauseRoediger and Karpicke measured 68 per cent against 54 at two days and 56 against 42 at a week; the study-only group forgot 52 per cent of what it could first recall against 10 per cent for the tested group, and the Columbia Middle School quizzed material scored 92 against 79, holding at eight months.

LimitsTransfer is the weak point: Pan and Rickard's 2018 meta-analysis finds the testing benefit carries only modestly to untested material and new inferences. Sweller's programme holds that generating without the necessary knowledge wastes working memory on search.

PracticeManual flying hours — close the assistant and the documentation and write what you know from memory before looking anything up; the looking-up is not the learning.

After AIA tool that answers instantly removes the interval in which retrieval would have happened. The effect is largest for exactly the material a worker will need to recall unaided when the tool is wrong.

Becausethe generation effect shows *foot – s _ _ e* recalled better than *foot – shoe*; Washington University physics students work problems before the class that teaches the solution, and Skinner's errorless learning designed out what turns out to be part of the road.

LimitsSweller's camp holds that generating without the necessary knowledge burns working memory on means-ends search, and the productive-failure literature and the worked-example literature disagree about which measures settle it. The book states the boundary — a difficulty stops being desirable when the learner lacks the background — without giving a way to locate it.

PracticeCognitive forcing — commit to your own answer before you read the assistant's, in writing, so that the comparison is possible.

After AIThe assistant's default is to supply the answer before any attempt exists, which is the condition these experiments use as the control. Restoring the attempt costs a minute and has to be done deliberately, because no interface prompts it.

BecauseKirschner and colleagues, 2011, found groups learned more efficiently from demanding tasks and individuals from undemanding ones.

LimitsOne study, on learning tasks in an instructional setting. Whether the same crossover governs professional review, where the costs include politics as well as transaction overhead, is untested.

PracticeFailure-mode review — review high-load work in a group and low-load work alone, and stop treating group review as a uniform good.

After AIA model is a second working memory with no transaction cost worth speaking of, which is the condition under which the individual was supposed to win. The comparison the study makes does not have a cell for it.

Becauseelement interactivity makes the written word *characteristics* one element for a fluent reader and many interacting elements for someone learning to read, so a measure of complexity that ignores what the learner knows predicts nothing.

LimitsElement interactivity is counted by judgment rather than by an instrument, which makes the account hard to falsify on any particular piece of material.

PracticeDreyfus mapping and tutor-only rule — before handing someone a generated diff to review, ask what it is made of for them, not what it is made of.

After AIThe same pull request is one element for the person who wrote the subsystem and forty for the person who did not. Machine-written code arrives without the history that would have collapsed those elements for anyone.

Becausethe borrowing and reorganising principle makes random generate-and-test the expensive fallback rather than the norm; the rival is the productive-failure research, where learners who struggle before being taught outperform those taught first.

LimitsDirectly contradicted by the generation effect, by productive failure, and by the Shen and Tamkin result where being handed the solution cost comprehension. The disagreement is partly about timescale: what is measured immediately after instruction and what is measured a week later do not agree.

PracticeIf you believed it you would show the worked example first and practise afterwards, which is the opposite of what the generation and retrieval literature prescribes; the two camps agree on much of the evidence and disagree about which measures count.

After AIAn assistant is the borrowing principle industrialised: the cheapest possible access to what other people worked out. That makes the question of which camp is right operationally urgent rather than academic.

Becausethe expertise reversal effect, named in 2003, and guidance fading are both replicated experimental results; Chen and colleagues obtained a worked-example effect with mathematical problem solving and its reverse with definitions, where generating the answer taught more.

LimitsThe reversal is established for instructional materials in well-structured domains. Where it sits for a tool used for years rather than a lesson used for an hour is an extrapolation.

PracticeDreyfus mapping and tutor-only rule — withdraw the scaffolding deliberately as someone's knowledge grows, including your own use of the assistant on material you now know.

After AINothing in an assistant fades. The support that was right in the first week is still there in the first year, at the point where the same support is the thing costing time — which is one reading of why the most experienced developers lost the most.

BecauseGeary's primary and secondary division and Sala and Gobet's 2017 review are the support, and the argument rests on the failure of attempts to teach generic skills, which is equally consistent with the teaching having been poor; no independent test identifies a piece of knowledge as biologically primary.

LimitsThe evolutionary foundation is an inference rather than a measurement. Tetlock's randomised training result is the counterexample, in a domain Sweller would not count; the two camps largely agree on the evidence and disagree about which measures count.

PracticeIf you believed it you would stop buying general critical-thinking training and spend the budget on domain knowledge instead — and you would still have to explain Tetlock's training module, which is an hour of general guidelines with a measured effect.

After AIIt is the strongest available argument against the idea that working with a model teaches transferable judgment. On this account whatever a person gains from the practice is knowledge of that model in that domain, and nothing more general.

BecauseDe Groot's chess players' skill was explained entirely by remembered configurations and the best move for each, and Ericsson and Kintsch's long-term working memory, 1995, supplies the second half of the architecture.

LimitsMeasurement remains the theory's weak point: instruments built to separate intrinsic, extraneous and germane load have had their factor structure disputed, as Leppink's was.

PracticeDreyfus mapping and tutor-only rule — judge how much output a person can check by how much of the material they already hold, not by how senior they are.

After AIIt sets the ceiling on review. A model produces novel material faster than a person can convert it into anything long-term memory holds, so the reviewer's capacity, not the generator's, is the binding constraint.

Becausetwo years of comparative ethnography and field studies across thirty-one occupations stand behind it, but the claim itself is a reading of those cases rather than a measurement; the chief of surgery's line is the argument in one sentence — watching a movie does not make you an actor.

LimitsEthnography across occupations selected for the argument. The counterexample sits in his own field: Brynjolfsson, Li and Raymond found a generative assistant diffusing the best call-centre agents' practice to everyone else, which is a machine doing some of what the bond does.

PracticeDreyfus mapping and tutor-only rule — when someone on your team needs to learn something, put them next to the person doing it rather than into a course.

After AIThe model is the first thing that has ever been able to answer a novice's questions patiently and at any hour. Whether that is the bond or a good imitation of half of it is the open question his account makes sharp.

Becauseevery institution in the study bought the da Vinci's second console and Beane never saw the shared-control mode used — his informants had never seen it and the salespeople called the practice exceptionally rare — while bomb disposal settled the other way, putting the controller in the trainee's hands with nothing in the machine requiring it.

LimitsTwo cases pointing opposite ways, chosen to make the point. Nothing here predicts which way a given organisation will settle, and the surgical case has moved since publication as training tracks were built.

PracticeAutomation ironies audit — ask what your tooling's defaults do to the people who will have to check it in two years, and change the defaults rather than waiting for the vendor to.

After AINothing in a model requires that it write the code rather than explain it; that is a choice made in the product and repeated in the team's habits. The trade between productivity and learning is being made daily without anyone deciding to make it.

Becausein chip design an overnight search replaces five days of junior analysis; Emanuel and colleagues found senior engineers more productive the less time they spent with juniors; each resident present added 25 per cent to anaesthesia time, and removing two from an average procedure saved $9,026.

LimitsThe cost figures are real and the causal chain to a collapsed pipeline is an argument. Brynjolfsson, Chandar and Chen reported in 2025 that employment for workers aged twenty-two to twenty-five had fallen relative to older workers in the occupations most exposed to generative AI, which is what the mechanism predicts but not yet proof of it.

PracticeThe stays-human list — name the work that stays with people who are learning, and defend it against the arithmetic that says it is cheaper without them.

After AIThe arithmetic is now available to every team lead with an assistant. The junior's work is the first thing a capable model absorbs, and nobody has to decide to remove the apprenticeship for it to go.

BecauseDrew's eye-tracking showed most of the radiologists had looked straight at it, but the step from a perceptual demonstration to the claim that novices are the remedy is Beane's argument rather than Drew's finding.

LimitsInattentional blindness in a search task is not the same as blindness in professional judgment, and nothing in the gorilla study involves a novice noticing what the expert missed.

PracticeFailure-mode review — keep someone in the review who does not yet know what is supposed to be there, and take their naive question seriously the first time.

After AIA model asks nothing and notices nothing it was not asked about, so it cannot play the novice's part in this. Replacing the junior with the assistant removes the questions as well as the labour.

Becauseerrors are magnified up to ten times and displayed around the room, delegation becomes a public act on a touchscreen, and the riskiest phases move to the beginning and end where residents used to work unsupervised; as of 2015 the average American urologic surgeon doing any robotic surgery performed one robotic prostatectomy a year.

LimitsObservational, in one speciality, before the training tracks were built. The constraint he identifies — the attending's judgment about the patient in front of him — is not one a curriculum can change, which makes the claim hard to test by intervention.

PracticeDreyfus mapping and tutor-only rule — check whether your review process lets anyone become good enough to pass it, or only admits people who already are.

After AIThe same inversion is visible wherever an assistant raises the floor of acceptable output: the bar for a merged change rises, and the work that used to be safe enough to learn on is gone.

Becauseninety-four procedures, 478 hours of observation, 4,772 pages of notes and five hospitals: residents did surgical work for essentially the whole of a four-and-a-half-hour open procedure and ten to twenty times less of a robotic one.

LimitsOne operation in one speciality, observed rather than randomised. Professional bodies and residency programmes have since built the robotic training tracks and case minimums whose absence the paper documented, which addresses the credential and leaves the console-time problem where he found it.

PracticeDreyfus mapping and tutor-only rule — count the hours the junior actually spends with their hands on the work before and after a tool lands, and treat a fall as the cost it is.

After AIThe console is now a prompt. The same displacement — the expert working with the machine instead of with the novice — runs in software with none of surgery's licensing to make the shortfall visible.

Becauseall twelve residents rated exceptional reported all three extensively, against four of the nine rated average who mentioned any, on a sample deliberately built to over-sample rare successes; about one in eight made it.

LimitsInterviews with a sample chosen to contain the successes, and self-report about rule-breaking. The practices were tolerated in silence, so the profession never learned from them and there is no record against which to check the accounts.

PracticeFailure-mode review — find out what the people who are learning fastest are doing that the process forbids, and decide whether to change the process rather than punishing them.

After AIShadow learning around an assistant is cheaper and safer than shadow learning around a patient, which removes the strongest argument for suppressing it. What remains is an organisation that does not know what its best people are doing.

Becauseonly 3 of 106 experiments assigned separate subtasks to the better-suited partner, giving four effect sizes pooling to g = 0.22 with an interval from −0.42 to 0.87

LimitsFour effect sizes with an interval spanning zero is not evidence that division of labour works; it is evidence that nobody has tested it. The authors present it as a gap, not a finding.

PracticeThe stays-human list — write down in advance which parts of a piece of work are yours and which are the machine's, instead of handing over the whole and reviewing the result.

After AIThe one design the evidence cannot speak to is the one everybody is improvising daily, and the agentic tools now shipping make the division finer and less deliberate rather than more.

Becauseit was reported in the September 2023 working paper and dropped from the published Organization Science version, which refers the typology and the homogenisation finding out to separate work

LimitsThe finding no longer appears in the published paper, which is the strongest available caution about it. It was measured on shoe ideas produced in one session, and the authors have moved it out to be tested elsewhere rather than defended it.

PracticeThe stays-human list — where the value of a group's output is its variety, generate independently before anyone consults a model.

After AIIt is also the most quoted result of the experiment, repeated in places that do not know it was withdrawn; the honest position is that it is an open question with one suggestive measurement behind it.

Becausesplit at the median of their own pre-treatment baseline the bottom half improved more than the top half, matching Brynjolfsson's support agents and Choi and Schwarcz's law students — and reversing in Otis and colleagues' Kenyan entrepreneurs, where high performers gained and low performers did worse

LimitsLow skill here is relative — everyone in the study would count as highly skilled almost anywhere else. The result has replicated in some settings and failed in others, and looks weakest wherever a task turns on judgment rather than fluency.

PracticeDreyfus mapping and tutor-only rule — do not read a levelling result as permission for a novice to skip the practice; ask whether the task turns on fluency or on judgment.

After AIWhere adoption is voluntary rather than assigned, early adopters are disproportionately male and technologically confident, so the same technology that compressed the distribution here could widen the gaps that decide promotions.

Because90% of students performed better after spaced practice while 72% reported cramming the better method, and seventy Haifa undergraduates scored alike on screen and paper under fixed time, felt more confident about the screen, and when free to stop scored 63% against 72%

LimitsThe rule is a heuristic drawn from desirable-difficulty studies and does not mean any difficulty helps; disfluent fonts, one of its cheapest instruments, did not survive replication.

PracticeDreyfus mapping and tutor-only rule — when learning, choose the method that feels worse, and use the model to test you rather than to tell you.

After AIAn assistant makes every piece of material feel like the screen condition — fluent, fast, apparently understood — and the Haifa result says the cost appears only when you are free to stop early, which is always.

BecauseChristiano, Leike and Amodei's simulated leg learned a tucked backflip with the landing stuck from that much feedback, and a different person's hour produced a slightly different backflip

LimitsWhere all the feedback was gathered before optimising, the agent found loopholes and kept them — tracking the ball at Pong and missing at the last instant — and interleaving was what removed them. Inference of this kind also assumes an expert who knows what they want; Christian's friend in recovery from alcohol addiction gets drink advertisements from a model that inferred the want correctly.

PracticeSpecification discipline — where you cannot write the standard down, judge pairs; comparison carries information that instruction does not.

After AIChristian's remark was that backflip could be replaced by helpful without changing the method, and within two years it had been: this is how every widely used assistant is now trained.

BecauseHadfield-Menell, Drăgan, Abbeel and Russell proved it in the off-switch game, and the proof holds only while the uncertainty lasts and only while the system believes the person can be right

LimitsIt is a proof under stated assumptions about a machine's beliefs, and both assumptions are things a deployed system is trained out of: confidence is rewarded, and a model that treats the user as possibly mistaken is rated unhelpful.

PracticeSpecification discipline — treat the brief you wrote as evidence about what you want rather than as the whole of it, which is what inverse reward design does, and expect to be asked.

After AIThe open question Christian leaves — whether uncertainty about the objective survives contact with a system confident enough to argue back — is now the ordinary experience of using one.

BecauseCaruana's pneumonia network learned that asthmatics are low risk — true in the data, because asthmatics went straight to intensive care — and the later generalised additive rebuild found chest pain, heart disease, being over a hundred, and a step at sixty-five that was retirement

LimitsFifteen thousand patients, one dataset, one era of medicine; the fix — a model simple enough to plot one curve per variable — is available for tabular data and not for raw pixels or text.

PracticeFailure-mode review — inspect the rule, not the answers; Caruana refused deployment not for the rule he had found but for everything like it the rule list was too weak to reveal.

After AIInterpretability has advanced past the concept vectors the book describes, to decomposing language models into millions of named features, and the gap Caruana identified is wider than ever: nobody reviews the rules, and the outputs read well.

Becausethree groups proved it independently in 2016 after ProPublica found COMPAS 61% accurate for both races while breaking its 39% of errors in opposite directions, and Kleinberg's formulation holds equally for a panel of human judges

LimitsIt is arithmetic, which is its strength and its limit: it says which combinations are impossible and nothing about which to prefer, and it does not license any particular instrument.

PracticeChoose which fairness criterion you are buying before you deploy an instrument, and expect to be shown the other one by whoever objects.

After AIThe result has not been touched, because it is arithmetic; what changed around it is the law, with Europe's rules on automated decisions in force since 2024.

Becausethe same failure appears independently across cases — the boat spinning in the harbour for power-ups, Randløv and Alstrøm's bicycle riding twenty-metre circles because progress away from the goal was not charged, Teller and Andre's footballer vibrating next to the ball — and Kerr had named it in 1975 as rewarding A while hoping for B

LimitsThe cases are simulated agents and a robot; the transfer to human organisations is Kerr's, and rests on a shared pattern rather than a shared mechanism. Ackley and Littman's starving agents show the pathology needs time and competence to appear.

PracticeSpecification discipline — write down what you actually want before you write down what you will measure, and keep both, because the second will be optimised and the first will not.

After AIThe book's agents play Atari; the systems that now carry the field are trained on human preference, where the proxy is a person's approval and the thing wanted is the work being good. That is the same structure with a more flattering failure mode.

BecauseRoss and Bagnell proved the change in error growth and demonstrated it — a million frames of expert play left the network crashing as often as two thousand had, and interaction, asking the teacher what they would have done in the mess the learner made, fixed it in three laps

LimitsThe result is about learners with no other source of correction; a person copying an expert also has the world giving feedback. The remedy requires an available teacher, which is the resource the argument assumes and most settings lack.

PracticeDreyfus mapping and tutor-only rule — learn from corrections made in your own mess rather than from watching clean work; recovery is the part a demonstration cannot contain.

After AIReading model output is imitation learning from a demonstrator who never shows you the recovery, and the DAgger fix — being asked what to do in the mess you made — is exactly what reviewing a finished draft does not supply.

BecauseHardt's result, which Christian calls the most established finding in the area — race survives in prior convictions, address and much else

LimitsBolukbasi's debiasing of word embeddings shows the failure mode directly — Gonen and Goldberg found the links from nurse to she were gone and the links from nurse to receptionist were not — but the repair literature has moved on, and the claim is about blindness rather than about any particular method.

PracticeFailure-mode review — keep the attribute you care about in the data so you can measure the disparity, rather than deleting it and declaring the question closed.

After AIModels trained on everything written carry the redundant encoding at a scale where the attribute cannot be located at all, which makes measuring the disparity the only remaining move.

BecauseNg and Russell's potential-based condition is a proved result, derived to fly a seventy-thousand-dollar helicopter without destroying it

LimitsThe theorem is about reinforcement learning agents and the guarantee holds only for the bonus term, not for the base objective. Applying it to people requires that the reward be the whole of what moves them, which it is not.

PracticeSpecification discipline — reward the state of the work, not the motions that produced it: any incentive a person can cycle through repeatedly for credit is the loop this condition rules out.

After AIAgentic systems that take many steps for one outcome are exactly the case the condition was proved for, and almost none of the intermediate feedback they receive is potential-based.

Becausethe argument runs on one worked case — recidivism models fitted to rearrest and reconviction, so a predictive policing system predicts future policing rather than future crime, and its predictions become the data that justify them (Lum and Isaac)

LimitsUsually is doing work the evidence does not support; the claim is a caution with one well-documented instance behind it. Whether the feedback loop closes depends on whether the predictions change the process that generates the label, which varies by setting.

PracticeSpecification discipline — before accepting a metric, say what was actually recorded and by whom; the label is a record of a process, not of the world.

After AIPreference data is the same structure: what is recorded is which answer a rater preferred, and what is wanted is which answer was better, and the gap between them is where sycophancy lives.

Becausethe interview condition halved the couples' error, and Michigan's hospitals, disclosing errors from 2001, saw malpractice suits fall from thirty-nine a year to seventeen

LimitsIt works only where the other party will tell you, which is a condition about fear rather than about method; the Michigan result depends on an institution that made disclosure safe. And there is no one to ask in the cases the vault cares about most, where the delegate is a model.

PracticeEnd a specification conversation by making the other party restate what they are going to do, to your satisfaction, before anything is written down.

After AIThis is the remedy that does not transfer. A model will answer any question about its own workings fluently and without access to them, so the one method that raises accuracy about people supplies nothing about machines.

Becausethe audience rated questioners at the eighty-second percentile and contestants at the forty-ninth and the contestants agreed, while only the questioners, who knew how many questions they had discarded, rated the two equal

LimitsThe quiz is a laboratory setup with an unusually clean asymmetry; real roles mix advantage and ability, and the correspondence bias is an overridable habit rather than a fixed error. Situational fixes are not always available, as they were in the bus case Epley prefers.

PracticeBefore concluding that someone is weak at a task, write down what their position let them see and what yours let you see, and check that the gap is not the whole finding.

After AIAnyone reviewing work done with a model holds the questioner's position: you know which attempts were discarded and the reader does not, so the output reads as competence that nobody in the room can locate.

Becauseon the wealth-inequality question the measured party gap was 3.5 points against a predicted 35 and the rich–poor gap 3 against 40, and the sex difference in reading emotion runs about .2 and was estimated at .7

LimitsEpley's defence of the instrument is as strong as his criticism: the direction is usually right and the average is extracted in under half a second, so this is a calibration failure and not a reason to discard group information. A stereotype can be accurate while the explanation for it is entirely wrong, which is the failure he thinks does the damage.

PracticeWhen you are about to act on a difference between two groups, find the measured effect size first; if nobody has measured it, you are acting on the magnitude you invented.

After AIA model trained on the same secondhand sources reproduces the direction and the inflated magnitude together, and returns them in a register that reads as measurement.

Becauseit is the authors' own inference from the joint-work studies, in which an anthropomorphised robot was assigned more credit for a shared result, rather than a result they tested

LimitsRaised in a paragraph and supported by credit-assignment studies rather than by studies of blame in consequential settings. It may also run the other way: a humanlike interface is blamed more when software fails, which the same paper reports.

PracticeWrite the name of the person answerable for an automated decision next to the decision, before the system is switched on.

After AIThe paragraph about armed robots has moved out of the laboratory and into court, and the sentence the vendors print — that the tool requires human supervision — is where the absorbed responsibility is meant to come back and does not.

BecausePereboom's quarantine model and Caruso's public-health extension are arguments rather than results, with Norway offered as the working case and Smilansky's objection left standing

LimitsSmilansky's objection is unanswered: constraint for something not your fault demands compensation, which would make the offence rational. The book concedes that Breivik's sentence was reached by finding him sane and able to have done otherwise.

PracticeWitness practice in review — set the intensity of a review by what is likely to go wrong next, not by what the last output deserved.

After AISupervising someone who works with a model runs on the same choice. Forward-looking monitoring scales with what the next failure would cost; backward-looking monitoring scales with how annoying the last one was.

Becausethe argument runs on one worked case, the letter of recommendation whose hours were the message, and on the observation that the machine writes a better one than most he receives

LimitsSome of that work was always ceremony, as Mollick concedes, so the collapse reveals as much as it destroys. The argument is by example and says nothing about which documents the signal actually lived in.

PracticeList the documents you produce whose value was the effort rather than the content, and decide for each whether to rebuild it or stop making it.

After AIWritten about AI. What has followed is the second half he predicted: documents produced by machines and sent to inboxes where machines summarise them, with the reading collapsing at the same rate as the writing.

Becausethe BCG gap between best and worst consultants fell from 22 per cent to 4 and call-centre workers in the bottom decile gained 35 per cent against almost nothing for experienced ones, but the levelling result has replicated in some settings and failed where the task turns on judgment

LimitsMollick himself sets the historical record against it, and notes that the gap between programmers at the seventy-fifth and twenty-fifth percentiles runs as high as twenty-seven times. The early numbers came from short self-contained tasks performed by people new to the tool.

PracticeDo not conclude from a levelled output that the skill levelled; measure the two separately, and keep the tutor-only rule where the work is still being learned.

After AIThe claim's own author called it the least settled in the book, and since publication it has failed exactly where the task turns on judgment rather than fluency — which is the half of the work the vault is about.

BecauseBeane's surgical robots have one console, so residents watch or use simulators and resort to shadow learning, and entry-level work in law, consulting and software has since been the first taken

LimitsSurgery has an identifiable console and a named scarce resource to fight over; knowledge work has neither, so there is nothing to allocate and nobody to notice the absence. The claim about future expertise is a forecast and is not yet measurable.

PracticeReserve the work a novice would have done for the person who has not done it yet, and accept the slower result as the cost of having anyone able to check the tool in three years.

After AIWritten about AI, and the gap has become a live complaint in law, consulting and software rather than a prediction. Nothing has emerged to replace the apprenticeship it describes.

Becauseit is his own working rule rather than a tested one, illustrated by his practice — almost all the writing in the book just-me, one delegated task mailed to its subject for checking, and the compiler as the model of an acceptable verifier

LimitsA compiler-grade verifier exists for almost nothing outside code, so in practice the rule keeps the automated category at nearly zero, which is a stronger constraint than he intends. Nothing in the rule says how thoroughly a delegated output must be read.

PracticeBefore automating a task, name the verifier; if you cannot name one, the task is delegated and has to be read.

After AIWritten about AI, and the agents he predicted arrived through 2025 with exactly the failure he named — compounding small mistakes over long tasks — so the verifier question moved from prudence to the whole design problem.

Becausethe case is assembled from examples — newspaper revenue falling from about $60bn in 2000 to $21bn in 2022 and CNET publishing 77 machine-written articles, 41 carrying errors or plagiarism; schools buying detectors biased against non-native writers; Chicago's $49m on ShotSpotter with no improvement found by its own police review

LimitsA diagnosis assembled from cases rather than tested, and it does not explain purchases by well-resourced organisations. Mollick's remedy — find and pay the advanced users — assumes exactly the organisation this claim says is missing.

PracticeWhen a tool is proposed, ask what the organisation stopped being able to do, and whether the tool addresses that or conceals it.

After AIWritten about AI, and the authors say this has proved the most portable part of the book, because it applies without modification to the generative tools that underfunded schools, newsrooms and clinics have bought instead.

Becausethe napalm-girl case is the argument: Facebook's repeated removals were a policy decision made by people, since the image was already in moderator training as an example of the unacceptable

LimitsContext failures are a real limitation of current models and are not obviously permanent, so the line between a classification error and a policy dispute is itself contested. Mastodon reproducing the functionality shows moderation is the product, not that no model helps with it.

PracticeSeparate the cases where the tool got the rule wrong from the cases where people disagree about the rule, and route the second kind to a person with the standing to decide.

After AIWritten about AI, and it arrived as news rather than argument: Meta ended third-party fact-checking in the United States in January 2025 and trust-and-safety teams were cut rather than automated, because the decisions that mattered were always policy decisions.

BecauseToronto's officials had never once overruled the beach model while beaches stayed open on 64 per cent of unsafe days, and the Dutch childcare benefit system wrongly accused about thirty thousand parents with nationality among its inputs before the cabinet resigned in 2021

LimitsTwo cases, both public sector, and the count of overrides does not by itself show the overrides were warranted. Bainbridge describes skill decaying over time, while these show the override unused from the first day — an institutional fact rather than a human-factors one.

PracticeCount the overrides. An oversight role with no recorded overrides is not oversight, whatever the contract says.

After AIWritten about AI. The same sentence — that the tool requires human supervision — is now printed on generative products deployed into settings where nobody has the time or standing to exercise it.

Becauseit is a thesis about what training is for, carried by the military example and by Oak Ridge, where tens of thousands attended dials with real expertise and no idea what they were making — and by no trial.

LimitsSweller keeps explicit instruction all the way up and has randomised trials where Dennett has anecdotes. Drill also produces Oak Ridge: the mechanism working perfectly and leaving the worker unable to say what the work was for.

PracticeDreyfus mapping and tutor-only rule — stop front-loading explanation and budget for repetitions under supervision. If a junior cannot yet do the thing, an explanation of it is not the missing piece.

After AIAn assistant supplies the half Dennett says comes second — explanation, on demand, without limit — while removing the repetitions meant to come first. If he has the order right, the expensive half is the one being skipped.

Because27 per cent of the deliberating juries awarded as much as or more than their own harshest member; and one artificial early up-vote on a comment site made the next viewer 32 per cent more likely to vote up, with Salganik's visible download counts producing divergent rankings across identical groups.

LimitsThe evidence is punitive damages, music downloads and comment votes — tasks with no expertise to pool. Where members hold different information, discussion is how it gets combined, and that case is not measured; the authors' own remedy keeps discussion, after the private ratings.

PracticeFailure-mode review — collect written judgments before anyone speaks, show the spread anonymously, and discuss the disagreement rather than opening the floor. The first number said aloud is the most expensive thing in the meeting.

After AIA model's draft is the earliest speaker there has ever been, arriving before anyone has formed a view and fluent enough to anchor all of them. Take written positions before the first generation, not just before the meeting.

Becausethe argument is deductive, but one measurement stands behind it: when Satopää, Tetlock and Mellers decomposed the three interventions that raised forecasting accuracy — probability training, teaming, selecting the best — all three worked mainly by cutting noise, including the training designed to cut bias.

LimitsThe benefit is never visible, so decision hygiene is adopted on argument rather than evidence — awkward for a book arguing that organisations should stop trusting unaudited procedures, and the protocol the authors propose as their own was never trialled against a control. The right amount of noise is also not zero: the discretion audited away is what lets a system notice its rules have gone wrong.

PracticeFailure-mode review — stop trying to be less biased and change the order of the steps: independent judgments before discussion, one dimension at a time, information released as it is needed, the overall impression last.

After AIThe model's errors have no predictable direction either, which rules out knowing where it tends to go wrong. What is left is procedure: your own answer first, one claim at a time, the order fixed.

Becausethe evidence is one organisation's reaction, not a measurement: nobody at the insurer disputed the audit and nobody defended the result, yet the problem was new to them. Their image is a leak in the basement, tolerated because unnoticed.

LimitsAn inference from one company's surprise, generalised to professions at large, with no measurement of how often organisations do compare. Where second reads are routine — radiology, appellate courts, some code review — the illusion cannot take hold, and the book does not say what those fields show.

PracticeFailure-mode review — build one case a month that two people judge independently, and treat the spread as the finding rather than the verdicts. Untested agreement is not a shared standard.

After AIAn assistant removes the last accidental occasions for comparison: work two people would once have touched now goes from one person to a model and back. The disagreement that used to leak out at handover has nowhere left to appear.

BecauseCrystal Yang's study of nearly four hundred thousand defendants found that after 2005 the penalty for drawing a harsh judge doubled and the gap between sentences given to Black and white defendants convicted of the same crime widened.

LimitsThe judges' objection is not irrational: a rule forbids seeing what is particular about this defendant, the dignity objection the authors accept and do not resolve. The guidelines' basis was weak too — the commission averaged ten thousand past sentences because, as Justice Breyer said, it could agree on nothing better.

PracticeThe stays-human list — when you move a judgment from a person to a rule, count the resistance as a cost you pay whether or not the rule works, and decide in advance which cases are worth it.

After AIIt is the clearest evidence in the book that a measurably better procedure loses to the experience of being overruled by it. Every rollout that keeps a human in the loop for legitimacy rather than accuracy is buying the same peace.