38Make It Stick

Peter C. Brown, Henry L. Roediger III and Mark A. McDaniel, Make It Stick: The Science of Successful Learning, Belknap Press, 2014; with Elizabeth and Robert Bjork, "Making things hard on yourself, but in a good way", 2011

Linked to 18 chapters: supports 7 · contradicts 3 · mechanism 13 · same claim 4 · pairing 2

Roediger and McDaniel are cognitive psychologists who have spent their careers on memory; Peter Brown is a writer. They came together after a collaboration funded in 2002 by the James S. McDonnell Foundation, which gave eleven psychologists a grant to carry laboratory findings about learning into schools, and the book is the report of that decade. Its argument is one inversion. The study methods almost everyone uses — rereading, highlighting, drilling one thing until it runs smooth — produce fast, visible gains that are almost entirely temporary. The methods that produce durable learning — retrieving from memory instead of reviewing, leaving gaps between sessions, mixing problem types, attempting an answer before being shown one — feel slow and unproductive while they are happening, which is the mechanism by which they stay unused: a learner judges learning by how the learning feels, and the feeling points the wrong way. Elizabeth and Robert Bjork, whose term desirable difficulties the book borrows, put the same point in the language of their own theory: what you observe during practice is performance, and performance is a poor index of learning. From that inversion everything else follows — why testing is a study tool rather than a measuring stick, why forgetting a little is a condition of learning more, why errors belong on the road to mastery, and why the last chapter can be a list of instructions that cost nothing.

Learning is misunderstood

The book opens with Matt Brown, a pilot who lost oil pressure in the right engine of a Cessna 401 at eleven thousand feet over Texas at night, shut the engine down, and made his approach in a wide left-hand turn because without power on the right it was only from a left turn that he had the lift to level out. Learning, in the definition taken from that story, is acquiring knowledge and skills and having them available from memory when a future problem calls for them. It requires memory, it goes on all our lives, and its most effective methods are counterintuitive.

The evidence that repetition alone does nothing goes back to Endel Tulving at Toronto in the mid-1960s. Participants read a list of paired items six times without expecting a test. Afterwards one group was asked to learn the nouns they had just read six times and another to learn nouns they had never seen, and the two learning curves were statistically indistinguishable. Prior exposure had bought nothing. Washington University researchers reported in 2008 on the same question for prose: a hundred and forty-eight students at two universities, strong readers and weak, read five passages from textbooks and Scientific American either once or twice in succession, and reading twice in close succession produced no benefit at all, in any condition tested. Rereading after a meaningful delay does help; rereading straight away does not, and it consumes the hours that would otherwise go to something that works. More than eighty per cent of college students in some surveys name rereading as their principal method.

The method survives because it feels as though it works. Growing familiarity with a page produces fluency, and fluency is mistaken for possession of the content. Against this the authors set testing, and ask that it be understood as something other than a dipstick. In a middle school in Columbia, Illinois, part of the eighth-grade science material was quizzed three times at low stakes during a semester and part was reviewed three times instead; a month later the students averaged an A minus on the quizzed material and a C plus on the rest. The claim is not that memorising substitutes for thinking. Knowledge is necessary rather than sufficient: as Robert Sternberg and two colleagues put it, you cannot apply in practice what you do not know.

To learn, retrieve

Mike Ebersold, a neurosurgeon who trained at the Mayo Clinic, was called to a Wisconsin emergency room in 2011 to see a deer hunter found unconscious in a cornfield, and recognised a gunshot wound. The slug had entered above a large venous sinus and corked the torn vein; when he removed it, two units of blood went in five minutes. The vein is flat and tears if you put a stitch round it, so he fell back on a repair he had worked out in earlier surgeries and never seen written anywhere: two small pieces of muscle cut from the opened scalp, with the torn ends of the vein stitched to them. The bleeding stopped in about a minute.

The authors use him twice. The repair came out of reflection — going home after a hard operation and asking what could have been done differently, a bigger bite with the needle or a smaller one, then trying it the next day. Reflection is retrieval of earlier training, connected to a new experience and mentally rehearsed, and it is a form of practice. The reflex came from repeated recall: unless you keep recalling the manoeuvre, Ebersold says, it will not become reflex, and in a bleed you act before you have time to think.

Psychologists have been plotting forgetting curves since 1885, and something like seventy per cent of what we hear or read is gone very quickly, so the problem of learning is the problem of interrupting forgetting. The tool for it has been known for a long time and used for almost none of it: Aristotle wrote that repeatedly recalling a thing strengthens the memory, and a study of more than three thousand Iowa sixth-graders in 1939 found that once a student had taken a test, forgetting nearly stopped. Testing then fell out of use as a research tool for the very reason that made it interesting: it interrupts what you are trying to measure.

In the experiments Roediger and Karpicke published in 2006, students who took a single recall test on a science passage recalled sixty-eight per cent of it after two days against fifty-four for those who restudied, and fifty-six against forty-two after a week; in a companion experiment the study-only group forgot fifty-two per cent of what they had first been able to recall and the repeatedly tested group forgot ten. Multiple tests immunise better than one, and spaced tests better than massed. Even the smallest demand helps: studying the pair foot – shoe produced worse recall than studying foot – s _ _ e, which the learner must complete. That is the generation effect, visible in the effort of filling one blank.

In 2005 the authors approached Roger Chamberlain, principal of Columbia Middle School in Illinois, with a proposal to try this in a real classroom. He was wary of anything that amounted to memorisation and protective of his teachers' methods, so the terms were that nothing would change except the addition of short quizzes: same textbooks, same lesson plans, same exams. Patrice Bain's sixth-grade social studies classes went first. A research assistant, Pooja Agarwal, built quizzes covering about a third of the material — once before the lesson, once after, once the day before the unit exam — scored for no grade, with the teacher out of the room so she would not know what had been tested. A second third was presented three times as plain statements of fact instead, so that mere re-exposure could be ruled out. The quizzed third came back a full grade level above the rest; the restudied third came back no better than the material nobody had touched. The eighth-grade science classes ran the same design over three semesters and averaged ninety-two per cent on quizzed material against seventy-nine on unquizzed, and the effect was still there in the year-end exams eight months later. Sixty-four per cent of those students said the quizzing reduced their anxiety about unit exams.

Formats that make you supply the answer beat formats that let you recognise it, and briefly delayed feedback beats immediate feedback, which functions like training wheels the learner comes to lean on. And almost nobody does any of it: in one survey eleven per cent of college students said they quizzed themselves, and most of those did it to find out what they did not know rather than because retrieval itself makes things stick. Retrieval ties the knot for memory, the authors write, and repeated retrieval snugs it up.

Mix up your practice

Eight-year-olds in a gym class practised tossing beanbags into buckets for twelve weeks. Half always threw at a bucket three feet away; half threw at buckets two and four feet away and never at three. Everyone was then tested at three feet, and the children who had never practised at that distance did better. The result, from a 1978 study by Kerr and Booth, is the book's standing rebuke to the idea that practice should look like the test.

Massed practice — the single-minded repetition of one thing until it is smooth — is the default everywhere: language boot camps, weekend professional seminars, sports drills, cramming. It works, in the sense that you can watch yourself improve. Researchers call that improvement momentary strength and distinguish it from underlying habit strength, and the techniques that build the second slow down the first. Thirty-eight surgical residents took the same four short lessons in microsurgery, half in a single day and half a week apart. A month later, on elapsed time, number of hand movements and success at reattaching the severed aortas of live rats, the spaced group won every measure; sixteen per cent of the massed group damaged the vessels beyond repair and could not finish.

Interleaving does the same within a session. Two groups of college students learned to find the volumes of four obscure solids — wedge, spheroid, spherical cone, half cone. One worked practice problems in blocks by type and averaged eighty-nine per cent during practice; the other worked the same problems shuffled and averaged sixty. On a test a week later the blocked group scored twenty per cent and the interleaved group sixty-three. The baseball team at California Polytechnic State University ran the same experiment on hitting: forty-five extra pitches twice a week for six weeks, one half receiving fifteen fastballs then fifteen curveballs then fifteen changeups, the other receiving the three types randomly interspersed. The blocked hitters improved visibly within each set of fifteen; the random hitters struggled through all six weeks and finished markedly the better hitters, though every one of them had been skilled before the training began. Hitting a curveball you know is coming is one skill; hitting one you do not is another, and only the second is the game.

Where interleaving earns most is discrimination. Mathematics textbooks are built in blocks — a chapter per problem type, twenty exercises, on to the next — and then the final exam mixes everything up and the student, who has never once had to ask which method this problem calls for, is asked exactly that. Studies of attributing paintings to artists were designed on the expectation that massing would win, since studying many works by one painter should best reveal that painter's style. They found the opposite: interleaving taught the differences between painters, which is what the test required, without costing the commonalities within one. The finding that ought to unsettle everyone is that students in these experiments went on preferring massed practice after taking the test that showed them they had been wrong. Douglas Larsen, who teaches pediatric neurology at Washington University Medical School, adds one caution: variety cannot be bought at the cost of repeated retrieval on the diseases a doctor must know cold.

Embrace difficulties

Mia Blundetto, a first lieutenant in the Marines who did not want to jump out of aeroplanes, went to the army's jump school at Fort Benning because the air delivery platoon was the billet everyone wanted, and the training is the book's picture of graded difficulty. No notebooks are allowed. The parachute landing fall is demonstrated in a gravel pit, then practised from a two-foot platform, then from a zip line with the instructor rather than the jumper choosing the moment of release so that the fall comes from an unpredictable direction, then from a thirty-four-foot tower with gear and a mock jump door. On her third real jump Blundetto came down on another jumper's canopy at four seconds and swam out of it, which she describes as unalarming, because her confidence rested on demonstrated performance under conditions close to the real one rather than on a feeling.

To explain why the hard version teaches more, the authors lay out three stages. Encoding turns perception into a memory trace, which is a note on a scratchpad and is mostly lost. Consolidation strengthens and reorganises that trace over hours or days, giving it meaning and tying it to prior knowledge; sleep appears to help. Retrieval brings it back, and the act of bringing it back makes it pliable again — reconsolidation — so that it can absorb new connections before it settles. Their analogy is drafting an essay: the first version is rangy and imprecise, you discover what you meant by trying to write it, and you come back two days later knowing which three points you are making.

Retrieval cues are the other half of the problem. Long-term storage is effectively unlimited, and the more you know the more places new knowledge has to attach; what is scarce is access, and what is lost is usually the cue rather than the knowledge, which is why a forgotten address can still be picked out of a list. Some forgetting is therefore necessary rather than incidental: to learn Italian in middle age you may have to lose the cue that keeps producing être.

This is where the Bjorks' theory earns its place. They separate storage strength — how entrenched and interconnected a memory is — from retrieval strength — how accessible it is right now, which depends on recency and on the cues that happen to be present. Current performance is entirely a matter of retrieval strength. Storage strength does not show up in performance at all; it shows up in how slowly the memory is lost and how quickly it is regained. The conditions that raise retrieval strength fastest are not those that build storage strength best, so a learner who reads performance as learning will reliably prefer the worse method. And when something is maximally accessible, further practice adds almost nothing: losing retrieval strength is what creates the opportunity to gain storage strength.

The Bjorks name four difficulties that are desirable, and the word is doing work. Varying the conditions of practice rather than keeping them fixed: studying the same material in two different rooms produces better recall than studying it twice in one, against all the standard advice about finding a quiet place and always using it. Spacing rather than massing, which they call one of the most general effects in the history of research on learning. Interleaving rather than blocking: in a 1979 experiment participants practised three movement patterns on a pinball-like apparatus, blocked or randomly interspersed, and the blocked group improved faster — but when both returned ten days later and were tested under random conditions, the blocked group appeared to have learned virtually nothing. And using tests rather than presentations as study events, with generation as its smaller cousin: any time you look up an answer you could have produced, you have thrown away a chance to learn. Retrieval, in Robert Bjork's phrase from 1975, is a memory modifier.

The metacognitive failure runs alongside each. Asked to predict their performance the next day, learners who had practised keystroke patterns under interleaved conditions predicted accurately and those who had practised blocked were markedly overconfident: they had read retrieval strength as storage strength. And a difficulty stops being desirable at the point where the learner lacks the background to overcome it. A textbook in Lithuanian is not a desirable difficulty, and nor is anxiety.

Inside that boundary the list is longer than the four. Elaboration — restating new material in your own words and tying it to what you already know, so that angular momentum arrives attached to a skater pulling in her arms — has no known ceiling, which is what separates it from mechanical repetition. And error, which B. F. Skinner's errorless learning tried to design out, is part of the road: given corrective feedback, learners do not learn their errors, and attempting a problem before being taught the solution beats being taught it first. The Bjorks end their own essay on the practical conclusion: spend less time on the input side and more on the output side, because "learning how to learn is the ultimate survival tool".

Avoid illusions of knowing

If judgement is the instrument that selects a study method, the book has to say how badly the instrument is made. It borrows Daniel Kahneman's two systems — the fast, unconscious, experience-driven one and the slow, analytic one — and its case in point is China Airlines Flight 006 in February 1985, a 747 at forty-one thousand feet over the Pacific. Engine four lost power; the correct action was to descend below thirty thousand feet and restart. Instead the crew held altitude on autopilot while the asymmetric thrust rolled the aircraft right. The captain could feel that the plane was decelerating but not that it was banking, because the trajectory gave him the sensation of level flight. The plane rolled past forty-five degrees, turned over and fell. The gauges showed the dive, but the angle was so improbable that the crew concluded the gauges had failed. They broke out of cloud at eleven thousand feet, pulled five gravities levelling off, bent the wings permanently upward, and landed at San Francisco.

Memory is no steadier than perception, and confidence tracks emotional vividness rather than accuracy: in a survey of fifteen hundred Americans about the September 11 attacks, revisited over ten years, the personal details respondents were most certain of were also the ones that had changed the most.

Two illusions bear directly on studying. The curse of knowledge makes an expert a poor judge of a novice's difficulty, so that the teacher who has bundled the steps into one model cannot see the student stuck among them. And fluency illusions do the rest: a lucid explanation of a hard idea leaves the impression that the idea is simple and was already known. The most direct evidence is David Dunning and Justin Kruger's work on people who are unskilled and unaware of it. Students given a test of logic and asked to rate themselves showed the pattern in its strongest form at the bottom: those who scored at the twelfth percentile placed their reasoning at the sixty-eighth. Shown their more competent peers' answers, the bottom quartile raised their estimates rather than lowering them. But ten minutes of training in how to test a syllogism made their self-assessments accurate — incompetence and the inability to see it are the same deficit, and teaching the skill repairs both.

The remedy proposed is not more introspection but external instruments. Frequent low-stakes quizzing, with the answers actually written down rather than waved at. Explaining a text, which requires recalling the salient points, putting them in your own words and saying why they matter. Eric Mazur's peer instruction, in which a lecture is broken by conceptual questions that small groups must settle among themselves. And corrective feedback, which Dunning argues is the route to self-insight, since the path to it runs through other people.

Two adjacent claims follow from the same commitment to evidence. In 2008 Harold Pashler, Mark McDaniel, Doug Rohrer and Robert Bjork were commissioned to review the learning styles literature. They set out what a study would have to look like to support the claim that matters — that people learn better when instruction matches their style, which requires each style group to do best under its own style and not merely to do well — and then looked for such studies. There were very few, almost none supported the claim, and several contradicted it. What does appear to matter is matching the mode to the subject: visual instruction for geometry, verbal for poetry, from which all learners benefit regardless of preference. A 2004 survey for Britain's Learning and Skills Research Centre had already counted more than seventy competing styles theories with their diagnostic instruments. What the authors put in their place is structure building — the habit of extracting central ideas and assembling them into a working model rather than carrying every detail — which separates learners measurably, and which embedded questions can teach.

The second claim is that ability itself is less fixed than most learners assume, and here too the authors are careful about limits. Measured intelligence has risen in every industrialised country since standardised sampling began in 1932 — eighteen points in sixty years in the United States — which is a fact about environments rather than genes; Carol Dweck's work shows that praising intelligence rather than effort makes children choose easier problems and collapse after a failure; and Anders Ericsson's deliberate practice is what the authors offer in place of talent. But brain-training products get short treatment: they rest largely on one Swiss study from 2008 with thirty-five participants, one training task, and no replication, while an attempt published in 2013 found no improvement in fluid intelligence, though participants nonetheless believed their capacities had grown. The brain is not a muscle; a pianist's myelination is specific to piano. What generalises is the habit of practising.

Make it stick

The last chapter stops describing and starts instructing, because early readers asked it to. Three strategies carry the weight, and each is presented with what intuition recommends instead and what the better method feels like while you are doing it, because the feeling is the obstacle.

Around these sit elaboration, generation, reflection and calibration. Elaboration is finding further layers of meaning — explaining new material to someone else in your own words, or finding the image that carries it. Generation is attempting the answer first, which is why the physics department at Washington University makes students work problems before the class in which the solution is taught. Reflection combines retrieval with elaboration: what happened, how did it work, what would I do differently. Calibration is the deliberate use of an objective instrument, and its discipline is writing the answer down rather than looking at a practice question and telling yourself you know it.

The people who demonstrate this are mostly not students. Michael Young entered medical school at Georgia Regents University without premedical coursework, scored sixty-five on his first exam, went and read the empirical literature rather than collecting opinions, and rebuilt his studying around stopping to recall rather than reading on, slowing down enough to attach meaning to a term, and spacing the attempts; by his second year he had come from the bottom of a class of two hundred to the top of it. Nathaniel Fuller, an actor at the Guthrie Theater who can go on for a lead at no notice, covers the page with a blank sheet and draws it down, speaking each line from memory before revealing it, then replaces the script with a recorder carrying the other actors' lines so that he must produce his own from the cue alone.

For teachers the instructions are to explain how learning works, to teach students how to study, to build desirable difficulties into the class, and to be transparent about why the class is harder than it needs to be. Mary Pat Wenderoth, who teaches physiology at the University of Washington, tells her students at the outset about the testing effect and the illusions of knowing, then models it: a question every five minutes, and when they reach for their notes, an instruction to stop and think instead, because opening the notes short-circuits the path they were about to make. She converts study groups into testing groups, where nobody opens a book and one student at the board is questioned by the rest. Comparing high-structure classes, built on daily and weekly low-stakes exercises, against low-structure ones built on lectures and two big exams, she and her colleagues found the failure rate in a gateway biology course significantly reduced and the gap between poorly prepared students and their peers narrowed; and exercises that counted toward the grade, even at very low stakes, outperformed identical exercises that did not.

West Point has been doing a version of this since Sylvanus Thayer: specific objectives for every course, the responsibility for meeting them placed on the student, and quizzing and recitation in every class. Michael Matthews tells his students that if they have read every word of the chapter they are not being efficient — start with questions and read for answers. Kiley Hunkler, one of them, prepared for the medical admission test by writing her own objectives for each block and taking a practice test every three days to see what she had wrong, which she calls shooting an azimuth: climbing to a height, sighting your landmark, and checking your heading before going back down into the trees. Trainers get the same principles in looser settings: Farmers Insurance brings new agents in for a week of exercises that cycle repeatedly through sales, marketing, business planning and the brand, so that a get-acquainted exercise on the first morning — ask about family, occupation, recreation and enjoyment — returns days later as a tool for qualifying a prospect.

The authors close by separating their subject from education. Education is an institutional question full of hard arguments about what should be taught and how outcomes should be measured. Learning is an individual responsibility, and the methods in the book require no structural reform, cost nothing, and can be started this afternoon.

Since 2014

The core has held up better than most of what was published in psychology that decade. Retrieval practice and spacing have survived meta-analysis repeatedly, and "retrieval practice" has become a phrase schoolteachers use without attribution; Pooja Agarwal, the research assistant who ran the Columbia quizzes, and Patrice Bain, the teacher who volunteered first, wrote a practitioners' book together in 2019. Interleaving has moved into randomised classroom trials — Doug Rohrer and colleagues ran interleaved mathematics assignments through seventh-grade classes and found a large advantage on a delayed test, the strongest evidence the effect has. Robert Bjork and Nicholas Soderstrom set out the learning-versus-performance distinction formally in 2015, and it is now the standard warning against reading a training metric as a learning metric.

Several claims have been narrowed. The transfer of retrieval practice is the largest: Steven Pan and Timothy Rickard's 2018 meta-analysis found that the benefit of testing transfers only modestly and inconsistently to changed formats, untested material and new inferences, which is more cautious than the book's suggestion that it carries across contexts. Growth mindset has had the hardest decade of any idea the book leans on: a 2018 meta-analysis found the average effect of mindset interventions on achievement close to nil, and the large national study published in Nature in 2019 found a small but real improvement concentrated among lower-achieving students in supportive schools — defensible, but much smaller than the book implies. Deliberate practice has been trimmed the same way, a 2014 meta-analysis across domains putting the share of performance variance explained by practice at about twelve per cent against Ericsson's much larger claim. Dunning and Kruger's effect has been contested on statistical grounds, critics arguing that much of the pattern follows from regression to the mean rather than from a specific incompetence about incompetence; the lesson the book draws from it, that self-assessment needs an external instrument, does not depend on how that argument resolves.

Two things have gone the book's way. The brain-training industry it dismissed was fined two million dollars by the Federal Trade Commission in 2016 for advertising claims it could not support, two years after a group of scientists had issued a consensus statement saying much the same. And learning styles have not recovered: no evidence for the matching claim has appeared, while surveys keep finding that large majorities of teachers and lecturers still believe in it, which makes that chapter more useful now than when it was written.

What remains is the inversion, and it has not been dented. Performance during practice is not learning. The methods that feel productive are the ones that leave least behind. A learner who wants to know whether something has been learned must ask for it back from memory, in the absence of the page, and look at what comes.