41Superforecasting

Philip E. Tetlock and Dan Gardner, Superforecasting: The Art and Science of Prediction, Crown, 2015

Linked to 33 chapters: supports 25 · contradicts 10 · mechanism 12 · same claim 3 · pairing 6

Tetlock is the researcher behind the joke about the dart-throwing chimpanzee, and he wrote this book partly to take it back. Between 1984 and 2004 he collected some twenty-eight thousand dated, checkable predictions from 284 professionals whose living depended on analysing political and economic events, and the average expert did about as well as random guessing. That was the punch line. The finding underneath it was that the experts fell into two statistically distinguishable groups, one that lost to chance and one that beat it, separated not by credentials, access or ideology but by style of thought. In 2011 the American intelligence community's research arm let him push that further, with a four-year tournament, a scoreboard and nearly five hundred questions about world affairs. Tetlock's volunteers won so decisively that the competing university teams were dropped, and among them were a few dozen amateurs whose accuracy outran professional analysts holding security clearances. The claim the book makes is small and uncomfortable: foresight over months and seasons is real, it is a skill, the skill is mostly a set of habits anyone can practise, and the reason so few people have it is that almost nobody keeps score.

An optimistic skeptic

Bill Flack lives in Kearney, Nebraska, is retired from the Department of Agriculture, and has answered roughly three hundred questions of the kind embassies and banks pay for: whether Russia will annex more Ukrainian territory in the next three months, whether any country will leave the eurozone within a year. Every answer has been dated, recorded and scored by independent observers, and his record is excellent. Tom Friedman also forecasts global events, from the Oval Office and Davos and the op-ed page, and his record cannot be described at all, because nobody has ever kept one. Baseball managers will not sign a player without consulting his statistics; the forecasts governments and voters act on carry none.

The skeptical half of Tetlock's position comes from the limits of prediction. On 17 December 2010 police in Sidi Bouzid confiscated the scales of a Tunisian fruit seller named Mohamed Bouazizi and slapped him; he set himself on fire, and within weeks a twenty-three-year dictatorship had ended and the Arab world was in revolt. The ingredients — poverty, unemployment, corruption, repression — had been in place for decades and had exploded on no previous morning. Edward Lorenz's 1972 paper asking whether a butterfly's wings in Brazil could set off a tornado in Texas had already moved science away from Laplace's dream of an intellect that, knowing every position and force, would see the future as clearly as the past. Reality contains clocks and clouds. Tides and eclipses are clockwork; the shape a particular cloud will take cannot be computed even in principle, and neither can the political weather three years out, which is what Tetlock's own data showed, accuracy decaying with the horizon until it reached chimpanzee level between three and five years.

The optimistic half comes from the weather, which is genuinely chaotic and genuinely better forecast than it was thirty years ago, because meteorologists make enormous numbers of forecasts, check them, adjust the models and try again. Forecast, measure, revise. Everywhere else the loop breaks at the first joint, for a reason that is commercial rather than mysterious: the buyers of forecasts do not ask for evidence of accuracy, so none is produced. What lets that stand is the substitution Tetlock calls bait and switch, answering without noticing an easier question than the one in front of you — as Archie Cochrane did in 1956, a man who had spent decades fighting the medical establishment over its refusal to test what it believed, when he accepted a surgeon's verdict of terminal cancer because the question he had really answered was is this the sort of man who would know? The pathologist's report, when it came, found no cancer at all.

Keeping score

Scoring forecasts is harder than collecting them. Steve Ballmer's 2007 dismissal of the iPhone never said what "significant market share" meant; the November 2010 open letter warning Ben Bernanke that quantitative easing risked currency debasement and inflation set no date and hedged on "risk". A forecast without a deadline is not a forecast, and a forecast in elastic words cannot be marked. Probability is the larger obstacle. In March 1951 a National Intelligence Estimate judged that a Soviet attack on Yugoslavia that year should be considered a serious possibility. Sherman Kent, who had built the American practice of intelligence analysis, had meant odds of about 65 to 35. When he asked his own team what they had agreed to, one analyst said 80 to 20, another said 20 to 80, and the rest were scattered between. He proposed a table mapping words to numbers, so that "probable" meant 63 to 87 per cent, and it was never adopted — partly because a number makes a subjective judgment look like a fact, partly on grounds of taste, to which his answer has survived him: "I'd rather be a bookie than a goddamn poet." The deeper reason is the wrong-side-of-maybe fallacy: a seventy-per-cent forecast is called wrong whenever the thing fails to happen, so vagueness is the safe play, since a "fair chance" can be stretched afterwards if it happens and shrunk if it does not.

What breaks the deadlock is volume. One probabilistic forecast cannot be judged; hundreds can. If the things a forecaster calls seventy per cent happen seventy per cent of the time she is calibrated, and calibration plots against a diagonal. But calibration alone is not accuracy, because someone who never leaves the zone between forty and sixty can be perfectly calibrated and useless. The second quality is resolution: the willingness to say ninety when ninety is warranted, and to be right.

Both are caught by a measure Glenn Brier worked out in 1950 for weather forecasts. A Brier score is the average squared distance between what you said and what happened, and it behaves like a golf score: zero is perfect, half a point is what random guessing produces in the aggregate, and claiming certainty and being wrong every time costs the maximum of two. A number alone still says nothing, since 0.2 would disgrace a forecaster in Phoenix and mark a world-class one in Springfield, Missouri. So a score needs benchmarks: the mindless rule of predicting no change, and other forecasters answering identical questions over identical periods. That second requirement is why trawling old newspapers for pundits' predictions settles nothing, and why a tournament is necessary.

Tetlock's own study met those conditions. Its 284 participants averaged forty-three years old and twelve years of relevant experience, and half held doctorates; they answered questions one, five and ten years out, inside and outside their specialities, under a guarantee of anonymity, because even experts well short of celebrity would not risk their reputations for nothing. When the results appeared in 2005 the aggregate matched chance, but the split underneath it did not. One group organised its thinking around a Big Idea — any Big Idea, from environmental scarcity to free markets — and squeezed each new problem into it, reaching for "impossible" and "certain" and saying just wait when events went the other way. The other used whatever tool the problem seemed to want, drew on many sources, spoke in "however" and "on the other hand", and changed their minds. Isaiah Berlin's scrap of Archilochus supplied the names, and the foxes beat the hedgehogs on calibration and resolution both, so they were not merely hedging. Worse for the hedgehogs, their accuracy fell on the subjects they knew best, because more information seen through one lens yields more confidence and no more truth. Fame ran inversely to accuracy too, since producers hunt for hedgehogs, who tell tight, confident stories and make better television.

What the foxes do is aggregation inside one skull, the trick Francis Galton stumbled on at a country fair in 1906 when the averaged guesses of hundreds of villagers put the dressed weight of an ox within a pound of the truth.

The tournament

The event that made the tournament possible was the worst intelligence failure in modern American history. National Intelligence Estimate 2002-16HC, released in October 2002, was the consensus of sixteen agencies employing about twenty thousand analysts, and it stated flatly that Iraq had chemical and biological weapons. Robert Jervis concluded afterwards that the judgment had been sincere and reasonable on the evidence available, and that the damning part was its certainty: there were no red teams and no devil's advocates, and, as the presidential investigation observed, nobody even raised the possibility that Saddam had ended his programmes. Had the estimate said sixty or seventy per cent rather than "slam dunk", the congressional vote might have gone differently. The community had accountability for process — did you consider alternatives, did you look for contrary evidence — but never for accuracy, which is like checking that physicians wash their hands without asking whether the treatment works.

That shock produced the Intelligence Advanced Research Projects Activity, and in the summer of 2010 two of its officials told Tetlock and Barbara Mellers that the recommendation to test analytical methods, which he had predicted would gather dust, would be acted on. Five research teams would compete, submitting forecasts by nine each morning from September 2011 to June 2015 on the same questions at the same time, against an independent control group whose combined judgment they had to beat by twenty per cent in the first year and fifty by the fourth. The questions were pitched where an attentive newspaper reader could not simply know the answer and nobody could be certain of it, and they ran weeks to months rather than years. About three thousand volunteers came through the screening and began forecasting for what Tetlock and Mellers named the Good Judgment Project; over four years nearly five hundred questions produced more than a million judgments.

The project won the first year by sixty per cent and the second by seventy-eight, beat the teams from Michigan and MIT by thirty to seventy per cent, and after two years IARPA stopped funding the competition and kept only the winner. The method was not exotic: take the crowd's average, weight the best forecasters more heavily, then extremize — push a 70 towards 85 and a 30 towards 15, on the reasoning that no member of a crowd holds all the information the crowd collectively holds, and each would be more confident if they did.

Doug Lorch, a retired IBM programmer in Santa Barbara, spent about an hour a day at his dining table with a laptop and the newspaper. In the first year he answered 104 questions, filed roughly a thousand separate forecasts as he revised them, and finished fifth of some 2,800 with a Brier score of 0.22. In the second, on a team of superforecasters, he scored 0.14 and finished first, beat the tournament's own prediction market by forty per cent, and exceeded single-handed the fourth-year target IARPA had set for multimillion-dollar research programmes. Fifty-eight others joined him at the top of that first table, and as a group the superforecasters scored 0.25 against 0.37 for everyone else. The gap widened: by the end they were beating regular forecasters by more than sixty per cent, and seeing three hundred days ahead as accurately as others managed at a hundred.

Tetlock then argues against himself, because a few dozen outliers among three thousand people is what luck produces: ask 2,800 people to call 104 coin tosses and a tail of apparent geniuses appears, different people each time. The test is regression to the mean, slow where skill dominates and fast where chance does, and across the second and third years the superforecasters extended their lead instead of regressing. Individually the year-to-year correlation is about 0.65, a little higher than between the heights of fathers and sons, so roughly thirty per cent of superforecasters drop out of the top two per cent each year and seventy per cent stay.

Supersmart, superquants, supernewsjunkies

Three obvious explanations are tested and all come back qualified. Intelligence and knowledge matter but run out: regular forecasters scored above about seventy per cent of the American population on tests of fluid intelligence and general knowledge, superforecasters above about eighty, so the large jump is from the public to the volunteers, not from the volunteers to the stars. Mathematics matters less than it looks, since explicit models are rare; Lionel Levine, a Cornell mathematician and one of the best of them, set out to prove he could forecast well without using any. Attentiveness is not the answer either, since their very first forecasts, before any updating, were already at least fifty per cent more accurate. What they have instead is an order of operations.

The first step is Fermi's. Enrico Fermi taught his students to estimate how many piano tuners work in Chicago by asking what they would need to know — how many pianos there are, how often a piano is tuned, how long a tuning takes, how many hours a tuner works in a year — and to break each of those down again until the guesses are small enough to inspect. Guessing is not eliminated. It is dragged into daylight, where it can be argued with.

When IARPA asked whether the French or Swiss inquiries would find elevated polonium in Yasser Arafat's exhumed remains, most people felt an immediate answer to a different question — did Israel poison Arafat. Bill Flack broke the real one apart. He established first that polonium could be detected in a body dead eight years, then asked what could have put it there: Israel was one route, Arafat's Palestinian enemies another, contamination of the remains afterwards a third, each extra route raising the probability of a positive test — and only one of the two laboratories had to find anything for the answer to be yes. None of that required knowing much about the Middle East.

The second step is the outside view before the inside one, and the order is not a preference: any number seen first becomes an anchor and we adjust from it too little, so the first number had better mean something. Days after the killings at Charlie Hebdo, asked whether Islamist militants would attack one of eight western European countries before the end of March 2015, the superforecaster David Rogg counted attacks in those countries over five years, got six, and set the base rate at 1.2 a year. Then he came inside: he dropped the years predating the rise of ISIS to lift it to 1.5, weighed new threats against tightened security, and settled on 1.8. Sixty-nine days remained, so sixty-nine over 365 times 1.8 gave 0.34, and he posted thirty-four per cent — then asked his teammates what they thought, because he wanted more lenses.

The third step is to say the number finely. Ordinary forecasters cluster on the tens; a third of superforecasters' judgments used single percentage points, and they will argue about whether something is five per cent or one, which matters when the difference is between an improbable thing and an impossible one. Mellers tested whether the precision was real by rounding everyone's forecasts and recalculating: superforecasters lost accuracy from rounding to the nearest five points, while regular forecasters barely lost anything from rounding four times as coarsely. The granularity was carrying information. Its opposite is a mental dial with three settings — yes, no, maybe — which Tetlock finds in the White House meeting where estimates that the tall man in the Abbottabad compound was Osama bin Laden ran from thirty to ninety-five per cent and the president closed the discussion by calling it a flip of a coin. It was not; the crowd's median was around seventy. Superforecasters also score lowest of any group tested for belief that events happen for a reason, and the more a forecaster inclines towards fate, the worse the Brier score.

The fourth step is updating, where most forecasts are won and lost. When the Swiss team said it needed more unspecified testing on Arafat's remains, Flack read the delay as a sign that they had found polonium and were checking whether decaying lead had produced it, and moved from sixty to sixty-five per cent; the tournament's prediction market closed the question at 4.27 per cent. Polonium was found, and Flack's score of 0.36 was five times better than the market's.

The two failures are symmetrical. Underreaction is usually commitment: Flack thought a visit to the Yasukuni Shrine made so little sense for Shinzo Abe that he discounted a leak saying Abe would go, and afterwards diagnosed himself exactly — the question he had answered was whether he would visit the shrine if he were prime minister. Beliefs sit in a tower of Jenga blocks, and those holding up your sense of who you are do not come out at all. Overreaction is the absence of commitment: Lorch swung from fifty-five to ninety-five per cent on an Arctic sea ice question on one month-old report, and the ice stopped melting.

Minto, a software engineer in Vancouver who won the third season with a score of 0.15, forms a first judgment in minutes, returns the next day for a second opinion, and trawls for contrary evidence five days a week; on one question he filed seventy-seven forecasts. What saves that from overreaction is the size of the steps: a forecaster who ignores new information loses its value, while one captured by it loses everything he knew before. Minto knows Bayes's theorem, has never used it in a forecast, and could not write it out from memory; most superforecasters are Bayesians who do not use Bayes's theorem. The exception is Lorch's: when the report a forecast rests on is visibly failing, throw it out and start again rather than retreating by degrees.

Perpetual beta

Mary Simpson has a doctorate in economics and did not grasp the financial crisis until Lehman Brothers filed for bankruptcy on 15 September 2008. The failure is why she volunteered. Tetlock's name for the disposition behind that response is Carol Dweck's growth mindset, and his exemplar is Keynes, who was nearly wiped out on currency in 1920, lost heavily again in 1929, converted each disaster into a revision of method, and took pride in being caught out. What Keynes could not do is transfer the skill by describing it, any more than the physics of a bicycle teaches anyone to ride.

Practice without feedback teaches nothing either: police officers who interrogate suspects for decades grow more confident at detecting lies without growing more accurate, because they rarely learn who was lying. Elastic language blocks feedback by stretching to fit any outcome, and delay blocks it because hindsight closes over the gap. In 1988 Tetlock asked experts how likely it was that the Communist Party would lose its monopoly on power within five years; asked in 1992 to recall their own answers, they remembered numbers on average thirty-one percentage points higher than the ones they gave. A dated number defeats both, which is why the cycle of trying, failing, analysing and adjusting can run at all — and because it runs slowly and in noise, it also needs what Angela Duckworth calls grit. Programmers have a name for software that is never finished but used, analysed and improved without end, and Tetlock takes it for the whole composite: superforecasters are perpetual beta. Commitment to belief updating and self-improvement is the strongest predictor the project found of who rises into their ranks, about three times as strong as intelligence.

Other people matter as much, and they cut both ways: Irving Janis, later one of Tetlock's doctoral advisers, coined groupthink for what happened at the Bay of Pigs, where nobody in Washington reconsidered a plan even after the New York Times printed the secret training base on its front page. So the tournament tested teams rather than assuming them. Several hundred forecasters were randomly assigned to work alone and several hundred in teams, warned about groupthink and about its opposite, and taught Andy Grove's constructive confrontation and precision questioning — dissecting a vague claim by asking what exactly is meant, on what evidence, over what period. Teams were twenty-three per cent more accurate than individuals, and in the second year, when the superforecasters were grouped into teams of a dozen, effort rose instead of being divided away: being named a superforecaster and placed on a superteam made a person fifty per cent more accurate, and the result replicated the following year. Tetlock's sharper observation is that a team's open-mindedness is not the average of its members' but a property of how they talk to each other.

The last objection is that none of this describes a leader, who is expected to be confident, decisive and visionary, while the superteams were, in Tetlock's phrase, little anarchist cells. His answer is Helmuth von Moltke's Auftragstaktik — for a commander who held that everything in war is uncertain, the way to be decisive is to tell subordinates the goal and not the method, which is why the fort at Eben Emael still fell on 10 May 1940 after the glider carrying the officer commanding the assault was forced down a hundred kilometres away. As for humility, Annie Duke supplies the distinction: humility in the face of the game is not humility in the face of your opponents. Good judgment needs only the first kind, and it sits perfectly well beside a large opinion of one's own abilities.

The guidelines Tetlock prints at the back of the book are the tournament's training module, the hour of reading that raised accuracy by roughly ten per cent for a whole year.

Are they really so super?

In the summer of 2014 Daniel Kahneman put the question to Tetlock directly: are superforecasters different kinds of people, or people who do different kinds of things? Tetlock answers that it is a bit of both and that the second matters more. Kahneman's standing worry is that correcting fast intuition by slow reasoning is exhausting and cannot be kept up, and that even accomplished people slip — as the retiring head of the Defense Intelligence Agency did in 2014 when he called the international situation unprecedented in its chaos, a claim that is checkable and false, since interstate wars have been declining since the 1950s and civil wars since the early 1990s.

So Kahneman and Mellers tested the superforecasters, using the bias Kahneman had documented thirty years earlier. Toronto residents would pay about ten dollars to clean up the lakes of a small Ontario region and about ten dollars to clean up all 250,000 lakes in the province; two thousand drowning migratory birds and two hundred thousand of them both drew about eighty. People price the image, not the quantity, and in forecasting the quantity is the time frame. One random half of the forecasters was asked the probability that Assad's regime would fall within three months and the other half within six, and nobody saw both. Regular forecasters said forty per cent and forty-one — flat, exactly as Kahneman predicted. The superforecasters said fifteen and twenty-four, which is not perfect sensitivity to scope but was enough to surprise him. Tetlock's explanation is habituation, a correction practised often enough to stop being deliberate; he notes that the superforecasters were raising scope problems in their forums before the researchers thought to study them.

The harder objection is Nassim Taleb's, and Tetlock states it at full strength. If history is made by events nobody could have conceived of beforehand, then a tournament of three-month questions is a fool's errand — "What matters can't be forecast and what can be forecast doesn't matter" — and worse than useless, because it teaches complacency. Tetlock replies four times and concedes the second. On the strict definition most nominated black swans are grey: a plot to crash an airliner into the Eiffel Tower was broken up in 1994, and in August 2001 a Harvard terrorism scholar told an official that the tactic was under consideration and would be used sooner rather than later. But on the looser definition — a highly improbable event with large consequences — the criticism holds entirely. If "highly improbable" means one in a hundred or one in a million, accumulating enough resolved questions to score anyone would take decades or millennia, so the tournament tells us nothing whatever about how good superforecasters are at spotting rare catastrophes; they might be as clueless as anybody, and we should not pretend to know. Third, history crawls as well as jumps, and one per cent average annual global growth in the nineteenth century and two in the twentieth is what turned eighteenth-century squalor into modern wealth. Fourth, what makes a shock a black swan is its consequences, and consequences unfold over months in tournament-shaped questions: whether the Taliban would hand over bin Laden, whether he would flee before the invasion.

Where all of them agree is the horizon. Accuracy decayed towards chance five years out in Tetlock's own data, and there is no evidence anyone can usefully forecast a decade ahead, which does not stop Congress requiring the Pentagon to produce a twenty-year outlook every four years. Beyond the horizon the sensible response is resilience rather than prediction — which Tetlock accepts while pointing out that resilience is expensive, that priorities must be set, and that setting priorities is forecasting again. His last concession is the shape of the distribution: human height is bell-shaped and wealth is not, and if catastrophes are distributed like wealth, an official in the summer of 1914 who assumed war deaths clustered around a hundred thousand was wrong in the way that makes the real possibility look like raving.

The closing chapter is about the questions a tournament cannot score. Paul Saffo told Tetlock that asking whether North Korea would launch a multistage rocket in a given window was trivial, and that the question worth asking is how all this turns out; his own long answer to it remains too vague to mark today. Tetlock's way through is clustering — the big question is made of many small ones whose answers accumulate like the dots of a pointillist canvas until a pattern appears. He also concedes that asking the right question is a separate talent, probably belonging to different people: Tom Friedman's September 2002 column asked whether Iraq was the way it was because Saddam was the way he was, or the reverse, which is the question that governed what followed in 2003, and he asked it while supporting an invasion he expected to go otherwise. The superquestioner pitches; the superforecaster bats.

Since 2015

The machinery outlived the tournament. Good Judgment Inc. was spun out of the project and sells forecasts commercially while running a public tournament anyone can enter, so the superforecaster label is now an accredited thing rather than a research artefact. The two findings that have held up best are the ones the book leans on hardest: that individual accuracy is consistent enough across years to be a skill rather than a run of luck, and that an hour with the training guidelines buys a measurable improvement lasting a season. Some of the supporting psychology has fared worse. Carol Dweck's mindset research and Angela Duckworth's grit both stood near the centre of the replication reckoning that swept social psychology: large pre-registered trials have found mindset effects on achievement that are real but small, and meta-analysis has found grit to be largely a relabelling of conscientiousness. None of that touches the tournament's own measurements, which were made on the forecasts themselves; it does weaken the chapter that explains them by reaching for traits.

The world then supplied the tests Taleb had in mind: the Brexit referendum in 2016, the American election that November, the pandemic of 2020. Where scores were kept, aggregated crowds went on beating individual commentators, which is the modest claim Tetlock actually makes; where they were not kept, the loudest forecasters paid no price, exactly as before. The signatories of the 2010 letter to Bernanke had already told Bloomberg in October 2014 that they would not change a word.

Tetlock's own work since has gone where the book says it must, to the questions too large and too slow to mark. The Forecasting Research Institute, which he helped found, ran an Existential Risk Persuasion Tournament in 2022, putting superforecasters and domain specialists on the same catastrophic-risk questions out to 2030, 2050 and 2100, with months of structured argument and money at stake. The forecasters gave far lower probabilities of catastrophe than the specialists, and the two groups finished about as far apart as they began — a discouraging result for the ambition the book ends on, which was to use tournaments to depolarise arguments that have gone tribal, and also the honest version of it, reached by the method the book recommends: state the disagreement precisely, attach a number and a date, and wait. What has lasted is not the superforecasters but the instrument — a question with an unambiguous resolution date, an answer expressed as a number, a score computed afterwards, and a benchmark to compare it against. Tetlock's closing line is a demand rather than a prophecy. All we have to do is get serious about keeping score.