Superforecasting
Philip E. Tetlock & Dan Gardner
Superforecasting reports what Philip Tetlock learned from a US intelligence forecasting tournament in which ordinary volunteers, the best two percent of them, beat professional analysts with classified information. The superforecasters were not geniuses; they had habits. They started from the outside view, broke hard questions into parts, stated probabilities as numbers, updated often and in small increments, kept score against reality, and treated their beliefs as hypotheses to be tested rather than treasures to be guarded. The book's claim is that these habits are learnable.
Pundits who are confidently wrong keep their jobs. A group of amateurs who said 63 percent instead of 'probably', and checked, beat the professionals. This is how they did it.
The book in essence
Tetlock's earlier research had found that expert political forecasts were barely better than chance, and that the more famous the expert, the worse the forecast. The Good Judgment Project, funded by the intelligence community's research arm from 2011, tested whether anyone could do better. The top two percent could, consistently, and the book anatomises them. They think in probabilities and are comfortable saying 63 rather than 'likely'. They begin with the base rate and only then adjust for the case. They decompose questions the way Fermi estimated piano tuners. They update frequently and by small amounts, avoiding both stubbornness and overreaction. They are measured by the Brier score and want to be. Above all they are, in Tetlock's phrase, in perpetual beta: they treat being wrong as information.
Co-written with the journalist Dan Gardner and published in 2015, it is both a report on the research and a manual, with the Ten Commandments for aspiring superforecasters as an appendix.
Who it's for
- Anyone who has to predict, plan or budget and wants to be less wrong
- Managers who reward confidence in meetings and suspect they should not
- Readers of Kahneman who want to know what the calibrated few actually do
Who it's not for
- — Readers who want certainty; the book's whole point is that the honest answer is a number below one hundred
- — Anyone looking for a quick read; the research is laid out in full
Key lessons
01
Start from the outside view
Before the story of this case, ask what class of thing it is and how often that class turns out one way or the other. Anchor on the base rate, then adjust, less than instinct wants.
A baker deciding on a second branch asks four owners with second branches and learns two closed within two years. Her forecast drops below her excitement.
02
Say a number, not 'probably'
'Likely' means 70 to one person and 95 to another. A number cannot hide, can be checked, and lets you be wrong by a measurable amount rather than embarrassed in the abstract.
A manager writes 55 percent that a new branch breaks even in year one, dated. Nine months later the number can be marked against reality.
03
Update often, in small steps
Superforecasters changed their minds far more often than others, and each change was small: 65 to 68, not 65 to 90. New evidence is weighed against everything known, not swapped in for it.
After his manager cancels two meetings, a new hire moves his probability from 65 to 55 and writes why, rather than to 30.
04
Keep score
Without a written, dated forecast marked against what happened, memory turns every prediction into 'I called it'. The score is what makes improvement possible.
A team that logs its delivery estimates finds its 90 percent confident dates come in on time 60 percent of the time, and starts adding buffers.
Try this today
Take one prediction you are making this week, write it as a number with a date, and note the base rate for cases like it. Put a reminder in your calendar to mark it against what happened.
Quick check
What most distinguished superforecasters?
What is the outside view?
Selected quotes
“For superforecasters, beliefs are hypotheses to be tested, not treasures to be guarded.”
“Unpack the question into components. Distinguish as sharply as you can between the known and unknown and leave no assumptions unscrutinized.”