Concept
Keeping score
Keeping score means writing every forecast down as a dated number and marking it later against what happened. Without the mark, memory turns every prediction into 'I basically called it'; with it, you discover which way your numbers run and adjust.
Tetlock's tournament measured every forecast with the Brier score, which punishes both confidence in wrong answers and timidity in right ones, and the forecasters who improved most were those who wanted to be measured. The mathematics is optional; the habit is not. A written, dated probability is the only thing that can later be compared with reality, and comparison is the only route to calibration. Annie Duke adds the social half: a truth-seeking group of two or three people who reward accuracy over confidence and ask 'how sure are you and why?' turns keeping score from a private embarrassment into a shared practice.
A real-life example
A manager sends his monthly probability and reasoning to two friends who ask him only what would have to happen for it to be 40. After a year of entries he sees that his work forecasts are honest and his family forecasts run ten points kind.
How to use it
- 1Write the number and the date before the outcome is known.
- 2Mark it against reality on the date you set, and note the direction you were off.
- 3Find two people who will ask 'how sure and why?' instead of 'great call'.