Concept
Keeping score
Keeping score means writing every forecast down as a dated number and marking it later against what happened. Without the mark, memory turns every prediction into 'I basically called it'; with it, you discover which way your numbers run and adjust.
Tetlock's tournament measured every forecast with the Brier score, which punishes both confidence in wrong answers and timidity in right ones. The mathematics is optional; the habit is not. A written, dated probability is the only thing that can later be compared with reality, and comparison is the only route to calibration. Annie Duke adds the social half: a truth-seeking group of two or three people who reward accuracy over confidence and ask 'how sure are you and why?' turns keeping score from a private embarrassment into a shared practice.
A real-life example
Every Monday, a buyer for a small chain of shops writes a dated probability that each new product will sell out within a month. Six months later she marks forty forecasts against what happened. Her 80 percents came true about half the time, mostly on products she liked herself. She now writes the number before she sees the samples.
How to use it
- 1Write the number and the date before the outcome is known.
- 2Mark it against reality on the date you set, and note the direction you were off.
- 3Find two people who will ask 'how sure and why?' instead of 'great call'.