Concept
Calibration
Calibration is the match between your stated probabilities and reality: when you say 70 percent, being right about 70 percent of the time. Most people are overconfident, so their 90 percents come true noticeably less often than nine times in ten, and only a written record reveals it.
Tetlock's superforecasters were distinguished less by knowing more than by knowing how much they knew. A calibrated forecaster's 60 percents come in at 60 and 90 percents at 90; an overconfident one's 90s might come in at 70. Calibration cannot be felt from the inside; it exists only if you write probabilities down, date them and mark them later against what happened. The Brier score does this formally, but the habit matters more than the formula. Once you see how often your 90s really come true, you adjust, and the adjustment is the whole improvement.
A real-life example
A team logs its delivery estimates for a quarter and finds that dates it was 90 percent confident about were met 60 percent of the time. From then on the team quotes ranges and adds a buffer, and the customers stop being surprised.
How to use it
- 1Write every probability as a number with a date.
- 2Mark it later against what happened and note which direction you were off.
- 3After ten or more, look for the pattern and adjust your numbers.