BookYol

Concept

Brier Score

The Brier score measures how good your probability forecasts are by averaging the squared gap between the probability you gave and what actually happened, so 0 is perfect and lower is always better.

A single event can't prove a "70% chance" right or wrong, but a set of forecasts can be scored. For each prediction, write the outcome as 1 if it happened and 0 if it didn't. Subtract that from your probability and square the result. Average those squares across many predictions, and that average is your Brier score. Squaring punishes confident misses hard. A 90% call that fails costs 0.81, while a 60% call that fails costs 0.36. Someone who says 50% on everything always scores 0.25, so that is the bar to beat. One number captures two skills. The first is calibration: your 70% claims come true about seven times in ten. The second is boldness: you move away from 50% when you genuinely know something. Philip Tetlock used this score in his forecasting tournaments to find superforecasters. His version adds both sides of the outcome, so it runs from 0 to 2.

A real-life example

Leyla, a product manager, predicts whether each sprint will ship on time. Over ten sprints she says 90% every time, and six ship. Each hit costs 0.01 and each miss costs 0.81, so her Brier score is 0.33. That is worse than someone who always says 50%. Next quarter she gives 85% to clean sprints and 45% to sprints with open dependencies. Her score falls to 0.20.

How to use it

  1. 1Log every forecast as a dated number, such as "70% the client signs by Friday", not "probably."
  2. 2When the forecast resolves, record 1 or 0 and compute (probability − outcome)². One spreadsheet column does this for you.
  3. 3After 20 or more forecasts, compare your average to 0.25. Then check each confidence band separately to see whether you run overconfident or underconfident.