Concept
System Noise
System noise is the unwanted variability in judgments that should be identical, such as different sentences, prices or diagnoses for the same case, depending on who makes the judgment and when.
In Noise, Daniel Kahneman, Olivier Sibony and Cass R. Sunstein separate two kinds of judgment error. Bias pushes judgments in one direction; noise scatters them. System noise has two main parts: level noise, when some judges are harsher or more generous across the board, and pattern noise, when judges react differently to particular cases. Pattern noise includes occasion noise, the same person judging the same case differently on another day. In the authors' error equation, overall error is bias squared plus noise squared, so noise counts as much as bias. Noise is easy to overlook because each case is usually judged once, and it can be measured without knowing the right answer: if several judgments of one case differ, they cannot all be correct.
A real-life example
At an insurer in Ankara, a water-damage claim is priced by two adjusters by mistake, one at 18,000 lira and one at 31,000. The claims manager has four adjusters re-price ten closed files without seeing the original figures, and finds wide gaps on almost every file. The team had never compared its judgments before.
How to use it
- 1Pick a judgment your organisation makes often and ask whether the outcome would change if a different colleague had handled the case.
- 2Separate the question of bias (are we off in one direction?) from the question of noise (are we scattered?).
- 3Measure noise by having several people judge the same cases independently and comparing the spread.