Concept
Noise Audit
A noise audit is a test in which several professionals judge the same realistic cases independently, so that an organisation can measure how much their judgments differ without needing to know the right answers.
The noise audit is the main diagnostic in Noise by Daniel Kahneman, Olivier Sibony and Cass R. Sunstein. The method is simple: choose a judgment the organisation makes repeatedly, prepare a few realistic cases, ask several of the people who normally make that judgment to assess them alone, and compare the answers. The book describes audits at an insurance company where executives expected a modest difference between two underwriters' quotes for the same case and found a gap several times larger. A practical measure is the difference between two randomly chosen judgments of the same case as a share of their average. An audit stays honest when the cases are realistic, the judgments are strictly independent, and the purpose is announced as improving the system rather than ranking individuals.
A real-life example
The head of procurement at a hospital group in İzmir removes supplier names and old scores from six past bids and asks five buyers to score them alone. On one bid the scores run from 48 to 81 out of 100. She shares the results without names, and the team agrees what each score should mean.
How to use it
- 1Choose three to six real past cases from one recurring judgment and remove the outcomes and earlier answers.
- 2Ask several colleagues to judge them alone, with no conferring, and collect every answer before anyone sees the others.
- 3Report the spread without names and discuss what would make the judgments converge.