Concept
Law of Small Numbers
The law of small numbers is Tversky and Kahneman's ironic name for the belief that a small sample will look like the population it comes from, which leads people to read real causes into results that small samples produce by chance.
The genuine law of large numbers says that large samples reliably resemble their population. Amos Tversky and Daniel Kahneman showed that people, trained researchers included, behave as if the same held for small ones. It does not: the fewer the cases, the more extreme the results can be, in both directions, by chance alone. Kahneman's best-known illustration is a map of kidney cancer rates across US counties. The counties with the lowest rates were mostly rural and sparsely populated, and it is tempting to credit clean air and simple living. But the counties with the highest rates were also mostly rural and sparsely populated. Small populations produce both extremes, and no cause is needed. The same logic undid a popular reading of school results: the smallest schools appeared among the very best, which encouraged investment in small schools, yet small schools were also overrepresented among the very worst. System 1 is built to find causes and stories, and it pays little attention to sample size, which is a statistical fact rather than an event. So a striking result from ten customers, one branch or a single week feels as solid as one from ten thousand. Before explaining a result, ask how many cases it rests on.
A real-life example
Oğuz sells handmade copper pots online from Kayseri. A week after he changed the photo of one pot, its sales went from four to nine, and he booked a photographer to reshoot the whole catalogue. Then he asked how many cases the jump rested on: nine orders. He cancelled the booking, kept the new photo for a month, and the difference shrank to almost nothing.
How to use it
- 1Before explaining any surprising result, write down how many cases it rests on.
- 2Treat extreme results from small groups, such as one branch, one week or a handful of customers, as noise until a larger sample repeats them.
- 3When you rank groups of very different sizes, expect the smallest ones at both the top and the bottom of the list.