Concept
Vanity metrics
Vanity metrics are cumulative numbers, such as total sign-ups, downloads and page views, that rise as long as you keep spending time and money, and therefore cannot tell you whether the last thing you did worked.
A simple test follows from Ries's distinction: could this number go down because of a decision we made? Total registered users cannot; it only grows. So it measures effort and time, not the product, and a dashboard full of such numbers makes a failing company feel like a growing one. The opposite is an actionable metric, which ties a specific change to a specific difference in what customers do, and the tool that makes a metric actionable is the cohort: compare what the users who arrived this week did with what last week's users did before the change. Ries's own company, IMVU, watched total registrations climb for months while every new cohort converted at roughly the same rate as the one before; the improvements were changing nothing, and only the cohort view could show it. Vanity metrics are dangerous less because they lie than because they flatter.
A real-life example
A Turkish food-delivery startup celebrated 200,000 registered users at its first anniversary. When a new hire built a cohort chart, it showed that of every hundred people who ordered for the first time in a given week, nine ordered again within a month, and that number had not moved through eleven releases. The registrations were real; the product had not improved once.
How to use it
- 1Run every number on your dashboard through one question: could it fall because of a decision we made? If not, retire it from the dashboard.
- 2Replace totals with cohorts: what this week's new users did, next to what last week's did before the change.
- 3Judge marketing by customers acquired and kept, never by traffic or impressions.