Concept
Vanity metrics
Vanity metrics are cumulative numbers, such as total sign-ups, downloads and page views, that rise as long as you keep spending time and money, and therefore cannot tell you whether the last thing you did worked.
Ries's test for a vanity metric is simple: could this number go down because of a decision we made? Total registered users cannot; it only grows. So it measures effort and time, not the product, and a dashboard full of such numbers makes a failing company feel like a growing one. The opposite is an actionable metric, which ties a specific change to a specific difference in what customers do, and the tool that makes a metric actionable is the cohort: compare what the users who arrived this week did with what last week's users did before the change. Ries's own company, IMVU, watched total registrations climb for months while every new cohort converted at exactly the same rate as the one before; the improvements were changing nothing, and only the cohort view could show it. Vanity metrics are dangerous less because they lie than because they flatter.
A real-life example
A Turkish language-learning app celebrated 200,000 downloads at its first anniversary. When a new hire built a cohort chart, it showed that of every hundred people who installed the app in a given week, four were still using it a month later, and that number had not moved through eleven releases. The downloads were real; the product had not improved once.
How to use it
- 1Run every number on your dashboard through one question: could it fall because of a decision we made? If not, retire it from the dashboard.
- 2Replace totals with cohorts: what this week's new users did, next to what last week's did before the change.
- 3Judge marketing by customers acquired and kept, never by traffic or impressions.