Concept
Split Testing
Split testing is running two versions of something during the same period for customers assigned at random, so any systematic difference in their behaviour can be traced to the one thing you changed.
When you change something and a number rises, you rarely know why. A holiday week, a mention on a podcast or a different mix of visitors can move behaviour as much as your change did. Comparing this month to last month mixes all of that together. Eric Ries's answer in The Lean Startup is the split test. Show version A and version B during the same period, and let chance decide who sees which. Both groups live through the same week, the same weather and the same news. That leaves your change as the only systematic difference between them. Randomness is the point. Letting customers pick, or giving the new version to your best accounts, quietly brings the bias back. The second half matters just as much. Report the result so plainly that anyone can check it: per customer, with group sizes, before you decide what it means.
A real-life example
A small online bakery wants to know whether free delivery over $40 lifts orders. Instead of switching it on for everyone, the owner uses a coin-flip script to show the offer to half the visitors for three weeks. Without the offer, 1,200 visitors placed 48 orders. With it, 1,180 visitors placed 71 orders. A rainy weekend and a local newspaper mention hit both groups alike. The owner sends the table, with dates, to a friend, who reaches the same answer.
How to use it
- 1Before you start, pick one change and one per-customer metric, such as orders per visitor. Write down what result would make you drop the change.
- 2Split customers by a random rule, never by hand. Alternating sign-ups, a coin-flip script or the last digit of a customer ID all work. Run both versions over the same dates.
- 3Report it in one plain table: group sizes, the metric for each group, and the dates. If someone else can't recompute your conclusion from it, the report isn't finished.