When you run A/B tests, you need to answer three key questions:

- How long do I need to run the test?
- Is the observed difference statistically significant?
- Which treatment wins?

I have used spreadsheet templates from Visual Website Optimizer for this in the past. Check out their tools and spreadsheets.

You can plug in numbers and determine whether there is a winner, which treatment wins, and also estimate the required duration to call a winner, based on traffic and traffic routing to A and B. You can probably use the spreadsheet to extract the math if you want to implement these tests yourself.

If you prefer the formal treatment, here is the statistical aspect of how we calculate winners.

Advertisements