A/B Testing Tools

When you run A/B tests, you need to answer three key questions:

  1. How long do I need to run the test?
  2. Is the observed difference statistically significant?
  3. Which treatment wins?

I have used spreadsheet templates from Visual Website Optimizer for this in the past.  Check out their tools and spreadsheets.

Screen Shot 2014-12-19 at 5.16.14 PMYou can plug in numbers and determine whether there is a winner, which treatment wins, and also estimate the required duration to call a winner, based on traffic and traffic routing to A and B.  You can probably use the spreadsheet to extract the math if you want to implement these tests yourself.

If you prefer the formal treatment, here is the statistical aspect of how we calculate winners.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s