Statistical significance in A/B testing - a little tool to help

Rationale

I read a tweet recently by @tclaiborne about a great blog post on the subject of Easy Statistics For Adwords AB Testing And Hamsters. With a title like that, how could I not take a peek?

It so happened that I was working on a small project to build a tool in Javascript to enable easy analysis of two data sets to compare them for statisticaly significant differences, specifically in the context of A/B and MV Testing.

This post is introducing the prototype of that tool. Just to be clear, this tool is a mash up of Javascript snippets that have been published. I didn't write the whole thing so I'm not taking credit here - I'm just looking to share a cool tool!

A Simple Test to Introduce The Tool

Let's say we've run a test using Google Website Optimiser. We made a change to a page to increase the number of outcomes. We have 6 days worth of data. Here are the conversion rations for the 6 days for the original and the test variation:

Test	1	2	3	4	5	6
Original	6%	6%	5%	6%	7%	6%
Test Page	9%	6%	7%	6%	9%	8%

So, from 5 days worth of data, can we see if the difference in the conversion rates are significant? It's a small data set...the numbers seem to be different but as the blog post referred to earlier says, we humans are really bad at looking at data sets and making accurate judgements.

We need some stats. Enter jsstat.



So, we can drop in our two samples of data as comma separated values. They don't have to be the same size or integer values. Let's hit that 'oh-so-tempting' import button to see what wonders we can find:



Ah, such insights, knowledge and power are ours! We can deliver meaning and value to our clients! Ahem, enough whimsy - what the heck does this mean?

I'm going to keep this high level:

• The differences could have happened by chance.


• The green text tells us the truth


• The results are conclusive.


• The new page converts 1.5% better than the old on average

Try the test yourself using '1,2,3,4,5,6,7,8,9' as both data sets. NOT SIGNIFICANT!

Moving swiftly on

It's a prototype okay? It might not work in crufty old browsers. Stick with a later version Chrome or FireFox to be safe. The graphing is adding little value right now but box plots are coming!

I'd really like a direct export from Google analytics or Website Optimiser into something like this...Hmmm.

Now, the new Google Analytics Intelligence functionality is very similar to this. It's great, don't get me wrong! Different in some ways but based on the same theory...mostly.

We are looking at taking this tool a lot further to supplement Multi-Variate testing results analysis and click stream data analysis.

I'll keep you posted.