I read a tweet recently by @tclaiborne about a great blog post on the subject of Easy Statistics For Adwords AB Testing And Hamsters. With a title like that, how could I not take a peek?
A Simple Test to Introduce The Tool
Let's say we've run a test using Google Website Optimiser. We made a change to a page to increase the number of outcomes. We have 6 days worth of data. Here are the conversion rations for the 6 days for the original and the test variation:
So, from 5 days worth of data, can we see if the difference in the conversion rates are significant? It's a small data set...the numbers seem to be different but as the blog post referred to earlier says, we humans are really bad at looking at data sets and making accurate judgements.
We need some stats. Enter jsstat.
So, we can drop in our two samples of data as comma separated values. They don't have to be the same size or integer values. Let's hit that 'oh-so-tempting' import button to see what wonders we can find:
Ah, such insights, knowledge and power are ours! We can deliver meaning and value to our clients! Ahem, enough whimsy - what the heck does this mean?
I'm going to keep this high level:
- The differences could have happened by chance.
- The green text tells us the truth
- The results are conclusive.
- The new page converts 1.5% better than the old on average
Try the test yourself using '1,2,3,4,5,6,7,8,9' as both data sets. NOT SIGNIFICANT!
Moving swiftly on
It's a prototype okay? It might not work in crufty old browsers. Stick with a later version Chrome or FireFox to be safe. The graphing is adding little value right now but box plots are coming!
I'd really like a direct export from Google analytics or Website Optimiser into something like this...Hmmm.
Now, the new Google Analytics Intelligence functionality is very similar to this. It's great, don't get me wrong! Different in some ways but based on the same theory...mostly.
We are looking at taking this tool a lot further to supplement Multi-Variate testing results analysis and click stream data analysis.
I'll keep you posted.