Estimated read time: 6 min
What’s in it for you: learn the factors that drive how quickly you can run new online tests.
You’re going to run A/B or MVT tests. You got lots of great ideas – hero images, offerings, value propositions, etc. What are the factors that determine how quickly you can run new online tests? When you test one landing page against another, what factors are involved in making sure the results are sufficient enough to scrap the loser and run a new challenger?
Here it is:
The number of visitors you are getting.
The higher the traffic to your site, the faster you’ll be able reach statistical significance. If amazon.com sees a 10% increase in overall visits one day, that’s huge! If they get 1,000,000 visits in one day, a 10% increase is 100,000 more visits. Let’s say I see a 10% increase in visits to my site. If I only get 10 visits to my site a day, I can’t really say that 11 visits on a random day is significant. That 1 visit over average could be totally random versus that 100,000 visits to amazon.com is very unlikely to be some random day. Something had to have happened to drive that change.
2. Conversion Rate
A conversion is whatever you decide is a key metric of success on your site. This can be a sale, order , a lead, bounce rate, a sign up, etc. A conversion rate is the total conversion divided by total number of users or visits to your site. If you have 10 conversions for 100 visits in a day, your conversion rate is 10%.
The higher the conversion rate, the faster you’ll reach significance. Why? If you only convert 1 user for every 10,000 visits to your site, versus 1 user for every 10, you’re going to require a whole heck of a lot less traffic to see whether your tests are really effective or not. Hence, higher the conversion rate, the faster you’ll reach significance.
3. Percent Difference in Conversion Rate
Huh? The bigger the difference in conversion rate from Landing Page A vs Landing Page B, the more confident you will be that one works better than the other. It’s pretty straight forward.
If you are selling lemonade at your lemonade stand and one day you sell 150 lemonades versus your average daily 100, you’re much more confident that something significant happened compared to a day when you sell 102.
4. Number of Tests
Simply the more tests you run at the same time, the less traffic you are running to each test. So based off the factors I mentioned above, this makes common sense.
5. Confidence Level
It’s the level of certainty that a statistical prediction is accurate.
Basically, it’s how confident you want to be in your test results. 95% is a popular confidence level. At a 95% confidence level, you can be 95% certain that the user base would pick the test results you ran.
I’m not going to go into detail here. Google “Confidence Level and Confidence Interval” and you’ll find lots of information on the topic.
The X factor.
This is something that cannot be ‘measured’ or overlooked. The great thing about testing and using analytics is that it takes out a lot of the ‘gut feeling’ BS. A danger, however, in my opinion is that people can go overboard with just relying on data.
Let’s say you run tests Page A vs Page B. After the tests go live, the conversion rates are pretty similar and you haven’t reached statistical significance quite yet. However, you start looking at Page B and it’s just not jiving with the brand you want to portray. Shut it down. Don’t prolong the tests if you believe the new test page you are running just isn’t right. This is assuming.. Page B isn’t getting killed to Page A.
Below are charts that boil the above down.
Most testing platforms have reporting interfaces that tell you the conversion rates and confidence level of the results. Some upper level platforms offer estimated duration of tests based on initial results.
Regardless, it is important to know the 5 factors and the X factor mentioned in this post if you want to get serious about testing. It will give you a good guideline for how you will approach testing.