Test Results Need To Normalize: Be Patient My Little Grasshopper
21 Oct
Estimated Read Time: 3 min
What’s In It For You: you might save yourself some embarrassment or from mistaken happiness.
Keep reading if:
- You’re kind of impatient
- You love challenges
- You love winning
- You’ve at least heard of landing page testing (A/B or MVT)
If you run online tests on landing pages, regardless of which tools you use such as Google Website Optimizer, Test&Target, Visual Website Optimizer, etc, you have to let your tests run for more than three or four days REGARDLESS of if the testing tools claim +160% in conversion rate at 99% confidence on the first day.
First, think about what you’re testing. Is a few simple modification to your homepage really going to create a +160% lift in conversion rate? Assuming your homepage is the highest viewed page, are you really seeing a record breaking sales day because of this +160% lift? As exciting as it is to see these too good-to-be-true lift, don’t pop the champagne bottle yet. Just like most things in life, lift doesn’t come easy.
Below is a great example an actual test currently running and why to keep the lid on the bubbly for a little while. Here’s the first three days of reporting for a control/A/B/C landing page test.
Day One:
Day Two:
Day Three:
The reason why you need to be patient and not immediately shut off the losing experiences (cells) is the data needs to normalize. Not getting into technical detail, the testing tools take some time to get the results under control.
Here’s a good example of how tests normalize over time. It’s a bit wild at first, but usually settles down.
What you need to know:
Monitor closely, let the tests ride and normalize. I recommend about at least four or five days.
A few quotes from this post I want to claim!
“Just like most things in life, lift doesn’t come easy”
“Let the tests ride and normalize”
Related Posts:
The posts here are my opinion. I hope you found it helpful or at least fun to read. My passion is in web analytics, A/B testing, MVT, LPO, CRO, etc. All things related to driving more conversions for you buck!
Find out more about me here.
No comments yet