[This is an ongoing series of business issues I hear from clients or users getting into optimization and testing]

There are lots of tools out there ranging from bare bones to bells & whistles and from testing for dummies easy to dance dance revolution for uncoordinated people tough. The most popular enterprise level testing tool, Adobe Test&Target, packs a big punch but is definitely not the easiest testing tool out there. Newer generation tools like Visual Website Optimizer and Optimizely, though not as powerful as a T&T,  makes getting A/B tests up and running seem all too easy.

Besides coming up with the testing strategy, ideas, creative copy and assets, there are two areas that can frustrate users trying to setup a test campaign. These are tagging the pages and the actual setup of the campaign within in the tool or platform.

Tagging -- almost all testing tools require Javascript tagging. They range from one universal tag to many per page. T&T requires an mbox tagged on all ‘areas’ of the page that need to be tested. For example, if you want to test four different areas of the home page, four mbox tags are required plus the standard mbox.js. For a VWO, only one universal tag needs to be placed on the page.

If you are using a tool like T&T and parts of the page where you want to test are not tagged with mboxes, it can take some time for you to get IT resources to tag the page. If you are using a tool like VWO, as long as all pages are tagged with its universal tag as standard protocol, there is no need to get further IT resources to test an area on the page. This is where the potential for the biggest difference in setup time can take place.

If you plan ahead and coordinate efforts with the IT team to get the pages tagged, tagging will not become the huge snag to getting your test up and running.

Campaign setup — here is where I believe there is a big misconception. If pages are tagged and creative assets are done, I can have a simple A/B campaign setup in T&T in minutes. Sounds silly, but the easy step by step setup process of tools like VWO or Optimizely can slow you down – if there is a timed race for campaign setup.

——————————————————————————

Bottom line: if you get comfortable with the testing tool, you can have tests setup and running in no time.

Recommendation: plan ahead by building a testing pipeline so if you need IT resources to test,

Estimated Read Time: 3min
What you will get out this post: get a high-level perspective of what it takes to build a kick-ass optimization culture in your organization.

This blog series is based on my presentation at the Web Analytics Wednesday hosted by Brooks Bell on 3/28/2012.

In my humble opinion, the reason companies keep hitting a brick wall on their optimization initiatives fundamentally comes down to culture. The tools out there are in most part being underutilized – regardless of if it’s the top enterprise testing tool like Test&Target. There are many other great platforms out there. However, if you don’t have the right optimization culture, you are not getting the most out of testing. I believe if optimization teams can embrace the 7 building blocks I mention below, they will find themselves more successful. I will go into detail about each one of the 7 building blocks in future posts and give you the reasons and examples of why I believe in these 7.

For now, here are the 7 building blocks to building a kick-ass optimization culture:

• Believe - everyone involved needs to believe in the power and value testing and optimization can deliver to the bottom line for your organization.
• Killer Process – to scale, be effective and efficient, your optimization program needs to have a great process.
• Different Skills – in order to have a fast moving and effective optimization program, it is extremely valuable to have analysts, marketers, creatives, and IT/developers on the optimization team.
• One Roof – many great ideas, insights, and analysis happen spontaneously outside of formal meeting settings, collaborative software, or over email/IM.
• Disagree – without having disagreements with others on the team over test ideas and strategies, you will end up leaving a lot on the table.
• Internal KPIs – just like all other departments have their KPIs – sales, orders, AOV, etc, your optimization group should have KPIs to track success of all the campaigns you run.
• Live & Breath Testing – it’s a good sign when members make cheesy a/b testing jokes like ‘we should test these different coffee beans’, celebrate and share campaign successes.

I have been fortunate enough to be a part of a company laser focused on just testing & optimization. Over the years, I have witnessed and learned from the highs and lows of what it takes to build a great optimization culture within an organization.  In this digital world, it’s truly a challenging, fun, and humbling space to be in.

Estimated read time: 3 min

What’s in it for you: get a simple definition and analogy of conversion rate optimization #cro

You and your boss go out to a bar.

He hands you a $20 and tells you to go get some beers. Let’s say the beer he likes is Guinness and it’s $5/pint. You order at the bar and get 4 pints of Guinness. You tip a few dollars from your own pocket.

Your boss is expecting 4 pints back.

Let’s say you’re a smooth talker and over time you’ve tested many lines and now you have that special line to get the bartender to hook you up. Wouldn’t your boss be surprised/impressed that you came back with 5 pints instead of 4?

I feel this often resembles the real world of marketing departments. A budget is given for a marketing department and the team is responsible for using that money to hit the projected numbers. What if you took that same budget and delivered more than expected? This is just like taking that $20 and getting 5 pints instead of 4.

Conversion rate optimization is simply just that. Take the same resources given and get more out of it by testing what works better than what you currently have going. Through A/B or multivariate testing on the pages of your site, you can start squeezing more conversions from the same amount of traffic. If you can squeeze an extra 10% of sales from your PPC campaigns, and you can prove your case, how happy will your boss be? He gives you $1MM expecting $2MM in sales and you get him $2.2MM because you did something smart about it.

More pints. More sales. More props.

Conversion rate optimization for the win!

Estimated Read Time: 3 min

What’s In It For You: you might save yourself some embarrassment or from mistaken happiness.

Keep reading if:

• You’re kind of impatient
• You love challenges
• You love winning
• You’ve at least heard of landing page testing (A/B or MVT)

Estimated read time: 4min

Value to you: get your mind thinking about ways to analyze and present a conversion funnel test.

Before reading this post, if you haven’t read the “Definition: Conversion Funnel” post, please read that first. It will only take a minute and the beer bong picture is worth the laugh.

If you’re an analyst at an organization, you will have to tell and sell the story to the decision makers of what is happening in the conversion funnel.

How to create funnels: I’ll write a post on this.. Once you know where to set them up, it’s a very easy process.

Below are examples of funnel reports from SiteCatalyst and Google Analytics:

Using this report, you can see where the big drop off steps are and which is important.

—————

In this post, I want to show how to effectively compare two paths being tested against one another and tell the story.

Recently a test was run where the Brooks Bell team created a test path against a control. A testing tool was used to split the traffic so UserA will see the control path and UserB will see the new path. It was a four page registration process.

Below are some numbers from the funnel report. When comparing two funnel paths against another, I like to put the numbers into excel and make my own comparison charts.

—————

This first chart shows control vs test where the drop offs are happening. At each step in the funnel, it shows what percentage of the initial traffic made it through to the different steps.

—————

The below chart shows the drop off percentage between each step, which is similar to the standard funnel reports in the web analytics tools.

—————

The below chart is my favorite comparing the two paths. This is the one that stands out and really gets people’s attention.

I prefer to use all three charts in sequence so it starts out more or less as a “so what?” thought on the first slide for people reviewing and builds up to a crystal clear takeaway.

Now you need to do the most important step of driving actionable solutions to what you just presented.

Takeaway: don’t settle for the standard reports. Use the data and see if you can slice it up in ways that packs more impact especially from a visual perspective. Start at the basic “so what?” slide and finish on the “ooohh.. I see” slide!

]]> http://www.naoshiyamauchi.com/testing/testing-the-conversion-funnel-and-telling-the-story/feed/ 0 http://www.naoshiyamauchi.com/tracking/definition-online-conversion-funnel/?utm_source=rss&utm_medium=rss&utm_campaign=definition-online-conversion-funnel http://www.naoshiyamauchi.com/tracking/definition-online-conversion-funnel/#comments Fri, 03 Jun 2011 02:00:34 +0000 Naoshi http://www.naoshiyamauchi.com/?p=1303

Estimated read time: 1 min

What is in it for you: Get a simple definition and analogy for an online conversion funnel. Beer bong funnel wha?

What is a conversion funnel?

My basic web marketing definition: The main steps a user goes through on your site to get to the end conversion goal. An example is a registration step process. The first step/page might be filling in name, second is filling bio, third is payment info, and fourth and final page is where you fill out your current address before hitting ‘register now’ button. So in this case, the ‘registration’ is the defined conversion and the funnel is the four pages before the user must go through to register. The funnel is the starting point on the first page all the way down to the end conversion. Along the funnel, certain percentage of users drop off, hence you may have started with 100 visitors to the first page but only 10 people finished registration. Depending on the web analytics tool, the actual name might be different than using “funnel.” For example, SiteCatalyst calls one of their funnel reports the “Fallout” report.

The Common Folk analogy: Think of it as a beer bong. The beer is the traffic. The top funnel is the start of your conversion funnel. You might spill some like this guy starting at the top. Let’s pretend there are holes in the tube so some beer squirt out along the way. Some beer might spill out of your mouth. The conversion is beer in your belly. So literally, the funnel is starting from pouring the beer (traffic) in at the top all the way down into your belly (conversion). The better you can funnel the beer and eliminate the spills and leaks, the more beer you’ll consume.

This is the exact same concept for your website. If you set up goals for your site, assess the spills and leaks and do your best to patch them up so you can maximize your end conversion!

(My original post is below that describes the exam). I finally got my ACE Kit from Adobe. Very similar to the Omniture Kit I got for SiteCatalyst, the only difference is the logo. Same shirt, picture frame, mints, and business card holder. Again, a very nice gesture and appreciation for becoming accredited in the Adobe product.

Here’s the pics. This time they sent me a Mens shirt instead of a Womens.. so that was nice. Thanks Amberly!

—— Original Post ——

I recently took the Adobe Test&Target certification exam. I think it’s a better test than SiteCatalyst’s certification exam. The Test&Target exam had a good balance of testing the understanding of the tool and the basic knowledge of testing.

My recommendations:

• Take the 3 day Power User Training course – had Joaquin as the instructor and he was great
• Go through some of the online videos – especially the OnSite one
• Thoroughly study the workbook
• Understand the high level concepts and strategies of testing and optimization
• Know how to create campaigns from scratch starting with lazy loading mboxes and understanding the basic differences of each campaign types
• Know campaign life cycle, segments, parameters, targeting, reporting and analysis, success metrics, steps, MVT, Onsite basics, Ad Testing, and some of the additional features.

Although “old” in terms of testing tools out there, I still feel that Test&Target is the most powerful tool out there if used to its potential. The depth of segmentation, targeting, and the integration with SiteCatalyst make it standout amongst the rest – especially for those that already are using SiteCatalyst.

The exam is fair and makes certain you know the basics of Test&Target. It’s 75 questions long, so make sure you get your coffee and concentrate throughout.

Estimated Read Time: 6 min

What’s in it for you: get a logical explanation of why testing and segmentation together is key to your online marketing success.

Testing + Segmentation = Dynamic Duo in Optimization

I am going to walk through why the two together pack the most punch in getting the most out of your marketing campaign.

Testing

A/B testing is great – simple and effective. You test one page vs another test page. For example, you can test a completely different layout of your homepage, test different messaging, etc to determine which version of the page generates better results.

Will a homepage with womens apparel sell more than a homepage with mens apparel?

This is a good question. It’s not a great question. It should be more along the lines of “for women coming to the homepage, which apparel generates best results” and same for men.

(I am using A/B as an example for this post. There is multivariate testing as well.)

This leads into…

Segmentation

Toyota Prius is one of the best selling cars of all time.. at least in Japan. Knowing this, would it make sense for a Toyota dealership salesman to present the Prius as the first pitch for a car to every single customer that walks through the door?

NO!!!!! That makes no sense. A “good” salesman sees who their target customers are and tries to present the right car and the right information to that person at the right time.

If I roll in with my family of six and the salesman will probably show me a Sienna van, and go through all the details of the automatic sliding doors, safety features, entertainment system for the kids, etc. If a young hipster walks through the door, he/she will probably be shown the new Scion – get the greatness of the sound system, alloy wheels, etc. Right? Assuming the salesman is good.. YES!

So what does this have anything to do with online marketing and segmentation? EVERYTHING! Knowing your customers and targeting your pitch to sell your product is critical to your success. Just like the car sales analogy I just went through.

By the way, what is “segmentation” that is constantly thrown around in the online marketing world?

Segments are basically a group of users who have similar characteristics - very similar to my example of the van people versus the Scion people. You can segment users on your site based on traffic sources like Google vs Yahoo! vs Bing, new vs returning users, registered users vs non-registered users, men vs women etc etc.

Note, the only segments worth paying attention to are the ones that make a positive impact on your business and those that you can target. If I come into the dealership, the salesman is not going to waste his time pitching my two year old twins – they don’t make the decision!.. well.. let’s just say no for this exercise.

Expand the image below:

The above is testing a segment. If your site captures which page sections your users are viewing, you can use testing tools to serve up tests just for that particular segment. The example above shows a test to see for of those that checked out the Sienna pages on Toyota.com in their previous site visit, which image is more effective on the homepage to get them to click through.

Now we’re talking!

If there was no segmentation, a lot of “waste” of time may have happened by testing out different Prius images. I say “waste” because you could have been optimizing for each segment instead of spending time trying to optimize for the taste of the general mass.

Take away: Before coming up with your testing strategy, think about your key customer segments. If able, target tests to each of these segments differently and you will see greater results!

Estimated read time: 6 min

What’s in it for you: learn the factors that drive how quickly you can run new online tests.

You’re going to run A/B or MVT tests. You got lots of great ideas – hero images, offerings, value propositions, etc. What are the factors that determine how quickly you can run new online tests? When you test one landing page against another, what factors are involved in making sure the results are sufficient enough to scrap the loser and run a new challenger?

Here it is:

1. Traffic

The number of visitors you are getting.

The higher the traffic to your site, the faster you’ll be able reach statistical significance. If amazon.com sees a 10% increase in overall visits one day, that’s huge! If they get 1,000,000 visits in one day, a 10% increase is 100,000 more visits. Let’s say I see a 10% increase in visits to my site. If I only get 10 visits to my site a day, I can’t really say that 11 visits on a random day is significant. That 1 visit over average could be totally random versus that 100,000 visits to amazon.com is very unlikely to be some random day. Something had to have happened to drive that change.

2. Conversion Rate

A conversion is whatever you decide is a key metric of success on your site. This can be a sale, order , a lead, bounce rate, a sign up, etc. A conversion rate is the total conversion divided by total  number of users or visits to your site. If you have 10 conversions for 100 visits in a day, your conversion rate is 10%.

The higher the conversion rate, the faster you’ll reach significance. Why? If you only convert 1 user for every 10,000 visits to your site, versus 1 user for every 10, you’re going to require a whole heck of a lot less traffic to see whether your tests are really effective or not. Hence, higher the conversion rate, the faster you’ll reach significance.

3. Percent Difference in Conversion Rate

Huh? The bigger the difference in conversion rate from Landing Page A vs Landing Page B, the more confident you will be that one works better than the other. It’s pretty straight forward.

If you are selling lemonade at your lemonade stand and one day you sell 150 lemonades versus your average daily 100, you’re much more confident that something significant happened compared to a day when you sell 102.

4. Number of Tests

Simply the more tests you run at the same time, the less traffic you are running to each test. So based off the factors I mentioned above, this makes common sense.

5. Confidence Level

It’s the level of certainty that a statistical prediction is accurate.

Basically, it’s how confident you want to be in your test results. 95% is a popular confidence level. At a 95% confidence level, you can be 95% certain that the user base would pick the test results you ran.

I’m not going to go into detail here. Google “Confidence Level and Confidence Interval” and you’ll find lots of information on the topic.

The X factor.

Your intuition

This is something that cannot be ‘measured’ or overlooked. The great thing about testing and using analytics is that it takes out a lot of the ‘gut feeling’ BS. A danger, however, in my opinion is that people can go overboard with just relying on data.

Let’s say you run tests Page A vs Page B. After the tests go live, the conversion rates are pretty similar and you haven’t reached statistical significance quite yet. However, you start looking at Page B and it’s just not jiving with the brand you want to portray. Shut it down. Don’t prolong the tests if you believe the new test page you are running just isn’t right. This is assuming.. Page B isn’t getting killed to Page A.

Below are charts that boil the above down.

Most testing platforms have reporting interfaces that tell you the conversion rates and confidence level of the results. Some upper level platforms offer estimated duration of tests based on initial results.

Regardless, it is important to know the 5 factors and the X factor mentioned in this post if you want to get serious about testing. It will give you a good guideline for how you will approach testing.

Take away: traffic, conversion rate, percent difference in test results, number of tests, degree of confidence, and the X factors make up how quickly you can test and optimize.

Estimated read time: 5 min

What’s in it for you? Learn what the difference is between ‘Top Content’ and ‘Top Landing Pages’ and how to get the RIGHT reports in Google Analytics.

What is the difference?

Top Content:

My definition: Which pages on your site are getting how many pageviews.

So what? It shows you which pages are popular and which are not.

How do I find it in Google Analytics?

Top Landing Pages:

My definition: Pages users directly come into on your site. In more simple talk – the first page a user sees on your site. So it won’t necessarily show you all the pages on your site if there are some pages that people aren’t coming directly into.

So what? Shows you which pages you should focus on testing & improving.

Reality is, as you start to generate more traffic, users are going to start coming into your site besides the homepage.

Think of it as your home and your front door entrance.

As you start to build close friends, maybe they’ll start coming in through your side entrance, back entrance, or maybe even the garage door entrance. Same thing with your site. The more users that start coming to your site, the more likely you’ll get users entering your site not through the homepage.

If you start to realize that your garage door entrance is the most popular entrance for your friends, you’ll want to clean the garage up or put a beer fridge next to it for added welcome points!

Same thing with your Top Landing Pages report. Find out which pages users are coming into your site on and optimize.

How do I find it in Google Analytics?

#1) Quick & easy way:

It’s easy to get the report. The problem… it only gives you this much!

#2) The better way. Although it’s a few more steps, once it’s setup, you’re good to go:

Drag and drop. Select the ‘Landing Page’ and then drop any metrics you like. Some of my top recommendations are ‘entrances,’ ‘bounces,’ and ‘goals.’

So there you have it. “Top Content” shows popularity of pages versus “Top Landing Pages” focuses on the ‘first’ page a user sees coming into your site. Although it may seem similar, “Top Landing Pages” will give you more insight into how to increase that first impression on the popular entry pages.