How to Run Your First A/B Split Test

Introduction

When asked who the most successful e-com retailer is right now, Amazon usually comes to mind. But can you remember the last time Amazon redesigned their site? It seems like Amazon has had the same site for years, right? The ‘real’ answer to this question is that Amazon redesigns their site on a daily basis, and because of that, you don’t notice all these small changes over time.

Amazon’s site seems fairly static because you simply don’t notice the small changes that take place each day and week as the result of all the A/B split tests they are running. Amazon uses an evolutionary approach to conversion rate optimization. Instead of making drastic changes all at once, Amazon instead tests very small changes by A/B testing, and then deploys those winning results on a regular basis.

The reason Amazon does this is simple. It works. Not only does it work, but it allows them to quantify every single change they make, whether it’s the position of a button or a call-to-action phrase. They know how much incremental revenue they are getting from each new change they make.

eConsultancy.com ran a survey a few years back that showed for every $92 spent on visitor acquisition, only $1 was spent on conversion rate optimization (abbreviated at CRO). Chances are quite good that your competitors are not A/B testing. Allocating a portion of your marketing budget to CRO strategies can be a strong competitive advantage.

So how does one dip their toes into the waters of A/B testing without investing a lot of time or money? That’s what we’ll be discussing in the rest of the article.

Four Steps for Running an A/B Split Test

Step 1: Define a business question

Step 2: Formulate a hypothesis

Step 3: Run an experiment

Step 4: Analyze & interpret the results

Let’s briefly talk more about each of these steps.

Step 1: Define a Business Question

Before you can even think of a test hypothesis, take a step back, analyze the web page you want to improve, and ask a specific business question. In the example below from a website offering auto loans, we asked the question:

“How can we get more visitors to click on the ‘Apply Online’ button?”

Figure 1: Our goal is to get more visitors clicking on the Apply button

Step 2: Formulate a Hypothesis

This step forces you to create a logical expression for the business question you defined in step one. This expression should take the form of:

“If we change X, then we expect Y.”

In our auto loans example, here was the hypothesis we came up with.

“If we add a stronger value proposition above the button and also change the button call-to-action, we’ll motivate more users to click on the Apply button.”

Note that the above hypothesis incorporates the goal of the test – more clicks on the blue Apply button.

Here’s the test variation we created from the above hypothesis.

Figure 2: The test variation added a new headline and button call-to-action

It’s important to keep things very simple for your first test. Stick with changing things like an image or a headline or the text on a specific button. As you get a few tests under your belt, you can start thinking of more complicated tests like partial page redesigns.

Step 3: Run an Experiment

In this step, you will use a split testing platform to set up and run your experiment. As you are just getting started with testing, my recommendation would be to register for a free 30-day trial for one of the many split testing platforms that are out there. Two of the most popular are Optimizely and Visual Website Optimizer, both of which offer free trials.

The nice thing about these split testing platforms is that they allow you to make simple changes directly in a visual editor. This means that changing things like headlines and images are very quick and easy and require no programming. The page variation isn’t “real” – it’s created on the fly and served up within the testing platform. The below image shows the button text being edited within the testing platform.

Figure 3: The A/B testing platform will allow you to easily change text on a button

Once your test variation is created within the selected testing platform, you can start up your test. The platform will randomly split visitors to the page so that 50% see the original page and the other 50% see the test variation. Below is a diagram showing how this works.

Figure 4: The split testing platform automatically splits 50% of traffic to the test variation

The testing platform will also let you know when you can stop your test based on the statistical significance of your test variation – no math degree required!

Step 4: Analyze & Interpret the Results

The testing platform you use will conduct all the back-end analysis you will need to determine whether your test variation wins, loses, or is a tie against the original page. Once the testing platform has determined an outcome, you are then ready to analyze the results. Below is an example of how test results look in Optimizely, one of the most popular testing platforms available.

Figure 5: The back-end stats engine from the Optimzely A/B testing platform

For our auto loans test example above, the test variation showed a 15.8% improvement in “Apply in Minutes” clicks. The recommendation was to move to the variation as the new standard page.

Keep in mind that, on average, only 10-15% of your tests will produce positive results. The rest of them will either be statistical ties with the original, or underperform against the original. As you ramp up your testing program, your goal should be to launch 1-2 tests per week and then build from there. The more tests you run, the more winners you’ll have – and that means more revenue and leads coming your way.

Now is the best time to run your very first test. Testing tools like Visual Website Optimizer provide a very low-cost solution to any organization’s testing needs. Sign up for a trial account today and give us a call if you need help or would like to expand your testing program.