Site   Web

December 8, 2017

10 A/B Split Testing Mistakes That You Make All The Time

Yes, mistakes happen. There is no need to be surprised because A/B testing can be very tricky. There are some really critical A/B testing mistakes that we all tend to make while we are trying to figure out the best course for increasing conversions.

And truth be told, it is not just you who is prone to making these mistakes. There are many businesses online that get into testing alternatives at their disposal but end up ruining their website instead. A/B tests when done right can in fact save you a lot of time and money but, when gone wrong, eat up all your precious time and drain your resources. So, are you too making any of the following mistakes during an A/B test?

1. You Stopped the Test Too Early

It has been just two days and you have decided to stop the test. You fell prey to your impulse and you just clicked on the “Stop Test” button because you have found or at least you think you have found a winner! Don’t stop the test just yet —chances are that you will end up ignoring the significance of statistics. Always make it a point to aim for above the 95 percent mark before you decide upon a winner.

2. You Set the Test Up Wrong

What? Does that even happen? Yes, and if you make this mistake, it can mean a huge difference in the statistical significance of the two variants. For instance, if you introduce a new variant in the test that just doesn’t register as a part of the original test. Instead, if you decide to introduce a new variant, you should start the A/B test from scratch. Starting a test from scratch puts all variants on the same level and you can easily assess which variant achieves the highest statistical significance.

3. Stopping the Test Too Late

Not stopping the test on time is bad because you end up wasting time. In that time that you wasted you could have started a whole new lineup of tests. Quite frankly, you don’t need to run an A/B test that runs thousands of times over a period of three or four months. You can easily reach a high level of statistical significance long before that. What you should do instead is wait and check the statistical significance each day and also devise a plan to go backward from a goal. This way you don’t waste too much time running a test too long.

4. There Is a Thing Called Too Much Testing

Yes that happens too. This is one of the most common mistakes you will make when you first start an A/B test. You’re obviously excited about optimizing your conversions and you just created two different popup ads for your blog page. You ran the test and one of them has come out a clear winner. But there is a catch; you have made so many changes that the two ads look staggeringly different from each other. You just can’t figure out what worked for your user. So yes start by changing one thing or element per test to know and understand what caused more conversions.

5. Too Many Tests Running On A Page

A/B testing is tricky, as I said that before. Many times a few tests may fail and then a few of them turn out to be not the big winners you were expecting them to be. This is what makes you feel like you need to run a lot of A/B tests. So you run one A/B test and then another and then another on the same page. This way you fall victim to opportunity cost and local maximum and the way out of it is to determine the duration of your A/B test. Know when your webpage hits its local maximum and move on to the next page immediately.

6. Testing the Variation Without Control

Do not even think about deleting the original variant of the site unless you have made proper comparisons and assessments. You will get conversions on the variant for sure but you need to test the statistics simultaneously against those of the original to see which one is doing better.

7. Making a Generalization

This happens a lot, especially when you are running an A/B test that shows a 30 percent increase in conversions from the variant over the control. You would think that it is time to retire the good old design because your conversions are high and you don’t need anything else or better. Don’t go about making assumptions and implementing without running proper tests against the control.

8. A/B Test Tool Installed Wrong

There are tons of tools online that allow you to perform A/B testing. The problem is that it is very difficult to find one particular tool that will be suitable for all kinds of elements on your site. And then there are other mistakes such as:

  • Putting the tracking code in the wrong page.
  • Putting the wrong code or making an error while writing one.
  • Not syncing up the tracking code with your A/B test.
  • Failure of your A/B test tool to track the test on the current plug-in.

The best and the most time saving solution to it is to find a marketing tool that has efficient A/B testing built in.

Testing Insignificant Elements

Do you really think that changing the color of your call to action button or that of the help popup menu would enhance your conversions? A lot of people end up wasting a lot of time on futile things such as these in the hopes of bringing more conversions. The solution is to change your call to action content, edit the headlines and upload more client testimonials through linking with your social media pages and the like. Test things that actually matter.

10. Not Outlining Your Goal

Why are you performing an A/B test in the first place? Don’t do it just because all the big-time marketers out there are telling you to do it. You need to decide what you need to test and how that is going to affect your conversions. If there’s no real goal in place, you will end up just testing something that is not even there and more likely pleases your psychology. Remember, you don’t want any mental satisfaction but statistical facts to measure the success of your site. Long story short, it is not a random test that you perform for the sake of it. You have to understand its importance and purpose that it would solve in the long run for your business.


Scott Keever is a prominent entrepreneur and the founder of Scott Keever SEO, a renowned SEO company in Tampa. With enormous experience in Internet marketing, Scott Keever has been consistently serving needs of clients into diverse verticals for a long time. As a passionate entrepreneur, Scott Keever loves working with Fortune 300 companies and helping them reach the heights of success.