RevaHealth.com Blog

At RevaHealth, we do A/B or multi-variant testing on our website frequently. We have found that this is the best way of maintaing a website that consistently performs better over time. What’s fascinating about this form of testing is how often gut feel and professional advice turn out to be wrong. In our experience, nothing beats getting the pages out there and testing them with real users.

A couple of weeks ago we decided to A/B test our conversion page.

This page hadn’t been examined properly in quite a while and had kind of organically changed as functionality was added to the product. When we looked at it we were shocked by how bad it was. I was amazed that anyone had the patience to deal with it at all!

There were loads of problems with the form including:

1. Lots of distractions. Forms that convert well typically don’t feature anthing that could distract the user from the job at hand. Amazon take this so far as to even mute their own branding.
2. The ‘call to action’ button was in the wrong place. Ideally it should be on the left hand side, directly under the fields.
3. There were too many fields. You should always ruthlessly cut out unnecessary fields. In this instance we didn’t really need to have the consumer repeat their email address and we didn’t need them to tell us what treatments they needed as they typically included this information in the description field.

So we redesigned the page and we came up with the following:

It was a much simpler form and because it was shorter we were able to put the labels directly above the fields and still fit the call to action above the fold. I was sure it would perform better (Phil wasn’t, but he is an argumentative sort!). So we put it out to test, splitting 50% of our traffic to the new form and leaving 50% of our traffic to the old form.

Results came back indicating that the old form was performing better. I was aghast and examined the page in all contexts to discover what was wrong. It was then that I noticed something that invalidated the whole test. Something that should have been obvious right from the start. Take a good look at the two forms and see if you can spot it (click on the screen shots and they will open to full screen).

On the right had side of the first form you will spot the clinic’s phone number. But it’s not on the new form. This means that when someone lands on the old form they can do three things:

1. They can fill out the form.
2. The can go back.
3. They can look up the phone number of the clinic, call the clinic and never even fill out the form (this is not necessarily a bad thing for us).

However, with the new form they could only do two of these things. This difference totally invalidated the test.

The moral of the story is, when you are A/B testing make sure you are testing like with like. We have redesigned the new form so as to do so. I’ll post back here in a few days with results.

2 Responses to “How to Invalidate your Own A/B Testing”

1. Richard Hearne says:

   May 31, 2009 at 2:17 pm

   Interesting. Not sure why this difference invalidates the test though? It’s still a valid test, with the presence of the phone number being a variable tested.

   I’d also say that the hero shot may have been a strong variable, but as you mention opinions are more often than not wrong. Do you measure the relative importance of each element on the outcomes you derive?

   Rgds
   Richard

   Reply
   • admin says:

     June 2, 2009 at 12:00 pm

     Hi Richard

     Thanks for dropping by and for the comment. You are right that it is a valid test, however it wasn’t a valid test for what we were trying to measure. We were trying to test designs to find a better converting form, however by introducing a new varibable it wasn’t possible to test how well the design affected form conversion though one A/B test run. Using different testing methodologies it would have, however this particular test wasn’t set up that way.

     We did not mearure the importance of each element, this was straight forward, boil in the bag A/B testing. We also do multi-variant testing but not in this instance.

     Reply