RevaHealth.com Blog

Most web developers realise the importance of A/B testing in ensuring their products are functional and user friendly. We recently carried out testing on our site and found the results and lessons learnt to be really useful. We think the case study shows how invaluable A/B testing is to our business. Hopefully you’ll find it interesting too.

Historically RevaHealth.com has always had a fairly unusual and unique User Interface. Whenever a user searched on our site, we would return a list of results on the left hand side on the page and when a user clicked on a search result, the brochure would appear on the right hand side. We built it this way because we thought that it would allow the user
to be able to compare clinics quickly without having to reload the page.

We were never entirely happy with this interface. We worried that, because it was so unique, users wouldn’t know how to use it. Nonetheless we stuck with the basic premise for over a year and concentrated on fine tuning it.

When we finally secured funding we decided to look at it again. We solicited feedback from a lot of people including some leading design agencies. What we got back was nearly universal – everyone hated our UI. It was too cluttered, they didn’t know how to use it and it was too unusual. So, we decided we needed to change to more traditional  pproach, with search results on one page and brochure information on a second page.

Our original plan was to A/B test this new layout against the original design, but unfortunately we ran out of time on  he project. We decided to push it out anyway and hoped that it would be an obvious improvement and that A/B testing would be unnecessary.

What happened surprised everyone – it was a disaster. Our bounce rate went from 45% to 51%, page views reduced by 25% and worst of all our conversion rate went from 7.2% down to 2.4%. IT WAS KILLING 75% OF OUR EVENUE! After a week of counting the cost we pulled it out and brought back the original design.

So that was the end of it then? Well … no. You see, when we looked at the stats for that week, we saw that there had
been a dramatic increase in traffic from the UK. Not only that, but we hadn’t had time to tweak the new design, as we had done with the original design for over a year. So after licking our wounds we decided to try again, but this time
with proper A/B testing.

We made some obvious changes to both designs and attempted to make sure that we were testing like with like. We
split our traffic 50/50. Half our users saw the ‘narrow’ page layout (search results in a panel on the left hand side, brochures in a panel on right), and half saw the ‘wide’ version (search results fill the page, and brochures open
on a new page).

Unfortunately for us, the results were much the same as when we pushed out our previous attempt at the start of the year. The wide version was generating nearly 25% fewer searches than the narrow version, and it was getting about 12% fewer brochure views too. So, once again we were disappointed and confused.

Conventional wisdom says not to surprise your users with an unusual or non-standard UI. Our UI was very unconventional, so, why did shifting to a more standard visual experience hurt our numbers so badly? Were our users really so unusual that they preferred something so different to most of the search sites out there? We couldn’t believe that, so we stared at the numbers until our eyes bled. And the answer, as is always the case with A/B testing, was in the numbers. It turned out that the metric we were using at the start, the number of searches performed, wasn’t actually a proper measure of what we were interested in testing. A fall in the number of searches performed didn’t actually mean the newer version of the page was worse. In fact, around the same number of people ended up contacting clinics through both funnels, and since the number of searches performed in the new funnel was lower, users there were finding what they wanted faster! So what’s the conclusion? And more importantly, how can you avoid some of the mistakes we made? It’s very easy to get caught up on one simple metric and try to force it one way or another. Even a relatively simple interaction of users searching, narrowing, viewing and selecting is affected by many factors. It was too simple to look at our raw metrics and conclude that our users ‘like’ one UI over the other.

All you can do is refine and test, and refine and test again. After each test, look hard and deep at the numbers – all of the numbers. Because, while one change in the UI may seem simple, the effect it has on the system as a whole may be widespread and not so easy to understand. If you only focus on one metric changing, you may well miss the bigger picture.

Most importantly, make sure that the metric you are looking at is a proper measure of the action / reaction you are supposed to be testing!

After having a great day out at Bizcamp Dublin, Caelen is hoping to repeat the experience at BizCamp Limerick on Saturday 21st. This time he’ll be taking newbie sales guy Owen along, so keep an eye out for him and say hi!

Caelen’s going to be speaking about “Monitizing your Web App” (making spondoolicks on t’internet). In particular, how to make money from:

Pay per Click
pay per Action
Pay per Lead
White-labelling
Licensing

Watch this space for further info about his presentation over the next few days.

BizCamp Limerick is taking place at the Kemmy Business School at University of Limerick and you can register here. Click here to see who else is attending, including Keith Bohanna of dbTwang, Anton Mannering and Alan O’Rourke of Toddle.com.

Who else is going?

RevaHealth, along with IGO People, PutPlace, ByteSurgery, Toddle.com and Look and Taste has been chosen as a finalist in Innovate!Europe’s search for Europe’s Top Startup. (Apologies if we left anyone out!)

The grand prize for the eventual winner is a stay at Guidewire STUDIO, an in-residence business accelerator in Silicon Valley.

The next step in the process is a Masterclass in Zaragoza, Spain with some participants then going on to the Innovate!Europe trade mission in Silicon Valley

So, may the best start-up win!

Caelen spoke about fundraising at Bizcamp on Saturday, a great day out by all accounts. And now the lovely people at Channelship have added a video of his presentation to their blog.

They also recorded presentations from Campbell Scott of IGOPeople and Keith Bohanna of dbTwang which you can watch here. RTE were on hand also, you can watch Edel McAllisters’s report here, featuring interviews with the likes of Niall Harbison of Look and Taste, as well as a shot of the back of Caelen’s head!

The front of his head is visible here, snapped to accompany a great piece on the Silicon Republic site about Bizcamp- thanks very much guys!

Some usability study, which I can no longer find (I think it was from Yahoo) suggested that if you are going to put a picture in a search result that it should be on the right hand side of the search. This was because the picture typically represented ‘flavour’ and not data and therefore the user had to continually skip it when trying to get to the meat of the search results.

So being slaves to other people’s usability studies and we duly put our picture on the right hand side of our search results. We kept it this way for about 2 months; however we have finally decided to change the position of the picture to the left hand side of search results.

The reasoning is because of user feedback. I’m not saying the original usability study was incorrect, however our experience is that users expect the picture to be on the left hand side and using the design principles of ‘Don’t Make Me Think’, if the user expects something in a particular position then we shouldn’t disappoint them.

Here are both example, which do you think works better?

Image on the Right (NEW)

Image on the Right (OLD)

Later today Caelen and I will be attending the Enterprise Ireland High Potential Start Up (HPSU) company event at EI headquarters. RevaHealth was one of 71 companies supported by EI in 2008 and today gives us a chance to meet other start-up people and show off what we do best. Minister McGuinness (Trade and Commerce) will be in attendance for the obligiatory handshakes.

Are you attending?

I had a great time at bizcamp Dublin this Saturday (many thank to the team for organizing it). It was fantastic being around so many enthusiastic people intent on building new and exciting businesses. It is only through the persistence of people like this that we are going to build our way out of this recession.

During the day I talked to about a dozen aspiring young entrepreneurs about how to get funding.
This was obviously a popular theme as I was also on the funding panel (Niall Harbison, Campbell Scott, Ciaran Crean, Keith Bohana and EI) which had a great attendance.
Given the nature of the panel we didn’t have a lot of time to present so I thought I would write up my thoughts.

If you are looking for any type of sizable investment from an Angel it has to be from somebody that trusts you. The reason is that while Angels are perfectly willing to invest in a risky idea or business they don’t want to invest in somebody that might just piss their money away. They need to have the confidence that at a minimum you will use heir money sensibly. After all a good angel is investing in you rather than the idea, this is because ideas are cheap and the value is in the implementation.

In my experience this trust can be achieved in four ways:

1. You know the person.
   
2. You have a proven track record.
3. You have something tangible to demonstrate that proves you can actually implement your ideas.
4. Somebody that trusts you and knows the angel introduces and vouches for you.

Many of the people I met just want me to introduce them to my angels, hoping that they will be able to convert a imply introduction into an investment. The problem with the line of thinking is that I can’t vouch for them. The conversation would go something like this:

Me: I met this guy, John, and he has a great idea - I think you should meet with him

Angel: What is he like to work with?

Me: I don’t know I just met him down the pub, but he seems like a good guy

Angel: What’s his track record?

Me: Well he’s a young guy, not long enough in the tooth for much of a track record

Angel: Can I see his product

Me: He needs funding before he can build a product

— End of conversation —

When I went out looking for funding for RevaHealth.com I went to the person who knew my abilities best … my boss. I had been working for him for the previous 7 years and he knew exactly what I could and couldn’t do. I also knew that if I could get him on board I could get him to introduce me to other angels that he knew.

The problem for most young entrepreneurs is that they are too young to have much of a track record. They are also too young for their network to include the kind of person likely to be in a position to be an angel investor. This  undamentally means that they have to have something tangible to show that they can get an implementation up and running.

A couple of years ago this used to mean a prototype. With the worsening economic climate, however, it isn’t enough simply to show that you can get a technical implementation up and running. You also have to be able to show that you can execute at sales and/or marketing. What this normally means is that you can get some paying customers and deliver value to those customers.

This is a tough ask, but bear in mind that when you are out looking for investment that you are asking someone to trust you with their money. It is not unreasonable for them to demand evidence that you will use their money wisely.

So if you’ve a great idea and you don’t have the track record or network to get angel funding then build your product and get some customers. If you can’t do the development find some developer at the next bizcamp who can build it and give them enough equity to get them bought in that they will live on beans and toast. Get every bit of help you can from EI and the various programs that are out there such as CORD.

When you develop your application, remember that you are not trying to build a finished, polished application. Do the absolute minimum you need to sign up a few customers. Once you’ve got a few people paying you everything becomes much easier.

Any questions then follow me on twitter @revahealth