Nov 16

The Results Of Our Local Domain Search Experiment

By Marta Sawicka Stuff we've learned 9 Comments »

A few months ago, in the post Using Country Specific Domains For SEO we introduced you to the idea of redirecting half of our Irish and UK pages from RevaHealth’s .com domain to our local .ie and .co.uk domains. In this follow up post we describe how the experiment turned out to be successful, but with a very disappointing result.

The main goal of all search engines is to return results from the sites that are the most useful and relevant to a user query. Sites with country-coded top-level domains, such as .ie, are associated with a geographic region, in this case Ireland. By default, and all other things being equal, sites with a .ie domain are interpreted as more relevant for users in Ireland. For generic top-level domains such as .com Google allows site owners to manually assign a preferred country in their Webmaster Tools.

As our .com domain’s content covers many countries we can’t use this particular feature to optimize the site for a particular region. Instead, we were hoping that by redirecting our Irish and UK pages to their local domains that those pages would benefit from a boost in search results positioning on Google.ie and Google.co.uk as being targeted at their local audiences.

The initial results were terrible. The redirected pages dropped in the SERPs by about 3 positions. As our Irish pages previously ranked better than our UK ones they suffered even more. (Ireland was our first local area of interest and those pages were in the index much earlier.)

After a few weeks our results moved slightly in the right direction but were still behind the control group. We decided to let the experiment run for longer, almost four months in total, to be sure that the conclusion drawn was correct. In the end we turned off the redirections, just last week in fact. The experiment was successful in terms of achieving measurable and unambiguous results, but unfortunately they were very disappointing.

Our redirected .com pages disappeared from the index and over time their corresponding .ie and .co.uk pages were indexed instead. However, they never reached the search results positions we had had before with our .com pages. We speculated that our .com domain had more trust and authority than our country specific domains and that’s why pages from RevaHealth.com ranked better even in Google’s country specific search engines.

Now, just one week after undoing the redirects, most of the .com pages have regained their positions from before the experiment, which we think goes some way to validating our speculation. Our generic .com domain, which is crawled much more frequently than our local domains are, does seem to be a more trusted site, not only by Google.com but also by Google.ie and Google.co.uk.

Have you run any similar experiments that you can share with us?

Tagged with: • •
Nov 09

Set Your SPF Record Correctly Or Risk Being Marked As Spam

By David Roe Stuff we've learned No Comments »

Email is notoriously open to fraud. It’s built upon old protocols which tell the user very little that is definitely true about its content and source. The most basic thing about an email, the “from” address, can easily be spoofed by a sender who wants to pretend to be someone else, especially spammers.

SPF (Sender Policy Framework) is a protocol that is built on top of email. When a message arrives at the server receiving your email (Google, Yahoo, or your own company’s email server) the email claims to be from person@domain.com, but it might not really. All we know for sure is that it definitely comes from some IP address.

Using SPF, your email server can check with the domain that the email purports to be from to see if the IP address you got the email from is one that they use to send email. If domain.com says no, they don’t send emails from that IP address, then the email may be spam. If the domain says yes, then it’s probably not.

Probably; however, this isn’t certain. Not all email senders support SPF. It is voluntary but very widely used. So, just because the sender doesn’t support it doesn’t mean all emails from that domain are spam. Conversely, people send spam from perfectly respectable domain names all the time, so just because you do get a valid SPF record back that matches the “from” domain, doesn’t mean it’s definitely not spam. Still, it is a good indication, and some email servers and ISPs will mark your email as spam if the SPF record doesn’t match, or isn’t present.

So how can you avoid this fate?

First check if you already have a valid SPF record. Go to http://www.kitterman.com/spf/validate.html and enter the domain where you send email from. If your domain returns a valid SPF record, everything is fine. If not, you may find some email servers you send emails to may block them as spam.

Here’s what Gmail was showing for us:

Received-SPF: neutral (google.com: XX.XXX.XX.XXX is neither permitted nor denied by best guess record for domain of info@revahealth.com) client-ip= XX.XXX.XX.XXX;

Authentication-Results: mx.google.com; spf=neutral (google.com: XX.XXX.XX.XXX is neither permitted nor denied by best guess record for domain of info@revahealth.com) smtp.mail=info@revahealth.com

(To see this, simply send email to a Gmail account, and then select ‘See Original’ in the little menu at the top of the email message. You get to see all the headers for the email.)

What this is saying is that when they check the IP address we’re sending from, they get back neither a “confirm” nor a “deny” message. That is, there is no SPF record at all.

We used to have our SPF record for RevaHealth.com set correctly. I know, because I did it. I also could tell when it stopped working – when we moved our front end box from one server to another a few months ago. Of course, I couldn’t remember just what I’d done to set it correctly nearly three years ago.

The key to the answer is not who or what sends your email, but who owns the domain. The email receiver doesn’t check with the domain, but with the DNS (Domain Name Server). In the example above, Google GMail isn’t checking with RevaHealth.com, but with the Domain Name Server for RevaHealth.com. In our case, that’s Go Daddy.

Of course, I didn’t remember that at first. Thinking our own front end box would have the SPF record I looked in its own DNS entries and added it there. There’s a very handy SPF setup wizard here to help you to create your SPF record and save it in your DNS. However since our DNS is Go Daddy, this did me no good at all.

So, after going back and reading the very helpful SPF FAQ again, I realised that I should use our DNS to create the SPF record. And that’s when I realised what had gone wrong. When we moved our servers, we updated our DNS entry for RevaHealth.com and lost the SPF record in the process.

Editing your SPF record on a domain register depends on their interface. Thankfully for us, there is a helpful guide to creating  SPF records for domains hosted with Go Daddy.

I quickly added the SPF record to our RevaHealth.com domain entry, but this wasn’t the end of the story.  We send email from our hosting server. This looks like mail.si-svXXXX.com and that’s what an IP address lookup returns. When I entered this domain as an allowed domain to send email, I got nothing. Running the validation check failed.  However, this was because the SPF record should return only the domains it supports, not the sub-domains. Dropping the “mail.” and changing the record to just si-svXXXX.com brought our SPF records back to normal.

Now, Google reports itself happy with us again.

Received-SPF: pass (google.com: domain of support@revahealth.com designates XX.XXX.XX.XXX as permitted sender) client-ip=XX.XXX.XX.XXX;

Authentication-Results: mx.google.com; spf=pass (google.com: domain of support@revahealth.com designates XX.XXX.XX.XXX as permitted sender) smtp.mail=support@revahealth.com

Did it make a difference? Is this worth bothering about?

Yes. Several of our customers had not been receiving emails from us because their ISP was blocking anything without a valid SPF record, and these emails are now getting through okay. It probably reduces the overall spam score for emails we’re sending too, but that was always very low anyway. It’s definitely worth doing, it costs nothing, and the business cost of emails not arriving where they are supposed to can be massive.

Tagged with: • • •
Nov 03

Sample Sizes for A/B Testing

By Caelen King Stuff we've learned No Comments »

A/B testing is great for iteratively improving web applications. I have had loads of conversations with startups who are using it to test. However, they all seem to be making one fatal mistake – their sample sizes are too small.

An A/B test that uses a sample size that is too small is worse than no test at all. The figures just aren’t statistically significant, and if you had run the test longer the results may have reversed. This means that not only might you have the wrong result but you feel that you have proof in that wrong result. This results in in you defending your position later on despite contradictory evidence.

So how big does your sample size need to be? Unfortunately the answer is – ‘it depends’. Luckily it doesn’t get very complicated. The only thing it depends on is the gap between the winner and the loser. In general the smaller the gap the bigger of a sample size that you need.

Specifically divide the gap between your winner and loser in half and then square the result. The result needs to be bigger than your sample size to be statistically significant?

Below is a worked example where there is a 10% difference between A and B

a-b-testing-sample-sizes-10-percent-difference

So with a 10% difference between A and B you need a sample size of about 1500 to be statistically significant.

Let’s see how this works with a 2% difference between A & B

a-b-testing-sample-sizes-2-percent-difference

Now you can see that you need a sample size that is around 25 times bigger.

Tip: If you find your test needs a very high sample size (above about 20,000) then your users don’t really care much about your changes. Give up and test something more significant.

Tagged with: • •
preload preload preload