Entries tagged with “dns” from Search Engine CookBook

I've seen a few posts from people on forums over the last few weeks complaining that GoogleBot is misbehaving and chewing up silly amounts of bandwidth.

Unfortunately it's all too easy to pretend to be GoogleBot (or any other bot for that matter) simply by spoofing the UserAgent string ie. telling the world that you are something that you aren't.

So how can you tell if a bot really is from Google?

One of the official Google blogs has the answer - DNS lookups:


Basically you need to check that the forward and reverse DNS entries are valid ie. that the A and PTR records are from Google...

That's a bit of a pain, but will save you headaches if you're seeing strange activity in your logs...

Losing traffic through laziness or silly mistakes is simply unacceptable.

What am I referring to?

If your site resides at  www.domain.tld and you've been marketing it successfully both online and offline people will forget about the "www" part. It's only natural.

While back in the mid 90s most sites were ONLY available via www.domain.tld that's no longer the case.
(there was some odd RFC that a lot of people referred to for this reason)

So the first thing you should do is check that both www.domain.tld and domain.tld point to your site.

There's no technical reason why your hosting provider can't set that up for you. If they tell you that they can't then you should really look elsewhere.

The one possible problem that you might face is that robots might treat the "www" version of your site and the non-www one as two separate sites. They shouldn't, but it can happen.

The simple solution to this is to force people (and spiders) to use either one or the other using a redirect.

Richard has a case in point with regard to the NCH in Dublin that could so easily be fixed! Others have fixed theirs already.