I've discovered this:
- GoogleBot is not only a bunch of servers (obviously). It is a very big distributed cluster with hundreds of machines. My site is indexed from more than 900 different Google IP addresses every day.
- I've identified 7 different GoogleBot crawling clusters.
- They seem to connect to my site from 6 different locations.
- Almost all of them are in USA but one location is Europe.
With access to your web site log files you can "grep" the string "http://www.google.com/bot.html" on the referrer field and find out which IP GoogleBot is using when it pays you a visit. There are some other malicious crawlers that fake their referrer as GoogleBot but they're easily spotted. Google Inc. owns the Autonomous System AS15169 and its connections come from there. In my case I got connections from those IP ranges below, during the last six months:
| 66.249.72.xxx |
Google advises that the way to find out if an Origin IP belongs to GoogleBot is to do a reverse DNS resolution and look for crawl-xx-xxx-xx-xxx.googlebot.com in the result.
Applying that the final list to work with is:
|66.249.72.xxx 184.108.40.206.in-addr.arpa domain name pointer crawl-66-249-72-1.googlebot.com.|
66.249.73.xxx 220.127.116.11.in-addr.arpa domain name pointer crawl-66-249-73-1.googlebot.com.
66.249.74.xxx 18.104.22.168.in-addr.arpa domain name pointer crawl-66-249-74-1.googlebot.com.
66.249.75.xxx 22.214.171.124.in-addr.arpa domain name pointer crawl-66-249-75-1.googlebot.com.
66.249.76.xxx 126.96.36.199.in-addr.arpa domain name pointer crawl-66-249-76-1.googlebot.com.
66.249.77.xxx 188.8.131.52.in-addr.arpa domain name pointer crawl-66-249-77-1.googlebot.com.
66.249.78.xxx 184.108.40.206.in-addr.arpa domain name pointer crawl-66-249-78-1.googlebot.com.
I don't take into account the networks 66.249.80.xxx, 66.249.81.xxx, etc. because seem to be used by Feedfetcher-Google and Mediapartners-Google (AdSense) and that's out of the scope of this post.
Latency = Hint
Nowadays is tricky to know where a IP is located when it belongs to a big network. Anycast routing method (like the one used with the popular Google Public DNS Service 220.127.116.11) becomes a challenge if you want to be certain. Google Inc. IP addresses are administrative located at Mountain View, California and without any further analysis this is the conclusion you will get.
But when I ping those networks from my server (Paris, France), write the obtained round trip times on a table and give a look to the Google Data Centers map... One can guess and approximated geographic location for those GoogleBot clusters:
|IPv4 Network||Ping Round Trip||Location|
|USA East Coast ?|
|USA Mid West ?|
|USA West Coast ?|
|USA East Coast ?|
|18.104.22.168||(Not active since
(Not USA nor Europe ?)
|Dublin, Ireland ?|
Round Trip milliseconds is not an accurate method to place a system on the map but the answer I'm trying to answer here is whether GoogleBot is at California or not. As you see, there is not a short answer but at least we know that it is spread around different locations within the States and Europe.
Post a Comment