Next: Robot Wrangling
Previous: Common Problems with Robots

Redirecting Robots

There are three ways to redirect a search engine robot elsewhere, the first two are named after their HTTP response codes and are: 302 Moved Temporarily and 301 Move Permanently. It is also possible to include a Meta Refresh tag in the page doing the redirecting:

<head>
<meta http-equiv="REFRESH" content="0;URL=http://newsite.com/index.htm">
</head>

A refresh is a two stage process. The robot connects to the page and gets told about the new location. It may then act on this information to access the new page. This second access should be visible at some point in the server log files.

The effects of the 302 and Meta Refresh can be undesirable with both pages being indexed by search engines depending on internal and external inbound-links or one of the pages being dropped due to duplicate page algorithms. In the case of Google the results can vary depending on the exact search term entered. A number of leading companies, apple.com and microsoft.com included, incorrectly use 302 redirects to merge their domain.com and www.domain.com names as discussed in the uniform resource locator chapter. It seems that the search engines eventually sort it all out, at least in the case of these old and high PageRank sites.

The effect of a 301 (moved permanently) redirect is much clearer in the logs. Here google first fetches the page index.html but gets redirected to index.htm

crawl-66-249-65-131.googlebot.com - crawl-66-249-65-131.googlebot.com.5337109924
5954889 [31/Oct/2004:18:05:54 +0000] "GET /index.html HTTP/1.1" 301 242 "-
" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

crawl-66-249-65-131.googlebot.com - crawl-66-249-65-131.googlebot.com.5334109924
595576 [31/Oct/2004:18:05:55 +0000] "GET /index.htm HTTP/1.1" 200 4782 "
-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

It processes this redirect directly.

As redirects are a two step process they are wasteful of resources and can cause problems as we have discussed. In some cases it is better to use server side aliases to directly serve the new content directly.

Redirecting Robots

See Also