| User Agent |
Search Engine URL |
Comment |
| NetResearchServer |
Net Research |
|
| ASPseek |
www.aspseek.com |
|
| Aport |
www.aport.ru |
Russian Portal |
| BDFetch |
www.brandimensions.com |
Net Spooks |
| BTbot |
www.btbot.com |
Bit Torrent |
| Baiduspider |
www.baidu.com |
Chinese SE |
| BlingBlangBlog.com |
BlingBlangBlog.com |
Blog SE |
| BlogPulse |
BlogPulse.com |
Blog SE |
| Bumblebee@relevare.com |
|
|
| Butch__2.1.1 |
|
|
| CJNetworkQuality |
cj.com |
Affiliate Network |
| CarsCrawler |
|
|
| Clushbot |
www.clush.com |
Clustering SE |
| CosmixCrawler |
|
|
| CrawlConvera |
Convera |
Net Spooks |
| CreativeCommons |
www.nutch.org |
Open Source SE |
| Cyberz Communication Agent |
www.cyberz.co.jp/ |
|
| DataparkSearch |
www.dataparksearch.org |
|
| Dumbot |
www.dumbfind.com |
|
| FAST-WebCrawler |
www.alltheweb.com |
|
| Gaisbot |
gais.cs.ccu.edu.tw/robot.php |
|
| GeonaBot |
www.geona.com |
|
| GetRight |
|
|
| Gigabot |
|
|
| GoForIt.com |
www.GoForIt.com |
Meta Search Engine |
| Googlebot-Image |
www.google.com |
|
| Googlebot |
www.google.com |
|
| Googlebot/Test |
www.google.com |
|
| HenriLeRobotMirago |
www.miragorobot.com |
|
| IlTrovatore-Setaccio |
www.iltrovatore.it |
|
| Infoseek SideWinder |
www.infoseek.com |
|
| Jetbot |
|
|
| Kitenga |
|
|
| LWP::Simple/5.76 |
|
|
| LinkWalker |
|
|
| Look.com |
|
|
| Mackster |
www.ukwizz.com |
|
| Mediapartners-Google |
|
|
| Ask Jeeves/Teoma |
|
|
| Voilabot |
www.voila.com |
|
| Girafabot |
www.girafa.com |
|
| "Mozilla/3.01 (compatible;)" |
|
|
| Exotic Crawler |
|
|
| focuseekbot |
|
|
| grub-client |
www.grub.org |
Distributed Open Source SE |
| ZyBorg/1.0 (wn-1.zyborg@looksmart.net; |
www.wisenut.com |
|
| Exabot |
www.exava.com |
|
| Slurp |
www.inktomi.com |
|
| Yahoo! Slurp |
|
|
| "Mozilla/5.0 (compatible; Yahoo! Slurp; |
|
|
| NG |
|
|
| NITLE Blog Spider |
|
|
| NPBot |
www.nameprotect.com |
Net Spooks |
| NaverBot-1.0 |
naver.com |
|
| NetResearchServer |
www.loopimprovements.com |
|
| "NuSearch Spider www.nusearch.com" |
www.nusearch.com |
|
| NutchCVS |
www.nutch.org |
Open Source SE |
| "ObjectsSearch/0.01 (ObjectsSearch; http://www.ObjectsSearch.com/bot.html;
support@thesoftwareobjects.com)" |
|
|
| "OmniFind [lnx-ir]" |
|
|
| "Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)" |
|
|
| "Openbot/3.0+(robot@monkia.com.tw;+http://gais.cs.ccu.edu.tw/robot.php)" |
|
|
| "Openfind data gatherer, Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)" |
|
|
| "PEERbot www.peerbot.com" |
|
|
| "Pompos/1.3 http://dir.com/pompos.html" |
|
|
| "Python-urllib/1.15" |
|
|
| "QuepasaCreep ( crawler@quepasacorp.com )" |
|
|
| "Reaper/2.07 (+http://www.sitesearch.ca/reaper)" |
|
|
| "RobotAgent" |
|
|
| "Sauce Reader/1.6 (.NET CLR |
|
|
| "ScSpider/0.2" |
|
|
| Scooter |
www.altavista.com |
|
| Seekbot |
www.seekbot.net |
German SE |
| "Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html;
info@szukacz.pl)" |
|
|
| "TTN-WebCrawler" |
|
|
| "Tarka/0.09EF" |
|
|
| "Tarka/0.09LF" |
|
|
| "Technoratibot/0.6" |
|
|
| "The World as a |
|
|
| "TurnitinBot/2.0 (http://www.turnitin.com/robot/crawlerinfo.html)" |
|
|
| "TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html" |
|
|
| "Tutorial Crawler 1.4 (http://www.tutorgig.com/crawler)" |
|
|
| "UptimeBot" |
|
|
| "Uptimebot" |
|
|
| "Vagabondo/2.0 MT (webagent at |
|
|
| "Vagabondo/2.0 MT" |
|
|
| "Waypath Scout v2.1 - |
|
|
| "Waypath development crawler - |
|
|
| "WebFilter Robot 1.0" |
|
|
| "Wget/1.8.2" |
|
|
| "Wotbox/alpha0.6 (bot@wotbox.com; http://www.wotbox.com)" |
|
|
| Yahoo-MMCrawler |
www.yahoo.com |
Multimedia robot |
| "YahooFeedSeeker |
www.yahoo.com |
Yahoo newsfeed robot |
| "Zao/0.2 (http://www.kototoi.org/zao/)" |
|
|
| "Zeus 29054 Webster Pro |
|
|
| "Zippp.net" |
|
|
| "ZipppBot/0.11 (ZipppBot; http://www.zippp.net;
webmaster@zippp.net)" |
|
|
| "antibot-V1.2.0/redhat-linux-9" |
|
|
| "appie 1.1 (www.walhello.com)" |
|
|
| "augurfind V-1.8" |
|
|
| "augurnfind V-1.8" |
|
|
| "blogcrawler (blogcrawler@yahoo.com)" |
|
|
| "blogsnowbot http://www.blogsnow.com/bot.html" |
|
|
| "boitho.com-dc/0.4 ( http://www.boitho.com/dcbot.html )" |
|
|
| "boitho.com-dc/0.5 ( http://www.boitho.com/dcbot.html )" |
|
|
| "boitho.com-dc/0.51 ( http://www.boitho.com/dcbot.html )" |
|
|
| "boitho.com-dc/0.52 ( http://www.boitho.com/dcbot.html )" |
|
|
| "boitho.com-dc/0.54 ( http://www.boitho.com/dcbot.html )" |
|
|
| "boitho.com-dc/0.57 ( http://www.boitho.com/dcbot.html )" |
|
|
| "boitho.com-dc/0.58 ( http://www.boitho.com/dcbot.html )" |
|
|
| "boitho.com-dc/0.60 ( http://www.boitho.com/dcbot.html )" |
|
|
| "boitho.com-dc/0.61 ( http://www.boitho.com/dcbot.html )" |
|
|
| "boitho.com-dc/0.63 ( http://www.boitho.com/dcbot.html )" |
|
|
| "booch_1.0.7 (tankvit@e-mail.ru)" |
|
|
| "deepak-USC/ISI" |
|
|
| "deepak-USC/ISI(1.0)" |
|
|
| "feedfinder/1.2 Python-urllib/1.15 +http://diveintomark.org/projects/feed_finder/" |
|
|
| "google" |
|
|
| "htdig/3.1.0b2 (root@localhost)" |
|
|
| "htdig/3.1.5 (root@localhost)" |
|
|
| "http://www.almaden.ibm.com/cs/crawler [bv2m308]" |
|
|
| "http://www.almaden.ibm.com/cs/crawler [c01]" |
|
|
| "http://www.sygol.com" |
|
|
| "http://www.ultraknowledge.com/spider" |
|
|
| "ia_archiver" |
|
|
| "ia_archiver-web.archive.org" |
|
|
| "kinjabot beta2 (http://www.kinja.com)" |
|
|
| "kinjabot beta2" |
|
|
| "kinjabot" |
|
|
| "kinjabot/1.0 beta (http://www.kinja.com/help/)" |
|
|
| "kuloko-bot/0.5" |
|
|
| "larbin_2.6.3 (larbin2.6.3@unspecified.mail)" |
|
|
| "larbin_2.6.3_for_(http://cosco.hiit.fi/search/) (Tomi.Silander@hiit.fi)" |
|
|
| "larbin_2.6.3_for_(http://cosco.hiit.fi/search/) (tsilande@hiit.fi)" |
|
|
| "larbin_extended (larbin@oktie.com)" |
|
|
| "libwww-perl/5.53" |
|
|
| "lmspider (lmspider@scansoft.com)" |
|
|
| "lwp-trivial/1.38" |
|
|
| "mozDex/0.04-dev (mozDex; http://www.mozdex.com/bot.html;
spider@mozdex.com)" |
|
|
| "mozDex/0.05-dev (mozDex; http://www.mozdex.com/bot.html;
spider@mozdex.com)" |
|
|
| "msnbot-rss |
search.msn.com |
MSN newsfeed bot |
| msnbot |
search.msn.com |
MSN beta SE |
| "mxyz/1.0" |
|
|
| "my-robot/0.1" |
|
|
| "nuSearch Spider <a href='http://www.nusearch.com'>www.nusearch.com</a> |
|
|
| "obidos-bot (weblog bookwatch)" |
|
|
| "obidos-bot" |
|
|
| "psbot/0.1 (+http://www.picsearch.com/bot.html)" |
|
|
| "search.ch V1.4.2 (spiderman@search.ch; http://www.search.ch)" |
|
|
| "sohu-search" |
|
|
| "timboBot/0.9 http://www.breakingblogs.com/timbo_bot.html" |
|
|
| "vspider" |
|
|
| "webcrawl.net" |
|
|
| "wikia-robot/0.3" |
|
|
| "wwwster/1.2 (Beta)" |
|
|
| "wwwster/1.2 (Beta, mailto:gue@cis.uni-muenchen.de)" |
|
|
