蜘蛛啊爬虫啊太勤劳了。。

大前天写了一篇,抓取蜘蛛爬虫痕迹的文章,之后就根据写的代码在本站上测试。

经过两天的时间,来发表一下对一些搜索引擎的爬虫蜘蛛的看法。。

整天24小时几乎都有爬虫过来爬。。那些大咔的和小咔的。其中sogou最积极。。

下面截取上午到下午几小时内的记录:

 


[2010-07-14 10:40:55] Baidu    :220.181.7.35 | /tag/歌曲 | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 10:41:02] Baidu    :123.125.66.18 | /post/583 | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 10:41:59] YandexBot:87.250.255.242 | /post/532 | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
[2010-07-14 10:42:49] Soso     :124.115.0.109 | /post/524 | Sosospider+(+http://help.soso.com/webspider.htm)
[2010-07-14 10:44:14] Google   :203.208.60.76 | /post/485 | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
[2010-07-14 10:44:18] Baidu    :123.125.68.117 | /tag-周润发.html | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 10:45:08] Twiceler :67.218.116.163 | /tag/key | Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)
[2010-07-14 10:46:28] Soso     :124.115.0.14 | /sort/18 | Sosospider+(+http://help.soso.com/webspider.htm)
[2010-07-14 10:47:05] Google   :203.208.60.76 | /post/573 | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
[2010-07-14 10:49:22] Soso     :124.115.0.167 | /?page=18&sort=18 | Sosospider+(+http://help.soso.com/webspider.htm)
[2010-07-14 10:51:37] Google   :203.208.60.76 | /post/570 | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
[2010-07-14 10:51:40] Baidu    :220.181.7.22 | /tag/str | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 10:52:35] YandexBot:87.250.255.242 | /?tag=%E7%9B%AE%E5%BD%95 | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
[2010-07-14 10:52:39] Soso     :124.115.4.197 | /sort/16 | Sosospider+(+http://help.soso.com/webspider.htm)
[2010-07-14 10:56:40] Baidu    :123.125.68.118 | /tag-gbk.html | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 10:57:01] Baidu    :123.125.66.48 | / | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 10:57:06] Baidu    :123.125.66.115 | /tag/热汤 | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 10:57:08] Baidu    :123.125.66.38 | /tag/点钟 | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 10:57:09] Baidu    :123.125.66.38 | /tag/熬夜 | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 11:00:16] Twiceler :67.218.116.163 | /tag/%E8%A3%85%E9%80%BC | Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)
[2010-07-14 11:02:27] Baidu    :220.181.7.17 | /tag-知识.html | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 11:03:13] YandexBot:87.250.255.242 | /?tag=%E5%AE%8C%E6%95%B4 | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
[2010-07-14 11:05:25] Baidu    :123.125.66.23 | /record/201005 | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 11:10:40] YandexBot:87.250.255.242 | /post/468 | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
[2010-07-14 11:13:14] Baidu    :220.181.7.49 | /?record=201001&page=1 | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 11:20:45] Soso     :124.115.4.198 | /post/45 | Sosospider+(+http://help.soso.com/webspider.htm)
[2010-07-14 11:21:15] YandexBot:87.250.255.242 | /post/430 | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
[2010-07-14 11:24:15] Baidu    :220.181.7.33 | /tag-span.html | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 11:32:56] YandexBot:87.250.255.242 | /?tag=%E5%AD%97%E7%AC%A6 | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
[2010-07-14 11:34:57] Baidu    :220.181.7.46 | /post/322 | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 11:38:20] YandexBot:87.250.255.242 | /?tag=%E5%9D%9A%E6%8C%81 | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
[2010-07-14 11:47:53] YandexBot:87.250.255.242 | /?record=200912&page=6 | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
[2010-07-14 11:56:56] Baidu    :123.125.66.43 | / | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 11:57:09] Google   :203.208.60.76 | /tag/res | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
[2010-07-14 11:57:29] YandexBot:87.250.255.242 | /?tag=%E6%B4%BB%E7%9D%80 | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
[2010-07-14 12:01:22] Twiceler :67.218.116.163 | /tag/temp | Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)
[2010-07-14 12:02:03] Baidu    :123.125.66.112 | / | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 12:08:10] YandexBot:87.250.255.242 | /tag/%E7%AA%83%E8%80%85 | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
[2010-07-14 12:09:21] Soso     :124.115.4.193 | /post/169 | Sosospider+(+http://help.soso.com/webspider.htm)
[2010-07-14 12:11:57] Baidu    :123.125.66.43 | / | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 12:14:16] Baidu    :123.125.68.119 | / | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 12:16:45] YandexBot:87.250.255.242 | /?tag=%E6%96%87%E4%BB%B6&page=2 | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
[2010-07-14 12:16:59] Baidu    :123.125.66.110 | / | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 12:20:12] Soso     :124.115.0.28 | /post/182 | Sosospider+(+http://help.soso.com/webspider.htm)
[2010-07-14 12:23:10] YandexBot:87.250.255.242 | /?tag=%E5%8F%82%E6%95%B0&page=2 | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
[2010-07-14 12:23:51] Google   :203.208.60.76 | /post/84 | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
[2010-07-14 12:26:50] Baidu    :123.125.66.99 | / | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 12:30:47] Twiceler :67.218.116.163 | /tag/dreamweaver | Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)
[2010-07-14 12:31:52] Soso     :124.115.0.107 | /post/8 | Sosospider+(+http://help.soso.com/webspider.htm)
[2010-07-14 12:32:00] Baidu    :123.125.66.28 | / | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 12:33:50] YandexBot:87.250.255.242 | /?tag=%E4%B8%AD%E6%96%87 | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
[2010-07-14 12:41:55] Baidu    :123.125.66.115 | / | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 12:43:18] Soso     :124.115.0.26 | /page/1 | Sosospider+(+http://help.soso.com/webspider.htm)
[2010-07-14 12:45:39] YandexBot:87.250.255.242 | /post/241 | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
[2010-07-14 12:48:08] Google   :203.208.60.76 | /post/187 | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
[2010-07-14 12:48:59] Google   :203.208.60.76 | /post/46 | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
[2010-07-14 12:49:09] Google   :203.208.60.76 | /post/582 | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
[2010-07-14 12:49:21] Google   :203.208.60.76 | /post/186 | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
[2010-07-14 12:49:34] Google   :203.208.60.76 | /post/581 | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
[2010-07-14 12:49:47] Google   :203.208.60.76 | /post/583 | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
[2010-07-14 12:52:19] Msn      :65.55.3.195 | /?author=1&page=2 | msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
[2010-07-14 12:52:26] Msn      :65.55.3.195 | /?tag=%E4%BB%A3%E7%A0%81&page=5 | msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
[2010-07-14 12:52:37] Msn      :65.55.3.195 | /tag-%E5%8A%A8%E6%80%81.html | msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
[2010-07-14 12:52:52] Msn      :65.55.3.195 | /tag-%E5%8A%A0%E8%BD%BD.html | msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
[2010-07-14 12:52:59] Msn      :65.55.3.195 | /?sort=9&page=2 | msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
[2010-07-14 12:53:06] Msn      :65.55.3.195 | /post-95.html | msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
[2010-07-14 12:53:09] Msn      :65.55.3.195 | /post-231.html | msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
[2010-07-14 12:53:11] Msn      :65.55.3.195 | /tag-%E6%97%B6%E5%85%89.html | msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
[2010-07-14 12:53:23] Msn      :65.55.3.195 | /tag/%E8%87%A3%E6%B0%91 | msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
[2010-07-14 12:53:30] Msn      :65.55.3.195 | /tag/%C3%A7%C2%A6%C2%BB%C3%A5%C2%BC%E2%82%AC | msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
[2010-07-14 12:53:47] Msn      :65.55.3.195 | /tag/%E7%B3%BB%E7%BB%9F | msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
[2010-07-14 12:53:52] Msn      :65.55.3.195 | /tag/%E7%B3%BB%E5%88%97 | msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
[2010-07-14 12:53:55] Msn      :65.55.3.195 | /tag/%E7%9C%9F%E8%AF%9D | msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
[2010-07-14 12:54:07] Yodao    :61.135.249.202 | / | Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )
[2010-07-14 12:54:10] Msn      :65.55.3.195 | /tag/%E7%A8%8B%E5%BC%8F | msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
[2010-07-14 12:54:17] Msn      :65.55.3.195 | /tag/%E5%BE%AE%E8%BD%AF | msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
[2010-07-14 12:55:56] Baidu    :123.125.68.116 | /tag/iframe | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 12:56:22] YandexBot:87.250.255.242 | /post/496 | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
[2010-07-14 12:56:55] Baidu    :123.125.66.25 | / | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 12:58:29] Soso     :124.115.0.167 | /sort/11 | Sosospider+(+http://help.soso.com/webspider.htm)
[2010-07-14 13:01:56] Google   :203.208.60.76 | / | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
[2010-07-14 13:02:20] Twiceler :67.218.116.163 | /tag/%E8%BF%9E%E7%BA%BF | Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)
[2010-07-14 13:03:47] Soso     :124.115.4.195 | /record/201005 | Sosospider+(+http://help.soso.com/webspider.htm)
[2010-07-14 13:05:59] YandexBot:87.250.255.242 | /tag/%E4%BC%81%E4%B8%9A | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
[2010-07-14 13:06:00] Msn      :207.46.13.42 | / | msnbot/2.0b (+http://search.msn.com/msnbot.htm)
[2010-07-14 13:07:51] Soso     :124.115.0.158 | /post/583 | Sosospider+(+http://help.soso.com/webspider.htm)
[2010-07-14 13:15:40] Google   :203.208.60.76 | /tag/%E5%9B%BD%E5%AE%B6 | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
[2010-07-14 13:15:43] YandexBot:87.250.255.242 | /post/374 | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
[2010-07-14 13:23:17] YandexBot:87.250.255.242 | /tag/%E4%B8%8B%E8%BD%BD | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
[2010-07-14 13:24:11] Soso     :124.115.0.111 | /?page=17&sort=7 | Sosospider+(+http://help.soso.com/webspider.htm)
[2010-07-14 13:30:57] Google   :203.208.60.76 | /tag/%E8%B7%AF%E5%BE%84 | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
[2010-07-14 13:34:01] YandexBot:87.250.255.242 | /post/38 | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
[2010-07-14 13:38:13] Soso     :124.115.4.201 | /post/326 | Sosospider+(+http://help.soso.com/webspider.htm)
[2010-07-14 13:40:35] Google   :203.208.60.76 | /post/322 | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
[2010-07-14 13:44:51] YandexBot:87.250.255.242 | /post/9 | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
[2010-07-14 13:52:45]Baidu    :123.125.68.125 | / | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 13:54:04] Google   :203.208.60.76 | /post/575 | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
[2010-07-14 13:55:46] YandexBot:87.250.255.242 | /?action=Rtpl&a=classic | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
[2010-07-14 13:58:52] Google   :203.208.60.76 | /tag/%E5%8E%9F%E7%BD%AA | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
[2010-07-14 14:03:10] Soso     :124.115.0.138 | /record/200912 | Sosospider+(+http://help.soso.com/webspider.htm)
[2010-07-14 14:03:23] YandexBot:87.250.255.242 | /tag/location | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
[2010-07-14 14:12:09] Baidu    :123.125.68.121 | /post/174 | Baiduspider+(+http://www.baidu.com/search/spider.htm)
[2010-07-14 14:14:35] YandexBot:87.250.255.242 | /post/298 | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
[2010-07-14 14:18:05] Google   :203.208.60.76 | /tag/%E6%98%AF%E4%B8%8D%E6%98%AF | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
[2010-07-14 14:22:22] Soso     :124.115.0.14 | /post-363.html | Sosospider+(+http://help.soso.com/webspider.htm)
[2010-07-14 14:23:19] YandexBot:87.250.255.242 | /?sort=3&page=5 | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

蜘蛛爬虫种类繁多。。  其中sogou特积极积极到另我有点反感,所以今天一整天都被我屏蔽掉了。。其他的则差不多是隔断时间就过来爬一次。。

以后去深入学习一下搜索引擎的知识。  看能不能生一只小蜘蛛。。NND··   YY  `` `