小赖子的英国生活和资讯

修改 .htaccess 阻止不良爬虫访问

JustYY.com 小赖子的英国生活和资讯
阅读 桌面完整版

一些网络爬虫如 360 很霸道. 不管您服务器有少资源, 同时几百几千个线程并发爪取你的网站, 这样给服务器带来不少的压力.

APACHE 服务器可以通过在每个网站目录下添加一些规则来限制访问. 在根目录下加入以下规则就可以阻止大部分不良爬虫的访问 减少服务器的压力.

1
2
3
4
5

# 开始 - 阻止不良爬虫访问
SetEnvIfNoCase User-Agent "Abonti|aggregator|AhrefsBot|asterias|BDCbot|BLEXBot|BuiltBotTough|Bullseye|BunnySlippers|ca\-crawler|CCBot|Cegbfeieh|CheeseBot|CherryPicker|CopyRightCheck|cosmos|Crescent|discobot|DittoSpyder|DOC|DotBot|Download Ninja|EasouSpider|EmailCollector|EmailSiphon|EmailWolf|EroCrawler|Exabot|ExtractorPro|Fasterfox|FeedBooster|Foobot|Genieo|grub\-client|Harvest|hloader|httplib|HTTrack|humanlinks|ieautodiscovery|InfoNaviRobot|IstellaBot|Java/1\.|JennyBot|k2spider|Kenjin Spider|Keyword Density/0\.9|larbin|LexiBot|libWeb|libwww|LinkextractorPro|linko|LinkScan/8\.1a Unix|LinkWalker|LNSpiderguy|lwp\-trivial|magpie|Mata Hari|MaxPointCrawler|MegaIndex|Microsoft URL Control|MIIxpc|Mippin|Missigua Locator|Mister PiX|MJ12bot|moget|MSIECrawler|NetAnts|NICErsPRO|Niki\-Bot|NPBot|Nutch|Offline Explorer|Openfind|panscient\.com|PHP/5\.\{|ProPowerBot/2\.14|ProWebWalker|Python\-urllib|QueryN Metasearch|RepoMonkey|RMA|SemrushBot|SeznamBot|SISTRIX|sitecheck\.Internetseer\.com|SiteSnagger|SnapPreviewBot|Sogou|SpankBot|spanner|spbot|Spinn3r|suzuran|Szukacz/1\.4|Teleport|Telesoft|The Intraformant|TheNomad|TightTwatBot|Titan|toCrawl/UrlDispatcher|True_Robot|turingos|TurnitinBot|UbiCrawler|UnisterBot|URLy Warning|VCI|WBSearchBot|Web Downloader/6\.9|Web Image Collector|WebAuto|WebBandit|WebCopier|WebEnhancer|WebmasterWorldForumBot|WebReaper|WebSauger|Website Quester|Webster Pro|WebStripper|WebZip|Wotbox|wsr\-agent|WWW\-Collector\-E|Xenu|yandex|Zao|Zeus|ZyBORG|coccoc|Incutio|lmspider|memoryBot|SemrushBot|serf|Unknown|uptime files" bad_bot
SetEnvIfNoCase Referer "Abonti|aggregator|AhrefsBot|asterias|BDCbot|BLEXBot|BuiltBotTough|Bullseye|BunnySlippers|ca\-crawler|CCBot|Cegbfeieh|CheeseBot|CherryPicker|CopyRightCheck|cosmos|Crescent|discobot|DittoSpyder|DOC|DotBot|Download Ninja|EasouSpider|EmailCollector|EmailSiphon|EmailWolf|EroCrawler|Exabot|ExtractorPro|Fasterfox|FeedBooster|Foobot|Genieo|grub\-client|Harvest|hloader|httplib|HTTrack|humanlinks|ieautodiscovery|InfoNaviRobot|IstellaBot|Java/1\.|JennyBot|k2spider|Kenjin Spider|Keyword Density/0\.9|larbin|LexiBot|libWeb|libwww|LinkextractorPro|linko|LinkScan/8\.1a Unix|LinkWalker|LNSpiderguy|lwp\-trivial|magpie|Mata Hari|MaxPointCrawler|MegaIndex|Microsoft URL Control|MIIxpc|Mippin|Missigua Locator|Mister PiX|MJ12bot|moget|MSIECrawler|NetAnts|NICErsPRO|Niki\-Bot|NPBot|Nutch|Offline Explorer|Openfind|panscient\.com|PHP/5\.\{|ProPowerBot/2\.14|ProWebWalker|Python\-urllib|QueryN Metasearch|RepoMonkey|RMA|SemrushBot|SeznamBot|SISTRIX|sitecheck\.Internetseer\.com|SiteSnagger|SnapPreviewBot|Sogou|SpankBot|spanner|spbot|Spinn3r|suzuran|Szukacz/1\.4|Teleport|Telesoft|The Intraformant|TheNomad|TightTwatBot|Titan|toCrawl/UrlDispatcher|True_Robot|turingos|TurnitinBot|UbiCrawler|UnisterBot|URLy Warning|VCI|WBSearchBot|Web Downloader/6\.9|Web Image Collector|WebAuto|WebBandit|WebCopier|WebEnhancer|WebmasterWorldForumBot|WebReaper|WebSauger|Website Quester|Webster Pro|WebStripper|WebZip|Wotbox|wsr\-agent|WWW\-Collector\-E|Xenu|yandex|Zao|Zeus|ZyBORG|coccoc|Incutio|lmspider|memoryBot|SemrushBot|serf|Unknown|uptime files" bad_bot
Deny from env=bad_bot
# 结束 - 阻止不良爬虫访问
# 开始 - 阻止不良爬虫访问
SetEnvIfNoCase User-Agent "Abonti|aggregator|AhrefsBot|asterias|BDCbot|BLEXBot|BuiltBotTough|Bullseye|BunnySlippers|ca\-crawler|CCBot|Cegbfeieh|CheeseBot|CherryPicker|CopyRightCheck|cosmos|Crescent|discobot|DittoSpyder|DOC|DotBot|Download Ninja|EasouSpider|EmailCollector|EmailSiphon|EmailWolf|EroCrawler|Exabot|ExtractorPro|Fasterfox|FeedBooster|Foobot|Genieo|grub\-client|Harvest|hloader|httplib|HTTrack|humanlinks|ieautodiscovery|InfoNaviRobot|IstellaBot|Java/1\.|JennyBot|k2spider|Kenjin Spider|Keyword Density/0\.9|larbin|LexiBot|libWeb|libwww|LinkextractorPro|linko|LinkScan/8\.1a Unix|LinkWalker|LNSpiderguy|lwp\-trivial|magpie|Mata Hari|MaxPointCrawler|MegaIndex|Microsoft URL Control|MIIxpc|Mippin|Missigua Locator|Mister PiX|MJ12bot|moget|MSIECrawler|NetAnts|NICErsPRO|Niki\-Bot|NPBot|Nutch|Offline Explorer|Openfind|panscient\.com|PHP/5\.\{|ProPowerBot/2\.14|ProWebWalker|Python\-urllib|QueryN Metasearch|RepoMonkey|RMA|SemrushBot|SeznamBot|SISTRIX|sitecheck\.Internetseer\.com|SiteSnagger|SnapPreviewBot|Sogou|SpankBot|spanner|spbot|Spinn3r|suzuran|Szukacz/1\.4|Teleport|Telesoft|The Intraformant|TheNomad|TightTwatBot|Titan|toCrawl/UrlDispatcher|True_Robot|turingos|TurnitinBot|UbiCrawler|UnisterBot|URLy Warning|VCI|WBSearchBot|Web Downloader/6\.9|Web Image Collector|WebAuto|WebBandit|WebCopier|WebEnhancer|WebmasterWorldForumBot|WebReaper|WebSauger|Website Quester|Webster Pro|WebStripper|WebZip|Wotbox|wsr\-agent|WWW\-Collector\-E|Xenu|yandex|Zao|Zeus|ZyBORG|coccoc|Incutio|lmspider|memoryBot|SemrushBot|serf|Unknown|uptime files" bad_bot
SetEnvIfNoCase Referer "Abonti|aggregator|AhrefsBot|asterias|BDCbot|BLEXBot|BuiltBotTough|Bullseye|BunnySlippers|ca\-crawler|CCBot|Cegbfeieh|CheeseBot|CherryPicker|CopyRightCheck|cosmos|Crescent|discobot|DittoSpyder|DOC|DotBot|Download Ninja|EasouSpider|EmailCollector|EmailSiphon|EmailWolf|EroCrawler|Exabot|ExtractorPro|Fasterfox|FeedBooster|Foobot|Genieo|grub\-client|Harvest|hloader|httplib|HTTrack|humanlinks|ieautodiscovery|InfoNaviRobot|IstellaBot|Java/1\.|JennyBot|k2spider|Kenjin Spider|Keyword Density/0\.9|larbin|LexiBot|libWeb|libwww|LinkextractorPro|linko|LinkScan/8\.1a Unix|LinkWalker|LNSpiderguy|lwp\-trivial|magpie|Mata Hari|MaxPointCrawler|MegaIndex|Microsoft URL Control|MIIxpc|Mippin|Missigua Locator|Mister PiX|MJ12bot|moget|MSIECrawler|NetAnts|NICErsPRO|Niki\-Bot|NPBot|Nutch|Offline Explorer|Openfind|panscient\.com|PHP/5\.\{|ProPowerBot/2\.14|ProWebWalker|Python\-urllib|QueryN Metasearch|RepoMonkey|RMA|SemrushBot|SeznamBot|SISTRIX|sitecheck\.Internetseer\.com|SiteSnagger|SnapPreviewBot|Sogou|SpankBot|spanner|spbot|Spinn3r|suzuran|Szukacz/1\.4|Teleport|Telesoft|The Intraformant|TheNomad|TightTwatBot|Titan|toCrawl/UrlDispatcher|True_Robot|turingos|TurnitinBot|UbiCrawler|UnisterBot|URLy Warning|VCI|WBSearchBot|Web Downloader/6\.9|Web Image Collector|WebAuto|WebBandit|WebCopier|WebEnhancer|WebmasterWorldForumBot|WebReaper|WebSauger|Website Quester|Webster Pro|WebStripper|WebZip|Wotbox|wsr\-agent|WWW\-Collector\-E|Xenu|yandex|Zao|Zeus|ZyBORG|coccoc|Incutio|lmspider|memoryBot|SemrushBot|serf|Unknown|uptime files" bad_bot
Deny from env=bad_bot
# 结束 - 阻止不良爬虫访问

设置之后 如果 USER AGENT 是以上列表中的, 服务端就会给出一个 403 错误

403-forbidden-error

英文: How to Ban Bad Bots by User Agent in .htaccess?

强烈推荐

微信公众号: 小赖子的英国生活和资讯 JustYYUK

阅读 桌面完整版

相关文章：

  1. 可以用ChatGPT来轻松转换编程语言 阅读 桌面完整版 ChatGPT是今年最火的技术，人工智能AI，但是确实来说，是AGI，也就是通用人工智能 Artificial General Intelligence。ChatGPT可以用来做很多事情，我最近发现它可以拿来转换编程语言。比如我想学习一门新语言了，就可以把现有的程序让ChatGPT转换成新的编程语言。或者在力扣上换个新的编程语言学习一下。 下面这个例子ChatGPT转换后的代码我就可以直接在力扣上跑，就通过了所有的测试用例。 网上还有很多不同语言之间的转换工具，相信实现起来都不简单，但是如今有了ChatGPT这个通用人工智能，完全可以套个UI前端就可以实现不同语言之间的转换工具，而且转换逻辑完全不用自己实现，调用ChatGPT的API即可。 英文：Programming Language Conversion Tool based on ChatGPT...
  2. 测测你的幸运 – Linux Fortune-Teller 阅读 桌面完整版 LINUX 下有很好很好玩的命令,之前已经介绍过: figlet, rig, curl. 现在推荐另一个 命令 fortune 是用来随机显示一段(句)话的.fortune 在英文里就是幸运的意思. 这个命令可以不需要 参数 如果没有...
  3. 新的旅途 – 离别总是伤感的, 离开了一起创业的公司 阅读 桌面完整版 2周前, 正式离开了一起创业的公司, 这公司是我博士毕业后的第一份正式工作, 待了8年多了, 离别总是伤感的. 我是9月初提的离职, 三个月 Notice Period, 最后的几周交接完工作确实没有什么压力了. 11月30号, 在公司最后一天, 公司有个习惯,...
  4. GoPro运动相机可以拿来做什么? 阅读 桌面完整版 自从新年打折入了GoPro Hero+运动相机以来, 也没怎么用上, 基本上是几天的热性就和其它电子产品一样被冷落到一旁. GoPro运动相机可以拿来做什么, 主要的几个用途是: GoPro 不太适合拍人像 拍孩子孩子容易动, 特别容易就糊了, GoPRO的摄像参数可以调的很少, 所以不要期望有大光圈能背景虚化, 唯一能拍的可能是大长腿,...
  5. 按揭贷款(房贷,车贷) 每月还贷计算器 阅读 桌面完整版 去年给银行借了17万英镑 买了20万7500英镑的房子, 25年还清. 前2年是定率 Fix Rate 的合同 (年利率2.49%). 每个月大概是还 700多英镑. 有很多种还贷的计算方式, 定率/每月固定 是比较常用的....
  6. 如何通过 IFTTT 及时获得 STEEMIT上的比赛通知? 阅读 桌面完整版 @kenchung 举办了很多期数学编程竞赛, 但是有时候我发现的时候已经是好几天了, 错失了获奖良机. 有没有简单一点的办法能迅速的知道有比赛呢? 很简单, 这需要结合 IFTTT和 RSS即可. 首先, 获得比赛相关的RSS为: https://uploadbeta.com/api/steemit/wechat/feed/rss/?cached&allow_tags=contest 如果只想要参与...
  7. Minuet in C – 小步舞曲C 阅读 桌面完整版 Posted Youtube – 油管地址 孩子弹琴的时候最帅了. 我现在成了我儿子的粉丝了. Eric (Aged 6) is playing “Minuet in...
  8. 儿子问我软件工程师的工作体验是怎么样的? 阅读 桌面完整版 儿子问我软件工程师(Software Engineer)都是做什么的, 他很好奇我的工作内容, 我简单的说就是写代码+调试=解决问题. 正好那天是周五下午, 娃在上Papworth上钢琴课, 我一般都在车里剪视频利用起这个碎片时间. 我抱着笔记本在车里工作, 从年初就在忙一个大的改动, 忙了有两个多月, 终于差不多了, 两同事代码审核(Code Review)都通过了就差一些小改动,...
Exit mobile version