如何阻止不良身份不明的漫游器爬网我的网站? [英] How to block bad unidentified bots crawling my website?

问题描述

我该如何抵抗不良的身份不明的漫游器来抓取我的网站?一些名称不存在于Apache cPanel中的不良bot严重访问了我的网站带宽.

How can I resist the bad unidentified bots to crawl my website? Some bad bots whose name is not present in cPanel of Apache are badly accessing my website bandwidth.

我曾在batgap.com/robots.txt上尝试过robots.txt，但也使用.htaccess进行了阻止，但带宽使用率没有任何改善.我不知道这些漫游器的IP，因此无法通过IP地址阻止它们.这些漫游器占用了太多的站点带宽，因此，我需要从服务器上增加它.

I had tried robots.txt on batgap.com/robots.txt and also blocked with .htaccess but there is no improvement in bandwidth usage. I don't know the IP of those bots so unable to block them by IP address. These bots are consuming too much bandwidth of site and hence a result I need to increase it from server.

推荐答案

我来自Incapsula，我们会定期处理不良机器人.

I'm from Incapsula and we deal with bad bots on a regular basis.

我们最近发布了一项与机器人相关的研究，该研究提供了有关问题范围的见解(

We've recently release a bot-related research that provides insights of the scope of the problem ( http://www.incapsula.com/the-incapsula-blog/item/225-what-google-doesnt-show-you-31-of-website-traffic-can-harm-your-business ) and in light of this data I have to agree with @Leonard Challis - you simply can not handle bot protection manually.

话虽如此，但有一些机器人保护解决方案，甚至包括免费的(包括我们在内的)机器人解决方案都可以帮助您解决不良的机器人问题.

Having said that, there are bot protection solutions, even Free ones (us included) that can help you with bad bots.

顺便说一句-就像您提到的那样，不良的漫游器访问的副产品是带宽损失. 我们最近意识到，与机器人相关的巨大带宽使用确实是多么令人惊讶. 这本身就是一个有趣的话题. 我们认为，通过避免不良的漫游器流量，托管服务提供商实际上可以极大地提高其效率(希望使用它来降低成本或改善服务).一旦您想到了这种对社会和企业的影响，您就可以了解这个严重的机器人问题的真正范围，而这一问题已超出了立即造成的损害.

BTW - Just like you mentioned, one byproduct of bad bots visits is a loss of bandwidth. We`ve recently became aware of just how surprisingly HUGE bot-related bandwidth usage really is. This is an interesting topic by itself. We believe that by avoiding bad bot traffic, hosting providers can actually greatly improve their efficiency (hopefully using this to drop cost or to improve services). Once you imagine Social and Business implication of this you can understand the real scope of this bad bot problem that goes way beyond the immediate damage done.

这篇关于如何阻止不良身份不明的漫游器爬网我的网站?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！