阻止滥用机器人爬行? [英] Stop abusive bots from crawling?

问题描述

这是个好主意吗??

http://browsers.garykeith.com/stream.asp?RobotsTXT

滥用爬行是什么意思?这对我的网站有什么影响?

What does abusive crawling mean? How is that bad for my site?

推荐答案

并非如此.无论如何，大多数坏机器人"都会忽略 robots.txt 文件.

Not really. Most "bad bots" ignore the robots.txt file anyway.

滥用爬行通常意味着抓取.这些机器人会出现以获取电子邮件地址或更常见的内容.

Abuse crawling usually means scraping. These bots are showing up to harvest email addresses or more commonly, content.

至于如何阻止他们?这真的很棘手，而且往往不明智.反爬虫技术往往不完美，会给普通人带来问题.

As to how you can stop them? That's really tricky and often not wise. Anti-crawl techniques have a tendency to be less than perfect and cause problems for regular humans.

遗憾的是，就像零售业的缩水"一样，这是在网络上开展业务的成本.

Sadly, like "shrinkage" in retail, it's a cost of doing business on the web.

这篇关于阻止滥用机器人爬行?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！