使用 htaccess 阻止一个特殊目录的所有机器人/爬虫/蜘蛛 [英] Block all bots/crawlers/spiders for a special directory with htaccess

问题描述

我正在尝试阻止一个特殊目录的所有机器人/爬虫/蜘蛛.我怎样才能用 htaccess 做到这一点?我搜索了一下，通过基于用户代理的阻塞找到了解决方案:

I'm trying to block all bots/crawlers/spiders for a special directory. How can I do that with htaccess? I searched a little bit and found a solution by blocking based on the user agent:

RewriteCond %{HTTP_USER_AGENT} googlebot

现在我需要更多的用户代理(对于所有已知的机器人)并且规则应该只对我的单独目录有效.我已经有一个 robots.txt，但不是所有的爬虫都会查看它......通过 IP 地址阻止不是一个选项.或者有其他解决方案吗?我知道密码保护，但我必须先问问这是否是一个选项.尽管如此，我还是在寻找基于用户代理的解决方案.

Now I would need more user agents (for all bots known) and the rule should be only valid for my separate directory. I have already a robots.txt but not all crawlers take a look at it ... Blocking by IP address is not an option. Or are there other solutions? I know the password protection but I have to ask first if this would be an option. Nevertheless, I look for a solution based on the user agent.

推荐答案

您需要启用 mod_rewrite.将它放在该文件夹中的 .htaccess 中.如果放在其他地方(例如父文件夹)，则需要稍微修改 RewriteRule 模式以包含该文件夹名称).

You need to have mod_rewrite enabled. Placed it in .htaccess in that folder. If placed elsewhere (e.g. parent folder) then RewriteRule pattern need to be slightly modified to include that folder name).

RewriteEngine On

RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot|Baiduspider) [NC]
RewriteRule .* - [R=403,L]

1. 我只输入了几个机器人——你自己添加任何其他机器人(字母大小写无关紧要).
2. 对于此类请求，此规则将以403 Access Forbidden"结果代码进行响应.如果您确实需要，您可以更改为另一个响应 HTTP 代码(考虑到您的要求，此处 403 最合适).

这篇关于使用 htaccess 阻止一个特殊目录的所有机器人/爬虫/蜘蛛的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！