在允许的内容之间阻止文件夹 [英] Blocking folders inbetween allowed content

问题描述

我有一个具有以下结构的网站:

I have a site with the following structure:

http://www.example.com/folder1/folder2/folder3

我想禁止在 folder1 和 folder2 中建立索引.但我希望机器人能够索引 folder3 下的所有内容.

I would like to disallow indexing in folder1, and folder2. But I would like the robots to index everything under folder3.

有没有办法用 robots.txt 做到这一点?

Is there a way to do this with the robots.txt?

对于我读到的内容，我认为指定文件夹中的所有内容都是不允许的.

For what I read I think that everything inside a specified folder is disallowed.

以下能达到我的目标吗?

Would the following achieve my goal?

user-agent: *
Crawl-delay: 0

Sitemap: <Sitemap url>

Allow: /folder1/folder2/folder3
Disallow: /folder1/folder2/
Disallow: /folder1/
Allow: /

推荐答案

是的，它有效...但是 google 有一个工具可以测试您的 robots.txt 文件

Yes, it works... however google has a tool to test your robots.txt file

您只需要使用 google 网站管理员工具 (https://www.google.com/webmasters/工具/)

you only need to go on google webmaster tools (https://www.google.com/webmasters/tools/)

并打开站点配置->爬虫访问"部分

and open the section "site configuration -> crawler access"

这篇关于在允许的内容之间阻止文件夹的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！