Robots.txt:这个通配符规则有效吗? [英] Robots.txt: Is this wildcard rule valid?

问题描述

简单的问题.我想补充:

Simple question. I want to add:

Disallow */*details-print/

基本上，/foo/bar/dynamic-details-print形式的阻塞规则---本例中的foo和bar也可以是完全动态的.

Basically, blocking rules in the form of /foo/bar/dynamic-details-print --- foo and bar in this example can also be totally dynamic.

我认为这会很简单，但后来在 www.robotstxt.org 上有这样一条消息:

I thought this would be simple, but then on www.robotstxt.org there is this message:

另请注意，在 User-agent 或 Disallow 行中不支持通配符和正则表达式.User-agent 字段中的*"是一个特殊值，表示任何机器人".具体来说，你不能有像User-agent: bot"、Disallow:/tmp/*"或Disallow: *.gif"这样的行.

Note also that globbing and regular expression are not supported in either the User-agent or Disallow lines. The '*' in the User-agent field is a special value meaning "any robot". Specifically, you cannot have lines like "User-agent: bot", "Disallow: /tmp/*" or "Disallow: *.gif".

所以我们不能这样做?搜索引擎会遵守吗?但是，还有 Quora.com 的 robots.txt 文件:

So we can't do that? Do search engines abide by it? But then, there's Quora.com's robots.txt file:

Disallow: /ajax/
Disallow: /*/log
Disallow: /*/rss
Disallow: /*_POST

那么，谁是对的 -- 还是我误解了 robotstxt.org 上的文字?

So, who is right -- Or am I misunderstanding the text on robotstxt.org?

谢谢！

推荐答案

答案是视情况而定".robotstxt.org 中定义的 robots.txt标准"是机器人应支持的最低要求.Googlebot、MSNbot 和 Yahoo Slurp 支持一些常见的扩展，而其他 bot 支持什么真的不知道.有人说他们支持什么，有人不说.

The answer is, "it depends". The robots.txt "standard" as defined at robotstxt.org is the minimum that bots are expected to support. Googlebot, MSNbot, and Yahoo Slurp support some common extensions, and there's really no telling what other bots support. Some say what they support and others don't.

总的来说，您可以期待主要的搜索引擎机器人支持您编写的通配符，而您所拥有的通配符看起来会起作用.最好的办法是运行它通过一个或多个 这些 robots.txt 验证器或使用 Google 的网站管理员工具进行检查.

In general, you can expect the major search engine bots to support the wildcards that you've written, and the one you have there looks like it will work. Best bet would be to run it past one or more of these robots.txt validators or use Google's Webmaster tools to check it.

这篇关于Robots.txt:这个通配符规则有效吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！