如何设置只允许网站默认页面的robot.txt [英] How to set up a robot.txt which only allows the default page of a site
问题描述
说我在 http://example.com 上有一个网站.我真的很想让机器人看到主页,但是其他任何页面都需要阻止,因为它对蜘蛛毫无意义.换句话说
Say I have a site on http://example.com. I would really like allowing bots to see the home page, but any other page need to blocked as it is pointless to spider. In other words
http://example.com &应该允许 http://example.com/,但是 http://example.com/anything 和
http://example.com & http://example.com/ should be allowed, but http://example.com/anything and http://example.com/someendpoint.aspx should be blocked.
此外,如果我可以允许某些查询字符串传递到主页,那就太好了: http://example.com?okparam=true
Further it would be great if I can allow certain query strings to passthrough to the home page: http://example.com?okparam=true
但不是 http://example.com?anythingbutokparam=true
推荐答案
因此,在进行了一些研究之后,我发现了以下内容-主要搜索提供商可以接受的解决方案: yahoo & msn(我可以在这里找到验证器):
So after some research, here is what I found - a solution acceptable by the major search providers: google , yahoo & msn (I could on find a validator here) :
User-Agent: *
Disallow: /*
Allow: /?okparam=
Allow: /$
诀窍是使用$标记URL的结尾.
The trick is using the $ to mark the end of URL.
这篇关于如何设置只允许网站默认页面的robot.txt的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!