在Django应用程序中，robots.txt的建议伪指令是什么? [英] What are recommended directives for robots.txt in a Django application?

问题描述

目前，我的django项目具有以下结构.

Currently my django project has following structure.

./
../ 
app1/
app2/
django_project
manage.py
media
static
secret_stuff

和我的 robots.txt 看起来像这样:

User-agent: *
Allow: /
Sitemap: mysite.com/sitemaps.xml

我想知道以下几点:

1. 我应该添加到我的 robots.txt 中的推荐指令是什么?文件，因为Django文档对此主题一无所知.

1. What are the recommend directives should i add to my robots.txt file, as django documentation is saying nothing on this topic.

如何阻止漫游器到达(编入索引)其中的内容 secret_stuff 和 mysite.com/admin/目录?

How do i stop bots from reaching (indexing) contents of secret_stuff and mysite.com/admin/ directory ?

  Disallow: /secret_stuff      (Is that okay ?)
  Disallow: /admin            (Is that okay ?)

推荐答案

机器人指令与Django框架无关，这就是为什么您无法在Django文档中找到有关它的任何信息的原因.通常情况下，您可以决定在您的网站上允许什么以及不允许什么进行搜索.

Robots directives are not related to Django framework that is why you won't find any information about it in Django docs. Normally, it is up to you what to allow and what to disallow for searching on you website.

有多种方法可以将robots.txt包含在Django项目中.我个人使用的是 django-robots 应用程序，该应用程序简化了将robots.txt嵌入到项目中的方式.

There are different ways to include robots.txt into the Django project. I am personally using django-robots app which simplifies the way you can embed robots.txt into you project.

不必在每个项目中都使用它.如果您觉得更简单，则可以自己渲染txt文件.

It is not necessary to use it in every project. If you find it simpler you can just render txt file by yourself.

我为Django项目简化的robots.txt如下:

My simplified robots.txt for Django project looks like:

User-agent: *
Disallow: /*.pdf
Disallow: /*.ppt
Disallow: /*.doc
Disallow: /*.xls
Disallow: /*.txt

User-agent: Yandex
Allow: /events
Allow: /contests
Allow: /schools
Disallow: /admin
Crawl-delay: 3

User-agent: Googlebot
Allow: /events
Allow: /contests
Allow: /schools
Disallow: /admin
Crawl-delay: 3

Host: https://mysite.ru
Sitemap: https://mysite.ru/sitemap.xml

这篇关于在Django应用程序中，robots.txt的建议伪指令是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！