如何防止 Googlebot 淹没网站? [英] How to prevent Googlebot from overwhelming site?

问题描述

我在中间的专用服务器上运行一个内容丰富但流量很少的网站.

I'm running a site with a lot of content, but little traffic, on a middle-of-the-road dedicated server.

有时，Googlebot 会踩踏我们，导致 Apache 将其内存用尽，并导致服务器崩溃.

Occasionally, Googlebot will stampede us, resulting in Apache maxing out its memory, and causing the server to crash.

我怎样才能避免这种情况?

How can I avoid this?

推荐答案

• 在谷歌网站管理员工具上注册，验证您的网站并限制谷歌机器人
• 提交站点地图
• 阅读谷歌准则:(if-Modified-Since HTTP 标头)
• 使用robot.txt来限制从机器人到网站某些部分的访问
• 制作一个脚本，每$[一段时间]更改robot.txt，以确保机器人永远不会同时抓取太多页面，同时确保它可以抓取所有内容
• register at google webmaster tools, verify your site and throttle google bot down
• submit a sitemap
• read the google guildelines: (if-Modified-Since HTTP header)
• use robot.txt to restrict access from to bot to some parts of the website
• make a script that changes the robot.txt each $[period of time] to make sure the bot is never able to crawl too many pages at the same time while making sure it can crawl all the content overall

这篇关于如何防止 Googlebot 淹没网站?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！