创建网络爬虫时的关键考虑因素是什么? [英] What are the key considerations when creating a web crawler?

问题描述

我今天刚开始考虑创建/定制一个网络爬虫，对网络爬虫/机器人礼仪知之甚少.我发现的大多数有关礼仪的文章都显得陈旧而笨拙，因此我想从网络开发者社区中获得一些当前(和实用)的见解.

I just started thinking about creating/customizing a web crawler today, and know very little about web crawler/robot etiquette. A majority of the writings on etiquette I've found seem old and awkward, so I'd like to get some current (and practical) insights from the web developer community.

为了一个超级简单的目的，我想使用爬虫遍历网络"——站点 XYZ 的标记是否满足条件 ABC?".

I want to use a crawler to walk over "the web" for a super simple purpose - "does the markup of site XYZ meet condition ABC?".

这给我带来了很多问题，但我认为我需要首先解决的两个主要问题是:

This raises a lot of questions for me, but I think the two main questions I need to get out of the way first are:

• 从一开始就感觉有点不确定"——这种事情可以接受吗?
• 为了不让人们感到不安，抓取工具应采取哪些具体考虑?

推荐答案

遵守 robots.txt(不要像已经说过的那样过于激进).

Obey robots.txt (and not too aggressive like has been said already).

您可能需要考虑一下您的用户代理字符串 - 它们是预先了解您正在做什么以及如何与您联系的好地方.

You might want to think about your user-agent string - they're a good place to be up-front about what you're doing and how you can be contacted.

这篇关于创建网络爬虫时的关键考虑因素是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！