scrapy 允许所有域 [英] scrapy allow all domains

问题描述

我看到了这篇帖子来制作scrapy在没有域限制的情况下抓取任何网站.

I saw this post to make scrapy crawl any site without allowed domains restriction.

是否有更好的方法，例如在允许的域变量中使用正则表达式，例如-

Is there any better way of doing it, such as using a regular expression in allowed domains variable, like-

allowed_domains = ["*"]

我希望除了侵入scrapy框架之外还有其他方法可以做到这一点.

I hope there is some other way than hacking into scrapy framework to do this.

推荐答案

根本不要设置 allowed_domains.

Don't set allowed_domains at all.

看看这个scrapy文件中的get_host_regex()函数:

Look at the get_host_regex() function in this scrapy file:

https://github.com/scrapy/scrapy/blob/master/scrapy/contrib/spidermiddleware/offsite.py

这篇关于scrapy 允许所有域的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！