在爬虫蜘蛛中使用线程 [英] Using threads within a scrapy spider

问题描述

是否可以在爬虫蜘蛛中使用多个线程?例如，假设我构建了一个爬虫，它可以抓取博客主题并保存其中的所有消息.我想将每个主题耦合到池中的一个线程，该线程将抓取所有需要的信息.每个线程都会以这种方式抓取不同的主题..

Is it possible to use multiple threads within a scrapy spider? For example lets say i have built a spider which crawl blog topics and saves all the messages within. I would like to couple every topic to a thread from a pool, and the thread will crawl all the needed information. Each thread will crawl a different topic that way..

推荐答案

Scrapy 本身是单线程的，因此您不能在一个蜘蛛中使用多个线程.但是，您可以同时使用多个蜘蛛 (CONCURRENT_REQUESTS)，这可能对您有所帮助 (参见常见做法)

Scrapy itself is single-threaded, and resultantly you cannot use multiple threads within a spider. You can however, make use of multiple spiders at the same time (CONCURRENT_REQUESTS), which may help you (see Common Practices)

Scrapy 不使用多线程，因为它建立在 Twisted 之上，这是一个异步 http 框架.

Scrapy does not use multithreading as it is built on Twisted, which is an asynchronous http framework.

这篇关于在爬虫蜘蛛中使用线程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！