Scrapy 是单线程还是多线程? [英] is Scrapy single-threaded or multi-threaded?

问题描述

Scrapy 中的并发设置很少，例如 CONCURRENT_REQUESTS.这是否意味着 Scrapy 爬虫是多线程的?因此，如果我运行 scrapy crawl my_crawler 它实际上会并行触发多个并发请求?我问是因为，我读过 Scrapy 是单线程的.

There are few concurrency settings in Scrapy, like CONCURRENT_REQUESTS. Does it mean, that Scrapy crawler is multi-threaded? So if I run scrapy crawl my_crawler it will literally fire multiple simultaneous requests in parallel? Im asking because, I've read that Scrapy is single-threaded.

推荐答案

Scrapy 是单线程的，除了交互式 shell 和一些测试，参见 来源.

Scrapy is single-threaded, except the interactive shell and some tests, see source.

它建立在 Twisted 之上，它也是单线程的，并利用了它有自己的异步并发功能，例如 twisted.internet.interfaces.IReactorThreads.callFromThread，见来源.

It's built on top of Twisted, which is single-threaded too, and makes use of it's own asynchronous concurrency capabilities, such as twisted.internet.interfaces.IReactorThreads.callFromThread, see source.

这篇关于Scrapy 是单线程还是多线程?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！