使用队列以扭曲的方式下载文件 [英] Downloading files in twisted using queue

问题描述

我想使用扭曲和(例如)20 个客户端线程从队列中下载许多文件.有什么例子吗?

I want to download a many files from queue using twisted and (for example ) 20 clients-threads. Any example ?

推荐答案

from twisted.internet.defer import inlineCallbacks, DeferredQueue

@inlineCallbacks
def worker(queue):
    while 1:
        url = yield queue.get() # wait for a url from the queue

        if url is None: # insert None into the queue to kill workers
            queue.put(None)
            return # done

        data = yield download(url) # download the file
        process(data) # do stuff with it


queue = DeferredQueue() # your queue

# make workers
MAX = 20
workers = [worker(queue) for _ in range(MAX)] 

这篇关于使用队列以扭曲的方式下载文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！