python写爬虫如何实现任务队列？

问题描述

问 题

比如说我的爬虫要爬行http://xxx.com/page.php?id=1到http://xxx.com/page.php?id=88的数据，但是由于机器限制我只能开十个线程，那么我怎么做到把这88个抓取任务分配给十个线程，并且以最快的速度完成任务？（Ps：我不要用框架，我想自己原生实现）

解决方案

至少三种方法及相应的参考实现：

• multiprocessing.dummy:

from multiprocessing.dummy import Pool as ThreadPool


def worker(n):
    return n + 2

numbers = range(100)

pool = ThreadPool(processes=10)
result = pool.map(worker, numbers)
pool.close()
pool.join()
print(result)

• concurrent.futures.Executor:

import concurrent.futures


def worker(n):
    return n + 2

numbers = range(100)

with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
    result = executor.map(worker, numbers)
    print(list(result))

• queue + Threading:

from collections import deque
import queue
import threading


def do_work(n):
    return n + 2


def worker():
    while True:
        item = q.get()
        if item is None:
            break
        result.append(do_work(item))
        q.task_done()

q = queue.Queue()
result = deque()
num_worker_threads = 10
threads = []
for i in range(num_worker_threads):
    t = threading.Thread(target=worker)
    t.start()
    threads.append(t)

for item in range(100):
    q.put(item)

# block until all tasks are done
q.join()

# stop workers
for i in range(num_worker_threads):
    q.put(None)
for t in threads:
    t.join()

print(result)

官方文档：

• 17.2. multiprocessing — Process-based parallelism — Python 3.5.2 documentation

• 17.4. concurrent.futures — Launching parallel tasks — Python 3.5.2 documentation

• 17.7. queue — A synchronized queue class — Python 3.5.2 documentation

• 17.1. threading — Thread-based parallelism — Python 3.5.2 documentation

这篇关于python写爬虫如何实现任务队列？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！