具有多处理，线程或Asyncio的多任务，具体取决于方案 [英] Multi-tasking with Multiprocessing or Threading or Asyncio, depending on the Scenario

问题描述

我的代码一次可以准备1个，我想将其升级为精美的多任务处理程序.我正在寻求有关我可以用来实现目标的帮助.

I have my codes ready for 1 at a time performance, I wanna upgrade it to something fancy, multi-tasking. I am seeking helps about what I can use to achieve my goal.

我的代码按以下顺序执行:解析多页，解析多帖子，解析多图像.我试图用pool.map()进行多页处理，这是Daemonic的KeyError不能有子进程的结果.我对这种多任务处理程序的理解是，解析页面速度很快，解析帖子和图像的时间可能非常长.

my codes performs in this order: parsing multi-pages, parsing multi-posts, parsing multi-images. I tried to do multi-pages with multi-processing with pool.map(), it came out with KeyError of Daemonic can't have children processes. My understanding of this multi-tasking procedure is that parsing pages are fast, parsing posts and images can be really long.

如果我在一页上同时解析帖子和图片，该怎么办?

What if I do parsing posts and parsing images together on single page, can it be allowed?

我应该使用哪个模块?线?多处理?异步吗?最近，我经历了很多事情，我在为应该使用的东西而苦苦挣扎.

Which modules should i use to do so? thread? multiprocessing? asyncio? I went through a lot lately, I am struggling with what I should use.

推荐答案

因此，从我的头顶上可以看到两件事.

So off the top of my head you can look at 2 things.

1)Asyncio(请注意，此示例使用线程化，并且不是线程安全的，特别是asyncio.gather函数)

1) Asyncio (be careful this example uses threading and is not thread safe specifically the function asyncio.gather)

import asyncio
for work in [1,2,3,4,5]:
    tasks.append(method_to_be_called(work))

results = await asyncio.gather(*tasks)

2)Asyncio +多处理 https://github.com/jreese/aiomultiprocess

2) Asyncio + multiprocessing https://github.com/jreese/aiomultiprocess

这篇关于具有多处理，线程或Asyncio的多任务，具体取决于方案的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！