Python多处理循环 [英] Python Multiprocessing Loop

问题描述

我希望使用multiprocessing来加速缓慢的循环.但是，从我对多处理示例的了解中，我不确定这种实现是否是好的实践，可行或可能的.

I'm hoping to use multiprocessing to speed up a sluggish loop. However, from what I've seen of multiprocessing examples, I'm not sure if this sort of implementation is good practice, feasible or possible.

循环大致分为两部分:data ingestion和data processing.我希望在处理正在进行时开始进行数据摄取的下一部分，因此数据将尽快可用.

There are broadly two parts to the loop: data ingestion and data processing. I would like to have the next part of data ingestion starting while processing is going on, so the data is available as soon as possible.

伪代码:

d = get_data(n)
for n in range(N):
    p = process_data(d)
    d = get_data(n+1) #prepare data for next process loop

1. 多处理程序是否适合这种功能?
2. 这怎么办?

提前谢谢.

推荐答案

正如您所说，多处理基本上就是分派和收集工作. 正如您所阐明的，您基本上希望process_data和get_data并行工作.

As you said, multiprocessing is basically dispatching and collecting work. And as you clarified, you basically want process_data and get_data to work in parallel.

这是我为您提供的解决方案

Here's my solution for you

import multiprocessing as mp

# create pool for dispatching work
pool = mp.Pool()

# call your functions asynchronously
process_data_process = pool.apply_async(process_data, (d,))
get_data_process = pool.apply_async(get_data, (n+1,))

# After your functions are dispatched, wait for results
process_data_result = process_data_process.get()
get_data_result = get_data_process.get()

# Note: get_data_result will not be fetched till process_data_result is ready
# But that should be fine since you can't start the next batch
# till this batch is done

您可以将其包装在循环中. 希望能回答您的问题！

And you can just wrap this in your loop. Hope that answers your question!

这篇关于Python多处理循环的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！