Python多重处理附加清单 [英] Python Multiprocessing appending list
问题描述
使用Multiprocessing.Pool(),有一个关于多个进程之间共享变量的快速问题.
Have a quick question about a shared variable between multiple processes using Multiprocessing.Pool().
如果要从多个流程中更新全局列表,是否会遇到任何问题? IE.如果有两个过程试图同时更新列表.
Will I run in to any issues if I am updating a global list from within multiple processes? I.e. if two of the processes were to try to update the list at the same time.
我看过有关使用Lock进行类似操作的文档,但我想知道是否有必要.
I have seen documentation about using a Lock for similar things but I was wondering if it was necessary.
我共享此变量的方式是在回调函数中使用全局变量, 在目标函数完成后,我将所有成功的操作附加到成功"的地方:
The way I am sharing this variable is by using a global variable in my callback function, 'successes' in which i append all of the successful actions to after the target function has completed:
TOTAL_SUCCESSES = []
def func(inputs):
successes = []
for input in inputs:
result = #something with return code
if result == 0:
successes.append(input)
return successes
def callback(successes):
global TOTAL_SUCCESSES
for entry in successes:
TOTAL_SUCCESSES.append(entry)
def main():
pool = mp.Pool()
for entry in myInputs:
pool.apply_async(func, args=(entry,),callback=callback)
道歉任何语法错误,很快就写下来,但是程序正在运行,只是想知道是否添加共享变量是否会出现问题.
Apologize for any syntax errors, wrote this up quickly but the program is working just wondering if I add the shared variable if I will have issues.
提前谢谢!
推荐答案
使用当前代码,您实际上并没有在进程之间共享CURRENT_SUCCESSES
. callback
在结果处理线程的主进程中执行.只有一个结果处理线程,因此每个callback
将一次运行一次,而不是同时运行.因此,您编写的代码是进程/线程安全的.
With your current code, you're not actually sharing CURRENT_SUCCESSES
between processes. callback
is executed in the main process, in a result handling thread. There is only one result handling thread, so each callback
will be run one at a time, not concurrently. So your code as written is process/thread safe.
但是,您忘记了要修复的从func
返回的successes
.
However, you are forgetting to return successes
from func
, which you'll want to fix.
此外,使用map
可以更简洁地编写:
Also, this could be much more succinctly written using map
:
def func(inputs):
successes = []
for input in inputs:
result = #something with return code
if result == 0:
successes.append(input)
return successes
def main():
pool = mp.Pool()
total_successes = pool.map(func, myInputs) # Returns a list of lists
# Flatten the list of lists
total_successes = [ent for sublist in total_successes for ent in sublist]
这篇关于Python多重处理附加清单的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!