Python多重处理附加清单 [英] Python Multiprocessing appending list

问题描述

使用Multiprocessing.Pool()，有一个关于多个进程之间共享变量的快速问题.

Have a quick question about a shared variable between multiple processes using Multiprocessing.Pool().

如果要从多个流程中更新全局列表，是否会遇到任何问题? IE.如果有两个过程试图同时更新列表.

Will I run in to any issues if I am updating a global list from within multiple processes? I.e. if two of the processes were to try to update the list at the same time.

我看过有关使用Lock进行类似操作的文档，但我想知道是否有必要.

I have seen documentation about using a Lock for similar things but I was wondering if it was necessary.

我共享此变量的方式是在回调函数中使用全局变量， 在目标函数完成后，我将所有成功的操作附加到成功"的地方:

The way I am sharing this variable is by using a global variable in my callback function, 'successes' in which i append all of the successful actions to after the target function has completed:

TOTAL_SUCCESSES = []

def func(inputs):
    successes = []

    for input in inputs:
        result = #something with return code
        if result == 0:
            successes.append(input)
    return successes

def callback(successes):
    global TOTAL_SUCCESSES

    for entry in successes:
        TOTAL_SUCCESSES.append(entry)

def main():     
    pool = mp.Pool()
    for entry in myInputs:
         pool.apply_async(func, args=(entry,),callback=callback)         

道歉任何语法错误，很快就写下来，但是程序正在运行，只是想知道是否添加共享变量是否会出现问题.

Apologize for any syntax errors, wrote this up quickly but the program is working just wondering if I add the shared variable if I will have issues.

提前谢谢！

推荐答案

使用当前代码，您实际上并没有在进程之间共享CURRENT_SUCCESSES. callback在结果处理线程的主进程中执行.只有一个结果处理线程，因此每个callback将一次运行一次，而不是同时运行.因此，您编写的代码是进程/线程安全的.

With your current code, you're not actually sharing CURRENT_SUCCESSES between processes. callback is executed in the main process, in a result handling thread. There is only one result handling thread, so each callback will be run one at a time, not concurrently. So your code as written is process/thread safe.

但是，您忘记了要修复的从func返回的successes.

However, you are forgetting to return successes from func, which you'll want to fix.

此外，使用map可以更简洁地编写:

Also, this could be much more succinctly written using map:

def func(inputs):
    successes = []

    for input in inputs:
        result = #something with return code
        if result == 0:
            successes.append(input)
    return successes

def main():     
    pool = mp.Pool()
    total_successes = pool.map(func, myInputs) # Returns a list of lists
    # Flatten the list of lists
    total_successes = [ent for sublist in total_successes for ent in sublist]

这篇关于Python多重处理附加清单的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！