在Python中进行多处理后的后处理结果 [英] Post-processing results after multi-processing in Python

问题描述

因此，我有一个简单的MP代码，它的工作原理就像一个魅力.但是，当我对通过MP生成的数据进行非常简单的后处理时，该代码将不再起作用.它永远不会停止并且永远运行！这是代码(同样可以正常工作):

So I have a simple MP code and it works like a charm. However, when I do a very simple post processing on the data generated via MP, the code does not work anymore. It never stops and runs like forever! This is the code (and again it works perfectly):

import numpy as np
from multiprocessing import Pool

n = 4
nMCS = 10**5

def my_function(j):
    result = []
    for j in range(nMCS // n):
        a = np.random.rand(10,2)
        result.append(a) 
    return result

if __name__ == '__main__':
    __spec__ = "ModuleSpec(name='builtins', loader=<class '_frozen_importlib.BuiltinImporter'>)" # this is because I am using Spyder!

    pool = Pool(processes = n) 

    data = pool.map(my_function, [i for i in range(n)])

    pool.close()
    pool.join()

#final_result = np.concatenate(data)   ### this is what ruins my code! ###

同时，如果我在末尾添加final_result = np.concatenate(data)，它将永远无法工作！我正在使用 Spyder ，如果我在完成MP之后简单地在控制台中键入final_result = np.concatenate(data)，它会给我我想要的内容，即连接列表.但是，如果我最后在主程序中放了那条简单的线，那就行不通了.谁能告诉我如何解决这个问题?

Meanwhile, if I add final_result = np.concatenate(data) at the end, it never works! I am using Spyder and if I simply type final_result = np.concatenate(data) in the console AFTER MP is done, it gives me what I want i.e. a concatenated list. However, if I put that simple line in the main program at the very end, it just doesn't work. Could anyone tell me how to fix this?

P.S.这是我生成的一个非常简单的示例，因此您可以了解发生了什么.我真正的问题是方法更加复杂，用完MP后再也无法进行后期处理.

P.S. this is a very simple example I generated so you can understand what is going on; my real problem is way more complicated and there is no way I can do post processing after I am done with MP.

推荐答案

正如@Ares所暗示的那样，您可以通过将if __name__ == "__main__"语句以南的所有内容缩进if块来解决此问题.

As @Ares already implied, you fix the problem by indenting everything south the if __name__ == "__main__"-statement into the if-block.

仅供参考，这是在Windows上发生的，Windows不提供启动Unix-y系统等新进程的功能，而是使用"spawn"作为默认(唯一)启动方法. Spawn意味着，操作系统必须为每个工作进程从头开始使用解释器来启动新进程.

FYI, this happens on Windows which doesn't provide forking for starting up new processes like Unix-y systems, but uses 'spawn' as default (and only) start-method. Spawn means, the OS has to boot a new process with an interpreter from scratch for every worker-process.

您的工作进程将需要导入目标函数my_function.发生这种情况时，if __name__ == "__main__": -block中不受保护的所有内容也将在导入时在每个子进程中运行.

Your worker-processes will need to import your target function my_function. When this happens, everything not protected within the if __name__ == "__main__":-block will also run in every child-process on import.

这篇关于在Python中进行多处理后的后处理结果的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！