导入Keras破坏了多处理 [英] Importing Keras breaks multiprocessing

问题描述

在使用keras时，我发现我无法使用multiprocessing.Pool.经过一些故障排除后，我认为导入keras是问题的根源，并为此创建了一个简单的示例.

While using keras I found that I couldn't use multiprocessing.Pool. After some troubleshooting I think importing keras is the source of the problem and have created a simple example of this.

import keras
from multiprocessing import Pool

def foo(q,y):
    print("In foo")
    return q,y
def test(a, b):
    x = []
    if __name__ == '__main__':
        p = Pool(5)
        print("Starting")
        x = p.starmap(foo, [[a,2],[b,4]])
        print("Finished")
        p.close()
        p.join()
    print(x)

if __name__ == '__main__':
    test(1,3)

输出

> Starting

运行时，输出正在启动"，然后挂起.如果我删除了keras导入，它将正常运行并按预期输出[(1，2)，(3，4)].知道如何解决此问题或可能是什么原因吗?我还不完全了解多处理如何与python一起使用.谢谢！

When run it outputs "Starting" then hangs. If I remove the keras import it runs fine and outputs [(1, 2), (3, 4)] as expected. Any idea how I can resolve this issue or what might be causing it? I don't fully understand how multiprocessing works with python yet. Thanks!

我正在使用anaconda和spyder作为代码.

I am using anaconda and spyder for my code.

推荐答案

这是一个可能对我有用的解决方案，因为子进程不需要导入keras.

Here is a possible solution to this problem that worked for me since the child processes don't require keras to be imported.

if __name__ != '__mp_main__': #This line solves the problem
    import keras
from multiprocessing import Pool

def foo(q,y):
    print("In foo")
    return q,y
def test(a, b):
    x = []
    if __name__ == '__main__':
        p = Pool(5)
        print("Starting")
        x = p.starmap(foo, [[a,2],[b,4]])
        print("Finished")
        p.close()
        p.join()
    print(x)

if __name__ == '__main__':
    test(1,3)
    print(keras.backend)

我如何发现此解决方案很奇怪.我运行了原始代码，它像往常一样挂起.然后，即使挂起了，我也注释掉了导入行，并保存了文件以供下一次测试.我这样做后，代码就以某种方式完成了执行.我对此进行了几次测试，以确保它不是fl幸.编辑已经运行的程序的文件如何影响其操作?然后，我发现if __name__ =='__main__'可能与c中的if(fork()== 0)相似，因此我将其放在导入周围，因此子进程将不会运行它.就像我在运行时手动注释掉并保存一样.我对其进行了测试，并且有效.

How I found this solution was very weird. I ran the original code and it hung as normal. Then even as it was hung I commented out the import line and saved the file for the next test. As soon as I did this the code finished executing somehow. I tested this several times to make sure it was no fluke. How can editing the file of an already run program affect its operation? I then figured the if __name__ == '__main__' might be similar to if(fork() == 0) in c so I put it around the import so the children processes would not run it. Just as if I was manually commenting it out and saving during the run time. I tested it and it works.

如果使用__name__ == __main__，则继承会中断，因为它们不是主要的. mp_main 是池工作程序的名称，它也可以正常工作

If you use if __name__ == __main__ then inheritance gets broken since they aren't main it seems. mp_main is the name of the pool workers and this works as well

这篇关于导入Keras破坏了多处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！