Tensorflow:将会话传递给 python 多进程 [英] Tensorflow: Passing a session to a python multiprocess

问题描述

我正在使用 tensorflow 来预处理一些大图像.我遇到了内存迅速崩溃的问题.我转而在 python 中使用多处理，这样我就可以随时完全释放内存.

I'm using tensorflow to preprocess some large images. I was having a problem where the memory was rapidly collapsing. I turned to use multiprocessing in python so the memory would free up entirely whenever I want.

问题是，我正在使用 python 的多进程队列，由于某种未知的原因，我无法将我的 tensorflow 会话从我的父进程传递给子进程.使用一些高级调试技术(即每隔几行打印一次)，我注意到 python 只是在我使用会话的行内闲置，它不会抛出错误消息.

The thing is, I'm using python's multiprocess queues and for some reason unknown I can't pass my tensorflow session from my parent process to the children. Using some advanced debugging techniques (i.e. printing something every few lines) I noticed that python just goes idle inside the line where I make use of the session, it doesn't throw an error message.

我的代码如下所示:

def subprocess(some_image, sess, q):
    with sess.as_default():
        # ... use sess and q ...
        print "All good and well" #This is printed
        some_image.eval() #Nothing happens here in console
        print "Still all good and well" #This is not printed

if __name__ == '__main__':
    # ... some initial operations ...
    some_image = read_some_image()

    sess = tf.Session()

    q = Queue()
    q.put(something)
    p = Process(target=subprocess, args=(some_image, sess, q))
    p.start()
    p.join()

可能是什么问题?非常感谢！

What could be the problem? Many thanks!

推荐答案

我不认为你可以分享 "state" 就像在这样的进程之间的 tf.Session() 中一样.我认为每个进程都需要自己的会话.

I don't think you can share "state" as in the tf.Session() between processes like that. I would think that each process needed it's own session.

这篇关于Tensorflow:将会话传递给 python 多进程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！