Keras + Tensorflow和Python中的多处理 [英] Keras + Tensorflow and Multiprocessing in Python

问题描述

我将Tensorflow作为后端使用Keras.

I'm using Keras with Tensorflow as backend.

我试图在主流程中保存模型，然后在另一个流程中加载/运行(即调用model.predict).

I am trying to save a model in my main process and then load/run (i.e. call model.predict) within another process.

我目前正尝试从文档中使用天真的方法来保存/加载模型:

I'm currently just trying the naive approach from the docs to save/load the model: https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model.
So basically:

1. model.save()在主要过程中
2. model = load_model()在子进程中
3. model.predict()在子进程中

1. model.save() in main process
2. model = load_model() in child process
3. model.predict() in child process

但是，它只是挂在load_model调用上.

However, it simply hangs on the load_model call.

四处寻找，我发现了这个可能相关的答案，表明Keras只能在一个过程中使用:使用带有theano的多处理程序，但是不确定这是否是正确的(似乎找不到很多东西).

Searching around I've discovered this potentially related answer suggesting that Keras can only be utilized in one process: using multiprocessing with theano but am unsure if this is true (can't seem to find much on this).

有没有一种方法可以实现我的目标?高度赞赏的描述或简短的示例.

Is there a way to accomplish my goal? A high level description or short example is greatly appreciated.

注意:我已经尝试过将图形传递给流程的方法，但是失败了，因为它似乎无法选择tensorflow图(相关的SO帖子在这里:

Note: I've attempted approaches along the lines of passing a graph to the process but failed since it seems tensorflow graphs aren't pickable (related SO post for that here: Tensorflow: Passing a session to a python multiprocess). If there is indeed a way to pass the tensorflow graph/model to the child process then I am open to that as well.

谢谢！

推荐答案

根据我的经验-问题在于将Keras加载到一个进程中，然后在keras加载到您的主环境中时生成一个新进程.但是对于某些应用程序(例如训练Keras模型的混合)，最好将所有这些都集中在一个过程中.因此，我建议您采取以下方法(有点麻烦-但为我工作):

From my experience - the problem lies in loading Keras to one process and then spawning a new process when the keras has been loaded to your main environment. But for some applications (like e.g. training a mixture of Kerasmodels) it's simply better to have all of this things in one process. So what I advise is the following (a little bit cumbersome - but working for me) approach:

1. 请勿将KERAS加载到您的主要环境中.如果要加载Keras/Theano/TensorFlow，请仅在功能环境中加载.例如. 不要这样做:

1. DO NOT LOAD KERAS TO YOUR MAIN ENVIRONMENT. If you want to load Keras / Theano / TensorFlow do it only in the function environment. E.g. don't do this:

import keras

def training_function(...):
    ...

但是请执行以下操作:

def training_function(...):
    import keras
    ...

在单独的过程中运行与每个模型相关的工作:我通常会创建正在从事这项工作的工作人员(例如培训，调整，评分)，然后运行他们在单独的过程中.很好的是，此过程完成后，此过程使用的全部内存已完全释放.这可以帮助您解决许多内存问题，这些问题通常是在使用多处理甚至在一个进程中运行多个模型时遇到的.所以看起来例如像这样:

Run work connected with each model in a separate process: I'm usually creating workers which are making the job (like e.g. training, tuning, scoring) and I'm running them in separate processes. What is nice about it that whole memory used by this process is completely freed when your process is done. This helps you with loads of memory problems which you usually come across when you are using multiprocessing or even running multiple models in one process. So this looks e.g. like this:

def _training_worker(train_params):
    import keras
    model = obtain_model(train_params)
    model.fit(train_params)
    send_message_to_main_process(...)

def train_new_model(train_params):
    training_process = multiprocessing.Process(target=_training_worker, args = train_params)
    training_process.start()
    get_message_from_training_process(...)
    training_process.join()

不同的方法只是为不同的模型动作准备不同的脚本.但这可能会导致内存错误，尤其是在模型消耗内存的情况下. 注意，由于这个原因，最好严格执行顺序执行.

Different approach is simply preparing different scripts for different model actions. But this may cause memory errors especially when your models are memory consuming. NOTE that due to this reason it's better to make your execution strictly sequential.

这篇关于Keras + Tensorflow和Python中的多处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！