不同线程中的预测和训练Keras Tensorflow [英] Predicting and Training in different threads Keras Tensorflow

问题描述

我正在使用Keras和Tensorflow进行某种在线学习，我会定期接收新数据，并使用这些新数据重新训练模型.我可以在".h5"文件中存储几个模型，以便在需要训练或预测时加载模型，然后执行必要的操作.

I am using Keras and Tensorflow to make a kind-of online learning, where I receive new data periodically and I retrain my models with this new data. I can have several models stored in ".h5" files so that when i need to train or predict I load the model and then I perform the necessary operations.

当前，我将训练和预测分为两个不同的线程，以便可以在其他线程训练时进行预测.对于锁，我尝试确保在同一模型中不会同时进行任何预测或训练(我认为这是可行的)，但是我知道keras对此并不准备.关于张量流的图形或会话，我总是会出现一些不同的错误，例如:

Currently I separated the training and the predictions in two different threads, so that predictions can be made while the other thread trains. With locks I try to make sure that no prediction or training is done in the same model at the same time (I think this works), but I am aware that keras is not so prepared for this. I always some different errors regarding the graph or session of tensorflow, for instance:

回溯(最近通话最近): wsgi_app中的文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ site-packages \ flask \ app.py"，第2292行 响应= self.full_dispatch_request() 在full_dispatch_request中的第1815行，文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ site-packages \ flask \ app.py" rv = self.handle_user_exception(e) 在handle_user_exception中的第1718行，文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ site-packages \ flask \ app.py" 加价(exc_type，exc_value，tb) 重新列出文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ site-packages \ flask_compat.py"，第35行 提高价值 在full_dispatch_request中，文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ site-packages \ flask \ app.py"，行1813 rv = self.dispatch_request() 在dispatch_request中的文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ site-packages \ flask \ app.py"，行1799 返回self.view_functionsrule.endpoint 文件"C:\ Users \ a703572 \ PycharmProjects \ ai-pred-eng \ src \ run_keras_server.py"，行859，在predict_times中 0] +'.h5') get_prediction中的文件"C:\ Users \ a703572 \ PycharmProjects \ ai-pred-eng \ src \ run_keras_server.py"，第164行 模型，缩放比例= self.load_model_file(self.graph_pred，self.session，路径) 在load_model_file中的文件"C:\ Users \ a703572 \ PycharmProjects \ ai-pred-eng \ src \ run_keras_server.py"，第114行 型号= load_model(路径) 在load_model中的文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ site-packages \ keras \ engine \ saving.py"，行419 模型= _deserialize_model(f，custom_objects，compile) _deserialize_model中的第287行"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ site-packages \ keras \ engine \ saving.py" K.batch_set_value(weight_value_tuples) 文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ site-packages \ keras \ backend \ tensorflow_backend.py"，行2470，位于batch_set_value中 get_session().run(assign_ops，feed_dict = feed_dict) get_session中的第206行的文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ site-packages \ keras \ backend \ tensorflow_backend.py" session.run(tf.variables_initializer(uninitialized_vars)) 文件"C:\ Users \ a703572 \ AppData \ Roaming \ Python \ Python36 \ site-packages \ tensorflow \ python \ ops \ variables.py"，第2831行，位于variables_initializer中 返回control_flow_ops.group(* [在var_list中v的v.initializer]，名称=名称) 组中的文件"C:\ Users \ a703572 \ AppData \ Roaming \ Python \ Python36 \ site-packages \ tensorflow \ python \ ops \ control_flow_ops.py"，行3432 返回_GroupControlDeps(dev，deps，name = name) _GroupControlDeps中的第3384行文件"C:\ Users \ a703572 \ AppData \ Roaming \ Python \ Python36 \ site-packages \ tensorflow \ python \ ops \ control_flow_ops.py" 返回no_op(name = name) 退出中的文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ contextlib.py"，第88行 下一个(self.gen) 设备中的文件"C:\ Users \ a703572 \ AppData \ Roaming \ Python \ Python36 \ site-packages \ tensorflow \ python \ framework \ ops.py"，行4249 self._device_function_stack.pop_obj() 文件"C:\ Users \ a703572 \ AppData \ Roaming \ Python \ Python36 \ site-packages \ tensorflow \ python \ framework \ traceable_stack.py"，行110，在pop_obj中 返回self._stack.pop().obj IndexError:从空列表中弹出

Traceback (most recent call last): File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\site-packages\flask\app.py", line 2292, in wsgi_app response = self.full_dispatch_request() File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\site-packages\flask\app.py", line 1815, in full_dispatch_request rv = self.handle_user_exception(e) File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\site-packages\flask\app.py", line 1718, in handle_user_exception reraise(exc_type, exc_value, tb) File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\site-packages\flask_compat.py", line 35, in reraise raise value File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\site-packages\flask\app.py", line 1813, in full_dispatch_request rv = self.dispatch_request() File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\site-packages\flask\app.py", line 1799, in dispatch_request return self.view_functionsrule.endpoint File "C:\Users\a703572\PycharmProjects\ai-pred-eng\src\run_keras_server.py", line 859, in predict_times 0] + '.h5') File "C:\Users\a703572\PycharmProjects\ai-pred-eng\src\run_keras_server.py", line 164, in get_prediction model, scaler = self.load_model_file(self.graph_pred, self.session, path) File "C:\Users\a703572\PycharmProjects\ai-pred-eng\src\run_keras_server.py", line 114, in load_model_file model = load_model(path) File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\engine\saving.py", line 419, in load_model model = _deserialize_model(f, custom_objects, compile) File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\engine\saving.py", line 287, in _deserialize_model K.batch_set_value(weight_value_tuples) File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\backend\tensorflow_backend.py", line 2470, in batch_set_value get_session().run(assign_ops, feed_dict=feed_dict) File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\backend\tensorflow_backend.py", line 206, in get_session session.run(tf.variables_initializer(uninitialized_vars)) File "C:\Users\a703572\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\variables.py", line 2831, in variables_initializer return control_flow_ops.group(*[v.initializer for v in var_list], name=name) File "C:\Users\a703572\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\control_flow_ops.py", line 3432, in group return _GroupControlDeps(dev, deps, name=name) File "C:\Users\a703572\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\control_flow_ops.py", line 3384, in _GroupControlDeps return no_op(name=name) File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\contextlib.py", line 88, in exit next(self.gen) File "C:\Users\a703572\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\ops.py", line 4249, in device self._device_function_stack.pop_obj() File "C:\Users\a703572\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\traceable_stack.py", line 110, in pop_obj return self._stack.pop().obj IndexError: pop from empty list

还是错误:

线程Thread-1中的异常: 追溯(最近一次通话): _bootstrap_inner中的第916行的文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ threading.py" self.run() 正在运行的文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ threading.py"，第1182行 self.function(* self.args，** self.kwargs) 火车上的文件"C:\ Users \ a703572 \ PycharmProjects \ ai-pred-eng \ src \ run_keras_server.py"，行632 self.update_prediction_historics_all() 文件"C:\ Users \ a703572 \ PycharmProjects \ ai-pred-eng \ src \ run_keras_server.py"，行649，在update_prediction_historics_all中 self.update_prediction_historics_dataset(new_dataset，loadModel = True) 文件"C:\ Users \ a703572 \ PycharmProjects \ ai-pred-eng \ src \ run_keras_server.py"，第672行，位于update_prediction_historics_dataset中 0] +".h5"，loadModel = loadModel)[ 文件"C:\ Users \ a703572 \ PycharmProjects \ ai-pred-eng \ src \ run_keras_server.py"，行198，位于get_predictions_sequential中 模型，缩放比例= self.load_model_file(self.graph_pred，self.session，路径) 在load_model_file中的文件"C:\ Users \ a703572 \ PycharmProjects \ ai-pred-eng \ src \ run_keras_server.py"，第114行 型号= load_model(路径) 在load_model中的文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ site-packages \ keras \ engine \ saving.py"，行419 模型= _deserialize_model(f，custom_objects，compile) _deserialize_model中的第225行的文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ site-packages \ keras \ engine \ saving.py" 模型= model_from_config(model_config，custom_objects = custom_objects) 在model_from_config中的行458中的文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ site-packages \ keras \ engine \ saving.py" 返回反序列化(config，custom_objects = custom_objects) 反序列化文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ site-packages \ keras \ layers__init __.py"，第55行 printable_module_name ='图层') 在deserialize_keras_object中的第145行中的文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ site-packages \ keras \ utils \ generic_utils.py" 列表(custom_objects.items())) 文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ site-packages \ keras \ engine \ sequential.py"，行301，位于from_config model.add(图层) 在文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ site-packages \ keras \ engine \ sequential.py"中，添加第181行 output_tensor = layer(self.outputs [0]) 调用中的文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ site-packages \ keras \ engine \ base_layer.py"，第431行 self.build(拆包单身(input_shapes)) 在构建中，文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ site-packages \ keras \ layers \ core.py"，第872行 约束= self.bias_constraint) 包装中的文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ site-packages \ keras \ legacy \ interfaces.py"，行91 return func(* args，** kwargs) 文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ site-packages \ keras \ engine \ base_layer.py"，第252行，位于add_weight中 约束=约束) 变量402行中的文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ site-packages \ keras \ backend \ tensorflow_backend.py" v = tf.Variable(值，dtype = tf.as_dtype(dtype)，名称=名称) 调用中的文件"C:\ Users \ a703572 \ AppData \ Roaming \ Python \ Python36 \ site-packages \ tensorflow \ python \ ops \ variables.py"，第183行 返回cls._variable_v1_call(* args，** kwargs) _variable_v1_call中的第146行的文件"C:\ Users \ a703572 \ AppData \ Roaming \ Python \ Python36 \ site-packages \ tensorflow \ python \ ops \ variables.py" 聚合=聚合) 文件"C:\ Users \ a703572 \ AppData \ Roaming \ Python \ Python36 \ site-packages \ tensorflow \ python \ ops \ variables.py"，第125行，在 previous_getter = lambda ** kwargs:default_variable_creator(None，** kwargs) 文件"C:\ Users \ a703572 \ AppData \ Roaming \ Python \ Python36 \ site-packages \ tensorflow \ python \ ops \ variable_scope.py"，行2444，位于default_variable_creator中 Expected_shape = expected_shape，import_scope = import_scope) 调用中的文件"C:\ Users \ a703572 \ AppData \ Roaming \ Python \ Python36 \ site-packages \ tensorflow \ python \ ops \ variables.py"，第187行 返回super(VariableMetaclass，cls).调用(* args，** kwargs) init 中的文件"C:\ Users \ a703572 \ AppData \ Roaming \ Python \ Python36 \ site-packages \ tensorflow \ python \ ops \ variables.py"，第1329行 约束=约束) 文件"C:\ Users \ a703572 \ AppData \ Roaming \ Python \ Python36 \ site-packages \ tensorflow \ python \ ops \ variables.py"，行1492，位于_init_from_args中 ops.add_to_collections(收藏夹，自己) 退出中的文件"C:\ Users \ a703572 \ AppData \ Local \ Programs \ Python \ Python36 \ lib \ contextlib.py"，第88行 下一个(self.gen) 文件"C:\ Users \ a703572 \ AppData \ Roaming \ Python \ Python36 \ site-packages \ tensorflow \ python \ framework \ ops.py"，行5347，位于init_scope中 屈服 退出中的文件"C:\ Users \ a703572 \ AppData \ Roaming \ Python \ Python36 \ site-packages \ tensorflow \ python \ framework \ ops.py"，行4369 self._graph._pop_control_dependencies_controller(自我) _pop_control_dependencies_controller中的文件"C:\ Users \ a703572 \ AppData \ Roaming \ Python \ Python36 \ site-packages \ tensorflow \ python \ framework \ ops.py"，行4390 断言self._control_dependencies_stack [-1]是控制器 AssertionError

Exception in thread Thread-1: Traceback (most recent call last): File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\threading.py", line 916, in _bootstrap_inner self.run() File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\threading.py", line 1182, in run self.function(*self.args, **self.kwargs) File "C:\Users\a703572\PycharmProjects\ai-pred-eng\src\run_keras_server.py", line 632, in train self.update_prediction_historics_all() File "C:\Users\a703572\PycharmProjects\ai-pred-eng\src\run_keras_server.py", line 649, in update_prediction_historics_all self.update_prediction_historics_dataset(new_dataset, loadModel=True) File "C:\Users\a703572\PycharmProjects\ai-pred-eng\src\run_keras_server.py", line 672, in update_prediction_historics_dataset 0] + ".h5", loadModel=loadModel)[ File "C:\Users\a703572\PycharmProjects\ai-pred-eng\src\run_keras_server.py", line 198, in get_predictions_sequential model, scaler = self.load_model_file(self.graph_pred, self.session, path) File "C:\Users\a703572\PycharmProjects\ai-pred-eng\src\run_keras_server.py", line 114, in load_model_file model = load_model(path) File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\engine\saving.py", line 419, in load_model model = _deserialize_model(f, custom_objects, compile) File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\engine\saving.py", line 225, in _deserialize_model model = model_from_config(model_config, custom_objects=custom_objects) File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\engine\saving.py", line 458, in model_from_config return deserialize(config, custom_objects=custom_objects) File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\layers__init__.py", line 55, in deserialize printable_module_name='layer') File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\utils\generic_utils.py", line 145, in deserialize_keras_object list(custom_objects.items()))) File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\engine\sequential.py", line 301, in from_config model.add(layer) File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\engine\sequential.py", line 181, in add output_tensor = layer(self.outputs[0]) File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\engine\base_layer.py", line 431, in call self.build(unpack_singleton(input_shapes)) File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\layers\core.py", line 872, in build constraint=self.bias_constraint) File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper return func(*args, **kwargs) File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\engine\base_layer.py", line 252, in add_weight constraint=constraint) File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\backend\tensorflow_backend.py", line 402, in variable v = tf.Variable(value, dtype=tf.as_dtype(dtype), name=name) File "C:\Users\a703572\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\variables.py", line 183, in call return cls._variable_v1_call(*args, **kwargs) File "C:\Users\a703572\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\variables.py", line 146, in _variable_v1_call aggregation=aggregation) File "C:\Users\a703572\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\variables.py", line 125, in previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs) File "C:\Users\a703572\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\variable_scope.py", line 2444, in default_variable_creator expected_shape=expected_shape, import_scope=import_scope) File "C:\Users\a703572\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\variables.py", line 187, in call return super(VariableMetaclass, cls).call(*args, **kwargs) File "C:\Users\a703572\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\variables.py", line 1329, in init constraint=constraint) File "C:\Users\a703572\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\ops\variables.py", line 1492, in _init_from_args ops.add_to_collections(collections, self) File "C:\Users\a703572\AppData\Local\Programs\Python\Python36\lib\contextlib.py", line 88, in exit next(self.gen) File "C:\Users\a703572\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\ops.py", line 5347, in init_scope yield File "C:\Users\a703572\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\ops.py", line 4369, in exit self._graph._pop_control_dependencies_controller(self) File "C:\Users\a703572\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\ops.py", line 4390, in _pop_control_dependencies_controller assert self._control_dependencies_stack[-1] is controller AssertionError

我的解决方案是使用图形进行预测，并使用图形进行训练，每次我要执行tf操作时，我都会使用:

My solution was using a graph for prediction and a graph for training, and every time I want to perform a tf operation I use:

with server_predict.graph_pred.as_default():
    with tf.Session(graph=server_predict.graph_pred) as sess:

我还添加了这一行:

        backend.set_session(sess)

尽管如此，我仍然会出现来自tf会话或图形的错误，因为似乎操作没有正确分开.另一个错误是我在此问题中写的关于tf的错误会议.使用k.clear_session()(k = keras后端)给出的解决方案对我不起作用.

Despite this, I keep having the errors coming from the tf session or graph, as It seems that the operations are not properly separated. Another error is the one I wrote in this issue that is still opened, regarding the tf session. Solution given using k.clear_session() (k = keras backend) did not work for me.

有人遇到过类似的问题或编写了类似的任务可能对我有帮助吗?

Does any one have had a similar problem or has programmed a similar task that might help me?

谢谢！

找到一个包装纸"来使这项工作生效.我没有在同一个类(自定义)上启动两个线程，而是拥有两个相同类的对象，一个对象专用于训练，另一个对象用于预测.这不是真正的多线程应用程序(即使这两个对象是从同一主程序启动的).在找到合适的多线程解决方案之前，这可能会有所帮助.

Found a "wrap" to make this work. Instead of launching two threads over the same class (custom), what I have is two objects of the same class, one is dedicated to training and the other to predict. This is not a real multithread app (even though the two objects are launched from the same main). Until I (we) find a proper multithread solution this might help.

但是我不知道我以前是怎么得到错误的，只是没有两个对象，即使这些对象在同一进程中运行也是如此. keras/tensorflow只能在一个图上进行操作，而在同一进程中为不同对象定义不同的图吗?

However I do not understand know how I got the errors before, and just by having two objects not, even if these objects run in the same process. Is it that keras/tensorflow can only make operations on only one graph but defines different graphs for different objects on the same process?

推荐答案

最简单的解决方案是拥有两个单独的keras模型-第一个在推理模式下运行，第二个在训练模式下运行.每次推理模型获取新的数据集进行预测时，它都会首先检查其是否具有最新的.h5文件，如果没有，则首先将其加载，然后运行预测.这样您可以避免锁等.

Easiest solution is to have two separate keras models - the first runs in inference mode, and the second runs in training mode. Every time the inference model gets a new dataset to predict on, it first checks to see if it has the most "up to date" .h5 file, if not then it loads it in first then runs the prediction. This way you can avoid locks and such.

很难针对您的情况提供建议，因为您想要的可能与我需要的不一样

It's hard to give advice specific to your case because what you want is likely not the same as what I need

• 这是我在使用Tensorflow Multiprocessing完成类似操作后的观点

这篇关于不同线程中的预测和训练Keras Tensorflow的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！