连续训练多个顺序模型会减慢速度 [英] Training multiple Sequential models in a row slows down

问题描述

我正在使用Keras/TensorFlow(GPU)创建时间序列预测模型.我有100倍的时间序列，想为每个时间序列训练一个网络.

I am using Keras/TensorFlow (GPU) to create a time series forecasting model. I have 100x of time series and want to train a network for each of them.

连续运行几个时间序列是可以的，但是一旦我运行100倍或1000倍，则每个模型的训练时间似乎会慢慢增加(但可以肯定).有一个简单的原因吗?

Running a few time series in a row is fine, but once I run 100x or 1000x then it appears that the training time of each model increase slowly (but surely). Is there a simple reason for this ?

下面是重现此问题的代码(请注意，可能需要一段时间才能运行).

Below is code to reproduce the issue (note that it could take a while to run).

https://gist.github.com/mannsi/c5666c4b786c35c3443beea6d13a32fe

在我的计算机上，第一次迭代耗时10s，#250迭代耗时16s，#500迭代耗时25s.

On my machine the first iteration takes 10s, iteration #250 takes 16s and iteration #500 takes 25s.

我是神经网络和Keras/TF的新手，所以也许这是完全正常的，但是在进行后台时间计算时我没有考虑到这一点.

I am new to Neural Networks and Keras/TF so maybe this is totally normal but I did not factor this in when doing my back-of-the-envelope time calculations.

系统信息:

• python 3.5
• 喀拉拉邦(1.2.2)
• tensorflow-gpu(1.0.0)

我在TensorFlow CPU后端测试了相同的代码，并且在那里看到了完全相同的行为.

I tested the same code on a TensorFlow CPU backend and I see the exact same behavior there.

推荐答案

在每次迭代中，计算图中可能会产生一些开销.使用Keras后端功能K.clear_session()重置每次运行之间的基础Tensorflow会话.

It's possible that there is some overhead building up in the computation graph over each iteration. Use the Keras backend function K.clear_session() to reset the underlying Tensorflow session between each run.

这篇关于连续训练多个顺序模型会减慢速度的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！