Keras(TensorFlow、CPU):在循环中训练序列模型会占用内存 [英] Keras (TensorFlow, CPU): Training Sequential models in loop eats memory

问题描述

我正在尝试循环训练 1000 倍的序列模型.在每个循环中，我的程序都会泄漏内存，直到用完并出现 OOM 异常.

I am trying to train 1000x of Sequential models in a loop. In every loop my program leaks memory until I run out and get an OOM exception.

我之前已经问过类似的问题(连续训练多个 Sequential 模型会变慢)

I already asked a similar question before (Training multiple Sequential models in a row slows down)

并且已经看到其他人遇到类似的问题(Keras: 进行超参数网格搜索时内存不足)

and have seen others in similar problems (Keras: Out of memory when doing hyper parameter grid search)

解决方案总是在您使用完模型后将 K.clear_session() 添加到您的代码中.所以我在上一个问题中这样做了，但我仍然在泄漏内存

and the solution is always to add K.clear_session() to your code after you have finished using the model. So I did that in my previous question and I am still leaking memory

这是重现问题的代码.

import random
import time
from keras.models import Sequential
from keras.layers import Dense
from keras import backend as K
import tracemalloc


def run():
    tracemalloc.start()
    num_input_nodes = 12
    num_hidden_nodes = 8
    num_output_nodes = 1

    random_numbers = random.sample(range(1000), 50)
    train_x, train_y = create_training_dataset(random_numbers, num_input_nodes)

    for i in range(100):
        snapshot = tracemalloc.take_snapshot()
        for j in range(10):
            start_time = time.time()
            nn = Sequential()
            nn.add(Dense(num_hidden_nodes, input_dim=num_input_nodes, activation='relu'))
            nn.add(Dense(num_output_nodes))
            nn.compile(loss='mean_squared_error', optimizer='adam')
            nn.fit(train_x, train_y, nb_epoch=300, batch_size=2, verbose=0)
            K.clear_session()
            print("Iteration {iter}. Current time {t}. Took {elapsed} seconds".
                  format(iter=i*10 + j + 1, t=time.strftime('%H:%M:%S'), elapsed=int(time.time() - start_time)))

        top_stats = tracemalloc.take_snapshot().compare_to(snapshot, 'lineno')

        print("[ Top 5 differences ]")
        for stat in top_stats[:5]:
            print(stat)


def create_training_dataset(dataset, input_nodes):
    """
    Outputs a training dataset (train_x, train_y) as numpy arrays.
    Each item in train_x has 'input_nodes' number of items while train_y items are of size 1
    :param dataset: list of ints
    :param input_nodes:
    :return: (numpy array, numpy array), train_x, train_y
    """
    data_x, data_y = [], []
    for i in range(len(dataset) - input_nodes - 1):
        a = dataset[i:(i + input_nodes)]
        data_x.append(a)
        data_y.append(dataset[i + input_nodes])
    return numpy.array(data_x), numpy.array(data_y)

run()

这是我从第一个内存调试打印得到的输出

Here is the output I get from the first memory debug print

/tensorflow/python/framework/ops.py:121: size=3485 KiB (+3485 KiB), count=42343 (+42343)/tensorflow/python/framework/ops.py:1400: size=998 KiB (+998 KiB), count=8413 (+8413)/tensorflow/python/framework/ops.py:116: size=888 KiB (+888 KiB), count=32468 (+32468)/tensorflow/python/framework/ops.py:1185: size=795 KiB (+795 KiB), count=3179 (+3179)/tensorflow/python/framework/ops.py:2354: size=599 KiB (+599 KiB), count=5886 (+5886)

/tensorflow/python/framework/ops.py:121: size=3485 KiB (+3485 KiB), count=42343 (+42343) /tensorflow/python/framework/ops.py:1400: size=998 KiB (+998 KiB), count=8413 (+8413) /tensorflow/python/framework/ops.py:116: size=888 KiB (+888 KiB), count=32468 (+32468) /tensorflow/python/framework/ops.py:1185: size=795 KiB (+795 KiB), count=3179 (+3179) /tensorflow/python/framework/ops.py:2354: size=599 KiB (+599 KiB), count=5886 (+5886)

系统信息:

• python 3.5
• keras (1.2.2)
• 张量流(1.0.0)

推荐答案

内存泄漏源于 Keras 和 TensorFlow 使用单个默认图"来存储网络结构，随着内部 for 循环.

The memory leak stems from Keras and TensorFlow using a single "default graph" to store the network structure, which increases in size with each iteration of the inner for loop.

调用 K.clear_session() 释放一些与迭代之间的默认图相关联的(后端)状态，但额外调用 tf.reset_default_graph() 需要清除 Python 状态.

Calling K.clear_session() frees some of the (backend) state associated with the default graph between iterations, but an additional call to tf.reset_default_graph() is needed to clear the Python state.

请注意，可能有更有效的解决方案:由于 nn 不依赖于任何一个循环变量，您可以在循环外定义它，并在循环内重用相同的实例.如果这样做，则无需清除会话或重置默认图形，性能应该会提高，因为您可以从迭代之间的缓存中受益.

Note that there might be a more efficient solution: since nn does not depend on either of the loop variables, you can define it outside the loop, and reuse the same instance inside the loop. If you do that, there is no need to clear the session or reset the default graph, and performance should increase because you benefit from caching between iterations.

这篇关于Keras(TensorFlow、CPU):在循环中训练序列模型会占用内存的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！