使用 Tensorflow/Keras 和内存问题训练神经网络，输入作为矩阵的滑动窗口 [英] Train a neural network with input as sliding windows of a matrix with Tensorflow / Keras, and memory issues
问题描述
我有一个数据集，它是一个形状为 (100 000, 2 000) 的大矩阵.
I have a dataset which is a big matrix of shape (100 000, 2 000).
我想用这个大矩阵的形状 (16, 2000) 的所有可能的滑动窗口/子矩阵来训练我的 Tensorflow 神经网络.
I would like to train my Tensorflow neural network with all the possible sliding windows/submatrices of shape (16, 2000) of this big matrix.
我使用:
from skimage.util.shape import view_as_windows
A.shape # (100000, 2000) ie 100k x 2k matrix
X = view_as_windows(A, (16, 2000)).reshape((-1, 16, 2000, 1))
X.shape # (99985, 16, 2000, 1)
...
model.fit(X, Y, batch_size=4, epochs=8)
不幸的是，这会导致内存问题:
Unfortunately, this leads to a memory problem:
W tensorflow/core/framework/allocator.cc:122] ...的分配超过系统内存的 10%.
W tensorflow/core/framework/allocator.cc:122] Allocation of ... exceeds 10% of system memory.
这是正常的，因为 X 有 ~ 100k * 16 * 2k 个系数，即超过 30 亿个系数！
This is normal, since X has ~ 100k * 16 * 2k coefficients, i.e. more than 3 billion coefficients!
但实际上，在内存中加载X是一种内存浪费，因为它高度冗余:它是由A<上的形状为(16, 2000)的滑动窗口组成的/代码>.
But in fact, it is a waste of memory to load X in memory because it is highly redundant: it is made of sliding windows of shape (16, 2000) over A.
问题:如何训练输入为宽度为 16 的所有滑动窗口超过 100k x 2k 矩阵的神经网络，而不浪费内存?
Question: how to train a neural network with input being all sliding windows of width 16 over a 100k x 2k matrix, without wasting memory?
skimage 的文档.util.view_as_windows 确实声明它的内存成本很高:
The documentation of skimage.util.view_as_windows states indeed that it's costly in memory:
当涉及到内存使用时，应该非常小心滚动视图.实际上，尽管视图"与其基本数组具有相同的内存占用，但在计算中使用此视图"时出现的实际数组通常比原始数组大(很多)，尤其是对于二维数组及以上.
One should be very careful with rolling views when it comes to memory usage. Indeed, although a ‘view’ has the same memory footprint as its base array, the actual array that emerges when this ‘view’ is used in a computation is generally a (much) larger array than the original, especially for 2-dimensional arrays and above.
例如，让我们考虑一个大小为 (100, 100, 100) 的 float64 的 3 维数组.[...] 滚动视图的假设大小(例如，如果要重塑视图)将是 8*(100-3+1)3*33，大约 203 MB！随着输入数组的维度变大，缩放变得更糟.
For example, let us consider a 3 dimensional array of size (100, 100, 100) of float64. [...] the hypothetical size of the rolling view (if one was to reshape the view for example) would be 8*(100-3+1)3*33 which is about 203 MB! The scaling becomes even worse as the dimension of the input array becomes larger.
timeseries_dataset_from_array 正是我正在寻找的，只是它仅适用于一维序列:
timeseries_dataset_from_array is exactly what I'm looking for except that it works only for 1D sequences:
import tensorflow
import tensorflow.keras.preprocessing
x = list(range(100))
x2 = tensorflow.keras.preprocessing.timeseries_dataset_from_array(x, None, 10, sequence_stride=1, sampling_rate=1, batch_size=128, shuffle=False, seed=None, start_index=None, end_index=None)
for b in x2:
print(b)
它不适用于二维数组:
x = np.array(range(90)).reshape(6, 15)
print(x)
x2 = tensorflow.keras.preprocessing.timeseries_dataset_from_array(x, None, (6, 3), sequence_stride=1, sampling_rate=1, batch_size=128, shuffle=False, seed=None, start_index=None, end_index=None)
# does not work
推荐答案
您可以使用生成器动态生成示例，而不是将它们存储在内存中.
You can use generator to yield examples on fly instead of storing them in memory.
您可以编写自定义 generator 或由 keras 提供的生成器，如 timeseries_dataset_from_array (docs) 也可以在 sequence_stride 等选项的帮助下产生窗口.
You can either write custom generator or generator provided by keras like timeseries_dataset_from_array (docs) that can yield windows as well with help of options like sequence_stride.
对于自定义生成器，您可以执行类似的操作
For custom generator, you can do something like
def generator_custom(df3):
for idex,row in df3.iterrows():
#some preprocessing
yield X,y
然后您可以使用 tf.data 将 128/64/32 的批次作为
And then you can use tf.data to take batch of 128/64/32 as
tf.data.Dataset.from_generator(lambda: generator_custom(df_train))
train_dataset = train_dataset.batch(128,drop_remainder=True)
回复你关于二维的评论
只是一个例子(例如我将 100000,2000 缩放到 1000.200，可以随意更改它们)
Just an example (I scale 100000,2000 to 1000.200 for example, feel free to change them)
import numpy as np
x = np.array(range(200000)).reshape(1000, 200)
x2 = tensorflow.keras.preprocessing.timeseries_dataset_from_array(x, None, 16, sequence_stride=1, sampling_rate=1, batch_size=128)
这给了你类似的东西
形状(128、16、200)
shapes (128, 16, 200)
形状(128、16、200)
shapes (128, 16, 200)
你想要哪个(16*2000)，对吧?(记住我们有 200 个只是为了显示目的)
Which is what you want (16*2000), right? (remember we have 200 just to showing purpose)
这篇关于使用 Tensorflow/Keras 和内存问题训练神经网络，输入作为矩阵的滑动窗口的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
