重塑使用 timeseries_dataset_from_array 预处理的 Tensorflow 数据集 [英] Reshape a Tensorflow dataset preprocessed with timeseries_dataset_from_array

问题描述

注意:我已经阅读了如何重塑 BatchDataset 类张量? 但该解决方案不适用于此处.

如使用 Tensorflow/Keras 和内存问题训练一个输入为矩阵滑动窗口的神经网络，我需要从一个形状矩阵训练一个具有所有形状 (16, 2000) 滑动窗口的神经网络(10000, 2000).(在我的另一篇文章中是 100k，但这里 10k 是可以的).

导入tensorflow将 numpy 导入为 npX = np.array(range(20000000)).reshape(10000, 2000)X = tensorflow.keras.preprocessing.timeseries_dataset_from_array(X，无，16，sequence_stride=1，sampling_rate=1，batch_size=32)对于 X 中的 b:print(b) # 形状张量 (32, 16, 2000)休息

问题是我需要将它输入到一个需要 (..., 16, 2000, 1) 形状的模型中.

model = Sequential()model.add(Conv2D(16, kernel_size=(9, 9), activation='relu', input_shape=(16, 2000, 1), padding='same'))...model.fit(X, Y, epochs=8)

我试过了

X = tensorflow.reshape(X, (-1, 16, 2000, 1))

没有成功.

如何做到这一点，即具有 (..., 16, 2000, 1) 形状的 timeseries_dataset_from_array 输出?

解决方案

要对 tf.data.Dataset 的每个元素应用转换，您应该使用 tf.data.Dataset.map 函数.在你的情况下，你可以使用 lambda 定义函数，使用 tf.expand_dims:

ds = tensorflow.keras.preprocessing.timeseries_dataset_from_array(X, None, 16, sequence_stride=1, sampling_rate=1, batch_size=32)ds_correct_shape = ds.map(lambda x: tf.expand_dims(x,axis=-1))

如果我们检查第一个元素的形状:

<预><代码>>>>对于 ds_correct_shape.take(1) 中的 elem:打印(f{elem.shape=}")elem.shape=TensorShape([32, 16, 2000, 1])

Note: I have already read How to reshape BatchDataset class tensor? but the solution doesn't apply here.

As mentioned in Train a neural network with input as sliding windows of a matrix with Tensorflow / Keras, and memory issues, I need to train a neural network with all sliding windows of shape (16, 2000) from a matrix of shape (10000, 2000). (In my other post it's 100k, but here 10k is ok).

import tensorflow
import numpy as np
X = np.array(range(20000000)).reshape(10000, 2000)  
X = tensorflow.keras.preprocessing.timeseries_dataset_from_array(X, None, 16, sequence_stride=1, sampling_rate=1, batch_size=32)
for b in X:
    print(b)  # Tensor of shape (32, 16, 2000)
    break

The problem is that I need to feed it into a model which requires a (..., 16, 2000, 1) shape.

model = Sequential()
model.add(Conv2D(16, kernel_size=(9, 9), activation='relu', input_shape=(16, 2000, 1), padding='same'))
...
model.fit(X, Y, epochs=8)

I tried

X = tensorflow.reshape(X, (-1, 16, 2000, 1))

without success.

How to do this, i.e. have the output of timeseries_dataset_from_array of shape (..., 16, 2000, 1)?

解决方案

To apply a transformation to each element of a tf.data.Dataset you should use the tf.data.Dataset.map function. In your case, you could define the function with a lambda, using tf.expand_dims:

ds = tensorflow.keras.preprocessing.timeseries_dataset_from_array(X, None, 16, sequence_stride=1, sampling_rate=1, batch_size=32)
ds_correct_shape = ds.map(lambda x: tf.expand_dims(x,axis=-1))

If we check the shape of the first element:

>>> for elem in ds_correct_shape.take(1):
        print(f"{elem.shape=}")
elem.shape=TensorShape([32, 16, 2000, 1])

这篇关于重塑使用 timeseries_dataset_from_array 预处理的 Tensorflow 数据集的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！