Keras自定义数据生成器提供多输入和多输出的尺寸错误(功能性api模型) [英] Keras custom data generator giving dimension errors with multi input and multi output( functional api model)
问题描述
我已经用Keras编写了一个生成器函数,然后从__getitem__
返回X,y之前,我仔细检查了X和Y的形状,它们都可以,但是生成器给出了尺寸不匹配数组和警告.
I have written a generator function with Keras, before returning X,y from __getitem__
I have double check the shapes of the X's and Y's and they are alright, but generator is giving dimension mismatch array and warnings.
(要复制的Colab代码: https://colab.research.google .com/drive/1bSJm44MMDCWDU8IrG2GXKBvXNHCuY70G?usp = sharing )
(Colab Code to reproduce: https://colab.research.google.com/drive/1bSJm44MMDCWDU8IrG2GXKBvXNHCuY70G?usp=sharing)
我的训练和验证生成器与
My training and validation generators are pretty much same as
class ValidGenerator(Sequence):
def __init__(self, df, batch_size=64):
self.batch_size = batch_size
self.df = df
self.indices = self.df.index.tolist()
self.num_classes = num_classes
self.shuffle = shuffle
self.on_epoch_end()
def __len__(self):
return int(len(self.indices) // self.batch_size)
def __getitem__(self, index):
index = self.index[index * self.batch_size:(index + 1) * self.batch_size]
batch = [self.indices[k] for k in index]
X, y = self.__get_data(batch)
return X, y
def on_epoch_end(self):
self.index = np.arange(len(self.indices))
if self.shuffle == True:
np.random.shuffle(self.index)
def __get_data(self, batch):
#some logic is written here
#hat prepares 3 X features and 3 Y outputs
X = [input_array_1,input_array_2,input_array_3]
y = [out_1,out_2,out_3]
#print(len(X))
return X, y
我是X的回音,y分别具有3个输入要素和3个输出要素,所以X的形状为(3,32,10,1)
I am return tupple of X,y from which has 3 input features and 3 output features each, so shape of X is (3,32,10,1)
我正在使用功能性api构建具有以下结构的模型(我具有诸如级联,多输入/输出等功能,而按顺序是不可能的)
I am using functional api to build model(I have things like concatenation, multi input/output, which isnt possible with sequential) with following structure
当我尝试使用以下代码将模型与生成器拟合时
When I try to fit the model with generator with following code
train_datagen = TrainGenerator(df=train_df, batch_size=32, num_classes=None, shuffle=True)
valid_datagen = ValidGenerator(df=train_df, batch_size=32, num_classes=None, shuffle=True)
model.fit(train_datagen, epochs=2,verbose=1,callbacks=[checkpoint,es])
我收到了这些警告和错误,这些警告和错误并没有消失
I get these warnings and errors, that dont go away
第1/2集 警告:tensorflow:为输入> Tensor("input_1:0&"; shape =(None,10),dtype = float32)构造了形状为(None,10)的模型,但在带有不兼容的形状(无,无,无).
Epoch 1/2 WARNING:tensorflow:Model was constructed with shape (None, 10) for input >Tensor("input_1:0", shape=(None, 10), dtype=float32), but it was called >on an input with incompatible shape (None, None, None).
警告:tensorflow:模型的形状为(None,10),可用于输入 Tensor("input_2:0&",shape =(None,10),dtype = float32),但它是 在形状不兼容的输入(无,无,无)上调用. 警告:tensorflow:模型的形状为(None,10),用于 输入Tensor("input_3:0&",shape =(None,10),dtype = float32),但它是 在形状不兼容的输入(无,无,无)上调用. ... ... 致电 返回超级(RNN,自我).调用(输入,** kwargs) /home/eduardo/.virtualenvs/kgpu3/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py:975 致电 input_spec.assert_input_compatibility(self.input_spec,输入, /home/eduardo/.virtualenvs/kgpu3/lib/python3.8/site-packages/tensorflow/python/keras/engine/input_spec.py:176 assert_input_compatibility 提高ValueError('Input'+ str(input_index)+'层'+
WARNING:tensorflow:Model was constructed with shape (None, 10) for input Tensor("input_2:0", shape=(None, 10), dtype=float32), but it was called on an input with incompatible shape (None, None, None). WARNING:tensorflow:Model was constructed with shape (None, 10) for input Tensor("input_3:0", shape=(None, 10), dtype=float32), but it was called on an input with incompatible shape (None, None, None). ... ... call return super(RNN, self).call(inputs, **kwargs) /home/eduardo/.virtualenvs/kgpu3/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py:975 call input_spec.assert_input_compatibility(self.input_spec, inputs, /home/eduardo/.virtualenvs/kgpu3/lib/python3.8/site-packages/tensorflow/python/keras/engine/input_spec.py:176 assert_input_compatibility raise ValueError('Input ' + str(input_index) + ' of layer ' +
ValueError: Input 0 of layer lstm is incompatible with the layer: expected ndim=3, found ndim=4. Full shape received: [None, None, None, 88]
我已经重新检查了整个代码,并且无法像警告或错误一样输入(无,无,无),我的输入尺寸是(3,32,10,1)
I have rechecked whole code and it isnt possible to have input (None,None,None) like in warning or in error, my input dimension is (3,32,10,1)
更新
我也试图用python编写一个生成器函数,并且得到了完全相同的错误.
I have also tried to write a generator function with python and got exactly same error.
我的生成器功能
def generate_arrays_from_file(batchsize,df):
#print(bat)
inputs = []
targets = []
batchcount = 0
while True:
df3 = df.loc[np.arange(batchcount*batchsize,(batchcount*batchsize)+batchsize)]
#Some pre processing
X = [input_array_1,input_array_2,input_array_3]
y = [out_1,out_2,out_3]
yield X,y
batchcount = batchcount +1
在内部使用keras似乎是错误的(可能是由于我使用的是功能性API)
It seems like it is something wrong internally wit keras (may be due to the fact I am using functional API)
更新2
我也尝试输出元组
X = (input1_X,input2_X,input3_X)
y = (output1_y,output2_y,output3_y)
,也命名为输入/输出,但是不起作用
and also named input/output, but it doesnt work
X = {"input_1": input1_X, "input_2": input2_X,"input_3": input3_X}
y = {"output_1": output1_y, "output_2": output2_y,"output_3": output3_y}
有关问题制定的说明:
将单个X要素更改为形状(32,10)而不是(32,10,1)可能有助于摆脱此错误,但这不是我想要的,这改变了我的问题(我不再有10个时间步长各有一个功能)
Changing the individual X features to shape (32,10) instead of (32,10,1) might help to get rid of this error but that is not what I want, it changes my problem(I no longer have 10 time steps with one feature each)
推荐答案
为解决此错误:
ValueError: Input 0 of layer lstm is incompatible with the layer: expected ndim=3, found ndim=4. Full shape received: [None, None, None, 88]
TrainGenerator应该以以下方式更改. 当前代码:
TrainGenerator should be changed in the following way. Current code:
input1_X = np.array(df3['input1_X'].to_list()).reshape(dlen,pad_len,1)
input2_X = np.array(df3['input2_X'].to_list()).reshape(dlen,pad_len,1)
input3_X = np.array(df3['input3_X'].to_list()).reshape(dlen,pad_len,1)
应更改为:
input1_X = np.array(df3['input1_X'].to_list()).reshape(dlen,pad_len)
input2_X = np.array(df3['input2_X'].to_list()).reshape(dlen,pad_len)
input3_X = np.array(df3['input3_X'].to_list()).reshape(dlen,pad_len)
原因是3个输入中的每一个都期望一个2维数组,但是生成器提供3维数组.预期的形状是(batch_size,10).
The reason is that each of the 3 Inputs expects a 2-dimensional array, but the generator provides a 3-dimensional one. The expected shape is (batch_size, 10).
这篇关于Keras自定义数据生成器提供多输入和多输出的尺寸错误(功能性api模型)的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!