Keras 函数式 API:将 CNN 模型与 RNN 结合以查看图像序列 [英] Keras functional API: Combine CNN model with a RNN to to look at sequences of images

问题描述

所以我被一个关于如何在 Keras 中结合 CNN 和 RNN 的问题所困扰.在发布问题时，有人指出我这是解决问题的正确方法.显然我只是忽略了原始代码中的一些东西，这让我回答了我自己的问题.

So i was stuck with a question on how to combine a CNN with a RNN in Keras. While posting the question someone pointed me out that this is the correct way to approach the problem. Apparently i just overlooked something in the original code, which made me answer my own question.

原问题如下:

如何在 Keras 中创建一个模型，将图像序列作为输入，让 CNN查看"每个单独的图像，并将 CNN 输出的序列输入到 RNN 中?

How do you create a model in Keras that has sequences of images as the input, with a CNN 'looking' at each individual image and the sequence of the CNN output being fed into a RNN?

为了更清楚:

模型一:查看单个图像的 CNN.
模型二:一个 RNN，它是模型一的 CNN 输出的序列.

Model one: a CNN that looks at single images.
Model two: a RNN that at the sequences of the output of the CNN from model one.

例如，CNN 应该看到 5 个图像，并且 CNN 的 5 个输出序列应该传递给 RNN.

So for example the CNN should see 5 images and this sequence of 5 outputs from the CNN should be passed on to the RNN.

输入数据格式如下:
(number_of_images, width, height, channels) = (4000, 120, 60, 1)

The input data is in the following format:
(number_of_images, width, height, channels) = (4000, 120, 60, 1)

推荐答案

这个问题的答案如下.

以这个过于简化的 CNN 模型为例:

Take this oversimplified CNN model:

cnn = Sequential()
cnn.add(Conv2D(16, (50, 50), input_shape=(120, 60, 1)))

cnn.add(Conv2D(16, (40, 40)))

cnn.add(Flatten()) # Not sure if this if the proper way to do this.

然后是这个简单的RNN模型:

Then there is this simple RNN model:

rnn = Sequential()

rnn = GRU(64, return_sequences=False, input_shape=(120, 60))

应该连接到密集网络:

dense = Sequential()
dense.add(Dense(128))
dense.add(Dense(64))

dense.add(Dense(1)) # Model output

请注意，为了便于阅读，已省略了激活函数等.

Notice that activation functions and such have been left out for readability.

现在剩下的就是组合这 3 个主要模型.

Now all that is left is combining these 3 main models.

main_input = Input(shape=(5, 120, 60, 1)) # Data has been reshaped to (800, 5, 120, 60, 1)

model = TimeDistributed(cnn)(main_input) # this should make the cnn 'run' 5 times?
model = rnn(model) # combine timedistributed cnn with rnn
model = dense(model) # add dense

最后

final_model = Model(inputs=main_input, outputs=model)

final_model.compile...
final_model.fit...

这篇关于Keras 函数式 API:将 CNN 模型与 RNN 结合以查看图像序列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！