端到端训练CNN-LSTLM? [英] Train a CNN-LSTLM end-to-end?

问题描述

有许多论文(特别是用于图像字幕的论文)将CNN和LSTM体系结构联合用于预测和生成任务.但是，他们似乎都独立于LSTM训练CNN.我一直在浏览Torch和TensorFlow(使用Keras)，但找不到为什么不可能进行端到端培训的原因(至少从架构设计的角度来看)，但是在那里似乎没有关于这种模型的任何文档.

There have been a number of papers (particularly for image captioning) that use CNN and LSTM architectures jointly for prediction and generation tasks. However, they all seem to train the CNN independently from the LSTM. I was looking through Torch and TensorFlow (with Keras), and couldn't find a reason why it shouldn't be possible to do end-to-end training (at least from an architecture design point-of-view), but there doesn't seem to be any documentation for such a model.

那么，可以做到吗? Torch或TensorFlow(甚至Theanos或Caffe)是否支持联合训练端到端CNN-LSTM神经网络?如果是这样，是否仅将CNN的输出链接到LSTM的输入并运行SGD一样简单?还是有更多的复杂性?

So, can it be done? Does Torch or TensorFlow (or even Theanos or Caffe) support jointly training an end-to-end CNN-LSTM neural network? If so, is it as simple as just linking the output from the CNN to the input to the LSTM and running SGD? Or is there more complexity to it?

推荐答案

可以使用tensorflow在端到端训练CNN-LSTM模型

CNN-LSTM model can be trained end-end using tensorflow

假设您有一个带有输入X的CNN模型M和一个LSTM模型LSTM.可以训练到终点

Suppose you have a CNN model M with input X and a LSTM model LSTM. This can be trained end-end

# here CNN is used to extract meaning features from the input data
features = M(X)
# CNN features are used as input to LSTM
y = LSTM(features)
cost = cost_function(ground_truths, y)

在 查看全文