如何使用嵌入了预训练的CNN的句子嵌入构建神经网络 [英] How to build a Neural Network with sentence embeding concatenated to pre-trained CNN

问题描述

我想构建一个神经网络，该神经网络将从CNN的最后一层(例如VGG或resnet)获取特征图，并连接一个附加向量(例如1X768 bert向量)，然后对最后一个向量进行重新训练分类问题. 因此，架构应类似于:

I want to build a neural network that will take the feature map from the last layer of a CNN (VGG or resnet for example), concatenate an additional vector (for example , 1X768 bert vector) , and re-train the last layer on classification problem. So the architecture should be like in:

但是我想在每个特征向量上附加一个向量(我有一个句子来描述每个帧).

but I want to concat an additional vector to each feature vector (I have a sentence to describe each frame).

我有5个可能的标签，并且在输入帧中有100帧.

I have 5 possible labels , and 100 frames in the input frames.

有人可以帮助我实现这种类型的网络吗?

Can someone help me as to how to implement this type of network?

推荐答案

我建议研究 Keras功能API .

与顺序模型(通常足以解决许多入门问题)不同，功能性API 允许您创建所需的任何非循环图.这意味着您可以有两个 输入分支，一个分支用于CNN(图像数据)，另一个分支用于您需要执行的任何NLP(与您提到的描述性句子有关).然后，您可以将这两个分支的组合输出馈送到网络的最后一层，并产生结果.

Unlike a sequential model (which is usually enough for many introductory problems), the functional API allows you to create any acyclic graph you want. This means that you can have two input branches, one for the CNN (image data) and the other for any NLP you need to do (relating to the descriptive sentence that you mentioned). Then, you can feed in the combined outputs of these two branches into the final layers of your network and produce your result.

即使您已经使用models.Sequential()创建了模型，使用功能性API对其进行重写也不应该太困难.

Even if you've already created your model using models.Sequential(), it shouldn't be too hard to rewrite it to use the functional API.

有关更多信息和实现细节，请在此处查看官方文档: https://keras .io/guides/functional_api/

For more information and implementation details, look at the official documentation here: https://keras.io/guides/functional_api/

这篇关于如何使用嵌入了预训练的CNN的句子嵌入构建神经网络的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！