如何在内置的预训练模型中放置自定义层? [英] How to place custom layer inside a in-built pre trained model?

问题描述

我们正在尝试在预先训练的imagenet模型内添加自定义图层.对于顺序模型或非顺序模型，我们可以轻松地做到这一点.但是这里有一些要求.

首先，我们不想公开整个图像网络模型，并处理所需的内层.假设对于 DenseNet ，我们需要以下几层，并进一步获得它们的输出形状以与一些自定义层连接.

  vision_model = tf.keras.applications.DenseNet121(input_shape =(224,224,3)，include_top = False，weights ='imagenet')对于我来说，枚举(vision_model.layers)中的层:如果layer.name在['conv3_block12_concat'，'conv4_block24_concat']中:打印(i，'\ t'，layer.trainable，'\ t:'，layer.name)如果layer.name =='conv3_block12_concat':print(layer.get_output_shape_at(0)[1:])#(28，28，512)如果layer.name =='conv4_block24_concat':print(layer.get_output_shape_at(0)[1:])#(14，14，1024) 

整个需求可以证明如下

绿色指示器基本上是密网的过渡层.

在上图中，密集网络模型(假设)有 5个块，在其中，我们要选择 block 3 和 block 4 ，并添加一些自定义图层，然后合并它们以生成最终输出.

此外， DenseNet 的块(第1到5块)也应尽可能采用其预先训练的imagenet权重来公开.我们希望在需要时可以控制冻结和解冻预训练的层.

如何使用 tf.keras 有效地实现?或者，如果您认为有更好的方法可以执行相同的操作，请提出建议.

---

比方说，自定义块就是这样

  class MLPBlock(tf.keras.layers.Layer):def __init __(self，kernel_num = 32，kernel_size =(3,3)，步幅=(1,1)，padding ='same'):超级(ConvModule，self).__ init __()#转换层self.conv = tf.keras.layers.Conv2D(kernel_num，kernel_size =内核大小，步幅=步幅，填充=填充)#批处理规范层self.bn = tf.keras.layers.BatchNormalization()def呼叫(self，input_tensor，training = False):x = self.conv(input_tensor)x = self.bn(x，training = training)返回tf.nn.relu(x) 

动机

我正在尝试实施

We're trying to add a custom layer inside a pre-trained imagenet model. For a sequential or non-sequential model, we can easily do that. But here are some requirements.

First of all, we don't wanna disclose the whole imagenet model and deal with the desired inside layer. Let's say for DenseNet we need the following layers and further get the output shape of theirs to connect with some custom layers.

vision_model = tf.keras.applications.DenseNet121(
    input_shape=(224,224,3),
    include_top = False, 
     weights='imagenet')

for i, layer in enumerate(vision_model.layers):
    if layer.name in ['conv3_block12_concat', 'conv4_block24_concat']:
        print(i,'\t',layer.trainable,'\t  :',layer.name)

        if layer.name == 'conv3_block12_concat':
            print(layer.get_output_shape_at(0)[1:])  # (28, 28, 512)

        if layer.name == 'conv4_block24_concat':
            print(layer.get_output_shape_at(0)[1:])  # (14, 14, 1024)

The whole requirement can be demonstrated as follows

The green indicator is basically the transition layer of the dense net.

In the above diagram, the dense net model has (let's say) 5 blocks and among them, we want to pick block 3 and block 4 and add some custom layers followed by merging them to lead the final output.

Also, the blocks of DenseNet (block 1 to 5), should be as disclose as possible with their pre-trained imagenet weights. We like to have control to freeze and unfreeze pre-trained layers when we need them.

How can we efficiently achieve with tf.keras? or, If you think there some better approach to do the same thing, please suggest.

---

Let's say, a custom block is something like this

class MLPBlock(tf.keras.layers.Layer):
    def __init__(self, kernel_num=32, kernel_size=(3,3), strides=(1,1), padding='same'):
        super(ConvModule, self).__init__()
        # conv layer
        self.conv = tf.keras.layers.Conv2D(kernel_num, 
                        kernel_size=kernel_size, 
                        strides=strides, padding=padding)
        # batch norm layer
        self.bn   = tf.keras.layers.BatchNormalization()

    def call(self, input_tensor, training=False):
        x = self.conv(input_tensor)
        x = self.bn(x, training=training)
        return tf.nn.relu(x)

Motivation

I'm trying to implement this paper-work where they did something like this. Initially, the paper was free to get but now it's not. But below is the main block diagram of their approach.

解决方案

I don't have access to the paper so I just build an example like the one your draw:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers, models

class ConvBlock(layers.Layer):
    def __init__(self, kernel_num=32, kernel_size=(3,3), strides=(1,1), padding='same'):
        super(ConvBlock, self).__init__()
        # conv layer
        self.conv = layers.Conv2D(kernel_num, 
                        kernel_size=kernel_size, 
                        strides=strides, padding=padding)
        # batch norm layer
        self.bn = layers.BatchNormalization()

    def call(self, input_tensor, training=False):
        x = self.conv(input_tensor)
        x = self.bn(x, training=training)
        return tf.nn.relu(x)

vision_model = keras.applications.DenseNet121(
    input_shape=(224,224,3),
    include_top = False,
    weights='imagenet')

# Control freeze and unfreeze over blocks
def set_freeze(block, unfreeze):
    for layer in block:
        layer.trainable = unfreeze

block_1 = vision_model.layers[:7]
block_2 = vision_model.layers[7:53]
block_3 = vision_model.layers[53:141]
block_4 = vision_model.layers[141:313]
block_5 = vision_model.layers[313:]

set_freeze(block_1, unfreeze=False)
set_freeze(block_2, unfreeze=False)

for i, layer in enumerate(vision_model.layers):
    print(i,'\t',layer.trainable,'\t  :',layer.name)

layer_names = ['conv3_block12_concat', 'conv4_block24_concat', 'conv5_block16_concat']
vision_model_outputs = [vision_model.get_layer(name).output for name in layer_names]

custom_0 = ConvBlock()(vision_model_outputs[0])
custom_1 = ConvBlock()(layers.UpSampling2D(2)(vision_model_outputs[1]))
cat_layer = layers.concatenate([custom_0, custom_1])

last_conv_num = 2
custom_2 = layers.UpSampling2D(4)(vision_model_outputs[2])
outputs = layers.concatenate([ConvBlock()(cat_layer) for i in range(last_conv_num)] + [custom_2])
model = models.Model(vision_model.input, outputs)

keras.utils.plot_model(model, "./Model_structure.png", show_shapes=True)

Run the code and you will see the block1 and block2 are frozen,

Because the plot of full model is long so I just post few snippet of it:

这篇关于如何在内置的预训练模型中放置自定义层?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！