冻结的Keras模型中失落层是否仍处于活动状态(即，trainable = False)? [英] Is dropout layer still active in a freezed Keras model (i.e. trainable=False)?

问题描述

我有两个训练有素的模型(model_A和model_B)，并且两个模型都有辍学层.我冻结了model_A和model_B并将它们与新的密集层合并以获得model_AB(但是我还没有删除model_A和model_B的退出层). model_AB的权重将是不可训练的，除了增加的致密层.

I have two trained models (model_A and model_B), and both of them have dropout layers. I have freezed model_A and model_B and merged them with a new dense layer to get model_AB (but I have not removed model_A's and model_B's dropout layers). model_AB's weights will be non-trainable, except for the added dense layer.

现在我的问题是:当我训练model_AB时，model_A和model_B中的辍学层是否处于活动状态(即神经元掉落)?

Now my question is: are the dropout layers in model_A and model_B active (i.e. drop neurons) when I am training model_AB?

推荐答案

简短答案:即使您设置了trainable属性设置为False.

Short answer: The dropout layers will continue dropping neurons during training, even if you set their trainable property to False.

长答案: Keras中有两种不同的概念:

Long answer: There are two distinct notions in Keras:

• 更新图层的权重和状态:这是使用该图层的trainable属性进行控制的，即，如果设置layer.trainable = False，则该图层的权重和内部状态不会被更新.

• Updating the weights and states of a layer: this is controlled using trainable property of that layer, i.e. if you set layer.trainable = False then the weights and internal states of the layer would not be updated.

在培训和测试阶段的行为:如您所知，在培训和测试阶段，辍学之类的图层可能会有不同的行为.使用keras.backend.set_learning_phase()设置Keras中的学习阶段.例如，当您调用model.fit(...)时，学习阶段会自动设置为1(即训练)，而当您使用model.predict(...)时，它将自动设置为0(即测试).此外，请注意，学习阶段1(即训练)并不一定意味着更新层的权重/状态.您可以在学习阶段为1(即训练阶段)的情况下运行模型，但是不会更新权重；只是层将切换到其训练行为(有关更多信息，请参见此答案).此外，还有另一种方法来设置每个阶段的学习阶段，方法是在张量上调用层时传递training=True参数(请参见 this答案以获取更多信息).

Behavior of a layer in training and testing phases: as you know a layer, like dropout, may have a different behavior during training and testing phases. Learning phase in Keras is set using keras.backend.set_learning_phase(). For example, when you call model.fit(...) the learning phase is automatically set to 1 (i.e. training), whereas when you use model.predict(...) it will be automatically set to 0 (i.e. test). Further, note that learning phase of 1 (i.e. training) does not necessarily imply updating the weighs/states of a layer. You can run your model with a learning phase of 1 (i.e. training phase), but no weights will be updated; just layers will switch to their training behavior (see this answer for more information). Further, there is another way to set learning phase for each individual layer by passing training=True argument when calling a layer on a tensor (see this answer for more information).

因此，根据以上几点，当您在辍学层上设置trainable=False并将其用于训练模式时(例如，通过调用model.fit(...)或手动将学习阶段设置为训练，如下例所示)，神经元将仍然被辍学层丢弃.

So according to the above points, when you set trainable=False on a dropout layer and use that in training mode (e.g. either by calling model.fit(...), or manually setting learning phase to training like example below), the neurons would still be dropped by the dropout layer.

这是一个可重现的示例，它说明了这一点:

Here is a reproducible example which illustrates this point:

from keras import layers
from keras import models
from keras import backend as K
import numpy as np

inp = layers.Input(shape=(10,))
out = layers.Dropout(0.5)(inp)

model = models.Model(inp, out)
model.layers[-1].trainable = False  # set dropout layer as non-trainable
model.compile(optimizer='adam', loss='mse') # IMPORTANT: we must always compile model after changing `trainable` attribute

# create a custom backend function so that we can control the learning phase
func = K.function(model.inputs + [K.learning_phase()], model.outputs)

x = np.ones((1,10))
# learning phase = 1, i.e. training mode
print(func([x, 1]))
# the output will be:
[array([[2., 2., 2., 0., 0., 2., 2., 2., 0., 0.]], dtype=float32)]
# as you can see some of the neurons have been dropped

# now set learning phase = 0, i.e test mode
print(func([x, 0]))
# the output will be:
[array([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]], dtype=float32)]
# unsurprisingly, no neurons have been dropped in test phase

这篇关于冻结的Keras模型中失落层是否仍处于活动状态(即，trainable = False)?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！