加权平均:自定义层权重在 TensorFlow 2.2.0 中不会改变 [英] Weighted Average: Custom layer weights don't change in TensorFlow 2.2.0

问题描述

我正在尝试在 TensorFlow 中实现两个张量之间的加权平均值，其中可以自动学习权重.遵循有关如何为 keras 模型设计自定义层的建议 此处，我的尝试如下:

I am trying to implement a weighted average between two tensors in TensorFlow, where the weight can be learned automatically. Following the advice on how to design a custom layer for a keras model here, my attempt is the following:

class WeightedAverage(tf.keras.layers.Layer):
    def __init__(self):
        super(WeightedAverage, self).__init__()

        init_value = tf.keras.initializers.Constant(value=0.5)

        self.w = self.add_weight(name="weight",
                                 initializer=init_value,
                                 trainable=True)

    def call(self, inputs):
        return tf.keras.layers.average([inputs[0] * self.w,
                                        inputs[1] * (1 - self.w)])

现在的问题是，在训练模型、保存并再次加载之后，w 的值仍然是 0.5.参数是否有可能没有收到任何梯度更新?打印模型的可训练变量时，会列出参数，因此在调用 model.fit 时应包含该参数.

Now the problem is that after training the model, saving, and loading it again, the value for w remains 0.5. Is it possible that the parameter does not receive any gradient updates? When printing the trainable variables of my model, the parameter is listed and should therefore be included when calling model.fit.

推荐答案

这里有一种可能在两个张量之间实现加权平均，权重可以自动学习.我还介绍了权重总和必须为 1 的约束.为了实现这一点，我们必须简单地对我们的权重应用 softmax.在下面的虚拟示例中，我将两个完全连接的分支的输出与此方法相结合，但您可以在其他所有场景中对其进行管理

Here is a possibility to implement a weighted average between two tensors, where the weight can be learned automatically. I also introduce the constrain that the weights must sum up to 1. To grant this we have to simply apply a softmax on our weights. In the dummy example below I combine with this method the output of two fully-connected branches but you can manage it in every other scenario

这里是自定义层:

class WeightedAverage(Layer):
    
    def __init__(self):
        super(WeightedAverage, self).__init__()
        
    def build(self, input_shape):
        
        self.W = self.add_weight(
                    shape=(1,1,len(input_shape)),
                    initializer='uniform',
                    dtype=tf.float32,
                    trainable=True)
        
    def call(self, inputs):

        # inputs is a list of tensor of shape [(n_batch, n_feat), ..., (n_batch, n_feat)]
        # expand last dim of each input passed [(n_batch, n_feat, 1), ..., (n_batch, n_feat, 1)]
        inputs = [tf.expand_dims(i, -1) for i in inputs]
        inputs = Concatenate(axis=-1)(inputs) # (n_batch, n_feat, n_inputs)
        weights = tf.nn.softmax(self.W, axis=-1) # (1,1,n_inputs)
        # weights sum up to one on last dim

        return tf.reduce_sum(weights*inputs, axis=-1) # (n_batch, n_feat) 

这里是回归问题的完整示例:

here the full example in a regression problem:

inp1 = Input((100,))
inp2 = Input((100,))
x1 = Dense(32, activation='relu')(inp1)
x2 = Dense(32, activation='relu')(inp2)
W_Avg = WeightedAverage()([x1,x2])
out = Dense(1)(W_Avg)

m = Model([inp1,inp2], out)
m.compile('adam','mse')

n_sample = 1000
X1 = np.random.uniform(0,1, (n_sample,100))
X2 = np.random.uniform(0,1, (n_sample,100))
y = np.random.uniform(0,1, (n_sample,1))

m.fit([X1,X2], y, epochs=10)

最后，您还可以通过这种方式可视化权重的值:

in the end, you can also visualize the value of the weights in this way:

tf.nn.softmax(m.get_weights()[-3]).numpy()

这篇关于加权平均:自定义层权重在 TensorFlow 2.2.0 中不会改变的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！