加权平均值:TensorFlow 2.2.0中的自定义图层权重不变 [英] Weighted Average: Custom layer weights don't change in TensorFlow 2.2.0

问题描述

我正在尝试在TensorFlow中的两个张量之间实现加权平均值，在那里可以自动学习权重.遵循有关如何为keras模型设计自定义图层的建议此处，这是我的尝试如下:

I am trying to implement a weighted average between two tensors in TensorFlow, where the weight can be learned automatically. Following the advice on how to design a custom layer for a keras model here, my attempt is the following:

class WeightedAverage(tf.keras.layers.Layer):
    def __init__(self):
        super(WeightedAverage, self).__init__()

        init_value = tf.keras.initializers.Constant(value=0.5)

        self.w = self.add_weight(name="weight",
                                 initializer=init_value,
                                 trainable=True)

    def call(self, inputs):
        return tf.keras.layers.average([inputs[0] * self.w,
                                        inputs[1] * (1 - self.w)])

现在的问题是，训练模型，保存并再次加载模型后，w的值仍为0.5.参数是否可能没有收到任何渐变更新?在打印模型的可训练变量时，将列出该参数，因此在调用model.fit时应将其包括在内.

Now the problem is that after training the model, saving, and loading it again, the value for w remains 0.5. Is it possible that the parameter does not receive any gradient updates? When printing the trainable variables of my model, the parameter is listed and should therefore be included when calling model.fit.

推荐答案

这里是在两个张量之间实现加权平均值的一种可能性，其中权重可以自动获知.我还介绍了权重必须等于1的约束.要实现这一点，我们必须简单地对权重应用softmax.在下面的虚拟示例中，我将这种方法与两个完全连接的分支的输出结合起来，但是您可以在所有其他情况下进行管理

here a possibility to implement a weighted average between two tensors, where the weight can be learned automatically. I also introduce the constrain that the weights must sum up to 1. To grant this we have to simply apply a softmax on our weights. In the dummy example below I combine with this method the output of two fully-connected branches but you can manage it in every other scenario

此处是自定义图层:

class WeightedAverage(Layer):

    def __init__(self, n_output):
        super(WeightedAverage, self).__init__()
        self.W = tf.Variable(initial_value=tf.random.uniform(shape=[1,1,n_output], minval=0, maxval=1),
            trainable=True) # (1,1,n_inputs)

    def call(self, inputs):

        # inputs is a list of tensor of shape [(n_batch, n_feat), ..., (n_batch, n_feat)]
        # expand last dim of each input passed [(n_batch, n_feat, 1), ..., (n_batch, n_feat, 1)]
        inputs = [tf.expand_dims(i, -1) for i in inputs]
        inputs = Concatenate(axis=-1)(inputs) # (n_batch, n_feat, n_inputs)
        weights = tf.nn.softmax(self.W, axis=-1) # (1,1,n_inputs)
        # weights sum up to one on last dim

        return tf.reduce_sum(weights*inputs, axis=-1) # (n_batch, n_feat)

这里是回归问题的完整示例:

here the full example in a regression problem:

inp1 = Input((100,))
inp2 = Input((100,))
x1 = Dense(32, activation='relu')(inp1)
x2 = Dense(32, activation='relu')(inp2)
x = [x1,x2]
W_Avg = WeightedAverage(n_output=len(x))(x)
out = Dense(1)(W_Avg)

m = Model([inp1,inp2], out)
m.compile('adam','mse')

n_sample = 1000
X1 = np.random.uniform(0,1, (n_sample,100))
X2 = np.random.uniform(0,1, (n_sample,100))
y = np.random.uniform(0,1, (n_sample,1))

m.fit([X1,X2], y, epochs=10)

最后，您还可以通过以下方式可视化权重的值:

in the end, you can also visualize the value of the weights in this way:

tf.nn.softmax(m.get_weights()[-3]).numpy()

这篇关于加权平均值:TensorFlow 2.2.0中的自定义图层权重不变的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！