在训练/动态正则化期间更改 keras 正则化器 [英] Change keras regularizer during training / dynamic regularization

问题描述

我正在寻找一种正确的方法来在训练期间实现层的权重动态正则化.作为 10 次调用后的示例，我想仅针对 MyLayer 的特定权重将 L2 正则化替换为 L1 正则化.下面显示了一个层实现的例子:

I am looking for a proper way to implement a dynamic regularization of weights for a layer during the training. As an example after 10 calls I want to replace the L2 regularization with the L1 regularization only for specific weights of MyLayer. The following shows an example of the layer implementation:

class MyLayer(tf.keras.layers.Layer):
    def __init__(...)
        some code
    def build(self, input_shape):
        self.regularizer = tf.keras.regularizers.L2() # this regularization should be changed after some steps
        self.my_weights = self.add_weight(name='myweights', shape=(self.input_dim, ),
                    initializer=tf.keras.initializers.Constant(1,),
                    regularizer= self.regularizer, trainable=True)
        self.counter = tf.Variable(0, dtype=tf.int32)
        ...
    def call(self, inputs):
        ... do some processing ...
        # for the following code i look for a proper implementation
        if self.counter > 10:
            self.regularizer = tf.keras.regularizers.L1()
        tf.keras.backend.update(self.counter,self.counter+1)

推荐答案

我们可以形成您的问题，而不是从 L1->L2，但将其形式更改为 L1L2 正则化器的参数，所以我们原则上可以改变正则化器

We could form your problem not as change the regularizer from L1->L2, but form it as change the L1L2 regularizer's parameters, so we can in principle change the regularizer

并且我们无法在编译模型后更改超参数(如果您自定义训练)，因为超参数在编译时内置于训练函数中

And we can't change the hyperparameters after compile the model (you can if you custom the training), because the hyperparameters are built-in to the training function when you compile

我们想在训练期间修改超参数，方法是在训练函数中使用后端变量并在训练期间更新这些变量

We want modify hyperparameters during training, and the way to do it is use backend variables in the training function and update those variables during training

因此我们可以定义以下自定义正则化器:

So we could define the following custom regularizer:

class L1L2_m(Regularizer):
    """Regularizer for L1 and L2 regularization.
    # Arguments
        l1: Float; L1 regularization factor.
        l2: Float; L2 regularization factor.
    """

    def __init__(self, l1=0.0, l2=0.01):
        with K.name_scope(self.__class__.__name__):
            self.l1 = K.variable(l1,name='l1')
            self.l2 = K.variable(l2,name='l2')
            self.val_l1 = l1
            self.val_l2 = l2
            
    def set_l1_l2(self,l1,l2):
        K.set_value(self.l1,l1)
        K.set_value(self.l2,l2)
        self.val_l1 = l1
        self.val_l2 = l2

    def __call__(self, x):
        regularization = 0.
        if self.val_l1 > 0.:
            regularization += K.sum(self.l1 * K.abs(x))
        if self.val_l2 > 0.:
            regularization += K.sum(self.l2 * K.square(x))
        return regularization

    def get_config(self):
        config = {'l1': float(K.get_value(self.l1)),
                  'l2': float(K.get_value(self.l2))}
        return config

添加您的自定义对象，以便在您想要导出模型时重新加载它不会有任何问题:

Add your custom object so that when you want to export your model you won't have any issue in reloading it:

from keras.utils.generic_utils import get_custom_objects
get_custom_objects().update({ L1L2_m.__name__: L1L2_m })

使用自定义对象 set_l1_l2 方法更新您的变量:

Update your variable using the custom object set_l1_l2 method:

class MyLayer(tf.keras.layers.Layer):
    def __init__(...)
        some code
    def build(self, input_shape):
        self.regularizer = L1L2_m() # this regularization should be changed after some steps
        self.my_weights = self.add_weight(name='myweights', shape=(self.input_dim, ),
                    initializer=tf.keras.initializers.Constant(1,),
                    regularizer= self.regularizer, trainable=True)
        self.counter = tf.Variable(0, dtype=tf.int32)
        ...
    def call(self, inputs):
        ... do some processing ...
        # for the following code i look for a proper implementation
        if self.counter == 10:
            self.regularizer.set_l1_l2(0.01,0.)
        tf.keras.backend.update(self.counter,self.counter+1)

参考:

如何在训练期间更改正则化参数?

这篇关于在训练/动态正则化期间更改 keras 正则化器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！