使用自定义张量作为变量的TensorFlow 2.0 Keras层 [英] TensorFlow 2.0 Keras layers with custom tensors as variables

问题描述

在TF 1.x中，可以使用自定义变量构建图层.这是一个示例:

In TF 1.x, it was possible to build layers with custom variables. Here's an example:

import numpy as np
import tensorflow as tf

def make_custom_getter(custom_variables):
    def custom_getter(getter, name, **kwargs):
        if name in custom_variables:
            variable = custom_variables[name]
        else:
            variable = getter(name, **kwargs)
        return variable
    return custom_getter

# Make a custom getter for the dense layer variables.
# Note: custom variables can result from arbitrary computation;
#       for the sake of this example, we make them just constant tensors.
custom_variables = {
    "model/dense/kernel": tf.constant(
        np.random.rand(784, 64), name="custom_kernel", dtype=tf.float32),
    "model/dense/bias": tf.constant(
        np.random.rand(64), name="custom_bias", dtype=tf.float32),
}
custom_getter = make_custom_getter(custom_variables)

# Compute hiddens using a dense layer with custom variables.
x = tf.random.normal(shape=(1, 784), name="inputs")
with tf.variable_scope("model", custom_getter=custom_getter):
    Layer = tf.layers.Dense(64)
    hiddens = Layer(x)

print(Layer.variables)

构造的密集层的打印变量将是我们在custom_variables字典中指定的自定义张量:

The printed variables of the constructed dense layer will be custom tensors we specified in the custom_variables dict:

[<tf.Tensor 'custom_kernel:0' shape=(784, 64) dtype=float32>, <tf.Tensor 'custom_bias:0' shape=(64,) dtype=float32>]

这允许我们创建直接使用custom_variables中提供的张量作为其权重的层/模型，以便我们可以相对于custom_variables可能依赖的任何张量进一步区分层/模型的输出(对于调制子网，元学习等).

This allows us to create layers/models that use provided tensors in custom_variables directly as their weights, so that we could further differentiate the output of the layers/models with respect to any tensors that custom_variables may depend on (particularly useful for implementing functionality in modulating sub-nets, parameter generation, meta-learning, etc.).

可变范围用于轻松地将带有自定义getter的所有图构建嵌套在范围内，并在提供的张量作为参数的基础上构建模型.由于在TF 2.0中不再建议使用会话和可变作用域(并且所有这些低级内容已移至tf.compat.v1)，因此，使用Keras和TF来实现上述目标的最佳做法 2.0?

Variable scopes used to make it easy to nest all off graph-building inside scopes with custom getters and build models on top of the provided tensors as their parameters. Since sessions and variable scopes are no longer advisable in TF 2.0 (and all of that low-level stuff is moved to tf.compat.v1), what would be the best practice to implement the above using Keras and TF 2.0?

(相关在GitHub上发布.)

推荐答案

根据下面的评论进行回答

鉴于您有

kernel = createTheKernelVarBasedOnWhatYouWant() #shape (784, 64)
bias = createTheBiasVarBasedOnWhatYouWant() #shape (64,)

做一个简单的函数，从Dense复制代码:

Make a simple function copying the code from Dense:

def custom_dense(x):
    inputs, kernel, bias = x

    outputs = K.dot(inputs, kernel)
    outputs = K.bias_add(outputs, bias, data_format='channels_last')
    return outputs

在Lambda层中使用该功能:

layer = Lambda(custom_dense)
hiddens = layer([x, kernel, bias])

警告:kernel和bias必须来自Keras层，或者来自kernel = Input(tensor=the_kernel_var)和bias = Input(tensor=bias_var)

Warning: kernel and bias must be produced from a Keras layer, or come from an kernel = Input(tensor=the_kernel_var) and bias = Input(tensor=bias_var)

---

如果上面的警告对您不利，您可以随时使用kernel和bias来自外部，例如:

---

If the warning above is bad for you, you can always use kernel and bias "from outside", like:

def custom_dense(inputs):
    outputs = K.dot(inputs, kernel) #where kernel is not part of the arguments anymore
    outputs = K.bias_add(outputs, bias, data_format='channels_last')
    return outputs

layer = Lambda(custom_dense)
hiddens = layer(x)

最后一个选项使保存/加载模型更加复杂.

This last option makes it a bit more complicated to save/load models.

您可能应该使用Keras Dense图层并以标准方式设置其权重:

You should probably use a Keras Dense layer and set its weights in a standard way:

layer = tf.keras.layers.Dense(64, name='the_layer')
layer.set_weights([np.random.rand(784, 64), np.random.rand(64)])

如果您需要这些权重不可训练，请在编译设置的keras模型之前:

If you need that these weights are not trainable, before compiling the keras model you set:

model.get_layer('the_layer').trainable=False

如果要直接访问张量变量，它们是:

If you want direct access to the variables as tensors, they are:

kernel = layer.kernel    
bias = layer.bias

---

还有许多其他选择，但这取决于您的确切意图，这在您的问题中尚不清楚.

---

There are plenty of other options, but that depends on your exact intention, which is not clear in your question.

这篇关于使用自定义张量作为变量的TensorFlow 2.0 Keras层的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！