具有附加动态参数的Keras损失函数 [英] Keras Loss Function with Additional Dynamic Parameter

问题描述

我正在努力为Deep-q网络实施优先体验回放，该规范的一部分是将梯度乘以重要性抽样(IS)权重.在下面的论文的第3.4节中讨论了梯度修改: https://arxiv.org/pdf/1511.05952. pdf 我正在努力创建一个自定义损失函数，该函数除y_true和y_pred之外还接受一系列IS权重.

I'm working on implementing prioritized experience replay for a deep-q network, and part of the specification is to multiply gradients by what's know as importance sampling (IS) weights. The gradient modification is discussed in section 3.4 of the following paper: https://arxiv.org/pdf/1511.05952.pdf I'm struggling with creating a custom loss function that takes in an array of IS weights in addition to y_true and y_pred.

这是我模型的简化版本:

Here's a simplified version of my model:

import numpy as np
import tensorflow as tf

# Input is RAM, each byte in the range of [0, 255].
in_obs = tf.keras.layers.Input(shape=(4,))

# Normalize the observation to the range of [0, 1].
norm = tf.keras.layers.Lambda(lambda x: x / 255.0)(in_obs)

# Hidden layers.
dense1 = tf.keras.layers.Dense(128, activation="relu")(norm)
dense2 = tf.keras.layers.Dense(128, activation="relu")(dense1)
dense3 = tf.keras.layers.Dense(128, activation="relu")(dense2)
dense4 = tf.keras.layers.Dense(128, activation="relu")(dense3)

# Output prediction, which is an action to take.
out_pred = tf.keras.layers.Dense(2, activation="linear")(dense4)

opt     = tf.keras.optimizers.Adam(lr=5e-5)
network = tf.keras.models.Model(inputs=in_obs, outputs=out_pred)
network.compile(optimizer=opt, loss=huber_loss_mean_weighted)

这是我的自定义损失函数，它只是Huber损失乘以IS权重的实现:

Here's my custom loss function, which is just an implementation of Huber Loss multiplied by the IS weights:

'''
 ' Huber loss: https://en.wikipedia.org/wiki/Huber_loss
'''
def huber_loss(y_true, y_pred):
  error = y_true - y_pred
  cond  = tf.keras.backend.abs(error) < 1.0

  squared_loss = 0.5 * tf.keras.backend.square(error)
  linear_loss  = tf.keras.backend.abs(error) - 0.5

  return tf.where(cond, squared_loss, linear_loss)

'''
 ' Importance Sampling weighted huber loss.
'''
def huber_loss_mean_weighted(y_true, y_pred, is_weights):
  error = huber_loss(y_true, y_pred)

  return tf.keras.backend.mean(error * is_weights)

重要的一点是is_weights是动态的，即每次调用fit()时它都是不同的.因此，我不能像下面描述的那样简单地关闭is_weights:在keras中自定义损失功能

The important bit is that is_weights is dynamic, i.e. it's different each time fit() is called. As such, I cannot simply close over is_weights as described here: Make a custom loss function in keras

我在网上找到了此代码，该代码似乎使用Lambda层来计算损失:

I found this code online, which appears to use a Lambda layer to compute the loss: https://github.com/keras-team/keras/blob/master/examples/image_ocr.py#L475 It looks promising, but I'm struggling to understand it/adapt it to my particular problem. Any help is appreciated.

推荐答案

确定.这是一个例子.

OK. Here is an example.

from keras.layers import Input, Dense, Conv2D, MaxPool2D, Flatten
from keras.models import Model
from keras.losses import categorical_crossentropy

def sample_loss( y_true, y_pred, is_weight ) :
    return is_weight * categorical_crossentropy( y_true, y_pred ) 

x = Input(shape=(32,32,3), name='image_in')
y_true = Input( shape=(10,), name='y_true' )
is_weight = Input(shape=(1,), name='is_weight')
f = Conv2D(16,(3,3),padding='same')(x)
f = MaxPool2D((2,2),padding='same')(f)
f = Conv2D(32,(3,3),padding='same')(f)
f = MaxPool2D((2,2),padding='same')(f)
f = Conv2D(64,(3,3),padding='same')(f)
f = MaxPool2D((2,2),padding='same')(f)
f = Flatten()(f)
y_pred = Dense(10, activation='softmax', name='y_pred' )(f)
model = Model( inputs=[x, y_true, is_weight], outputs=y_pred, name='train_only' )
model.add_loss( sample_loss( y_true, y_pred, is_weight ) )
model.compile( loss=None, optimizer='sgd' )
print model.summary()

请注意，由于您已经通过add_loss()添加了亏损，因此您不必通过compile( loss=xxx )进行亏损.

Note, since you've add loss through add_loss(), you don't have to do it through compile( loss=xxx ).

关于训练模型，除了将y_true移至输入端外，没有什么特别的.见下文

With regards to train a model, nothing is special except you move y_true to your input end. See below

import numpy as np 
a = np.random.randn(8,32,32,3)
a_true = np.random.randn(8,10)
a_is_weight = np.random.randint(0,2,size=(8,1))
model.fit( [a, a_true, a_is_weight] )

最后，您可以制作一个测试模型(在model中共享所有权重)，以方便使用，即

Finally, you can make a testing model (which share all weights in model) for easier use, i.e.

test_model = Model( inputs=x, outputs=y_pred, name='test_only' )
a_pred = test_model.predict( a )

这篇关于具有附加动态参数的Keras损失函数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！