Keras-如何获取非规范的logit而不是概率 [英] Keras - how to get unnormalized logits instead of probabilities

问题描述

我正在Keras中创建一个模型，并想计算自己的指标(困惑度).这需要使用未归一化的概率/对数.但是，keras模型仅返回softmax概率:

I am creating a model in Keras and want to compute my own metric (perplexity). This requires using the unnormalized probabilities/logits. However, the keras model only returns the softmax probabilties:

model = Sequential()
model.add(embedding_layer)
model.add(LSTM(n_hidden, return_sequences=False))
model.add(Dropout(dropout_keep_prob))
model.add(Dense(vocab_size))
model.add(Activation('softmax'))
optimizer = RMSprop(lr=self.lr)

model.compile(optimizer=optimizer, 
loss='sparse_categorical_crossentropy')

Keras常见问题解答提供了获取中间层输出的解决方案此处提供了另一种解决方案.但是，这些答案将中间输出存储在其他模型中，这不是我所需要的. 我想将logits用于我的自定义指标.自定义指标应包含在model.compile()函数中，以便在训练过程中对其进行评估和显示.因此，我不需要在其他模型中分隔Dense层的输出，而是将其作为我的原始模型的一部分.

The Keras FAQ have a solution to get the output of intermediate layers here. Another solution is given here. However, these answers store the intermediate outputs in a different model which is not what I need. I want to use the logits for my custom metric. The custom metric should be included in the model.compile() function such that it's evaluated and displayed during training. So I don't need the output of the Dense layer separated in a different model, but as part of my original model.

简而言之，我的问题是:

In short, my questions are:

• 在此处中定义的自定义指标时，使用def custom_metric(y_true, y_pred)进行y_pred包含logit或归一化的概率?

• When defining a custom metric as outlined here using def custom_metric(y_true, y_pred), does the y_pred contain logits or normalized probabilities?

如果它包含归一化的概率，如何获得未归一化的概率，即Dense层输出的logits?

If it contains normalized probabilities, how can I get the unnormalized probabilities, i.e. the logits output by the Dense layer?

推荐答案

我想我已经找到了解决方法

I think I have found a solution

首先，我将激活层更改为线性，以便接收@loannis Nasios概述的logit.

First, I change the activation layer to linear such that I receive logits as outlined by @loannis Nasios.

第二，为了仍然将sparse_categorical_crossentropy作为损失函数，我定义了自己的损失函数，将from_logits参数设置为true.

Second, to still get the sparse_categorical_crossentropy as a loss function, I define my own loss function, setting the from_logits parameter to true.

model.add(embedding_layer)
model.add(LSTM(n_hidden, return_sequences=False))
model.add(Dropout(dropout_keep_prob))
model.add(Dense(vocab_size))
model.add(Activation('linear'))
optimizer = RMSprop(lr=self.lr)


def my_sparse_categorical_crossentropy(y_true, y_pred):
    return K.sparse_categorical_crossentropy(y_true, y_pred, from_logits=True)

model.compile(optimizer=optimizer,loss=my_sparse_categorical_crossentropy)

这篇关于Keras-如何获取非规范的logit而不是概率的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！