Tensorflow estimator:切换到 care_interpolation 以获得模型的正确 PR-AUC [英] Tensorflow estimator: Switching to careful_interpolation to get the correct PR-AUC of a model

问题描述

在我的项目中，我使用的是预制的估算器 DNNClassifier.这是我的估算器:

In my project, I am using the premade estimator DNNClassifier. Here is my estimator:

model = tf.estimator.DNNClassifier(
        hidden_units=network,
        feature_columns=feature_cols,
        n_classes= 2,
        activation_fn=tf.nn.relu,
        optimizer=tf.train.ProximalAdagradOptimizer(
            learning_rate=0.1,
            l1_regularization_strength=0.001
        ),
        config=chk_point_run_config,
        model_dir=MODEL_CHECKPOINT_DIR
    )

当我使用 eval_res = model.evaluate(..) 评估模型时，我收到以下警告:

when I evaluate the model using eval_res = model.evaluate(..), I get the following warning:

警告:张量流:已知梯形规则会产生不正确的 PR-AUC；请改用careful_interpolation".

WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

如何切换到 care_interpolation 以从 evaluate() 方法获得正确的结果?

How I can switch to careful_interpolation to get the correct results from the evaluate() method?

Tensorflow 版本:1.8

Tensorflow version: 1.8

推荐答案

不幸的是，使用预制的估算器几乎没有定制评估过程的自由.目前，DNNClassifier 似乎没有提供一种方法来调整评估指标，对于其他估计器也是如此.

Unfortunately, the use of a pre-made estimator leaves little freedom for customizing the evaluation process. Currently, a DNNClassifier does not seem to provide a means to adjust the evaluation metrics, likewise for other estimators.

虽然不理想，但一种解决方案是使用 tf.contrib.metrics.add_metrics，如果将完全相同的键分配给新指标，它将替换旧指标:

Albeit not ideal, one solution is to augment an estimator with the desired metrics using tf.contrib.metrics.add_metrics, which will replace the old metric if the exact same key is assigned to the new one:

如果此指标与估算器现有指标之间存在名称冲突，则会覆盖现有指标.

If there is a name conflict between this and estimators existing metrics, this will override the existing one.

它具有适用于任何产生概率预测的估计器的优势，但代价是仍然为每个评估计算覆盖的指标.DNNClassifier 估计器在键 'logistic' 下提供逻辑值(介于 0 和 1 之间)(固定估计器中可能的键列表是 这里).对于其他估算器头，情况可能并非总是如此，但可能有替代方案:在使用 tf.contrib.estimator.multi_label_head，logistic 不可用，但 probabilities 可以改为使用.

It comes with the advantage of working for any estimator that produces probabilistic predictions, at the expense of still calculating the overridden metric for each evaluation. A DNNClassifier estimator provides logistic values (between 0 and 1) under the key 'logistic' (the list of possible keys in canned estimators are here). This might not always be the case for other estimator heads, but alternatives may be available: in a multi-label classifier built with tf.contrib.estimator.multi_label_head, logistic is not available, but probabilities can be used instead.

因此，代码如下所示:

def metric_auc(labels, predictions):
    return {
        'auc_precision_recall': tf.metrics.auc(
            labels=labels, predictions=predictions['logistic'], num_thresholds=200,
            curve='PR', summation_method='careful_interpolation')
    }

estimator = tf.estimator.DNNClassifier(...)
estimator = tf.contrib.estimator.add_metrics(estimator, metric_auc)

评估时，警告信息仍会出现，但稍后会调用经过仔细插值的AUC.将此指标分配给不同的键还可以让您检查两种求和方法之间的差异.我对多标签逻辑回归任务的测试表明，测量值确实可能略有不同:auc_precision_recall = 0.05173396，auc_precision_recall_careful = 0.05059402.

When evaluating, the warning message will still appear, but the AUC with careful interpolation will be called shortly afterwards. Assigning this metric to a different key would also allow you to check the discrepancy between the two summation methods. My tests on a multi-label logistic regression task show that the measurements may indeed be slightly different: auc_precision_recall = 0.05173396, auc_precision_recall_careful = 0.05059402.

尽管 文档 建议仔细插值是严格首选".正如 在拉取请求 #19079 中所评论的那样，该更改将显着向后不兼容.对同一拉取请求的后续评论建议采用上述解决方法.

There is also a reason why the default summation method is still 'trapezoidal', in spite of the documentation suggesting that careful interpolation is "strictly preferred". As commented in pull request #19079, the change would be significantly backwards incompatible. Subsequent comments on the same pull request suggest the workaround above.

这篇关于Tensorflow estimator:切换到 care_interpolation 以获得模型的正确 PR-AUC的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！