使用Vowpal Wabbit时计算AUC [英] Calculating AUC when using Vowpal Wabbit

问题描述

反正是在Vowpal Wabbit中计算AUC吗?

Is there anyway to compute AUC within Vowpal Wabbit?

我使用Vowpal Wabbit的原因之一是数据文件很大. 我可以使用Vowpal Wabbit的输出在Vowpal Wabbit环境之外计算AUC，但是如果数据文件很大，这可能会出现问题.

One of the reasons I am using Vowpal Wabbit is the large size of the data file. I can calculate the AUC outside of the Vowpal Wabbit environment using the output of Vowpal Wabbit but this might be problematic if the data file is large.

推荐答案

当前，大众汽车无法报告 AUC .更糟糕的是，它无法直接针对AUC进行优化.针对AUC进行优化与在线学习不兼容，但是有一些适合进行优化的AUC近似值.

Currently, VW cannot report AUC. What is worse, it cannot optimize directly for AUC. Optimizing for AUC is not compatible with online learning, but there are some approximations of AUC suitable for optimizing.

关于您的问题，您不需要将带有原始预测的中间文件存储在磁盘上.您可以将其直接传输到外部评估工具(在这种情况下，是 perf ):

Concerning your question, you don't need to store the intermediate file with raw predictions on disk. You can pipe it directly to the external evaluation tool (perf in this case):

vw -d test.data -t -i model.vw -r /dev/stdout | perf -roc -files gold /dev/stdin

约翰·兰福德(John Langford)确认通常可以通过更改误报率和误报率来优化AUC.负损失.在大众汽车中，这意味着为正面和负面的示例设置不同的重要性权重.您需要使用保留集(或交叉验证，或用于单次学习的渐进式验证损失)来调整最佳权重.

John Langford confirmed that AUC can generally be optimized by changing the ratio of false positive and false negative loss. In VW, this means setting a different importance weight for positive and negative examples. You need to tune the optimal weight using a hold out set (or cross validation, or progressive validation loss for one-pass learning).

这篇关于使用Vowpal Wabbit时计算AUC的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！