具有替代性能指标的 R 插入符号模型评估 [英] R caret model evaluation with alternate performance metric

问题描述

我正在使用 R 的 caret 包进行一些网格搜索和模型评估.我有一个自定义评估指标，它是绝对误差的加权平均值.在观察级别分配权重.

I'm using R's caret package to do some grid search and model evaluation. I have a custom evaluation metric that is a weighted average of absolute error. Weights are assigned at the observation level.

X <- c(1,1,2,0,1) #feature 1
w <- c(1,2,2,1,1) #weights
Y <- 1:5 #target, continuous

#assume I run a model using X as features and Y as target and get a vector of predictions

mymetric <- function(predictions, target, weights){

v <- sum(abs(target-predictions)*weights)/sum(weights) 
return(v)
}

这里举例说明如何使用summaryFunction来定义插入符号 train() 的自定义评估指标.引用:

Here an example is given on how to use summaryFunction to define a custom evaluation metric for caret's train(). To quote:

trainControl 函数有一个名为 summaryFunction 的参数，它指定了一个用于计算性能的函数.该函数应具有以下参数:

The trainControl function has a argument called summaryFunction that specifies a function for computing performance. The function should have these arguments:

data 是数据框或矩阵的引用，其中的列称为 obs并预测观察到的和预测的结果值(数字用于回归的数据或用于分类的字符值).目前，类概率不会传递给函数.这数据中的值是保留的预测(及其相关的参考值)用于单个调谐参数组合.如果trainControl 对象的 classProbs 参数设置为 TRUE，将出现包含该类的数据中的其他列概率.这些列的名称与类相同水平.lev 是具有结果因子水平的字符串取自训练数据.对于回归，NULL 值是传入函数.模型是模型的字符串正在使用(即传递给 train 的方法参数的值).

data is a reference for a data frame or matrix with columns called obs and pred for the observed and predicted outcome values (either numeric data for regression or character values for classification). Currently, class probabilities are not passed to the function. The values in data are the held-out predictions (and their associated reference values) for a single combination of tuning parameters. If the classProbs argument of the trainControl object is set to TRUE, additional columns in data will be present that contains the class probabilities. The names of these columns are the same as the class levels. lev is a character string that has the outcome factor levels taken from the training data. For regression, a value of NULL is passed into the function. model is a character string for the model being used (i.e. the value passed to the method argument of train).

我不太明白如何将观察权重传递给 summaryFunction.

I cannot quite figure out how to pass the observation weights to summaryFunction.

推荐答案

您不能将权重直接传递给汇总函数，这是一个疏忽，因为您可以将它们传递给建模函数.如果基础模型适应权重，它们将用于生成预测值.

You can't pass in weights directly to the summary function, which is an oversight since you can pass them to the modeling function. If the underlying model accommodates weights, they are used to produce the predicted values.

我会将其添加到下一个版本中.

I'll add that to the next release.

最大

这篇关于具有替代性能指标的 R 插入符号模型评估的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！