使用自定义指标的标准偏差选择带有插入符号的调整参数 [英] Selecting tuning parameters with caret using standard deviation of custom metric

问题描述

我正在使用带有自定义拟合指标的插入符号，但我不仅需要最大化该指标，还需要最大化其置信区间的下限.所以我想最大化类似 mean(metric) - k * stddev(metric) 的东西.我知道如何手动执行此操作，但是有没有办法告诉 caret 使用此函数自动选择最佳参数?

I'm using caret with custom fitting metric, but I need to maximize not just this metric but lower bound of it's confidence interval. So I'd like to maximize something like mean(metric) - k * stddev(metric). I know how to do this manually, but is there a way to tell caret to automatically select best parameters using this function?

推荐答案

是的，您可以通过trainControl"对象的summaryFunction"参数定义您自己的选择指标，然后使用您调用的metric"参数train().有关这方面的详细信息，请参阅 caret 模型调整页面的替代性能指标"部分:http://caret.r-forge.r-project.org/training.html

Yes, you can define your own selection metric through the "summaryFunction" parameter of your "trainControl" object and then with the "metric" parameter of your call to train(). Details on this are pretty well documented in the "Alternate Performance Metrics" section on caret's model tuning page: http://caret.r-forge.r-project.org/training.html

我认为您没有提供足够的信息让任何人准确地编写您要查找的内容，但这里有一个使用 twoClassSummary 函数中的代码的示例:

I don't think you gave enough information for anyone to write exactly what you're looking for, but here is an example using the code from the twoClassSummary function:

> library(caret)
> data(Titanic)
> 
> #an example custom function
> roc <- function (data, lev = NULL, model = NULL) {
+   require(pROC)
+   if (!all(levels(data[, "pred"]) == levels(data[, "obs"]))) 
+     stop("levels of observed and predicted data do not match")
+   rocObject <- try(pROC:::roc(data$obs, data[, lev[1]]), silent = TRUE)
+   rocAUC <- if (class(rocObject)[1] == "try-error") 
+     NA
+   else rocObject$auc
+   out <- c(rocAUC, sensitivity(data[, "pred"], data[, "obs"], lev[1]), specificity(data[, "pred"], data[, "obs"], lev[2]))
+   names(out) <- c("ROC", "Sens", "Spec")
+   out
+ }
> 
> #your train control specs
> tc <- trainControl(method="cv",classProb=TRUE,summaryFunction=roc)
> #yoru model with selection metric specificed
> train(Survived~.,data=data.frame(Titanic),method="rf",trControl=tc,metric="ROC")
32 samples
 4 predictors
 2 classes: 'No', 'Yes' 

No pre-processing
Resampling: Cross-Validation (10 fold) 

Summary of sample sizes: 28, 29, 30, 30, 28, 28, ... 

Resampling results across tuning parameters:

  mtry  ROC    Sens  Spec  ROC SD  Sens SD  Spec SD
  2     0.9    0.2   0.25  0.175   0.35     0.425  
  4     0.85   0.4   0.6   0.211   0.459    0.459  
  6     0.875  0.35  0.6   0.212   0.412    0.459  

ROC was used to select the optimal model using  the largest value.
The final value used for the model was mtry = 2. 

这篇关于使用自定义指标的标准偏差选择带有插入符号的调整参数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！