使用测试数据进行统计测试 [英] Statistical test with test-data
问题描述
如果我使用带插入符号的两种方法(NN 和 KNN)，然后我想提供显着性检验，我该如何进行 wilcoxon 检验.
If I am using two method (NN and KNN) with caret and then I want to provide significance test, how can I do wilcoxon test.
我提供的数据样本如下
structure(list(Input = c(25, 193, 70, 40), Output = c(150, 98,
27, 60), Inquiry = c(75, 70, 0, 20), File = c(60, 36, 12, 12),
FPAdj = c(1, 1, 0.8, 1.15), RawFPcounts = c(1750, 1902, 535,
660), AdjFP = c(1750, 1902, 428, 759), Effort = c(102.4,
105.2, 11.1, 21.1)), row.names = c(NA, 4L), class = "data.frame")
d=readARFF("albrecht.arff")
index <- createDataPartition(d$Effort, p = .70,list = FALSE)
tr <- d[index, ]
ts <- d[-index, ]
boot <- trainControl(method = "repeatedcv", number=100)
cart1 <- train(log10(Effort) ~ ., data = tr,
method = "knn",
metric = "MAE",
preProc = c("center", "scale", "nzv"),
trControl = boot)
postResample(predict(cart1, ts), log10(ts$Effort))
cart2 <- train(log10(Effort) ~ ., data = tr,
method = "knn",
metric = "MAE",
preProc = c("center", "scale", "nzv"),
trControl = boot)
postResample(predict(cart2, ts), log10(ts$Effort))
如何在此处执行 wilcox.test().
Warm regards
推荐答案
处理问题的一种方法是为 knn 和 NN 生成多个性能值，您可以使用统计测试进行比较.这可以使用嵌套重采样来实现.
One way to deal with your problem is to generate several performance values for knn and NN which you can compare using a statistical test. This can be achieved using Nested resampling.
在嵌套重采样中，您要多次执行训练/测试拆分并在每个测试集上评估模型.
In nested resampling you are performing train/test splits multiple times and evaluating the model on each test set.
例如让我们使用 BostonHousing 数据:
Lets for instance use BostonHousing data:
library(caret)
library(mlbench)
data(BostonHousing)
让我们为示例选择数字列以使其简单:
lets just select numerical columns for the example to make it simple:
d <- BostonHousing[,sapply(BostonHousing, is.numeric)]
据我所知，没有办法在开箱即用的插入符号中执行嵌套的 CV，因此需要一个简单的包装器:
As far as I know there is no way to perform nested CV in caret out of the box so a simple wrapper is needed:
为嵌套的 CV 生成外部折叠:
generate outer folds for nested CV:
outer_folds <- createFolds(d$medv, k = 5)
让我们使用引导重采样作为内部重采样循环来调整超参数:
Lets use bootstrap resampling as the inner resample loop to tune the hyper parameters:
boot <- trainControl(method = "boot",
number = 100)
现在循环外部折叠并使用训练集执行超参数优化并在测试集上进行预测:
now loop over the outer folds and perform hyper parameter optimization using the train set and predict on the test set:
CV_knn <- lapply(outer_folds, function(index){
tr <- d[-index, ]
ts <- d[index,]
cart1 <- train(medv ~ ., data = tr,
method = "knn",
metric = "MAE",
preProc = c("center", "scale", "nzv"),
trControl = boot,
tuneLength = 10) #to keep it short we will just probe 10 combinations of hyper parameters
postResample(predict(cart1, ts), ts$medv)
})
仅从结果中提取 MAE:
extract just MAE from the results:
sapply(CV_knn, function(x) x[3]) -> CV_knn_MAE
CV_knn_MAE
#output
Fold1.MAE Fold2.MAE Fold3.MAE Fold4.MAE Fold5.MAE
2.503333 2.587059 2.031200 2.475644 2.607885
例如对 glmnet 学习者做同样的事情:
Do the same for glmnet learner for instance:
CV_glmnet <- lapply(outer_folds, function(index){
tr <- d[-index, ]
ts <- d[index,]
cart1 <- train(medv ~ ., data = tr,
method = "glmnet",
metric = "MAE",
preProc = c("center", "scale", "nzv"),
trControl = boot,
tuneLength = 10)
postResample(predict(cart1, ts), ts$medv)
})
sapply(CV_glmnet, function(x) x[3]) -> CV_glmnet_MAE
CV_glmnet_MAE
#output
Fold1.MAE Fold2.MAE Fold3.MAE Fold4.MAE Fold5.MAE
3.400559 3.383317 2.830140 3.605266 3.525224
现在使用 wilcox.test 比较两者.由于两个学习器的性能是使用相同的数据拆分生成的，因此配对测试是合适的:
now compare the two using wilcox.test. Since the performance for both learners was generated using the same data splits a paired test is appropriate:
wilcox.test(CV_knn_MAE,
CV_glmnet_MAE,
paired = TRUE)
如果比较两种以上的算法，可以使用 Friedman.test
If comparing more than two algorithms one can use friedman.test
这篇关于使用测试数据进行统计测试的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
