删除高度相关的变量 [英] Remove highly correlated variables

查看:152
本文介绍了删除高度相关的变量的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我有一个巨大的数据框5600 X 6592,并且我想删除彼此之间相关性超过0.99的任何变量,我确实知道如何一步一步地做到这一点,即逐步形成相关矩阵,将值取整,删除相似的数据并使用索引再次获取我的精简数据。

I have a huge dataframe 5600 X 6592 and I want to remove any variables that are correlated to each other more than 0.99 I do know how to do this the long way, step by step i.e. forming a correlation matrix, rounding the values, removing similar ones and use the indexing to get my "reduced" data again.

cor(mydata)
mydata <- round(mydata,2)
mydata <- mydata[,!duplicated (mydata)]
## then do the indexing...

我想知道这是否可以用短命令或某些高级功能完成。我正在学习如何使用R语言中的强大工具,从而避免了这么长的不必要命令

I would like to know if this could be done in short command, or some advanced function. I'm learning how to make use of the powerful tools in the R language, which avoids such long unnecessary commands

我在想类似

mydata <- mydata[, which(apply(mydata, 2, function(x) !duplicated(round(cor(x),2))))]

对不起,我知道上面的命令不起作用,但是我希望我能够为此。

Sorry I know the above command doesn't work, but I hope I would be able to do this.

适用于该问题的播放数据:

a play-data that applies to the question:

mydata <- structure(list(V1 = c(1L, 2L, 5L, 4L, 366L, 65L, 43L, 456L, 876L, 
78L, 687L, 378L, 378L, 34L, 53L, 43L), V2 = c(2L, 2L, 5L, 4L, 
366L, 65L, 43L, 456L, 876L, 78L, 687L, 378L, 378L, 34L, 53L, 
41L), V3 = c(10L, 20L, 10L, 20L, 10L, 20L, 1L, 0L, 1L, 2010L, 
20L, 10L, 10L, 10L, 10L, 10L), V4 = c(2L, 10L, 31L, 2L, 2L, 5L, 
2L, 5L, 1L, 52L, 1L, 2L, 52L, 6L, 2L, 1L), V5 = c(4L, 10L, 31L, 
2L, 2L, 5L, 2L, 5L, 1L, 52L, 1L, 2L, 52L, 6L, 2L, 3L)), .Names = c("V1", 
"V2", "V3", "V4", "V5"), class = "data.frame", row.names = c(NA, 
-16L))

非常感谢

推荐答案

我相信有很多方法要做到这一点,当然比这要好一些,但这应该可行。我基本上只是将上三角设置为零,然后删除值超过0.99的所有行。

I'm sure there are many ways to do this and certainly some better than this, but this should work. I basically just set the upper triangle to be zero and then remove any rows that have values over 0.99.

> tmp <- cor(data)
> tmp[upper.tri(tmp)] <- 0
> diag(tmp) <- 0
# Above two commands can be replaced with 
# tmp[!lower.tri(tmp)] <- 0
#
> 
> data.new <- data[,!apply(tmp,2,function(x) any(x > 0.99))]
> head(data.new)
   V2 V3 V5
1   2 10  4
2   2 20 10
3   5 10 31
4   4 20  2
5 366 10  2
6  65 20  5

这篇关于删除高度相关的变量的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆