在 R 中的稀疏矩阵中直接创建虚拟变量集 [英] Directly creating dummy variable set in a sparse matrix in R

问题描述

假设您有一个包含大量列(1000 个因子，每个因子有 15 个级别)的数据框.您想创建一个虚拟变量数据集，但由于它太稀疏，您希望以稀疏矩阵格式保留虚拟变量.

Suppose you have a data frame with a high number of columns(1000 factors, each with 15 levels). You'd like to create a dummy variable data set, but since it would be too sparse, you would like to keep dummies in sparse matrix format.

我的数据集很大，步骤越少，对我来说越好.我知道如何做上述步骤；但我无法直接从初始数据集创建稀疏矩阵，即一步而不是两步.有什么想法吗?

My data set is quite big and the less steps there are, the better for me. I know how to do above steps; but I couldn't get my head around directly creating that sparse matrix from the initial data set, i.e. having one step instead of two. Any ideas?

一些评论要求进一步阐述，所以这里是:

Some comments asked for further elaboration, so here it goes:

其中 X 是我的原始数据集，包含 1000 列和 50000 条记录，每列有 15 个级别，

Where X is my original data set with 1000 columns and 50000 records, each column having 15 levels,

Step1:使用如下代码从原始数据集中创建虚拟变量；

Step1: Creating dummy variables from the original data set with a code like;

# Creating dummy data set with empty values
dummified <- matrix(NA,nrow(X),15*ncol(X))
# Adding values to this data set for each column and each level within columns
for (i in 1:ncol(X)){colFactr <- factor(X[,i],exclude=NULL)
  for (j in 1:l){
    lvl <- levels(colFactr)[j]
    indx <- ((i-1)*l)+j
    dummified[,indx] <- ifelse(colFactr==lvl,1,0)
  }
}

Step2:将那个巨大的矩阵转换成稀疏矩阵，代码如下；

Step2: Converting that huge matrix into a sparse matrix, with a code like;

sparse.dummified <- sparseMatrix(dummified)

但是这种方法仍然创建了这个临时的大矩阵，这需要很多时间&记忆，因此我问的是直接方法(如果有的话).

But this approach still created this interim large matrix which takes a lot of time & memory, therefore I am asking the direct methodology (if there is any).

推荐答案

感谢您澄清您的问题，试试这个.

Thanks for having clarified your question, try this.

这是两列分别具有三个和两个级别的示例数据:

Here is sample data with two columns that have three and two levels respectively:

set.seed(123)
n <- 6
df <- data.frame(x = sample(c("A", "B", "C"), n, TRUE),
                 y = sample(c("D", "E"),      n, TRUE))
#   x y
# 1 A E
# 2 C E
# 3 B E
# 4 C D
# 5 C E
# 6 A D

library(Matrix)
spm <- lapply(df, function(j)sparseMatrix(i = seq_along(j),
                                          j = as.integer(j), x = 1))
do.call(cBind, spm)
# 6 x 5 sparse Matrix of class "dgCMatrix"
#               
# [1,] 1 . . . 1
# [2,] . . 1 . 1
# [3,] . 1 . . 1
# [4,] . . 1 1 .
# [5,] . . 1 . 1
# [6,] 1 . . 1 .

<小时>

@user20650 指出 do.call(cBind, ...) 缓慢或因大数据而失败.所以这里有一个更复杂但更快更有效的方法:

---

@user20650 pointed out do.call(cBind, ...) was sluggish or failing with large data. So here is a more complex but much faster and efficient approach:

n <- nrow(df)
nlevels <- sapply(df, nlevels)
i <- rep(seq_len(n), ncol(df))
j <- unlist(lapply(df, as.integer)) +
     rep(cumsum(c(0, head(nlevels, -1))), each = n)
x <- 1
sparseMatrix(i = i, j = j, x = x)

这篇关于在 R 中的稀疏矩阵中直接创建虚拟变量集的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！