组合迭代器,如 expand.grid [英] Combinatorial iterator like expand.grid
问题描述
是否有一种快速的方法来遍历由 expand.grid
或 CJ
(data.table
) 返回的组合.当有足够的组合时,它们会变得太大而无法放入内存.itertools2
库(Python 的 itertools 的端口)中有 iproduct
,但它真的很慢(至少我使用它的方式 - 如下所示).还有其他选择吗?
Is there a fast way to iterate through combinations like those returned by expand.grid
or CJ
(data.table
). These get too big to fit in memory when there are enough combinations. There is iproduct
in itertools2
library (port of Python's itertools) but it is really slow (at least the way I'm using it - shown below). Are there other options?
这是一个示例,其想法是将函数应用于来自两个 data.frames
(上一篇).
Here is an example, where the idea is to apply a function to each combination of rows from two data.frames
(previous post).
library(data.table) # CJ
library(itertools2) # iproduct iterator
library(doParallel)
## Dimensions of two data
dim1 <- 10
dim2 <- 100
df1 <- data.frame(a = 1:dim1, b = 1:dim1)
df2 <- data.frame(x= 1:dim2, y = 1:dim2, z = 1:dim2)
## function to apply to combinations
f <- function(...) sum(...)
## Too big to expand with bigger dimensions (ie, 1e6, 1e5) -> errors
## test <- expand.grid(seq.int(dim1), seq.int(dim2))
## test <- CJ(indx1 = seq.int(dim1), indx2 = seq.int(dim2))
## Error: cannot allocate vector of size 3.7 Gb
## Create an iterator over the cartesian product of the two dims
it <- iproduct(x=seq.int(dim1), y=seq.int(dim2))
## Setup the parallel backend
cl <- makeCluster(4)
registerDoParallel(cl)
## Run
res <- foreach(i=it, .combine=c, .packages=c("itertools2")) %dopar% {
f(df1[i$x, ], df2[i$y, ])
}
stopCluster(cl)
## Expand.grid results (different ordering)
expgrid <- expand.grid(x=seq(dim1), y=seq(dim2))
test <- apply(expgrid, 1, function(i) f(df1[i[["x"]],], df2[i[["y"]],]))
all.equal(sort(test), sort(res)) # TRUE
推荐答案
我认为如果你给每个工作人员一个数据框的一个块,让他们每个人执行计算,然后你会获得更好的性能结合结果.这样可以提高计算效率并减少工作人员的内存使用量.
I think you'll get better performance if you give each of the workers a chunk of one of the data frames, have them each perform the computations, and then combine the results. This results in more efficient computation and reduced memory usage by the workers.
以下是使用 itertools
包中的 isplitRow
函数的示例:
Here is an example that uses the isplitRow
function from the itertools
package:
library(doParallel)
library(itertools)
dim1 <- 10
dim2 <- 100
df1 <- data.frame(a = 1:dim1, b = 1:dim1)
df2 <- data.frame(x= 1:dim2, y = 1:dim2, z = 1:dim2)
f <- function(...) sum(...)
nw <- 4
cl <- makeCluster(nw)
registerDoParallel(cl)
res <- foreach(d2=isplitRows(df2, chunks=nw), .combine=c) %dopar% {
expgrid <- expand.grid(x=seq(dim1), y=seq(nrow(d2)))
apply(expgrid, 1, function(i) f(df1[i[["x"]],], d2[i[["y"]],]))
}
我拆分 df2
因为它有更多行,但您可以选择其中之一.
I split df2
because it has more rows, but you could choose either.
这篇关于组合迭代器,如 expand.grid的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!