如何正确使用doSMP和foreach软件包? [英] How to use the doSMP and the foreach packages correctly?

问题描述

我正在尝试使用 doSMP软件包，该软件包提供了并行foreach软件包的后端.

I am trying to use the doSMP package that provides a parallel backend for the foreach package.

您能指出我做错了什么吗?确实，以这种方式使用foreach会显着增加计算时间...

Can you point out what I do wrong? Indeed, using foreach in that way increases significantly the computing time...

#------register doSMP to be used with foreach------
library(doSMP)
w <- startWorkers(4)
registerDoSMP(w)
#--------------------------------------------------

#------A simple function------
sim <- function(a, b)
{
    return(10 * a + b)
}
avec <- 1:200
bvec <- 1:400
#-----------------------------

#------The naive method------
ptime <- system.time({
mat <- matrix(NA, nrow=length(avec), ncol=length(bvec))
for(i in 1:length(avec))
{
    for(j in 1:length(bvec))
    {
         mat[i, j] <- sim(avec[i], bvec[j])
    }
}
})[3]
ptime

elapsed 
   0.36
#----------------------------

#------Using foreach------
ptime <- system.time({
mat2 <- foreach(b=bvec, .combine="cbind") %:%
         foreach(a=avec, .combine="c") %dopar%
     {
            sim(a, b)
    }
})[3]
ptime

elapsed 
  86.98
#-------------------------

编辑

该问题与该问题非常相似并已从stats.stackexchange迁移.

That question is very very similar to this one and has been migrated from stats.stackexchange.

推荐答案

我个人不喜欢doSMP软件包，因为它经常使我的R崩溃.它是为REvolution构建而开发的，因此无法在我的计算机上顺利运行.例如，您上面的代码未更改，只是使我的R崩溃了.

I personally don't like the doSMP package as it crashes my R often. It is developed for the REvolution build, and somehow fails to run smooth on my machine. For example, your code above, unaltered, just crashes my R.

此外，尝试在循环函数中使用并行化函数似乎很奇怪在外部循环中进行并行化是更多逻辑.嵌套并行计算中涉及的通信导致计算时间的急剧增加.您没有任何收获，因为您的sim函数非常快.实际上，保持内部循环序列化更有意义，因为在这种情况下，一个内核上的计算时间会比由于通信而产生的开销大.

Next to that, it seems strange to try to use a parallelized function within a loop function It is more logic to do the parallelization in the outer loop. The communication involved in nested parallel computing is causing the dramatic increase in calculation time. You don't gain anything, as your sim function is incredibly fast. In fact, keeping the inner loop serialized makes more sense, as in that situation the calculation time on one core gets bigger than the overhead due to communication.

使用降雪包并使用apply循环而不是for循环的插图.这也很幼稚，因为矢量化有很多好处(见下文).

An illustration using the snowfall-package and using apply for looping instead of for loops. This is also very naive as there is a lot to win with vectorization (see below).

library(snowfall)
sfInit(parallel=T,cpus=2)
#same avec, bvec, sim

system.time({
    out <- sapply(avec,function(i) {
      sapply(bvec,function(j){
        sim(i,j)
      })
    })
})[3]
elapsed 
   0.33 

sfExport("avec","bvec","sim")
system.time({
    out <- sfSapply(avec,function(i) { # this one is parallel
      sapply(bvec,function(j){ # this one is not, no sense in doing so
        sim(i,j)
      })
    })
})[3]
elapsed 
   0.17 

由于结构，两个维度都相等，除了维名称:

Both matrices are equal, apart from the dimension names due to the structure :

> all.equal(out1,out2)
[1] "Attributes: < Length mismatch: comparison on first 1 components >"

正确的R方法是:

system.time(
  out3 <- outer(avec*10,bvec,"+")
)[3]
elapsed 
   0.01 

显着提高了速度，并创建了一个相同的(尽管经过转置的)矩阵:

which is significantly faster, and creates an identical (though transposed) matrix :

> all.equal(out1,t(out3))
[1] TRUE

(作为参考，您的double for循环在我的系统上以0.73的经过时间运行...)

(as a reference, your double for-loop runs on 0.73 elapsed time on my system...)

这篇关于如何正确使用doSMP和foreach软件包?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！