doMC vs doSNOW vs doSMP vs doMPI:为什么"foreach"的各种并行后端在功能上不相同? [英] doMC vs doSNOW vs doSMP vs doMPI: why aren't the various parallel backends for 'foreach' functionally equivalent?

问题描述

我已经在多台机器上运行了一些测试代码，总是得到相同的结果.我认为各种do ...软件包背后的理念是它们可以互换用作foreach的％dopar％的后端.为什么不是这种情况?

I've got a few test pieces of code that I've been running on various machines, always with the same results. I thought the philosophy behind the various do... packages was that they could be used interchangeably as a backend for foreach's %dopar%. Why is this not the case?

例如，此代码段有效:

library(plyr)
library(doMC)
registerDoMC()
x <- data.frame(V= c("X", "Y", "X", "Y", "Z" ), Z = 1:5)
ddply(x, .(V), function(df) sum(df$Z),.parallel=TRUE)

虽然这些代码片段均失败:

While each of these code snippets fail:

library(plyr)
library(doSMP)
workers <- startWorkers(2)
registerDoSMP(workers)
x <- data.frame(V= c("X", "Y", "X", "Y", "Z" ), Z = 1:5)
ddply(x, .(V), function(df) sum(df$Z),.parallel=TRUE) 
stopWorkers(workers)

library(plyr)
library(snow)
library(doSNOW)
cl <- makeCluster(2, type = "SOCK")
registerDoSNOW(cl)
x <- data.frame(V= c("X", "Y", "X", "Y", "Z" ), Z = 1:5)
ddply(x, .(V), function(df) sum(df$Z),.parallel=TRUE) 
stopCluster(cl)

library(plyr)
library(doMPI)
cl <- startMPIcluster(count = 2)
registerDoMPI(cl)
x <- data.frame(V= c("X", "Y", "X", "Y", "Z" ), Z = 1:5)
ddply(x, .(V), function(df) sum(df$Z),.parallel=TRUE) 
closeCluster(cl)

在所有四种情况下，foreach(i = 1:3,.combine = "c") %dopar% {sqrt(i)}都会产生完全相同的结果，所以我知道我已经在每台经过测试的计算机上安装了程序包并可以正常工作.

In all four cases, foreach(i = 1:3,.combine = "c") %dopar% {sqrt(i)} yields the exact same result, so I know I have the packages installed and working properly on each machine I've tested them on.

doMC与doSMP，doSNOW和doMPI有何不同?

What is doMC doing differently from doSMP, doSNOW, and doMPI?

推荐答案

doMC派生当前R进程，以便它继承所有现有变量.所有其他do后端仅传递明确请求的变量.不幸的是，我没有意识到这一点，仅使用doMC进行了测试-我希望在下一版plyr中可以解决此问题.

doMC forks the current R process so it inherits all the existing variables. All the other do backends only pass on explicitly requested variables. Unfortunately I didn't realise that, and only tested with doMC - this is something I hope to fix in the next version of plyr.

这篇关于doMC vs doSNOW vs doSMP vs doMPI:为什么"foreach"的各种并行后端在功能上不相同?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！