在foreach循环中出现mclapply的R错误 [英] R error with mclapply in a foreach loop

问题描述

基于此处的帖子 ，我尝试编写脚本，如下所示:

library(parallel)
library(doParallel)

cl<-makeCluster(2,outfile='')
registerDoParallel(cl)

foreach(i=1:5, .packages='parallel') %dopar% {
    system.time(mclapply(1:10, function(x){rnorm(1e5)},mc.cores=2))
}

stopCluster(cl)

最初它可以工作，但是现在抛出错误代码:

Error in unserialize(node$con) : error reading from connection
Calls: <Anonymous> ... doTryCatch -> recvData -> recvData.SOCKnode -> unserialize
Execution halted
Error in unserialize(socklist[[n]]) : error reading from connection
Error in unserialize(node$con) : error reading from connection
Calls: <Anonymous> ... doTryCatch -> recvData -> recvData.SOCKnode -> unserialize
Execution halted

知道发生了什么吗?甚至可以将mclapply放在foreach循环中吗?

我也想说这是在一台8核计算机上，而不是一个集群上.

解决方案

我仅使用R 3.2.3中的"parallel"包就可以在Linux机器上重现您的问题:

library(parallel)
cl <- makeCluster(2)
clusterEvalQ(cl, library(parallel))
fun <- function(i) {
  mclapply(1:10, function(x) rnorm(1e5), mc.cores=2)
  0
}
clusterApplyLB(cl, 1:5, fun)

在我的调试会话中，看来主服务器和工作线程之间的套接字连接可能会损坏，这可能导致工作线程在尝试从已损坏的套接字连接中反序列化"数据时遇到错误时导致死亡. >

有趣的是，我可以使用多核"程序包而不是并行"程序来使此示例正常工作.我使用以下命令从RForge.net安装了多核0.1-8:

> install.packages('multicore',,'http://www.rforge.net/')  

然后，我在工作程序上加载了多核"而不是并行":

clusterEvalQ(cl, library(multicore))

然后该示例运行良好.您可以将foreach循环更改为使用.packages='multicore'选项.

就我所知.我的猜测是，并行"中由"mclapply"派生的子进程在某种程度上破坏了它们继承的套接字连接，但是我没有查看代码来看看该理论是否合理.

我想您的选择是:

1. 在"doParallel" foreach循环中不要使用"mclapply"
2. 使用多核0.1-8"中的"mclapply"代替并行"
3. 将此问题报告给R-Core

您需要做一些额外的工作才能将其报告给R-Core，但希望我的示例会有所帮助.

Based on this post here, I tried to write a script, seen here:

library(parallel)
library(doParallel)

cl<-makeCluster(2,outfile='')
registerDoParallel(cl)

foreach(i=1:5, .packages='parallel') %dopar% {
    system.time(mclapply(1:10, function(x){rnorm(1e5)},mc.cores=2))
}

stopCluster(cl)

It worked intially but is now throwing up error codes:

Error in unserialize(node$con) : error reading from connection
Calls: <Anonymous> ... doTryCatch -> recvData -> recvData.SOCKnode -> unserialize
Execution halted
Error in unserialize(socklist[[n]]) : error reading from connection
Error in unserialize(node$con) : error reading from connection
Calls: <Anonymous> ... doTryCatch -> recvData -> recvData.SOCKnode -> unserialize
Execution halted

Any idea what's going on? Can one even put mclapply in a foreach loop?

Edit: I also want to say this is on a single 8-core machine, not a cluster.

解决方案

I was able to reproduce your problem on my Linux machine using only the "parallel" package in R 3.2.3:

library(parallel)
cl <- makeCluster(2)
clusterEvalQ(cl, library(parallel))
fun <- function(i) {
  mclapply(1:10, function(x) rnorm(1e5), mc.cores=2)
  0
}
clusterApplyLB(cl, 1:5, fun)

From my debugging sessions, it appears that the socket connections between the master and the workers can get corrupted, which can cause the workers to die when they get an error trying to "unserialize" data from the corrupted socket connection.

Interestingly, I could get this example to work by using the "multicore" package instead of "parallel". I installed multicore 0.1-8 from RForge.net using the command:

> install.packages('multicore',,'http://www.rforge.net/')  

Then, I loaded "multicore" instead of "parallel" on the workers:

clusterEvalQ(cl, library(multicore))

Then the example worked fine. You could change your foreach loop to use the .packages='multicore' option.

That's as far as I've tracked it down. My guess is that the child processes forked by "mclapply" in "parallel" are somehow corrupting the socket connection that they've inherited, but I haven't looked at the code to see if that theory is plausible.

I guess your choices are:

1. Don't use "mclapply" in a "doParallel" foreach loop
2. Use "mclapply" from "multicore 0.1-8" instead of "parallel"
3. Report this issue to R-Core

You'll have to do additional work to report this to R-Core, but hopefully my example will help.

这篇关于在foreach循环中出现mclapply的R错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！