Windows 7上具有内联功能的doParallel问题(在Linux上有效) [英] doParallel issue with inline function on Windows 7 (works on Linux)

问题描述

我在Windows 7和Linux(SUSE Server 11(x86_64))上都使用R 3.0.1.以下示例代码在Windows上会产生错误，但在Linux上不会产生错误.两台机器上列出的所有工具箱都是最新的. Windows错误是:

I am using R 3.0.1 both on Windows 7 and Linux (SUSE Server 11 (x86_64)). The following example code produces an error on Windows but not on Linux. All the toolboxes listed are up-to-date in both machines. The Windows error is:

Error in { : task 1 failed - "NULL value passed as symbol address"

如果更改%dopar% to %do%，则Windows代码将运行而没有任何错误.我最初的猜测是，这与Windows中的某些配置问题有关，我尝试重新安装Rcpp和R，但这没有帮助.该错误似乎与作用域有关-如果我在f1中定义并编译函数cFunc，则%dopar%可以工作，但正如预期的那样，它非常慢，因为我们为每个任务调用一次编译器.

If I change %dopar% to %do%, the Windows code runs without any errors. My initial guess was that this relates to some configuration issue in Windows and I tried reinstalling Rcpp and R but that did not help. The error seems to be related to scoping - if I define and compile the function cFunc inside f1, then %dopar% works but, as expected, it is very slow since we are calling the compiler once for each task.

有人对错误发生的原因有什么见解，或对如何纠正错误有建议?

Does anyone have some insights on why the error happens or suggestions on how to fix it?

library(inline)
sigFunc <- signature(x="numeric", size_x="numeric")
code <- ' double tot =0;
for(int k = 0; k < INTEGER(size_x)[0]; k++){
tot += REAL(x)[k];
};
return ScalarReal(tot);
' 
cFunc <- cxxfunction(sigFunc, code)

f1 <- function(){
x <- rnorm(100)
a <- cFunc(x=x, size_x=as.integer(length(x)))
return(a)
}

library(foreach)
library(doParallel)
registerDoParallel()
# this produces an error in Windows but not in Linux
res <- foreach(counter=(1:100)) %dopar% {f1()}
# this works for both Windows and Linux
res <- foreach(counter=(1:100)) %do% {f1()}

# The following is not a practical solution, but I can compile cFunc inside f1 and then     this works in Windows but it is very slow
f1 <- function(){
library(inline)
sigFunc <- signature(x="numeric", size_x="numeric")

code <- ' double tot =0;
for(int k = 0; k < INTEGER(size_x)[0]; k++){
tot += REAL(x)[k];
};
return ScalarReal(tot);
' 
cFunc <- cxxfunction(sigFunc, code)
x <- rnorm(100)
a <- cFunc(x=x, size_x=as.integer(length(x)))
return(a)
}
# this now works in Windows but is very slow
res <- foreach(counter=(1:100)) %dopar% {f1()}

谢谢！ 古斯塔沃

推荐答案

错误消息作为符号地址传递的NULL值"是异常的，并且不是由于未将函数导出到工作程序而引起的. cFunc函数在被序列化，发送给工作程序并未序列化后才无法使用.从已保存的工作空间中加载它时也不起作用，这会导致相同的错误消息.这并不令我感到惊讶，这可能是inline软件包的已记录行为.

The error message "NULL value passed as symbol address" is unusual, and isn't due to the function not being exported to the workers. The cFunc function just doesn't work after being serialized, sent to a worker, and unserialized. It also doesn't work when it's loaded from a saved workspace, which results in the same error message. That doesn't surprise me much, and it may be a documented behavior of the inline package.

如前所述，您可以通过在工作线程上创建cFunc来解决该问题.为了有效地做到这一点，您只需要对每个工人执行一次.要使用doParallel后端做到这一点，我将定义一个 worker初始化函数，并使用clusterCall函数在每个worker上执行它:

As you've demonstrated, you can work-around the problem by creating cFunc on the workers. To do that efficiently, you need to do it only once on each of the workers. To do that with the doParallel backend, I would define a worker initialization function, and execute it on each of the workers using the clusterCall function:

worker.init <- function() {
  library(inline)
  sigFunc <- signature(x="numeric", size_x="numeric")
  code <- ' double tot =0;
  for(int k = 0; k < INTEGER(size_x)[0]; k++){
  tot += REAL(x)[k];
  };
  return ScalarReal(tot);
  '
  assign('cFunc', cxxfunction(sigFunc, code), .GlobalEnv)
  NULL
}

f1 <- function(){
  x <- rnorm(100)
  a <- cFunc(x=x, size_x=as.integer(length(x)))
  return(a)
}

library(foreach)
library(doParallel)
cl <- makePSOCKcluster(3)
clusterCall(cl, worker.init)
registerDoParallel(cl)
res <- foreach(counter=1:100) %dopar% f1()

请注意，您必须显式创建PSOCK群集对象才能调用clusterCall.

Note that you must create the PSOCK cluster object explicitly in order to call clusterCall.

您的示例在Linux上运行的原因是，在不带参数的情况下调用registerDoParallel时将使用mclapply函数，而在Windows上将创建集群对象并使用clusterApplyLB函数.使用mclapply时，函数和变量不会序列化并发送给工作程序，因此没有错误.

The reason that your example worked on Linux is that the mclapply function is used when you call registerDoParallel without an argument, while on Windows a cluster object is created and the clusterApplyLB function is used. Functions and variables aren't serialized and sent to the workers when using mclapply, so there is no error.

如果doParallel包括不需要使用clusterCall的初始化工作程序的支持，那就很好了，但是还没有.

It would be nice if doParallel included support for initializing the workers without the need for using clusterCall, but it doesn't yet.

这篇关于Windows 7上具有内联功能的doParallel问题(在Linux上有效)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！