R:foreach 循环如何找到应该调用的函数? [英] R: how does a foreach loop find a function that should be invoked?

问题描述

我在使用调用自定义函数的 foreach 循环(使用 %dopar%)时遇到问题.在Linux下确实没有问题，但是在Windows下却找不到自定义函数.这个问题很难用语言来解释，所以我写了一个小例子来说明.假设我有三个简单函数的集合，其中 FUN2(使用 %do%)和 FUN3(使用 %dopar%) 调用第一个 (FUN):

I have problems when I use a foreach loop (using %dopar%) which invokes a self-defined function. There is not really a problem when I work with Linux, but when I use Windows the self-defined function cannot be found. It is hard to explain the problem in words, so I composed a small example to show it. Assume I have a collection of three simple functions, where FUN2 (using %do%) and FUN3 (using %dopar%) invoke the first one (FUN):

FUN <- function(x,y,z) { x + y + z }
FUN2 <- function(a, b) {
  foreach(i=1:3) %do% FUN(i, a, b)
}
FUN3 <- function(a, b) {
  foreach(i=1:3) %dopar% FUN(i, a, b)
}

函数存储在名为 foreach_testfunctions.R 的脚本中.在另一个脚本 (foreach.test) 中，我提供了这些函数，使用 library(doParallel) 并尝试使用这些函数.首先我用 Linux 做，一切正常:

The functions are stored in a script called foreach_testfunctions.R. In another script (foreach.test) I source these functions, use library(doParallel) and try to use the functions. First I do it with Linux and all works fine:

source("foreach_testfunctions.R")
a <- 2
b <- 3
library(doParallel)
registerDoParallel()

foreach(i=1:3) %do% FUN(i, a, b)    ## works fine
FUN2(a, b)                          ## works fine
foreach(i=1:3) %dopar% FUN(i, a, b) ## works fine
FUN3(a, b)                          ## works fine 

然后我在 Windows 中执行此操作:

Then I do it in Windows:

source("foreach_testfunctions.R")
a <- 2
b <- 3
library(doParallel)
cl <- makeCluster(3)
registerDoParallel(cl)

foreach(i=1:3) %do% FUN(i, a, b)    ## works fine
FUN2(a, b)                          ## works fine
foreach(i=1:3) %dopar% FUN(i, a, b) ## works fine
FUN3(a, b)                          ## does not work
Error in FUN(i, a, b) : task 1 failed - "Could not find function "FUN""

结论:(1) %do% 没有问题.(2) 使用Windows时%dopar%的问题.我尝试在调用 FUN3 的行之前插入行 clusterExport(cl, varlist=c("FUN", "a", "b"), env=environment())code> 以确保函数 FUN 和变量 a 和 b 在适当的环境中找到，但错误仍然存​​在.

Conclusion: (1) No problems with %do%. (2) Problems with %dopar% when using Windows. I tried inserting the line clusterExport(cl, varlist=c("FUN", "a", "b"), env=environment()) before the line that invokes FUN3 to make sure that the function FUN and the variables a and b are found in the proper environment, but the error remains.

我的问题:尽管代码相同(除了不同的 registerDoParallel 语法)，为什么 Windows 的行为与 Linux 不同?当通过函数 FUN3 调用时，如何确保 Windows 确实找到函数 FUN?

My questions: Why does Windows behave different than Linux although the code is identical (apart from the different registerDoParallel syntax)? How can I make sure that Windows does find function FUN when invoked via function FUN3?

推荐答案

它们的行为不同，因为 registerDoParallel 在 Linux 上注册了一个 mclapply 后端，而它注册了一个 Windows 上的 clusterApplyLB 后端.使用 mclapply 后端时，基本上没有数据导出问题，因此它适用于 Linux.但是对于 clusterApplyLB，如果 foreach 没有自动导出所需的函数和数据，您可能会遇到问题.

They behave differently because registerDoParallel registers an mclapply backend on Linux, while it registers a clusterApplyLB backend on Windows. When using an mclapply backend, there are essentially no data exporting issues, so it works on Linux. But with clusterApplyLB, you can run into problems if foreach doesn't auto-export the functions and data that are needed.

您可以通过修改FUN3通过.export选项导出FUN来解决这个问题:

You can solve this problem by modifying FUN3 to export FUN via the .export option:

FUN3 <- function(a, b) {
  foreach(i=1:3, .export='FUN') %dopar% FUN(i, a, b)
}

此解决方案适用于 Linux 和 Windows，因为 .export 被 mclapply 后端忽略.

This solution works on both Linux and Windows, since .export is ignored by the mclapply backend.

正如 Hong Ooi 所指出的，您在使用 clusterExport 时出错，但我不会使用 clusterExport 来解决问题，因为它是后端特定的.

As pointed out by Hong Ooi, you have an error in your use of clusterExport, but I wouldn't use clusterExport to solve the problem since it is backend specific.

这篇关于R:foreach 循环如何找到应该调用的函数?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！