将整个包裹传递到雪团 [英] Passing an entire package to a snow cluster

问题描述

我正在尝试并行化(使用snow::parLapply)一些依赖于程序包(即，除snow以外的程序包)的代码.在parLapply调用的函数中引用的对象必须使用clusterExport显式传递给群集.有什么方法可以将整个程序包传递到群集，而不必在clusterExport中显式命名每个函数(包括用户功能调用的程序包的内部函数！)?

I'm trying to parallelize (using snow::parLapply) some code that depends on a package (ie, a package other than snow). Objects referenced in the function called by parLapply must be explicitly passed to the cluster using clusterExport. Is there any way to pass an entire package to the cluster rather than having to explicitly name every function (including a package's internal functions called by user functions!) in clusterExport?

推荐答案

在所有节点上安装程序包，并通过一条可用命令在所有节点上调用代码library(thePackageYouUse)，类似

Install the package on all nodes, and have your code call library(thePackageYouUse) on all nodes via one the available commands, egg something like

 clusterApply(cl, library(thePackageYouUse))

我认为最近R发行版随附的parallel软件包中有示例-例如，请参见help(clusterApply)中的示例，其中boot软件包随处可见:

I think the parallel package which comes with recent R releases has examples -- see for example here from help(clusterApply) where the boot package is loaded everywhere:

 ## A bootstrapping example, which can be done in many ways:
 clusterEvalQ(cl, {
   ## set up each worker.  Could also use clusterExport()
   library(boot)
   cd4.rg <- function(data, mle) MASS::mvrnorm(nrow(data), mle$m, mle$v)
   cd4.mle <- list(m = colMeans(cd4), v = var(cd4))
   NULL
 })

这篇关于将整个包裹传递到雪团的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！