每个工人具有多个内核的并行化 [英] Parallelization with Multiple Cores per Worker

问题描述

某些R软件包具有的功能可以在多个核可用的情况下并行执行它们的工作-例如，rstan软件包可以并行运行多个MCMC链.当我使用例如doSNOW和foreach彼此并行运行多个Stan进程时，我希望我的代码在两个级别上并行运行*.取而代之的是，Stan流程被分发给我的工作人员，并且似乎在那儿按顺序运行它们的链，好像一旦将它们分配到一个核心后，他们就看不到机器的其他核心，并认为它们在一个单一的-核心机器.

Some R packages have functions that can do their work in parallel if multiple cores are available - for example, the rstan package can run multiple MCMC chains in parallel. When I run a number of Stan processes in parallel to each other using, e.g., doSNOW and foreach, I'd like my code to operate in parallel at both levels*. Instead, the Stan processes get farmed out to my workers and seem to run their chains in sequence there, as if once they've been assigned to a core they can't see the machine's other cores and think they're on a single-core machine.

有没有一种方法可以创建4核节点的群集，可以将其传递给R中的某些并行化程序包，以便从计算机中获得最大的效率?

Is there a way to create clusters of 4-core nodes that I can pass to some parallelization package in R, so that I can get the maximum efficiency out of my machine?

*说我有36台核心机器，并且有9个Stan方案，每个方案有4条链.理想情况下，我可以同时运行36个进程.现在，我一次使用了9个内核，并且只要我希望它能用4倍.

*say I have a 36 core machine, and 9 Stan scenarios run with 4 chains each. Ideally, I have 36 processes that I could run all at once. Right now, I get 9 cores used at a time, and it takes 4x as long as I'm hoping it could.

推荐答案

假设您有充分的理由不直接创建36个并行工作器 (例如，如果您有计算集群或类似的集群)， 那么以下应该工作 (以doParallel为例)

Assuming you have a good reason not to create 36 parallel workers directly (e.g. if you have a computing cluster or something like that), then the following should work (using doParallel as an example):

library(doParallel)

# create "outer" workers
outer_workers <- makeCluster(2L)
# register outer_workers
registerDoParallel(outer_workers)

# create "inner" workers
clusterEvalQ(outer_workers, {
    library(doParallel)
    inner_workers <- makeCluster(2L)
    # register inner_workers
    registerDoParallel(inner_workers)

    NULL
})

# assuming you use foreach directly
foreach(i = 1L:2L) %dopar% {
    foreach(j = 1L:2L) %dopar% {
        # code
    }

    NULL
}

# stop inner workers
clusterEvalQ(outer_workers, {
    stopCluster(inner_workers)
    registerDoSEQ()
    NULL
})

stopCluster(outer_workers); registerDoSEQ()

此示例总共创建了4个流程， 2个外部和2个内部. 在您的示例中，您可以有9个外部过程和4个内部过程.

This example creates 4 processes in total, 2 outer and 2 inner. In your example you could have 9 outer and 4 inner processes.

这篇关于每个工人具有多个内核的并行化的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！