R 中的并行处理受限 [英] Parallel processing in R limited

问题描述

我正在运行带有 parallel 和 doParallel 包的 ubuntu 12.04 和 R 2.15.1.当我并行运行任何东西时，我仅限于 100% 的内核，而我应该有 800%，因为我使用 8 个内核运行它.系统监视器上显示的是每个子进程仅获得 12%.

I'm running ubuntu 12.04 and R 2.15.1 with the parallel and doParallel packages. when I run anything in parallel I'm limited to 100% of a core, when I should have up to 800%, since I am running it with 8 cores. What shows up on the system monitor is that each child process is getting only 12%.

发生了什么限制了我的执行速度?

What is going on that is limiting my execution speed?

推荐答案

问题可能是 R 进程被限制在一个核心(并且子进程继承它).

The problem may be that the R process is restricted to one core (and the subprocesses inherit that).

试试这个:

> system(sprintf("taskset -p 0xffffffff %d", Sys.getpid()))
pid 3064's current affinity mask: fff
pid 3064's new affinity mask: fff

现在，如果在您的机器上，当前关联掩码报告为 1，那么这就是问题所在.上面的行应该解决它(即第二行应该报告 fff(或类似).

Now, if on your machine, the current affinity mask reports a 1, then this was the problem. The line above should solve it (i.e. the second line should report fff (or similar).

Simon Urbanek 编写了一个函数 mcaffinity，允许对多核进行这种控制.据我所知，它还在 R-devel 中.

Simon Urbanek wrote a function mcaffinity that allows this control for multicore. As far as I know, it's still in R-devel.

有关详细信息，请参见例如这个 关于 R-sig-hpc 的讨论.

For details, see e.g. this discussion on R-sig-hpc.

更新，并补充新郭的回答:

Update, and addition to Xin Guo's answer:

如果您通过 openblas 和显式并行化(通过并行/雪地/多核)一起使用隐式并行化，您可能需要更改 openblas 使用的线程数，具体取决于您是否在显式并行部分内.
这是可能的(在 Linux 下使用 openblas，我不知道任何其他通常优化的 BLAS' 提供线程数功能)，请参阅 Simon Fuller 的博文 了解详情.

If you use implicit parallelization via openblas and explicit parallelization (via parallel/snow/multicore) together, you may want to change the number of threads that openblas uses depending on whether you are inside an explicitly parallel part or not.
This is possible (with openblas under Linux, I'm not aware of any other of the usual optimized BLAS' that provides a function to the number of threads), see Simon Fuller's blog post for details.

这篇关于R 中的并行处理受限的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！