使用降雪包，具有C-in-R功能的R中的并行计算.问题:Mac显示加载轮并几乎死机 [英] Parallel computing in R with C-within-R functions, using snowfall package. Issue: Mac shows loading wheels and almost freezes

问题描述

我有一个R包，其中包含C-within-R函数，名为myFun.我想在我的并行计算中称其为myFun.此myFun本身在Mac上可以正常运行，但是通过并行计算功能sfClusterApplyLB调用时，它表现出奇怪的行为:我的Mac显示装载轮，R几乎冻结.稍后，R停止冻结，并且sfClusterApplyLB返回并行化的结果.我真的想避免这种冻结状态，因为冻结时我什至无法向上/向下滚动R Console！

为了说明这一点，我有一个小的示例代码.

我有一个小的C代码，它会循环100次，同时每20秒打印一次迭代次数，并在每次迭代时休眠1秒:

 # include <R.h>
 # include <Rinternals.h>
 # include <Rmath.h>
 # include <R_ext/Linpack.h>
 # include <unistd.h>

 SEXP myC (SEXP j)
  {
   for (int i = 0; i < 100; i++)
      {
        R_FlushConsole(); 
        R_ProcessEvents();
        R_CheckUserInterrupt(); 
        sleep(1); // sleep one second at each iteration. this sleep is
        // replaced by something in my code
        if (i%20==0) Rprintf("\v%d iterations are done...",i);
    }
  return (R_NilValue);
  }

我创建了一个临时R包"myRpack"，其中包含此myC函数及其R包装函数myFun，其定义为:

 myFun <- function(x)
   {
    .Call("myC",
       as.integer(x),
       "myRpack")
    }

"myRpack"是通过终端中的命令R CMD INSTALL myRpack安装到我的Mac的.

此函数myCfun在独立运行时可以正常工作.看

 library(myRpack)
 myFun(1)

现在，我想使用降雪包并行计算此myFun.这是用于此目的的并行计算的R代码:

 library("snowfall")
 sfInit(parallel=TRUE,cpus=3,type="SOCK")
 x <- 1 : 100
 res.list <- sfClusterApplyLB(x,myFun)

然后R冻结！

P.S.前一段时间，我在执行C-in-R函数(不使用并行计算)时遇到了这个问题.我向 Rcpp询问了此问题:Mac显示了加载轮并几乎死机了.当时的解决方案是添加行

    R_FlushConsole(); 
    R_ProcessEvents();
    R_CheckUserInterrupt();  

在我的C文件中(我也在我的代码中做了).但是，该解决方案在并行计算环境中无济于事.

我将不胜感激.

PSS 即使我将myC函数定义为:

# include <R.h>
# include <Rinternals.h>
# include <Rmath.h>
# include <R_ext/Linpack.h>
# include <unistd.h>

SEXP myC (SEXP j)
{
  const int input=INTEGER(j)[0];
  Rprintf("\n input %d",input);
  for (int i = 0; i < 100; i++)
    {
  R_FlushConsole(); 
  R_ProcessEvents();
      R_CheckUserInterrupt(); 
  sleep(1); // sleep one second at each iteration. this sleep is
      // replaced by something in my code
  if (i%20==0) Rprintf("\v%d iterations are done...",i);
}
  return (R_NilValue);
}

存在问题.

解决方案

我遇到了同样的问题，偶然发现了一种可能无法解决您的问题的解决方法.

tl; dr:我意识到我正在运行的脚本会使用较低级别的C代码并行调用这些函数，这些脚本将挂在R GUI中，并会在终端的R实例中正确执行(每个实例一次) —随后的source() ings会挂起).

---

更多背景信息，对于面临相同问题的其他人:我遇到了它，它调用函数gbm.step()

我只是开始在Mavericks上遇到此问题，所以我想知道这是否与Mavericks的

I created a temporary R package "myRpack" which contains this myC function as well as its R wrapper function myFun, which is defined as:

 myFun <- function(x)
   {
    .Call("myC",
       as.integer(x),
       "myRpack")
    }

"myRpack" was installed to my Mac via the command R CMD INSTALL myRpack in terminal.

This function myCfun works fine when it is run independently. To see,

 library(myRpack)
 myFun(1)

Now, I want to parallel compute this myFun using snowfall package. Here is the R-code for parallel computing for this purpose:

 library("snowfall")
 sfInit(parallel=TRUE,cpus=3,type="SOCK")
 x <- 1 : 100
 res.list <- sfClusterApplyLB(x,myFun)

Then the R freezes!

P.S. I experienced this problem a while ago when I was executing C-within-R function (with NO parallel computing). I asked this question to Rcpp: Mac shows loading wheel and almost freeze. The solution that time was to add the lines

    R_FlushConsole(); 
    R_ProcessEvents();
    R_CheckUserInterrupt();  

in my C file (which I did in my code, too). However, this solution does not help in parallel computing environment...

I would appreciate any help.

PSS Even if I define the myC function as:

# include <R.h>
# include <Rinternals.h>
# include <Rmath.h>
# include <R_ext/Linpack.h>
# include <unistd.h>

SEXP myC (SEXP j)
{
  const int input=INTEGER(j)[0];
  Rprintf("\n input %d",input);
  for (int i = 0; i < 100; i++)
    {
  R_FlushConsole(); 
  R_ProcessEvents();
      R_CheckUserInterrupt(); 
  sleep(1); // sleep one second at each iteration. this sleep is
      // replaced by something in my code
  if (i%20==0) Rprintf("\v%d iterations are done...",i);
}
  return (R_NilValue);
}

The problem is present.

解决方案

I encountered the same problem, and stumbled upon a workaround which may or may not work in your case.

tl;dr: I realized that scripts I was running, which called functions using lower-level C code in parallel, that would hang in the R GUI, would execute properly in an R instance in the Terminal (once time per instance—subsequent source()ings would hang).

---

More background, for anyone else facing the same issue: I ran into it calling the function gbm.step() the dismo package in parallel. I was using doParallel and foreach packages to parallelize it. The `gbm.step() function a C function to fit boosted regression tree models, and when I'd run it in parallel, it'd freeze (upon examination, the majority of CPU usage was System, not User).

I only started encountering this problem on Mavericks, so I wonder if it has to do with Mavericks' compressed memory or something similar.

(My ultimate workaround was to start running this code on a Linux server, so take that for what it's worth.)

这篇关于使用降雪包，具有C-in-R功能的R中的并行计算.问题:Mac显示加载轮并几乎死机的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！