使用外部指针的并行化函数 (XPtr) [英] Parallelize function taking external pointers (XPtr)

问题描述

这个问题既不是这个问题的重复，也不是这个，是关于返回外部指针的函数.

This question is neither a duplicate of this one nor of this one, which were about functions returning external pointers.

问题来了.此后的 Rcpp 代码定义了两个函数，一个创建 XPtr，另一个可以在 XPtr 上工作.

Here's the issue. The Rcpp code hereafter defines two functions, one which creates an XPtr, and another one which can work on the XPtr.

#include <Rcpp.h>
using namespace Rcpp;

//[[Rcpp::export]]
SEXP f(int n) {
  std::vector<int> * v = new std::vector<int>;

  for(int i = 0; i < n; i++)
    v->push_back(i);

  XPtr< std::vector<int> > p(v, true);
  return p;
}

//[[Rcpp::export]]
int g(XPtr< std::vector<int> > p, int i) {
  return (*p)[i];

它工作正常:

> x <- f(100)
> g(x, 45)
[1] 45

让我们尝试并行化对 g 的调用.这有效:

Let's try to parallelize calls to g. This works:

require(parallel)
test1 <- function(a) {
  cl <- makeForkCluster(nnodes=2)
  r <- parLapply(cl, 1:5, function(i) g(a,i) )
  stopCluster(cl)
  return(r)
}

预期行为:

> unlist( test1(x) )
[1] 1 2 3 4 5

但这不起作用:

test2 <- function(a) {
  cl <- makeForkCluster(nnodes=2)

  p <- g(a, 0)
  r <- parLapply(cl, 1:5, function(i) g(a,i) )
  stopCluster(cl)
  return(r)
}

意外行为:

> test2(x)
Error in checkForRemoteErrors(val) : 
  2 nodes produced errors; first error: external pointer is not valid

问题似乎是由于在调用集群中的从属之前在函数中使用了一次外部指针.什么解释了这种行为，是否有解决方法?非常感谢.

The problem seems to arise from the fact that the external pointer is used once in the function before calling the slaves in the cluster. What explains this behavior, and is there a workaround? Many thanks in advance.

推荐答案

在函数的开头，a 是一个承诺，即表示在特定环境中评估特定表达式的东西.当您访问该变量时，会计算表达式，因此现在 a 是一个指针，并且该指针特定于特定的 R 实例.您可以使用 pryr::promise_info:

At the beginning of your function, a is a promise, i.e. something that says evaluate a certain expression in a certain environment. When you access the variable, the expression is evaluated, so now a is an pointer, and that pointer is specific to the particular R instance. You can look at this using pryr::promise_info:

test2 <- function(a) {
  cl <- makeForkCluster(nnodes = 2)
  print(pryr::promise_info(a))
  p <- g(a, 0)
  print(pryr::promise_info(a))
  stopCluster(cl)
  return(r)
}

输出:

$code
x

$env
<environment: R_GlobalEnv>

$evaled
[1] FALSE

$value
NULL

$code
x

$env
NULL

$evaled
[1] TRUE

$value
<pointer: 0x565295e3a410>

一种解决方法是使用 eval(substitute(a)):

One way around is to use eval(substitute(a)):

test2 <- function(a) {
  cl <- makeForkCluster(nnodes = 2)
  print(pryr::promise_info(a))
  p <- g(eval(substitute(a)), 0)
  print(pryr::promise_info(a))
  r <- parLapply(cl, 1:5, function(i) g(a,i) )
  stopCluster(cl)
  return(r)
}

我相信有更好的方法.非标准评价对我来说还是有点陌生​​...

I am sure there are better ways. Non-standard evaluation is still a bit foreign to me ...

这篇关于使用外部指针的并行化函数 (XPtr)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！