如何在R中的并行方法中使用徒劳的记录器进行记录? [英] How to log using futile logger from within a parallel method in R?

问题描述

我正在R中使用徒劳的记录器进行记录. 我有一个使用R中的降雪实现的并行算法.并行进程的每个核心都在记录器中记录一个中间输出.但是此输出未显示在记录器中吗?

I am using futile logger in R for logging. I have a parallel algorithm implemented using snowfall in R. Each core of the parallel process logs an intermediate output in the logger. But this output is not showing up in the logger?

我们是否可以在降雪的并行作业中使用徒劳的记录器记录日志?

Can we log using futile logger from within a parallel job using snowfall?

我的具体情况有些不同.我正在使用创建的共享库从R调用C函数.该函数是一个迭代算法，我需要每隔几次迭代记录一次输出.我对从C函数记录到徒劳的记录器很感兴趣.为什么徒劳的记录器?因为这是Web应用程序的一部分，所以使用户会话的所有输出都具有一致的格式是有意义的.

My specific case was a bit different. I am calling a C function from R using a shared object that I created. The function is an iterative algorithm and I need output to be logged every few iterations. I was interested in logging from the C function to futile logger. Why futile logger? Because this is part of a web-application and it makes sense to have all output for a user session to come in a consistent format.

这是我根据公认的答案所遵循的一般方法.

This is the general approach I followed based on the accepted answer.

# init script
# iter logger namespace global variable
assign("MCMC_LOGGER_NAMESPACE", "iter.logger", envir = .GlobalEnv)  

loginit <- function(logfile) { 
  require('futile.logger')
  flog.layout(layout.simple, name = ITER_LOGGER_NAMESPACE)  
  flog.threshold(TRACE, name = ITER_LOGGER_NAMESPACE)
  flog.appender(appender.file(logfile), name = ITER_LOGGER_NAMESPACE)   
  NULL
}

parallel_funct_call_in_R <- function(required args) {    
require('snowfall')  
sfSetMaxCPUs() 
sfInit(parallel = TRUE, cpus = NUM_CPU) 
sfLibrary(required libs)
sfExport(required vars including logger namespace variable ITER_LOGGER_NAMESPACE)
iterLoggers = sprintf(file.path(myloggingdir, 'iterativeLogger_%02d.log', fsep = .Platform$file.sep), seq_len(NUM_CPU))
sfClusterApply(iterLoggers, loginit)  
sfSource(required files)
estimates <- sfLapply(list_to_apply_over, func_callling_C_from_R, required args)
sfStop()  
return(estimates)
}

iterTrackNumFromC <- function(numvec){
# convert numvec to json and log using flog.info
# the logger namespace has already been registered in the individual cores
flog.info("%s", toJSON(numvec), name = ITER_LOGGER_NAMESPACE) 
}

func_callling_C_from_R <- function(args){
 load shared obh using dyn.load
estimates = .C("C_func", args, list(iterTrackNumFromC)) # can use .Call also I guess
return(estimates)
}

现在有了C函数

void C_func(other args, char **R_loggerfunc){ // R_loggerfunc is passed iterTrackNumFromC    
// do stuff
// call function that logs numeric values to futile.logger
logNumericVecInR();
}

void logNumericVecInR (char *Rfunc_logger, double *NumVec, int len_NumVec){        
    long nargs = 1;        
    void *arguments[1];
    arguments[0] = (double*)NumVec;    
    char *modes[1];
    modes[0] = "double";        
    long lengths[1];
    lengths[0] = len_NumVec;        
    char *results[1];
    //    void call_R(char *func, long nargs, void **arguments, char **modes, long *lengths, char **names, long nres, char **results)    
    call_R(Rfunc_logger, nargs, arguments, modes, lengths, (char**)0, (long)1, results);
}

希望这会有所帮助.如果R和C有更简单的方法来共享通用记录器，请告诉我.

Hope this helps. If there is an easier way for R and C to share a common logger, please let me know.

推荐答案

在降雪程序中使用futile.logger包的一种简单方法是使用sfInit slaveOutfile=''选项，以使工作输出不会被重定向

A simple way to use the futile.logger package from a snowfall program is to use the sfInit slaveOutfile='' option so that worker output isn't redirected.

library(snowfall)
sfInit(parallel=TRUE, cpus=3, slaveOutfile='')
sfLibrary(futile.logger)
work <- function(i) {
  flog.info('Got task %d', i)
  i
}
sfLapply(1:10, work)
sfStop()

这是降雪makeCluster outfile=''选项的降雪界面.它可能无法与Rgui之类的GUI界面一起正常使用，具体取决于它们如何处理进程输出，但它确实可以在Windows中使用Rterm.exe来工作.

This is the snowfall interface to the snow makeCluster outfile='' option. It may not work properly with GUI interfaces such as Rgui, depending on how they handle process output, but it does work on Windows using Rterm.exe.

我认为最好为每个工作人员指定不同的日志文件.这是一个示例:

I think that it's better to specify different log files for each worker. Here's an example of that:

library(snowfall)
nworkers <- 3
sfInit(parallel=TRUE, cpus=nworkers)

loginit <- function(logfile) {
  library(futile.logger)
  flog.appender(appender.file(logfile))
  NULL
}
sfClusterApply(sprintf('out_%02d.log', seq_len(nworkers)), loginit)

work <- function(i) {
  flog.info('Got task %d', i)
  i
}
sfLapply(1:10, work)
sfStop()

这避免了所有额外的输出都来自大雪，并将每个工作人员的日志消息放入一个单独的文件中，这样可以减少混乱.

This avoids all of the extra output coming from snow and puts each worker's log messages into a separate file which can be less confusing.

这篇关于如何在R中的并行方法中使用徒劳的记录器进行记录?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！