在并行计算过程中如何写出日志？如何调试并行计算？ [英] how to write out log during parallel computation? how to debug parallel computation?

问题描述

我发现并行计算中有多个打印功能，只有最后一个打印功能将显示在控制台上。所以我设置outfile选项，希望我可以得到每个打印的结果。这是R代码：

  cl<  -  makeCluster（3，type =SOCK，outfile =log.txt ）
 
 abc<<  -  123 
 
 clusterExport（cl，abc）
 
 clusterApplyLB（cl，1：6，
函数（y）{
 print（paste（before：，abc））; 
 abc<< -y; 
打印（粘贴（after：，abc ））; 
} 
）
 stopCluster（cl）
  

但是我只得到三条记录：

  localhost的启动工作器：11888 
类型：EXEC 
类型： EXEC 
 [1]index：3
 [1]之前：123
 [1]之后：2
类型：EXEC 
 [1 ]index：6
 [1]之前：2
 [1]之后：6
类型：DONE 
  pre> 
 
解决方案
看起来你只是从log.txt中的一个工作人员得到输出。我经常想知道是否会发生这种情况，因为当您指定 outfile =log.txt时，每个工作人员将打开log.txt进行追加，然后调用 sink 。以下是当 outfile 不是空字符串时由工作进程执行的代码：
  ##所有的工作人员登录到同一个文件。 
 outcon<  -  file（outfile，open =a）
 sink（outcon）
 sink（outcon，type =message）
  
这让我很紧张，因为我不确定所有的工作人员打开同一个文件以同时添加可能发生什么。它可能是操作系统或文件系统的依赖，它可能解释了为什么你只得到一个工作人员的输出。
 
 
 
因此，我倾向于使用 outfile =，在这种情况下，此代码不会执行，允许输出操作正常发生，而不用 sink 函数。但是，在Windows上，如果您使用Rgui，则不会看到输出，因此使用Rterm。
 
 
 
多个打印语句不应该有问题在一个任务中，但如果您没有设置 outfile ，则不应该看到任何输出，因为在这种情况下，所有输出都被重定向到/ dev / null。 > 
I found if there are more than one print function during the parallel computation, only the last on will display on the console. So I set outfile option and hope I can get the result of every print. Here is the R code:
cl <- makeCluster(3, type = "SOCK",outfile="log.txt") 

abc <<- 123

clusterExport(cl,"abc")

clusterApplyLB(cl, 1:6,  
         function(y){
                     print(paste("before:",abc));
                     abc<<-y;
                     print(paste("after:",abc));
         }
)
stopCluster(cl)
But I just get three records:
starting worker for localhost:11888 
Type: EXEC 
Type: EXEC 
[1] "index: 3"
[1] "before: 123"
[1] "after: 2"
Type: EXEC 
[1] "index: 6"
[1] "before: 2"
[1] "after: 6"
Type: DONE 

解决方案
It looks like you're only getting the output from one worker in log.txt.  I've often wondered if that could happen, because when you specify outfile="log.txt", each of the workers will open log.txt for appending and then call sink. Here is the code that is executed by the worker processes when outfile is not an empty string:
## all the workers log to the same file.
outcon <- file(outfile, open = "a")
sink(outcon)
sink(outcon, type = "message")
This makes me nervous because I'm not certain what might happen with all of the workers opening the same file for appending at the same time. It may be OS or file system dependent, and it might explain why you're only getting the output from one worker.

For this reason, I tend to use outfile="", in which case this code isn't executed, allowing the output operations to happen normally without redirecting them with the sink function. However, on Windows, you won't see the output if you're using Rgui, so use Rterm instead.

There shouldn't be a problem with multiple print statements in a task, but if you didn't set outfile, you shouldn't see any output since all output is redirected to /dev/null in that case.

                        
这篇关于在并行计算过程中如何写出日志？如何调试并行计算？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！