“反序列化中的错误” - 用SOCK（窗口）foreach / doSNOW / snow [英] "Error in unserialize" - foreach/doSNOW/snow with SOCK (windows)

问题描述

我正在使用本地机器上的工作人员的SOCK群集运行并行操作。如果我限制我正在迭代的集合（在一个测试中使用70而不是完整的135个任务），那么一切正常。如果我去全套，我得到errror反序列化错误（socklist [[n]]）：从连接读取错误。

• 我已经在Windows防火墙（无论是输入/输出）中取消阻止端口，并允许Rscript / R的所有访问。 / p>




• 由于套接字超时设置为365天，所以不能成为超时问题。它不是一个任何特定的任务的问题，因为我可以顺序运行就好（如果我把数据集分成两半，并行运行也很好）并行运行）




• 我能想到的最好的方式是通过套接字传输太多的数据。看起来没有一个集群选项来限制数据限制。

继续。有没有人见过这个问题，或者可以建议修复？




以下是我用来设置群集的代码：



  cluster = makeCluster（degreeOfParallelism，type =SOCK，outfile =）
 registerDoSNOW（cluster）
  

编辑

虽然这个问题与整个数据集有关，时间缩短的数据集。这可能表明，这不仅仅是一个数据限制问题。



编辑2

我挖得更深一点事实证明，我的函数实际上有一个随机的组件，使得有时候有时会是一个任务会引发一个错误。如果我连续运行任务，那么在操作结束时我会被告知哪个任务失败。如果我平行运行，那么我得到反序列化的错误。我试着用error = function（e）{stop（e）}来包装tryCatch调用中每个任务执行的代码，但是也会产生unserialize错误。我很困惑，因为我认为雪把错误传递给了主人呢？


解决方案

我已经报告过这个问题但是SNOW的作者却不幸没有回复。



编辑

我在一段时间内没有看到这个问题。我转移到并行/ doParallel。另外，我现在使用try（）来并行执行任何代码。我不能重复原来的问题。




I'm running a parallel operation using a SOCK cluster with workers on the local machine. If I limit the set I'm iterating over (in one test using 70 instead of a full 135 tasks) then everything works just fine. If I go for the full set, I get the errror "Error in unserialize(socklist[[n]]) : error reading from connection".

• I've unblocked the port in Windows Firewall (both in/out) and allow all access for Rscript/R.

• It can't be a timeout issue because the socket timeout is set to 365 days.

• Its not an issue with any particular task because I can run sequentially just fine (also runs fine in parallel if I split the dataset in half and do two separate parallel runs)

• The best I can come up with is that there is too much data being transferred over the sockets. There doesn't seem to be a cluster option to throttle data limits.

I'm at a loss on how to proceed. Has anyone seen this issue before or can suggest a fix?

Here's the code I'm using to setup the cluster:

cluster = makeCluster( degreeOfParallelism , type = "SOCK" , outfile = "" )
registerDoSNOW( cluster )

Edit
While this issue is constent with the entire dataset, it also appears from time-to-time with a reduced dataset. That might suggest that this isn't simply a data limit issue.

Edit 2
I dug a little deeper and it turns out that my function in fact has a random component that makes it so that sometimes a task will raise an error. If I run the tasks serially then at the end of the operation I'm told which task failed. If I run in parallel, then I get the "unserialize" error. I tried wrapping the code that gets executed by each task in a tryCatch call with error = function(e) { stop(e) } but that also generates the "unserialize" error. I'm confused because I thought that snow handles errors by passing them back to the master?

解决方案

I have reported this issue to the author of SNOW but unfortunately there has been no reply.

Edit
I haven't seen this issue in a while. I moved to Parallel/doParallel. Also, I'm now using try() to wrap any code that gets executed in parallel. I can't repro the original issue.

这篇关于“反序列化中的错误” - 用SOCK（窗口）foreach / doSNOW / snow的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！