Windows 集群上的并行 R [英] Parallel R on a Windows cluster

问题描述

我有一个 Windows HPC 服务器，在后端运行一些节点.我想使用后端的多个节点运行 Parallel R.我认为 Parallel R 可能在 Windows 上使用 SNOW，但不太确定.我的问题是，我是否还需要在后端节点上安装 R?假设我想使用两个节点，每个节点 32 个核心:

I've got a Windows HPC Server running with some nodes in the backend. I would like to run Parallel R using multiple nodes from the backend. I think Parallel R might be using SNOW on Windows, but not too sure about it. My question is, do I need to install R also on the backend nodes? Say I want to use two nodes, 32 cores per node:

cl <- makeCluster(c(rep("COMP01",32),rep("COMP02",32)),type="SOCK")

现在，它只是挂起.

我还需要做什么?后端节点是否需要运行某种 sshd 才能相互通信?

What else do I need to do? Do the backend nodes need some kind of sshd running to be able to communicate each other?

推荐答案

在 Windows 集群上设置 snow 相当困难.每台机器都需要安装 R 和 snow，但这很容易.要启动 SOCK 集群，您需要在每台工作机器上运行一个 sshd 守护程序，但您仍然会遇到麻烦，因此除非您擅长调试和 Windows 系统管理，否则我不推荐它.

Setting up snow on a Windows cluster is rather difficult. Each of the machines needs to have R and snow installed, but that's the easy part. To start a SOCK cluster, you would need an sshd daemon running on each of the worker machines, but you can still run into troubles, so I wouldn't recommend it unless you're good at debugging and Windows system administration.

我认为您在 Windows 集群上的最佳选择是使用 MPI.我自己没有在 Windows 上使用 MPI 的任何经验，但我听说有人在 Windows 的 MPICH 和 DeinoMPI MPI 发行版上取得了成功.在集群上安装 MPI 后，您还需要在每台工作机器上从源代码安装 Rmpi 包.然后，您将使用 makeMPIcluster 函数创建集群对象.这是很多工作，但由于 Windows 上的 ssh/sshd 问题，我认为它最终比尝试使用 SOCK 集群更有可能奏效.

I think your best option on a Windows cluster is to use MPI. I don't have any experience with MPI on Windows myself, but I've heard of people having success with the MPICH and DeinoMPI MPI distributions for Windows. Once MPI is installed on your cluster, you also need to install the Rmpi package from source on each of your worker machines. You would then create the cluster object using the makeMPIcluster function. It's a lot of work, but I think it's more likely to eventually work than trying to use a SOCK cluster due to the problems with ssh/sshd on Windows.

如果您迫切希望在 Windows 集群上运行一次或两次并行作业，您可以尝试使用手动模式.它允许您在没有 ssh 的情况下创建 SOCK 集群:

If you're desperate to run a parallel job once or twice on a Windows cluster, you could try using manual mode. It allows you to create a SOCK cluster without ssh:

workers <- c(rep("COMP01",32), rep("COMP02",32))
cl <- makeSOCKluster(workers, manual=TRUE)

makeSOCKcluster 函数将提示您启动每个 worker，显示每个 worker 使用的命令.您必须在指定的机器上手动打开命令窗口并执行指定的命令.它可能非常乏味，尤其是对于许多工人来说，但至少它并不复杂或棘手.与 outfile='' 选项结合使用时，它对于调试也非常有用.

The makeSOCKcluster function will prompt you to start each one of the workers, displaying the command to use for each. You have to manually open a command window on the specified machine and execute the specified command. It can be extremely tedious, particularly with many workers, but at least it's not complicated or tricky. It can also be very useful for debugging in combination with the outfile='' option.

这篇关于Windows 集群上的并行 R的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！