在多台主机上运行并行 R [英] Running parallel R on multiple hosts

问题描述

您能否提供一个脚本，以便在 Ubuntu Linux 机器上从头开始在 2 台主机(amazon ec2)上运行并行集群?

Can you please provide a script to run parallel cluster on 2 hosts ( amazon ec2) from scratch on Ubuntu Linux Machine ?

规格

1. 主机通过名为 amazon_key.pem 的身份密钥从本地计算机连接
2. 主机名应该通过亚马逊云提供的内部 IP 地址连接
3. 在设置云时将主机名保留为 rserver1 和 rserver2

推荐答案

所有这些命令都是从本地系统运行的.这是以这样一种方式编写的，以便人们可以根据自己的需要自动执行此代码.

All these commands are run from local system. This has been written in such a way so that one can automate this code based on their need.

   HOST1=ip_of_server1
   HOST2=ip_of_server2

没有pem文件的人可以避免这种情况.否则，请输入本地系统中 pem(key) 文件的确切位置.

People who do not have pem file can avoid this. Else put your exact location of the pem(key) file present in your local system.

pem_file_loc="~"

为了查找私有IP，我的地区是eu-west-1，请替换为您所在的地区.已经知道私有IP的人可以忽略这个，直接填写PIP1和PIP2

for finding out the private IP , my region is eu-west-1 , please replace with your region. people who already knows private IP can ignore this and simply fill PIP1 and PIP2

PIP1=$(aws ec2 describe-instances --region eu-west-1 --filter "Name=ip-address,Values=${HOST1}" --query 'Reservations[].Instances[].[PrivateIpAddress]' --output text) #for finding out the private IP
PIP2=$(aws ec2 describe-instances --region eu-west-1 --filter "Name=ip-address,Values=${HOST2}" --query 'Reservations[].Instances[].[PrivateIpAddress]' --output text) 

只是为了方便使用.根据您启动 ec2 实例的方式，有些人可能不需要这个.

for ease of usage only. some people may not need this based on how you spin up your ec2 instance.

SSH_ARGS="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${pem_file_loc}/amazon_key.pem" 

将pem文件复制到亚马逊机器上，如果pem文件不存在则忽略这个

copy the pem file to the amazon machine , if pem file is not present then ignore this

rsync -e "ssh ${SSH_ARGS}" ${pem_file_loc}/amazon_key.pem ubuntu@${HOST1}:~/
rsync -e "ssh ${SSH_ARGS}" ${pem_file_loc}/amazon_key.pem ubuntu@${HOST2}:~/

运行此命令在HOST1上设置ssh认证，没有pem文件的人，可以直接登录运行两个EOF之间的步骤.另请注意，为了便于编码，我正在设置 rserver1 和 rserver2.已经有很多主机的人可以避免这种情况，而是使用私有 IP 执行以下所有命令.

Run this command to set up the ssh authentication on HOST1 , people without pem file , can directly login and run the steps between the two EOF . Also note that i am setting up rserver1 and rserver2 for the ease of coding. People who already have many host machines can avoid this and instead use the private IP for all below commands.

ssh -T $SSH_ARGS ubuntu@${HOST1} <<EOF
sudo sh -c 'echo ${PIP1} rserver1 >> /etc/hosts'
sudo sh -c 'echo ${PIP2} rserver2 >> /etc/hosts'
rm -rf ~/.ssh/id_rsa.pub ~/.ssh/id_rsa
ssh-keygen -t rsa -N '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub | ssh -i amazon_key.pem -o StrictHostKeyChecking=no ubuntu@rserver2 'cat >> ~/.ssh/authorized_keys'
cat ~/.ssh/id_rsa.pub | ssh -i amazon_key.pem  -o StrictHostKeyChecking=no ubuntu@rserver1 'cat >> ~/.ssh/authorized_keys' #required for clustering
EOF

为 HOST2 运行相同的代码

Run the same for HOST2

ssh -T $SSH_ARGS ubuntu@${HOST2} <<EOF
sudo sh -c 'echo ${PIP1} rserver1 >> /etc/hosts'
sudo sh -c 'echo ${PIP2} rserver2 >> /etc/hosts'
rm -rf ~/.ssh/id_rsa.pub ~/.ssh/id_rsa
ssh-keygen -t rsa -N '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub | ssh -i amazon_key.pem  -o StrictHostKeyChecking=no ubuntu@rserver1 'cat >> ~/.ssh/authorized_keys'
EOF 

这些是我从博客中获取的命令 http://www.win-vector.com/blog/2016/01/running-r-jobs-quickly-on-many-machines/.

These are the commands i took from the blog http://www.win-vector.com/blog/2016/01/running-r-jobs-quickly-on-many-machines/ .

在 host1 的 R 服务器上运行这些命令.交叉验证您要保留的核心数.我的情况我用作 11 .根据经验，保持 detectCores() - 1

Run these commands on R server of host1. Cross verify the number of cores you want to keep. My case i used as 11 . empirically it is good to keep as detectCores() - 1

machineAddresses <- list(
  list(host='rserver1',user='ubuntu',
       ncore=11),
  list(host='rserver2',user='ubuntu',
       ncore=11)
)

spec <- lapply(machineAddresses,
               function(machine) {
                 rep(list(list(host=machine$host,
                               user=machine$user)),
                     machine$ncore)
               })

spec <- unlist(spec,recursive=FALSE)

library("doParallel")
cl <- makeCluster(type='PSOCK',master=primary,spec=spec)
registerDoParallel(cl)
#this is purely based on your need , there are many articles on how to run parallel loops , the focus is mainly on multiple hosts
clusterExport(cl, varlist=ls(.GlobalEnv)) 

print(cl)
##run your commands
stopCluster(cl)

以防万一命令挂起，请通过运行确认设置system("ssh ubuntu@rserver1") 和 system("ssh ubuntu@rserver2").如果 ssh 设置正确，这些命令应该可以工作.

Just in case the command hangs , confirm the setup by running system("ssh ubuntu@rserver1") and system("ssh ubuntu@rserver2") . These commands should work if ssh is properly setup.

这篇关于在多台主机上运行并行 R的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！