用雪(降雪)与AWS的在研发并行处理 [英] Using snow (and snowfall) with AWS for parallel processing in R

查看:177
本文介绍了用雪(降雪)与AWS的在研发并行处理的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

在关系到我的前面相似,所以怀疑,我试着使用AWS上的并行计算的雪/降雪。

In relation to my earlier similar SO question , I tried using snow/snowfall on AWS for parallel computing.

我所做的是:

  • sfInit()功能,我提供的公共DNS来 socketHosts 像这样的参数 sfInit(平行= TRUE,socketHosts =列表(ec2-00-00-00-000.compute-1.amazonaws.com))
  • 错误返回权限被拒绝(公钥)
  • 然后我跟着指示(I presume正确!)在的http://在www.imbi.uni-freiburg.de/parallel/ 无密码安全外壳(SSH)登录部分
  • 在我的猫,我创建的AWS到AWS实例的〜/ .ssh / authorized_keys中我想从我的主人AWS实例连接到和主AWS实例以及该.pem文件<内容/ LI>
  • In the sfInit() function, I provided the public DNS to socketHosts parameter like so sfInit(parallel=TRUE,socketHosts =list("ec2-00-00-00-000.compute-1.amazonaws.com"))
  • The error returned was Permission denied (publickey)
  • I then followed the instructions (I presume correctly!) on http://www.imbi.uni-freiburg.de/parallel/ in the 'Passwordless Secure Shell (SSH) login' section
  • I just cat the contents of the .pem file that I created on AWS into the ~/.ssh/authorized_keys of the AWS instance I want to connect to from my master AWS instance and for the master AWS instance as well

有什么我失踪了呢? 我会很感激,如果用户可以在AWS上使用的积雪分享他们的经验。

Is there anything I am missing out ? I would be very grateful if users can share their experiences in the use of snow on AWS.

非常感谢你对你的建议。

Thank you very much for your suggestions.

更新: 我只是想更新我发现我的具体问题的解决方案:

UPDATE: I just wanted to update the solution I found to my specific problem:

  • 在我以前StarCluster设置我的AWS集群: StarCluster
  • 在安装包降雪集群的所有节点上
  • 从主节点发出以下命令
  • hostslist&LT; - list("ec2-xxx-xx-xxx-xxx.compute-1.amazonaws.com","ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com")
  • sfInit(平行= TRUE,处理器= 2,类型=​​袜子,socketHosts = hostslist)
  • L&LT; - sfLapply(1:2,功能(X)系统(使用ifconfig,实习生= T))
  • lapply(L,函数(X)×[2])
  • sfStop()
  • IP信息证实了AWS节点正在使用
  • I used StarCluster to setup my AWS cluster : StarCluster
  • Installed package snowfall on all the nodes of the cluster
  • From the master node issued the following commands
  • hostslist <- list("ec2-xxx-xx-xxx-xxx.compute-1.amazonaws.com","ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com")
  • sfInit(parallel=TRUE, cpus=2, type="SOCK",socketHosts=hostslist)
  • l <- sfLapply(1:2,function(x)system("ifconfig",intern=T))
  • lapply(l,function(x)x[2])
  • sfStop()
  • The ip information confirmed that the AWS nodes were being utilized

推荐答案

我相信@Anatoliy是正确的:你使用的X.509证书。对于precise要采取的步骤添加SSH密钥,看的 EC2起动指南

I believe @Anatoliy is correct: you're using an X.509 certificate. For the precise steps to take to add the SSH keys, look at the "Types of credentials" section of the EC2 Starters Guide.

要上传自己的SSH密钥,一起来看看这个网页Alestic

To upload your own SSH keys, take a look at this page from Alestic.

这是起初有点混乱,但你要保持清醒这是您的访问密钥,您的证书和密钥对,它可能出现在文本文件DSA或RSA。

It is a little confusing at first, but you'll want to keep clear which are your access keys, your certificates, and your key pairs, which may appear in text files with DSA or RSA.

这篇关于用雪(降雪)与AWS的在研发并行处理的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆