Spark 1.2.1独立集群模式spark-submit不起作用 [英] Spark 1.2.1 standalone cluster mode spark-submit is not working

问题描述

我有3个节点的火花群集

node1，node2和node 3

我在节点1上的以下命令中运行以部署驱动程序

/usr/local/spark-1.2.1-bin-hadoop2.4/bin/spark-submit --class com.fst.firststep.aggregator.FirstStepMessageProcessor --master spark://ec2-xx-xx- xx-xx.compute-1.amazonaws.com:7077-部署模式群集-监督文件:///home/xyz/sparkstreaming-0.0.1-SNAPSHOT.jar/home/xyz/config.properties

驱动程序在群集中的节点2上启动.但是在节点2上遇到异常，它试图绑定到节点1 ip.

2015-02-26 08:47:32 DEBUG AkkaUtils:63 - In createActorSystem, requireCookie is: off 
2015-02-26 08:47:32 INFO  Slf4jLogger:80 - Slf4jLogger started 
2015-02-26 08:47:33 ERROR NettyTransport:65 - failed to bind to ec2-xx.xx.xx.xx.compute-1.amazonaws.com/xx.xx.xx.xx:0, shutting down Netty transport 
2015-02-26 08:47:33 WARN  Utils:71 - Service 'Driver' could not bind on port 0. Attempting port 1. 
2015-02-26 08:47:33 DEBUG AkkaUtils:63 - In createActorSystem, requireCookie is: off 
2015-02-26 08:47:33 ERROR Remoting:65 - Remoting error: [Startup failed] [ 
akka.remote.RemoteTransportException: Startup failed 
        at akka.remote.Remoting.akka$remote$Remoting$$notifyError(Remoting.scala:136) 
        at akka.remote.Remoting.start(Remoting.scala:201) 
        at akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:184) 
        at akka.actor.ActorSystemImpl.liftedTree2$1(ActorSystem.scala:618) 
        at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:615) 
        at akka.actor.ActorSystemImpl._start(ActorSystem.scala:615) 
        at akka.actor.ActorSystemImpl.start(ActorSystem.scala:632) 
        at akka.actor.ActorSystem$.apply(ActorSystem.scala:141) 
        at akka.actor.ActorSystem$.apply(ActorSystem.scala:118) 
        at org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$doCreateActorSystem(AkkaUtils.scala:121) 
        at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:54) 
        at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:53) 
        at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1765) 
        at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) 
        at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1756) 
        at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:56) 
        at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:33) 
        at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala) 
Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to: ec2-xx-xx-xx.compute-1.amazonaws.com/xx.xx.xx.xx:0 
        at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272) 
        at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:393) 
        at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:389) 
        at scala.util.Success$$anonfun$map$1.apply(Try.scala:206) 
        at scala.util.Try$.apply(Try.scala:161) 
        at scala.util.Success.map(Try.scala:206) 

请建议

谢谢

解决方案

花了很多时间之后.我得到了答案.我做了以下更改

1. 删除SPARK_LOCAL_IP和SPARK_MASTER_IP的条目
2. 在etc/hosts中添加其他节点的主机名和私有IP地址.
3. 使用--deploy-mode群集--supervise

仅此而已，它与完整的HA组件(主站，从站和驱动程序)完美配合

谢谢

I have 3 node spark cluster

node1 , node2 and node 3

I running below command on node 1 for deploying driver

/usr/local/spark-1.2.1-bin-hadoop2.4/bin/spark-submit --class com.fst.firststep.aggregator.FirstStepMessageProcessor --master spark://ec2-xx-xx-xx-xx.compute-1.amazonaws.com:7077 --deploy-mode cluster --supervise file:///home/xyz/sparkstreaming-0.0.1-SNAPSHOT.jar /home/xyz/config.properties

driver gets launched on node 2 in cluster. but getting exception on node 2 that it is trying to bind to node 1 ip.

2015-02-26 08:47:32 DEBUG AkkaUtils:63 - In createActorSystem, requireCookie is: off 
2015-02-26 08:47:32 INFO  Slf4jLogger:80 - Slf4jLogger started 
2015-02-26 08:47:33 ERROR NettyTransport:65 - failed to bind to ec2-xx.xx.xx.xx.compute-1.amazonaws.com/xx.xx.xx.xx:0, shutting down Netty transport 
2015-02-26 08:47:33 WARN  Utils:71 - Service 'Driver' could not bind on port 0. Attempting port 1. 
2015-02-26 08:47:33 DEBUG AkkaUtils:63 - In createActorSystem, requireCookie is: off 
2015-02-26 08:47:33 ERROR Remoting:65 - Remoting error: [Startup failed] [ 
akka.remote.RemoteTransportException: Startup failed 
        at akka.remote.Remoting.akka$remote$Remoting$$notifyError(Remoting.scala:136) 
        at akka.remote.Remoting.start(Remoting.scala:201) 
        at akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:184) 
        at akka.actor.ActorSystemImpl.liftedTree2$1(ActorSystem.scala:618) 
        at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:615) 
        at akka.actor.ActorSystemImpl._start(ActorSystem.scala:615) 
        at akka.actor.ActorSystemImpl.start(ActorSystem.scala:632) 
        at akka.actor.ActorSystem$.apply(ActorSystem.scala:141) 
        at akka.actor.ActorSystem$.apply(ActorSystem.scala:118) 
        at org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$doCreateActorSystem(AkkaUtils.scala:121) 
        at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:54) 
        at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:53) 
        at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1765) 
        at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) 
        at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1756) 
        at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:56) 
        at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:33) 
        at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala) 
Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to: ec2-xx-xx-xx.compute-1.amazonaws.com/xx.xx.xx.xx:0 
        at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272) 
        at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:393) 
        at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:389) 
        at scala.util.Success$$anonfun$map$1.apply(Try.scala:206) 
        at scala.util.Try$.apply(Try.scala:161) 
        at scala.util.Success.map(Try.scala:206) 

kindly suggest

Thanks

解决方案

after spending lot more time.i got the answer.i did below changes

1. remove entry of SPARK_LOCAL_IP and SPARK_MASTER_IP
2. add host name and private ip address of each other nodes in etc/hosts.
3. use --deploy-mode cluster --supervise

thats all and it works perfectly with fully HA components(Master,Slaves and Driver)

Thanks

这篇关于Spark 1.2.1独立集群模式spark-submit不起作用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！