Apache Spark 错误:无法连接到 akka.tcp://sparkMaster@ [英] Apache Spark error : Could not connect to akka.tcp://sparkMaster@

问题描述

这是我们使用 apache spark 和 hadoop 等大数据的第一步.

This is our first steps using big data stuff like apache spark and hadoop.

我们已经安装了 Cloudera CDH 5.3.我们选择从 cloudera manager 安装 spark.Spark 在集群中的一个节点中启动并运行良好.

We have a installed Cloudera CDH 5.3. From the cloudera manager we choose to install spark. Spark is up and running very well in one of the nodes in the cluster.

在我的机器上，我制作了一个小应用程序，它可以连接读取存储在 hadoop HDFS 上的文本文件.

From my machine I made a little application that connects to read a text file stored on hadoop HDFS.

我正在尝试从 Eclipse 运行应用程序并显示这些消息

I am trying to run the application from Eclipse and it displays these messages

15/02/11 14:44:01 INFO client.AppClient$ClientActor: 连接到 master spark://10.62.82.21:7077...15/02/11 14:44:02 警告 client.AppClient$ClientActor:无法连接到 akka.tcp://sparkMaster@10.62.82.21:7077:akka.remote.InvalidAssociation:无效地址:akka.tcp://sparkMaster@10.62.82.21:707715/02/11 14:44:02 警告远程处理:试图关联无法访问的远程地址 [akka.tcp://sparkMaster@10.62.82.21:7077].地址现在被门控 5000 毫秒，所有发送到该地址的消息都将被发送到死信中.原因:连接被拒绝:无更多信息:/10.62.82.21:7077

应用程序有一个类，使用以下行创建上下文

The application is has one class the create a context using the following line

JavaSparkContext sc = new JavaSparkContext(new SparkConf().setAppName("Spark Count").setMaster("spark://10.62.82.21:7077"));

这里的IP是spark工作的机器的IP.

where this IP is the IP of the machine spark working on.

然后我尝试使用以下行从 HDFS 读取文件

Then I try to read a file from HDFS using the following line

sc.textFile("hdfs://10.62.82.21/tmp/words.txt")

当我运行应用程序时，我得到了

When I run the application I got the

推荐答案

检查您的 Spark 主日志，您应该看到类似:

Check your Spark master logs, you should see something like:

15/02/11 13:37:14 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkMaster@mymaster:7077]
15/02/11 13:37:14 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkMaster@mymaster:7077]
15/02/11 13:37:14 INFO Master: Starting Spark master at spark://mymaster:7077

然后当你连接到master时，一定要使用与上面日志中找到的完全相同的主机名(不要使用IP地址):

Then when your connecting to the master, be sure to use exactly the same hostname as found in the logs above (do not use the IP address):

.setMaster("spark://mymaster:7077"));

Spark Standalone 对这个主机名/IP 内容有点挑剔.

Spark standalone is a bit picky with this hostname/IP stuff.

这篇关于Apache Spark 错误:无法连接到 akka.tcp://sparkMaster@的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！