在EMR 4.0错误启动星火 [英] Error starting Spark in EMR 4.0

问题描述

我创建了所有可用的应用程序，包括星火在AWS的 EMR 4.0 实例。我手工做的，通过AWS控制台。我开始集群和SSHed到主节点时，它涨。在那里，我跑了 pyspark 。我收到以下错误，当 pyspark 试图创建 SparkContext ：

I created an EMR 4.0 instance in AWS with all available applications, including Spark. I did it manually, through AWS Console. I started the cluster and SSHed to the master node when it was up. There I ran pyspark. I am getting the following error when pyspark tries to create SparkContext:

2015年9月3日19：36：04195错误线程3 spark.SparkContext   （Logging.scala：LOGERROR（96）） - -ec2用户，访问=写，   索引节点=/用户：HDFS：Hadoop的的。drwxr-XR-X在

2015-09-03 19:36:04,195 ERROR Thread-3 spark.SparkContext (Logging.scala:logError(96)) - -ec2-user, access=WRITE, inode="/user":hdfs:hadoop:drwxr-xr-x at

org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:271)

org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:271)

我还没有添加任何自定义应用程序，也没有引导和预期都没有错误的工作。不知道这是怎么回事。任何建议将大大AP preciated。

I haven't added any custom applications, nor bootstrapping and expected everything to work without errors. Not sure what's going on. Any suggestions will be greatly appreciated.

推荐答案

登录的用户Hadoop的（<一href="http://docs.aws.amazon.com/ElasticMa$p$pduce/latest/ManagementGuide/emr-connect-master-node-ssh.html" rel="nofollow">http://docs.aws.amazon.com/ElasticMa$p$pduce/latest/ManagementGuide/emr-connect-master-node-ssh.html).它具有所有适当的环境和预期工作的相关设置。您收到的错误是由于登录为EC2用户。

Login as the user "hadoop" (http://docs.aws.amazon.com/ElasticMapReduce/latest/ManagementGuide/emr-connect-master-node-ssh.html). It has all the proper environment and related settings for working as expected. The error you are receiving is due to logging in as "ec2-user".

这篇关于在EMR 4.0错误启动星火的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！