如何提交申请纱线集群所以在包装罐也被复制？ [英] How to submit applications to yarn-cluster so jars in packages are also copied?
本文介绍了如何提交申请纱线集群所以在包装罐也被复制？的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
我想提出一个火花的工作指定火花CSV 包作为一个依赖：
火花/斌/火花提交--packages com.databricks：火花csv_2.10：1.0.3 --deploy模式集群--master纱线集群script.py
但我得到下面的异常（片断）
15/05/05 22点23分46秒INFO yarn.Client：源文件和目标文件系统是相同的。不复制/home/hadoop/.ivy2/jars/spark-csv_2.10.jar
异常线程mainjava.io.FileNotFoundException：文件不存在：HDFS：//172.31.13.205：9000 /家庭/ Hadoop的/ .ivy2 /罐/火花csv_2.10.jar
的火花集群安装和使用下面的脚本配置：
AWS EMR创建集群--name沙箱--ami版本3.6 --instance型m3.xlarge --instance数3 \\
--ec2的属性KeyName是=沙盒\\
--applications名称=蜂巢\\
--bootstrap-行动路径= S3：//support.elasticma$p$pduce/spark/install-spark \\
--log-URI S3：// mybucket /火花日志\\
- 脚步 \\
Name=SparkHistoryServer,Jar=s3://elasticma$p$pduce/libs/script-runner/script-runner.jar,Args=s3://support.elasticma$p$pduce/spark/start-history-server \\
Name=SparkConfigure,Jar=s3://elasticma$p$pduce/libs/script-runner/script-runner.jar,Args=[s3://support.elasticma$p$pduce/spark/configure-spark.bash,spark.default.parallelism=100,spark.locality.wait.rack=0]
这应该是广泛适用，因为我想象利用EMR星火星火开发商是不是一种罕见的工作流程，我没有做任何事情太复杂了。
下面是扩展堆栈跟踪：
星火装配已建成蜂巢，包括类路径DataNucleus的罐子
常春藤默认缓存设置为：/home/hadoop/.ivy2/cache
存储在包的罐子：/home/hadoop/.ivy2/jars
::加载设置:: URL = jar:file:/home/hadoop/.versions/spark-1.3.0.d/lib/spark-assembly-1.3.0-hadoop2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
com.databricks＃火花csv_2.10添加为依赖
::解决依赖:: org.apache.spark＃火花提交父母; 1.0
confs：[默认]
发现com.databricks＃火花csv_2.10; 1.0.3在中央
发现org.apache.commons＃公地CSV 1.1中部
::分辨率报告::解决238MS ::文物DL的8ms
::在使用的模块：
com.databricks＃火花csv_2.10;中央1.0.3 [默认]
org.apache.commons＃公地CSV 1.1中央在[默认]
-------------------------------------------------- -------------------
| |模块||文物|
| CONF |号|搜索| dwnlded |驱逐||号| dwnlded |
-------------------------------------------------- -------------------
|默认| 2 | 0 | 0 | 0 || 2 | 0 |
-------------------------------------------------- -------------------
::检索:: org.apache.spark＃火花提交父
confs：[默认]
0文物复制，2已检索（0KB / 10毫秒）
15/05/05 22点07分23秒INFO client.RMProxy：连接到ResourceManager中的/172.31.13.205:9022
15/05/05 22点07分23秒INFO yarn.Client中请求集群2 NodeManagers一个新的应用
15/05/05 22点07分23秒INFO yarn.Client：验证我们的应用程序没有要求不是群集的最大内存容量更大（11520每个集装箱MB）
15/05/05 22点07分23秒INFO yarn.Client：将分配AM容器，具有896 MB内存，包括384 MB的开销
15/05/05 22点07分23秒INFO yarn.Client：设置容器背景下推出为我们AM
15/05/05 22点07分23秒INFO yarn.Client：preparing我们的AM容器资源
15/05/05 22点07分24秒INFO yarn.Client：上传资源文件：/home/hadoop/.versions/spark-1.3.0.d/lib/spark-assembly-1.3.0-hadoop2.4.0.jar - ＆GT; hdfs://172.31.13.205:9000/user/hadoop/.sparkStaging/application_1430862769169_0005/spark-assembly-1.3.0-hadoop2.4.0.jar
15/05/05 22点07分24秒INFO metrics.MetricsSaver：MetricsConfigRecord disabledInCluster：假instanceEngineCycleSec：60 clusterEngineCycleSec：60 disableClusterEngine：假的
15/05/05 22点07分24秒INFO metrics.MetricsSaver：创建MetricsSaver J-3C91V87M8TXWD：I-e4bd8f2d：SparkSubmit：05979期：60 /mnt/var/em/raw/i-e4bd8f2d_20150505_SparkSubmit_05979_raw.bin
15/05/05 22点07分25秒INFO yarn.Client：源文件和目标文件系统是相同的。不复制/home/hadoop/.ivy2/jars/spark-csv_2.10.jar
异常线程mainjava.io.FileNotFoundException：文件不存在：HDFS：//172.31.13.205：9000 /家庭/ Hadoop的/ .ivy2 /罐/火花csv_2.10.jar
在org.apache.hadoop.fs.Hdfs.getFileStatus（Hdfs.java:129）
在org.apache.hadoop.fs.AbstractFileSystem.resolvePath（AbstractFileSystem.java:460）
在org.apache.hadoop.fs.FileContext $ 23.next（FileContext.java:2120）
在org.apache.hadoop.fs.FileContext $ 23.next（FileContext.java:2116）
在org.apache.hadoop.fs.FSLinkResolver.resolve（FSLinkResolver.java:90）
在org.apache.hadoop.fs.FileContext.resolve（FileContext.java:2116）
在org.apache.hadoop.fs.FileContext.resolvePath（FileContext.java:591）
在org.apache.spark.deploy.yarn.Client.copyFileToRemote（Client.scala：203）
在org.apache.spark.deploy.yarn.Client$$anonfun$$p$ppareLocalResources$4$$anonfun$apply$1.apply(Client.scala:285)
在org.apache.spark.deploy.yarn.Client$$anonfun$$p$ppareLocalResources$4$$anonfun$apply$1.apply(Client.scala:280)
在scala.collection.IndexedSeqOptimized $ class.foreach（IndexedSeqOptimized.scala：33）
在scala.collection.mutable.ArrayOps $ ofRef.foreach（ArrayOps.scala：108）
在org.apache.spark.deploy.yarn.Client $$ anonfun $prepareLocalResources $ 4.适用（Client.scala：280）
在org.apache.spark.deploy.yarn.Client $$ anonfun $prepareLocalResources $ 4.适用（Client.scala：278）
在scala.collection.immutable.List.foreach（List.scala：318）
在org.apache.spark.deploy.yarn.Client prepareLocalResources（Client.scala：278）。
在org.apache.spark.deploy.yarn.Client.createContainerLaunchContext（Client.scala：384）
在org.apache.spark.deploy.yarn.Client.submitApplication（Client.scala：102）
在org.apache.spark.deploy.yarn.Client.run（Client.scala：619）
在org.apache.spark.deploy.yarn.Client $。主要（Client.scala：647）
在org.apache.spark.deploy.yarn.Client.main（Client.scala）
在sun.reflect.NativeMethodAccessorImpl.invoke0（本机方法）
在sun.reflect.NativeMethodAccessorImpl.invoke（NativeMethodAccessorImpl.java:57）
在sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）
在java.lang.reflect.Method.invoke（Method.java:606）
在org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
在org.apache.spark.deploy.SparkSubmit $ .doRunMain $ 1（SparkSubmit.scala：166）
在org.apache.spark.deploy.SparkSubmit $ .submit（SparkSubmit.scala：189）
在org.apache.spark.deploy.SparkSubmit $。主要（SparkSubmit.scala：110）
在org.apache.spark.deploy.SparkSubmit.main（SparkSubmit.scala）
15/05/05 22点07分25秒INFO metrics.MetricsSaver：保存3：3记录/mnt/var/em/raw/i-e4bd8f2d_20150505_SparkSubmit_05979_raw.bin
命令与RET退出'1'
解决方案
我觉得这可能是一个Apache星火错误，虽然我没有看到它在的星火JIRA 。 <一href=\"http://apache-spark-user-list.1001560.n3.nabble.com/Resources-not-uploaded-when-submitting-job-in-yarn-client-mode-td21516.html\" rel=\"nofollow\">http://apache-spark-user-list.1001560.n3.nabble.com/Resources-not-uploaded-when-submitting-job-in-yarn-client-mode-td21516.html似乎然而描述了同样的情况。根据该职位的问题是，在展开安装星火错误地认为目标系统相同的客户端系统，所以它foregoes复制：
15/05/05 22点07分25秒INFO yarn.Client：源文件和目标文件系统是相同的。不复制/home/hadoop/.ivy2/jars/spark-csv_2.10.jar
我建议尝试
- 罐子而不是- 包（见的提交申请）。如果这样的作品，请提交错误这个问题！I'm trying to submit a spark job specifying the
spark-csvpackage as a dependency:spark/bin/spark-submit --packages com.databricks:spark-csv_2.10:1.0.3 --deploy-mode cluster --master yarn-cluster script.pyBut I get the following exception (snippet)
15/05/05 22:23:46 INFO yarn.Client: Source and destination file systems are the same. Not copying /home/hadoop/.ivy2/jars/spark-csv_2.10.jar Exception in thread "main" java.io.FileNotFoundException: File does not exist: hdfs://172.31.13.205:9000/home/hadoop/.ivy2/jars/spark-csv_2.10.jarThe
sparkcluster is installed and configured with the following script:aws emr create-cluster --name sandbox --ami-version 3.6 --instance-type m3.xlarge --instance-count 3 \ --ec2-attributes KeyName=sandbox \ --applications Name=Hive \ --bootstrap-actions Path=s3://support.elasticmapreduce/spark/install-spark \ --log-uri s3://mybucket/spark-logs \ --steps \ Name=SparkHistoryServer,Jar=s3://elasticmapreduce/libs/script-runner/script-runner.jar,Args=s3://support.elasticmapreduce/spark/start-history-server \ Name=SparkConfigure,Jar=s3://elasticmapreduce/libs/script-runner/script-runner.jar,Args=[s3://support.elasticmapreduce/spark/configure-spark.bash,spark.default.parallelism=100,spark.locality.wait.rack=0]This should be widely applicable to Spark developers as I imagine using EMR with Spark is not an uncommon workflow and I'm not doing anything too complex.
Here's the extended stack trace:
Spark assembly has been built with Hive, including Datanucleus jars on classpath Ivy Default Cache set to: /home/hadoop/.ivy2/cache The jars for the packages stored in: /home/hadoop/.ivy2/jars :: loading settings :: url = jar:file:/home/hadoop/.versions/spark-1.3.0.d/lib/spark-assembly-1.3.0-hadoop2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml com.databricks#spark-csv_2.10 added as a dependency :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 confs: [default] found com.databricks#spark-csv_2.10;1.0.3 in central found org.apache.commons#commons-csv;1.1 in central :: resolution report :: resolve 238ms :: artifacts dl 8ms :: modules in use: com.databricks#spark-csv_2.10;1.0.3 from central in [default] org.apache.commons#commons-csv;1.1 from central in [default] --------------------------------------------------------------------- | | modules || artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| --------------------------------------------------------------------- | default | 2 | 0 | 0 | 0 || 2 | 0 | --------------------------------------------------------------------- :: retrieving :: org.apache.spark#spark-submit-parent confs: [default] 0 artifacts copied, 2 already retrieved (0kB/10ms) 15/05/05 22:07:23 INFO client.RMProxy: Connecting to ResourceManager at /172.31.13.205:9022 15/05/05 22:07:23 INFO yarn.Client: Requesting a new application from cluster with 2 NodeManagers 15/05/05 22:07:23 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (11520 MB per container) 15/05/05 22:07:23 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead 15/05/05 22:07:23 INFO yarn.Client: Setting up container launch context for our AM 15/05/05 22:07:23 INFO yarn.Client: Preparing resources for our AM container 15/05/05 22:07:24 INFO yarn.Client: Uploading resource file:/home/hadoop/.versions/spark-1.3.0.d/lib/spark-assembly-1.3.0-hadoop2.4.0.jar -> hdfs://172.31.13.205:9000/user/hadoop/.sparkStaging/application_1430862769169_0005/spark-assembly-1.3.0-hadoop2.4.0.jar 15/05/05 22:07:24 INFO metrics.MetricsSaver: MetricsConfigRecord disabledInCluster: false instanceEngineCycleSec: 60 clusterEngineCycleSec: 60 disableClusterEngine: false 15/05/05 22:07:24 INFO metrics.MetricsSaver: Created MetricsSaver j-3C91V87M8TXWD:i-e4bd8f2d:SparkSubmit:05979 period:60 /mnt/var/em/raw/i-e4bd8f2d_20150505_SparkSubmit_05979_raw.bin 15/05/05 22:07:25 INFO yarn.Client: Source and destination file systems are the same. Not copying /home/hadoop/.ivy2/jars/spark-csv_2.10.jar Exception in thread "main" java.io.FileNotFoundException: File does not exist: hdfs://172.31.13.205:9000/home/hadoop/.ivy2/jars/spark-csv_2.10.jar at org.apache.hadoop.fs.Hdfs.getFileStatus(Hdfs.java:129) at org.apache.hadoop.fs.AbstractFileSystem.resolvePath(AbstractFileSystem.java:460) at org.apache.hadoop.fs.FileContext$23.next(FileContext.java:2120) at org.apache.hadoop.fs.FileContext$23.next(FileContext.java:2116) at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) at org.apache.hadoop.fs.FileContext.resolve(FileContext.java:2116) at org.apache.hadoop.fs.FileContext.resolvePath(FileContext.java:591) at org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:203) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$4$$anonfun$apply$1.apply(Client.scala:285) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$4$$anonfun$apply$1.apply(Client.scala:280) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$4.apply(Client.scala:280) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$4.apply(Client.scala:278) at scala.collection.immutable.List.foreach(List.scala:318) at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:278) at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:384) at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:102) at org.apache.spark.deploy.yarn.Client.run(Client.scala:619) at org.apache.spark.deploy.yarn.Client$.main(Client.scala:647) at org.apache.spark.deploy.yarn.Client.main(Client.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 15/05/05 22:07:25 INFO metrics.MetricsSaver: Saved 3:3 records to /mnt/var/em/raw/i-e4bd8f2d_20150505_SparkSubmit_05979_raw.bin Command exiting with ret '1'
解决方案I think this may be an Apache Spark bug, though I don't see it reported in the Spark JIRA. http://apache-spark-user-list.1001560.n3.nabble.com/Resources-not-uploaded-when-submitting-job-in-yarn-client-mode-td21516.html seems to describe the same situation, however. According to that post the issue is that in your deployment setup Spark mistakenly believes that the destination system is the same as the client system, so it foregoes the copying:
15/05/05 22:07:25 INFO yarn.Client: Source and destination file systems are the same. Not copying /home/hadoop/.ivy2/jars/spark-csv_2.10.jar
I'd recommend trying
--jarsinstead of--packages(see Submitting Applications). If that works, please file a bug about this issue!这篇关于如何提交申请纱线集群所以在包装罐也被复制？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
