无法将本地jar提交给Spark集群:java.nio.file.NoSuchFileException [英] Failed to submit local jar to spark cluster: java.nio.file.NoSuchFileException

问题描述

~/spark/spark-2.1.1-bin-hadoop2.7/bin$ ./spark-submit --master spark://192.168.42.80:32141 --deploy-mode cluster file:///home/me/workspace/myproj/target/scala-2.11/myproj-assembly-0.1.0.jar

Running Spark using the REST application submission protocol.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/06/20 16:41:30 INFO RestSubmissionClient: Submitting a request to launch an application in spark://192.168.42.80:32141.
17/06/20 16:41:31 INFO RestSubmissionClient: Submission successfully created as driver-20170620204130-0005. Polling submission state...
17/06/20 16:41:31 INFO RestSubmissionClient: Submitting a request for the status of submission driver-20170620204130-0005 in spark://192.168.42.80:32141.
17/06/20 16:41:31 INFO RestSubmissionClient: State of driver driver-20170620204130-0005 is now ERROR.
17/06/20 16:41:31 INFO RestSubmissionClient: Driver is running on worker worker-20170620203037-172.17.0.5-45429 at 172.17.0.5:45429.
17/06/20 16:41:31 ERROR RestSubmissionClient: Exception from the cluster:
java.nio.file.NoSuchFileException: /home/me/workspace/myproj/target/scala-2.11/myproj-assembly-0.1.0.jar
    sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:526)
    sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253)
    java.nio.file.Files.copy(Files.java:1274)
    org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$copyRecursive(Utils.scala:608)
    org.apache.spark.util.Utils$.copyFile(Utils.scala:579)
    org.apache.spark.util.Utils$.doFetchFile(Utils.scala:664)
    org.apache.spark.util.Utils$.fetchFile(Utils.scala:463)
    org.apache.spark.deploy.worker.DriverRunner.downloadUserJar(DriverRunner.scala:154)
    org.apache.spark.deploy.worker.DriverRunner.prepareAndRunDriver(DriverRunner.scala:172)
    org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:91)
17/06/20 16:41:31 INFO RestSubmissionClient: Server responded with CreateSubmissionResponse:
{
  "action" : "CreateSubmissionResponse",
  "message" : "Driver successfully submitted as driver-20170620204130-0005",
  "serverSparkVersion" : "2.1.1",
  "submissionId" : "driver-20170620204130-0005",
  "success" : true
}

spark-worker的日志:

Log from spark-worker:

 2017-06-20T20:41:30.807403232Z 17/06/20 20:41:30 INFO Worker: Asked to launch driver driver-20170620204130-0005
2017-06-20T20:41:30.817248508Z 17/06/20 20:41:30 INFO DriverRunner: Copying user jar file:///home/me/workspace/myproj/target/scala-2.11/myproj-assembly-0.1.0.jar to /opt/spark/work/driver-20170620204130-0005/myproj-assembly-0.1.0.jar
2017-06-20T20:41:30.883645747Z 17/06/20 20:41:30 INFO Utils: Copying /home/me/workspace/myproj/target/scala-2.11/myproj-assembly-0.1.0.jar to /opt/spark/work/driver-20170620204130-0005/myproj-assembly-0.1.0.jar
2017-06-20T20:41:30.885217508Z 17/06/20 20:41:30 INFO DriverRunner: Killing driver process!
2017-06-20T20:41:30.885694618Z 17/06/20 20:41:30 WARN Worker: Driver driver-20170620204130-0005 failed with unrecoverable exception: java.nio.file.NoSuchFileException: home/me/workspace/myproj/target/scala-2.11/myproj-assembly-0.1.0.jar 

知道为什么吗?谢谢

更新

以下命令正确吗?

./spark-submit --master spark://192.168.42.80:32141 --deploy-mode cluster file:///home/me/workspace/myproj/target/scala-2.11/myproj-assembly-0.1.0.jar

更新

我想我对火花有更多的了解，为什么我会遇到这个问题以及 spark-submit错误:ClassNotFoundException .关键在于，尽管此处在REST URL: spark://127.0.1.1:6066 (cluster mode)中使用了REST一词，但提交后应用程序jar不会上传到集群，这与我的理解有所不同.因此，Spark集群找不到应用程序jar，并且无法加载主类.

I think I understand a little more about the spark and why I had this problem and spark-submit error: ClassNotFoundException. The key point is that though the word REST used here REST URL: spark://127.0.1.1:6066 (cluster mode), the application jar will not be uploaded to the cluster after submission, which is different with my understanding. so, the spark cluster cannot find the application jar, and cannot load the main class.

我将尝试找到如何设置Spark集群并使用集群模式提交应用程序.不知道客户端模式是否将使用更多资源进行流作业.

I will try to find how to setup the spark cluster and use the cluster mode to submit application. No idea whether client mode will use more resources for streaming jobs.

推荐答案

Blockquote 更新

Blockquote UPDATE

我想我对火花有更多的了解，为什么我会遇到这个问题以及> spark-submit错误:ClassNotFoundException.关键点在于，尽管此处使用了> REST单词，即REST URL:spark://127.0.1.1:6066(集群模式)，但是提交后应用程序> jar将不会上传到集群，这与>我的理解不同.因此，spark集群找不到应用程序jar，并且>无法加载主类.

I think I understand a little more about the spark and why I had this problem and >spark-submit error: ClassNotFoundException. The key point is that though the word >REST used here REST URL: spark://127.0.1.1:6066 (cluster mode), the application >jar will not be uploaded to the cluster after submission, which is different with >my understanding. so, the spark cluster cannot find the application jar, and >cannot load the main class.

这就是为什么您必须在jar节点中找到jar文件，或者在提交火花之前将其放入hdfs中的原因.

That's why you have to locate the jar-file in the master node OR put it into the hdfs before the spark submit.

这是怎么做的: 1.)使用ubuntu命令将文件传输到主节点

This is how to do it: 1.) Transfering the file to the master node with ubuntu command

$ scp <file> <username>@<IP address or hostname>:<Destination>

例如:

$ scp mytext.txt tom@128.140.133.124:~/

2.)将文件传输到HDFS:

2.) Transfering the file to the HDFS:

$ hdfs dfs -put mytext.txt

希望我能为您提供帮助.

Hope I could help you.

这篇关于无法将本地jar提交给Spark集群:java.nio.file.NoSuchFileException的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！