如何在jar中与主类进行火花提交? [英] How to spark-submit with main class in jar?

查看：44
本文介绍了如何在jar中与主类进行火花提交?的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

关于 ClassNotFoundException 的问题很多，但我还没有看到任何适合此特定情况的问题.我正在尝试运行以下命令:

There are a ton of questions about ClassNotFoundException but I haven't seen any (yet) that fit this specific case. I am attempting to run the following command:

spark-submit --master local [*] --class com.stronghold.HelloWorld scala-ts.jar

它引发以下异常:

\u@\h:\w$ spark_submit --class com.stronghold.HelloWorld scala-ts.jar                                                                                                                                                                                                                                                                               ⬡ 9.8.0 [±master ●●●] 
2018-05-06 19:52:33 WARN  Utils:66 - Your hostname, asusTax resolves to a loopback address: 127.0.1.1; using 192.168.1.184 instead (on interface p1p1)                               
2018-05-06 19:52:33 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address                                                                                       
2018-05-06 19:52:33 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable                                
java.lang.ClassNotFoundException: com.stronghold.HelloWorld                               
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)                     
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)                          
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)                          
        at java.lang.Class.forName0(Native Method)                                        
        at java.lang.Class.forName(Class.java:348)                                        
        at org.apache.spark.util.Utils$.classForName(Utils.scala:235)                     
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:836)                                                                  
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)        
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)             
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)               
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)                    
2018-05-06 19:52:34 INFO  ShutdownHookManager:54 - Shutdown hook called                   
2018-05-06 19:52:34 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-e8a77988-d30c-4e96-81fe-bcaf5d565c75

但是，广口瓶显然包含此类:

However, the jar clearly contains this class:

1     " zip.vim version v28                                                                                                                                                                                                                                                                                                                                               
    1 " Browsing zipfile /home/[USER]/projects/scala_ts/out/artifacts/TimeSeriesFilter_jar/scala-ts.jar
    2 " Select a file with cursor and press ENTER
    3  
    4 META-INF/MANIFEST.MF
    5 com/
    6 com/stronghold/
    7 com/stronghold/HelloWorld$.class
    8 com/stronghold/TimeSeriesFilter$.class
    9 com/stronghold/DataSource.class
   10 com/stronghold/TimeSeriesFilter.class
   11 com/stronghold/HelloWorld.class
   12 com/stronghold/scratch.sc
   13 com/stronghold/HelloWorld$delayedInit$body.class

通常，这里的挂断是关于文件结构的，但是我很确定这是正确的:

Typically, the hang up here is on file structure, but I am pretty sure that's correct here: 

../
scala_ts/
| .git/
| .idea/
| out/
| | artifacts/
| | | TimeSeriesFilter_jar/
| | | | scala-ts.jar
| src/
| | main/
| | | scala/
| | | | com/
| | | | | stronghold/
| | | | | | DataSource.scala
| | | | | | HelloWorld.scala
| | | | | | TimeSeriesFilter.scala
| | | | | | scratch.sc
| | test/
| | | scala/
| | | | com/
| | | | | stronghold/
| | | | | | AppTest.scala
| | | | | | MySpec.scala                                                                                                                                                                                                                                                                                                                                                  
| target/
| README.md
| pom.xml

我已经在工作中以相同的结构(因此，在不同的环境中)进行了其他工作.我现在正尝试通过家庭项目获得更多的便利，但这似乎是一个早期的挂断.

I have run other jobs with the same structure at work (so, a different environment). I am now trying to gain some more facility with a home project, but this seems to be an early hang up.

简而言之，我只是错过了显而易见的东西吗?

In a nutshell, am I just missing something glaringly obvious?

APPENDIX

对于那些感兴趣的人，这是我的pom:

For those that are interested, here is my pom:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.stronghold</groupId>
  <artifactId>scala-ts</artifactId>
  <version>1.0-SNAPSHOT</version>
  <inceptionYear>2008</inceptionYear>
  <properties>
    <scala.version>2.11.8</scala.version>
  </properties>

  <repositories>
    <repository>
      <id>scala-tools.org</id>
      <name>Scala-Tools Maven2 Repository</name>
      <url>http://scala-tools.org/repo-releases</url>
    </repository>
  </repositories>

  <pluginRepositories>
    <pluginRepository>
      <id>scala-tools.org</id>
      <name>Scala-Tools Maven2 Repository</name>
      <url>http://scala-tools.org/repo-releases</url>
    </pluginRepository>
  </pluginRepositories>

  <dependencies>
    <dependency>
      <groupId>org.scala-lang</groupId>
      <artifactId>scala-library</artifactId>
      <version>2.11.8</version>
    </dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.9</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.scala-tools.testing</groupId>
      <artifactId>specs_2.10</artifactId>
      <version>1.6.9</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.11</artifactId>
      <version>2.2.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.11</artifactId>
      <version>2.2.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-catalyst_2.11</artifactId>
      <version>2.2.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-common</artifactId>
      <version>2.7.3</version>
    </dependency>
  </dependencies>

  <build>
    <sourceDirectory>src/main/scala</sourceDirectory>
    <testSourceDirectory>src/test/scala</testSourceDirectory>
    <plugins>
      <plugin>
        <groupId>org.scala-tools</groupId>
        <artifactId>maven-scala-plugin</artifactId>
        <executions>
          <execution>
            <goals>
              <goal>compile</goal>
              <goal>testCompile</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <scalaVersion>${scala.version}</scalaVersion>
          <args>
            <arg>-target:jvm-1.5</arg>
          </args>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-eclipse-plugin</artifactId>
        <configuration>
          <downloadSources>true</downloadSources>
          <buildcommands>
            <buildcommand>ch.epfl.lamp.sdt.core.scalabuilder</buildcommand>
          </buildcommands>
          <additionalProjectnatures>
            <projectnature>ch.epfl.lamp.sdt.core.scalanature</projectnature>
          </additionalProjectnatures>
          <classpathContainers>
            <classpathContainer>org.eclipse.jdt.launching.JRE_CONTAINER</classpathContainer>
            <classpathContainer>ch.epfl.lamp.sdt.launching.SCALA_CONTAINER</classpathContainer>
          </classpathContainers>
        </configuration>
      </plugin>
    </plugins>
  </build>
  <reporting>
    <plugins>
      <plugin>
        <groupId>org.scala-tools</groupId>
        <artifactId>maven-scala-plugin</artifactId>
        <configuration>
          <scalaVersion>${scala.version}</scalaVersion>
        </configuration>
      </plugin>
    </plugins>
  </reporting>
</project>

更新

道歉，缺乏明确性.我从与 .jar (/home/[USER]/projects/scala_ts/out/artifacts/TimeSeriesFilter_jar/)相同的目录中运行命令.就是说，为了清楚起见，指定完整路径不会改变结果.

Apologies for the lack of clarity. I ran the command from within the same directory as the .jar (/home/[USER]/projects/scala_ts/out/artifacts/TimeSeriesFilter_jar/). That said, just to be clear, specifying the full path does not change the outcome.

还应注意，我可以在Intellij中运行HelloWorld，它使用相同的类引用( com.stronghold.HelloWorld ).

It should also be noted that I can run HelloWorld from within Intellij, and it uses the same class reference (com.stronghold.HelloWorld).

推荐答案

害怕这些都不是问题.我以前曾尝试删除项目中的所有内容并重新开始，但是那也不起作用.一旦想到启动一个完全不同的项目，它就可以正常工作.显然，Intellij(我是我的粉丝)决定在某个地方创建一个隐藏的问题.

Afraid none of these were the issue. I had previously tried deleting everything in the project and starting over, but that didn't work either. Once it occurred to me start an entirely different project, it worked just fine. Apparently Intellij (of which I am a fan) decided to create a hidden problem somewhere.

这篇关于如何在jar中与主类进行火花提交?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
Java开发最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆