将 jsoup 添加为提供的依赖项后，Storm 命令失败并显示 NoClassDefFoundError [英] Storm command fails with NoClassDefFoundError after adding jsoup as provided dependency

问题描述

我在我的项目中使用 JSoup，并且我已经在我的 POM 文件中声明了依赖项.它编译得很好，也运行得很好，但只有当我使用 jar 和所有依赖项 并将依赖项的范围更改为 compiled 时.

如果我将此范围更改为 provided，那么我仍然可以很好地编译，但不能运行它.它给了我 ClassNotFoundException.我已经在 classpath 和 path 变量中包含了必要的 JAR 文件，但我仍然面临这个问题.

我可以使用编译选项，但我真的很恼火，为什么我无法使用提供的选项运行它，如果有人，我真的会很感激可以帮我弄清楚原因.

以下是我看到的错误:

java.lang.NoClassDefFoundError: Lorg/jsoup/nodes/Document;在 java.lang.Class.getDeclaredFields0(Native Method)在 java.lang.Class.privateGetDeclaredFields(Class.java:2300)在 java.lang.Class.getDeclaredField(Class.java:1882)在 java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1605)在 java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:50)在 java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:423)在 java.security.AccessController.doPrivileged(Native Method)在 java.io.ObjectStreamClass.(ObjectStreamClass.java:411)在 java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308)在 java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1114)在 java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)在 backtype.storm.utils.Utils.serialize(Utils.java:52)在 backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:94)在 com.yahoo.amit.wordstorm.WordStormTopology.main(WordStormTopology.java:25)引起:java.lang.ClassNotFoundException:org.jsoup.nodes.Document在 java.net.URLClassLoader$1.run(URLClassLoader.java:202)在 java.security.AccessController.doPrivileged(Native Method)在 java.net.URLClassLoader.findClass(URLClassLoader.java:190)在 java.lang.ClassLoader.loadClass(ClassLoader.java:306)在 sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)在 java.lang.ClassLoader.loadClass(ClassLoader.java:247)……还有 14 个

以下是我的 POM 文件:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"><modelVersion>4.0.0</modelVersion><groupId>com.yahoo.amit.wordstorm</groupId><artifactId>wordstorm</artifactId><version>1.0-SNAPSHOT</version><包装>罐</包装><name>wordstorm</name><url>http://maven.apache.org</url><存储库><存储库><id>clojars.org</id><url>http://clojars.org/repo</url></repository></repositories><属性><project.build.sourceEncoding>UTF-8</project.build.sourceEncoding></属性><依赖项><依赖><groupId>storm</groupId><artifactId>storm</artifactId><version>0.8.2</version><scope>提供</scope></依赖><依赖><groupId>junit</groupId><artifactId>junit</artifactId><version>3.8.1</version><范围>测试</范围></依赖><依赖><groupId>org.jsoup</groupId><artifactId>jsoup</artifactId><version>1.7.2</version><scope>提供</scope></依赖></依赖项><构建><插件><!--将 maven-assembly-plugin 绑定到 package 阶段这将创建一个没有风暴依赖项的 jar 文件适合部署到集群.--><插件><artifactId>maven-assembly-plugin</artifactId><配置><descriptorRef>jar-with-dependencies</descriptorRef></descriptorRefs><存档><清单><mainClass></mainClass></清单></存档></配置><执行><执行><id>make-assembly</id><phase>包</phase><目标><目标>单</目标></目标></执行></执行></插件></plugins></build></项目>

以下是我的系统变量:

<块引用>

echo $PATH/用户/程序员/夏天项目/apache-maven-3.0.5/bin/:/Users/programmerman/Summer项目/storm-0.8.2/bin/:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Users/programmerman/Summer项目/CLASSPATH/jsoup-1.7.2.jar:/Users/programmerman/Summer项目/类路径/*回声 $CLASSPATH/用户/程序员/夏天项目/storm-0.8.2/storm-0.8.2.jar:/Users/programmerman/Summer项目/storm-0.8.2/lib/*:/Users/programmerman/Summer项目/storm-0.8.2/conf/storm.yaml:/Users/programmerman/SummerProject/storm-starter-masterPOM/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar:/Users/程序员/夏天项目/CLASSPATH/jsoup-1.7.2.jar:/Users/programmerman/Summer项目/类路径/*

解决方案

这既是关于 Maven 的问题，也是关于 Storm 及其部署模型的问题.你必须查看storm命令的实际作用.首先，它实际上是一个最终调用java的Python脚本.

如果您查看函数 get_classpath(extrajars)，您会注意到它根本不使用 $CLASSPATH 环境变量.相反，它会加载核心 Storm jar 和您在相对于您的工作目录的 lib/ 目录下的任何 jar，以及 ~/.storm<下的配置文件/p>

(您会发现忽略 $CLASSPATH 在许多 Java 应用程序中非常普遍.通常启动脚本"所做的第一件事就是覆盖 CLASSPATH 或根本不使用它.这是为了防止未知/不受支持/较早版本的 jar 导致应用程序出现问题).

当jsoup被声明为provided"时你的应用程序会失败:当你将jar声明为一个provided依赖时，它不会被打包到你的jar with dependencies"程序集中.请参阅此问题以获得一个很好的解释:maven 范围之间的差异编译并提供给 JAR 打包

tl;dr 的解释是编译范围与您的 uber-jar 一起提供，前提是范围不是，因为它应该由您要部署到的容器提供".通常，容器"是一个 Java Web 服务器，例如 Tomcat(因此，您永远不必将 JSP 或 Servlet jar 与 Java 中的 Web 应用程序一起发送).在这种情况下，您期望提供"类的容器"是 Storm.但是，Storm 不提供 jsoup，因此您的错误.

编译范围类仍然需要与您的应用程序一起提供，因为您的应用程序将实例化/使用接口、枚举等.

我的建议是只声明 jsoup编译"范围并继续.另一种方法是编写您自己的定制部署脚本和/或程序集，将 jsoup 放在 lib/ 下 - 最终本质上是相同的.

I'm using JSoup in my project and I've declared the dependency in my POM file. It compiles just fine and runs fine too, but only when I used the jar with all dependencies and change the have the scope of the dependency to compiled.

If I change this scope to provided, then I can still compile just fine, but not run it. It gives me the ClassNotFoundException. I have included the necessary JAR file in the classpath and also the path variables but I'm still facing this problem.

I can get working with the compile option but it's really irking me at the back of my mind why I can't get it running with the provided option, and I would really appreciate it if someone could help me figure why.

Following is the error I am seeing:

java.lang.NoClassDefFoundError: Lorg/jsoup/nodes/Document;
    at java.lang.Class.getDeclaredFields0(Native Method)
    at java.lang.Class.privateGetDeclaredFields(Class.java:2300)
    at java.lang.Class.getDeclaredField(Class.java:1882)
    at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1605)
    at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:50)
    at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:423)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:411)
    at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1114)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
    at backtype.storm.utils.Utils.serialize(Utils.java:52)
    at backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:94)
    at com.yahoo.amit.wordstorm.WordStormTopology.main(WordStormTopology.java:25)
Caused by: java.lang.ClassNotFoundException: org.jsoup.nodes.Document
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    ... 14 more

Following is my POM file:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.yahoo.amit.wordstorm</groupId>
  <artifactId>wordstorm</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>wordstorm</name>
  <url>http://maven.apache.org</url>

    <repositories>
        <repository>
            <id>clojars.org</id>
            <url>http://clojars.org/repo</url>
        </repository>
    </repositories>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <dependencies>
    <dependency>
            <groupId>storm</groupId>
            <artifactId>storm</artifactId>
            <version>0.8.2</version>
            <scope>provided</scope>
        </dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>

    <dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.7.2</version>
    <scope>provided</scope>
</dependency>

  </dependencies>
  <build>
    <plugins>
            <!--
            bind the maven-assembly-plugin to the package phase
            this will create a jar file without the storm dependencies
            suitable for deployment to a cluster.
             -->
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                    <archive>
                        <manifest>
                            <mainClass></mainClass>
                        </manifest>
                    </archive>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>

            </plugin>
        </plugins>
        </build>
</project>

Following are my system variables:

echo $PATH

/Users/programmerman/Summer
Project/apache-maven-3.0.5/bin/:/Users/programmerman/Summer
Project/storm-0.8.2/bin/:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Users/programmerman/Summer
Project/CLASSPATH/jsoup-1.7.2.jar:/Users/programmerman/Summer
Project/CLASSPATH/*

echo $CLASSPATH

/Users/programmerman/Summer
Project/storm-0.8.2/storm-0.8.2.jar:/Users/programmerman/Summer
Project/storm-0.8.2/lib/*:/Users/programmerman/Summer
Project/storm-0.8.2/conf/storm.yaml:/Users/programmerman/SummerProject/storm-starter-masterPOM/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar:/Users/programmerman/Summer
Project/CLASSPATH/jsoup-1.7.2.jar:/Users/programmerman/Summer
Project/CLASSPATH/*

解决方案

This is as much a question about Maven as it is about Storm and its deployment model. You have to check out what the storm command actually does. First of all, it's actually a Python script that ultimately calls java.

If you look at the function get_classpath(extrajars), you'll note that it does not use the $CLASSPATH evironment variable at all. Rather, it loads the core Storm jars and any jars that you have under a directory lib/ relative to your working directory, as well as config files under ~/.storm

(You will find that ignoring $CLASSPATH is very common in many Java applications. Usually the first thing a "launch script" does is overwrite the CLASSPATH or not use it at all. This is to prevent unknown / unsupported / earlier versions of your jars from causing problems in your application).

As to your application fails when jsoup is declared "provided": when you declare the jar as a provided dependency, it will not be packaged in your "jar with dependencies" assembly. See this question for a good explanation: Difference between maven scope compile and provided for JAR packaging

The tl;dr explanation is that compile scope is shipped with your uber-jar, provided scope isn't, because it's expected to be "provided" by the container you are deploying to. Typically, the "container" is a Java web server, like Tomcat (hence, you should never have to ship JSP or Servlet jars with your web apps in Java). In this case, the "container" that you are expecting to "provide" classes is Storm. However, jsoup is not provided by Storm, hence your error.

Compile-scope classes still need to be shipped with your application because your application will be instantiating / using interfaces, enums, etc.

My recommendation is to just declare jsoup "compile" scope and move on. The alternative will be to write your own bespoke deployment script and/or assembly that puts jsoup under lib/ - essentially the same thing in the end.

这篇关于将 jsoup 添加为提供的依赖项后，Storm 命令失败并显示 NoClassDefFoundError的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！