发现多个 defaults.yaml 资源 [英] Found multiple defaults.yaml resources

问题描述

当我尝试提交拓扑时，我发现了这个

when i tried to submit the topology i found this

Exception in thread "main" java.lang.RuntimeException: Found multiple defaults.yaml resources. You're probably bundling the Storm jars with your topology jar.
at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:115)
at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:135)
at backtype.storm.utils.Utils.readStormConfig(Utils.java:155)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:61)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:40)
at trident.myproject.main(myproject.java:288)

但是这个错误是在pom.xml中更新后出现的

But this error appeared after updated in pom.xml by

<代码><范围>编译而不是 <scope>provided</scope>

<scope>compile</scope> instead of <scope>provided</scope>

那是因为我错了

An exception occured while executing the Java class. storm/trident/state/StateFactory

这里是pom文件

<plugins>
    <plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <configuration>
            <descriptorRefs>
                <descriptorRef>jar-with-dependencies</descriptorRef>
            </descriptorRefs>
            <archive>
                <manifest>
                    <mainClass>trident.myproject</mainClass>
                    <!-- <mainClass>crawler.Crawler</mainClass> -->
                </manifest>
            </archive>
        </configuration>

pom 文件的第 2 部分

part 2 of pom file

<executions>
    <execution>
        <id>make-assembly</id>
        <phase>package</phase>
        <goals>
            <goal>single</goal>
        </goals>
    </execution>
</executions>

pom 文件的第 3 部分

part 3 of pom file

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
        <configuration>
            <source>1.6</source>
            <target>1.6</target>
        </configuration>
    </plugin>
</plugins>

推荐答案

在 LocalCluster 或通过 StormSubmitter 远程运行拓扑(这是项目中的默认设置).

There is a fundamental difference in running a topology in LocalCluster or remotely via StormSubmitter (which is the default setting in the project).

storm-core 的范围设置为 provided 是默认值，因为无论如何这些类文件在集群中都是可用的.provided 告诉 maven，那些类一定不能包含在组装的 jar 文件中，从而减少了 jar 的大小.此外，如果文件被多次提供，这可以避免冲突——如果将范围更改为 compile，default.yaml 就会发生这种情况.对于这种情况，storm-core 中的所有文件都会打包到您的 jar 中并提交给集群.Storm 在本地"(即，在集群中的工作机器上)和您的 jar 中找到文件 defaults.yaml.因此，Storm 不知道该使用哪个并引发错误.

The scope of storm-core is set to <scope>provided</scope> be the default because those class files are available in the cluster anyway. provided tells maven, that those classes must not be included in the jar file that is assembled, thus reducing the size of the jar. Furthermore, this avoids conflicts if files are provided multiple times -- that is what happens with default.yaml if you change the scope to compile. For those case, all files from storm-core are packaged into you jar and submitted to the cluster. Storm finds the file defaults.yaml "locally" (ie, locally on the worker machine in the cluster) and in your jar. Thus, Storm does not know which one to use and raises an error.

但是，如果您也在本地运行，provided 会排除这些类文件.当然，在本地，这些文件不是自动可用的，而是必须在启动本地 JVM 时包含在 CLASSPATH 中.由于 provided 从 storm-core 中排除了文件，你会得到 ClassNotFound 异常.

However, provided excludes those class files if you run locally, too. Of course, locally those files are not available automatically but must be included in CLASSPATH when starting up the local JVM. As provided excludes the files from storm-core you get the ClassNotFound exception.

作为每次要提交到不同环境时更改范围的替代方法，您可以将范围设置为 compile 并在 maven-jar-plugin 设置.这种显式包含会自动从 jar 中排除所有其他文件，即 storm-core 中的所有文件.

As an alternative to change the scope each time you want to submit to a different environment, you can set the scope to compile and include your topology Main/Bolt/Spout classes explicitly in your maven-jar-plugin settings. This explicit inclusion automatically excludes all other files from the jar, ie, all files from storm-core.

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-jar-plugin</artifactId>
  <version>2.6</version>

  <executions>
    <execution>
      <id>MyTopology</id>
      <phase>package</phase>
      <goals>
        <goal>jar</goal>
      </goals>
      <configuration>
        <includes>
          <include>my/topology/package/**/*.class</include>
        </includes>
      </configuration>
    </execution>
  </executions>
</plugin>

这篇关于发现多个 defaults.yaml 资源的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！