适用于Maven的Spark 2.1和Pureconfig 0.8解决方法 [英] Spark 2.1 with Pureconfig 0.8 workaround for Maven

问题描述

从 Spark无法与pureconfig配合使用的以下解决方案似乎是可行的sbt的解决方案，但为此很难找出Maven版本.尝试使用spark-submit使pureconfig 0.8与spark 2.1配合使用，但仍然在IntelliJ之外得到讨厌的Exception in thread "main" java.lang.NoSuchMethodError: shapeless.Witness$.mkWitness(Ljava/lang/Object;)Lshapeless/Witness;错误.

The below solution taken from Spark not working with pureconfig seems to be the working solution for sbt but having a hard time figuring out a maven version for doing this. Trying to get pureconfig 0.8 working with spark 2.1 using spark-submit but still getting the pesky Exception in thread "main" java.lang.NoSuchMethodError: shapeless.Witness$.mkWitness(Ljava/lang/Object;)Lshapeless/Witness; error outside of IntelliJ.

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("shapeless.**" -> "shadeshapless.@1")
    .inLibrary("com.chuusai" % "shapeless_2.11" % "2.3.2")
    .inLibrary("com.github.pureconfig" %% "pureconfig" % "0.7.0")
    .inProject
)

还从此处尝试了建议的解决方案使用Pureconfig进行火花-正确的maven树荫插件配置，但仍然没有运气.

Have also tried proposed solution from here Spark with Pureconfig - proper maven shade plugin configuration but still no luck.

如果我使用创建的uber jar，这是最终的配置，但是我不确定我是否完全了解maven着色的工作原理，是否可以避免创建其他重命名的jar?理想情况下，我只想将jar与要创建的依赖项一起使用，而不用以下内容创建其他第三个jar:

This is the final configuration that has worked if I use the uber jar that gets created but I'm not sure I fully understand how maven shading works and is there a way to avoid having to create an additional renamed jar? Ideally I want to just use the jar with dependencies that gets created and not create an additional third jar with the below:

   <plugin>
       <groupId>org.apache.maven.plugins</groupId>
       <artifactId>maven-shade-plugin</artifactId>
       <version>3.0.0</version>
       <executions>
           <execution>
               <phase>package</phase>
               <goals>
                   <goal>shade</goal>
               </goals>
           </execution>
       </executions>
       <configuration>
           <createDependencyReducedPom>false</createDependencyReducedPom>
           <relocations>
               <relocation>
                   <pattern>shapeless</pattern>
                   <shadedPattern>com.shaded.shapeless</shadedPattern>
               </relocation>
           </relocations>
           <filters>
               <filter>
                   <artifact>*:*</artifact>
                   <excludes>
                       <exclude>META-INF/*.SF</exclude>
                       <exclude>META-INF/*.DSA</exclude>
                       <exclude>META-INF/*.RSA</exclude>
                   </excludes>
               </filter>
           </filters>
           <finalName>uber-${project.artifactId}-${project.version}</finalName>
       </configuration>
   </plugin>

推荐答案

很抱歉回答得这么晚. 我真的不确定是什么引起了问题.我将我的pureconfig版本更改为0.8.0(正在使用0.7.2)，并且一切似乎仍然有效.

Sorry for answering so late. I'm really not sure what's causing the problem. I changed my pureconfig version to 0.8.0 (was using 0.7.2) and everything still seems to work.

由于这似乎并不容易，所以我将尝试为您提供有关我的项目结构的更多见解，也许这会有所帮助.哦，顺便说一句，我正在使用Maven 3.3.9，但我怀疑这很重要(谁知道).

As it does not seem easy, I'll try to give you more insight about how my project is structured, maybe this will somehow help. Oh and btw, I'm using maven 3.3.9 but I doubt it matters (who knows though).

因此，我的项目由子模块组成，即，我在其中指定了<modules></modules>的顶级pom，并且每个模块都有自己的pom.

So, my project consists of submodules, i.e. I have the top level pom where <modules></modules> are specified, and every module has its own pom.

现在，在顶级pom中，我使用了依赖项管理以及插件管理，所以看起来像这样:

Now, in the top level pom, I use dependency management as well as plugin management, so it looks like this:

</dependencyManagement>
  <dependencies>
    <dependency>
        <!-- some dependency -->
    </dependency>
  </dependencies>
</dependencyManagement>

<build>
    <pluginManagement>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>3.0.0</version>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.0.0</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <createDependencyReducedPom>false</createDependencyReducedPom>
                    <relocations>
                        <relocation>
                            <pattern>shapeless</pattern>
                            <shadedPattern>com.matek.chuusai.shapeless</shadedPattern>
                        </relocation>
                    </relocations>
                </configuration>
            </plugin>
        </plugins>
    </pluginManagement>
</build>

现在在子模块中，配置依赖项以及添加Maven阴影插件如下所示:

Now in the submodules, configuring the dependencies as well as adding maven shade plugin looks like this:

    <dependencies>
      <dependency>
          <!-- some dependency -->
      </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-shade-plugin</artifactId>
                <groupId>org.apache.maven.plugins</groupId>
            </plugin>
        </plugins>
    </build>

如果您的项目具有子模块，请尝试以这种方式进行配置.希望这会有所帮助.

If your project has submodules, try to configure it this way. Hope this helps (somehow).

这篇关于适用于Maven的Spark 2.1和Pureconfig 0.8解决方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！