使用Pureconfig进行Spark-正确的Maven Shade插件配置 [英] Spark with Pureconfig - proper maven shade plugin configuration

问题描述

我遇到的问题与此处所述完全相同: Spark无法与pureconfig配合使用.上面问题的唯一答案似乎是合理的，但是我使用的是Maven而不是sbt，并且无法将发布的解决方案从sbt转换为Maven.

I have exactly the same problem as described here: Spark not working with pureconfig. The one and only answer to the question above seems reasonable, but I am working with Maven instead of sbt and I'm failing to translate the posted solution from sbt to Maven.

我尝试了以下操作:

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.0.0</version>
<executions>
    <execution>
        <phase>package</phase>
        <goals>
            <goal>shade</goal>
        </goals>
    </execution>
</executions>
<configuration>
    <createDependencyReducedPom>false</createDependencyReducedPom>
    <relocations>
        <relocation>
            <pattern>com.chuusai:shapeless_2.11:2.3.2</pattern>
            <shadedPattern>com.matek.shaded.com.chuusai:shapeless_2.11:2.3.2</shadedPattern>
        </relocation>
        <relocation>
            <pattern>com.chuusai:shapeless_2.11:2.0.0</pattern>
            <shadedPattern>com.matek.shaded.com.chuusai:shapeless_2.11:2.0.0</shadedPattern>
        </relocation>
        <relocation>
            <pattern>com.github.pureconfig</pattern>
            <shadedPattern>com.matek.shaded.com.github.pureconfig</shadedPattern>
            <excludes>
                <exclude>com.chuusai:shapeless_2.11:2.3.2</exclude>
            </excludes>
            <includes>
                <include>com.matek.shaded.com.chuusai:shapeless_2.11:2.3.2</include>
            </includes>
        </relocation>
    </relocations>
</configuration>

但并不奇怪，这是行不通的(我什至不确定它是否正确). 如何指定Maven Shade插件配置以使其与Spark Submit一起使用?

But not surprisingly, this does not work (I am not even sure whether it is correct). How to specify the maven shade plugin configuration to make it work with spark submit?

推荐答案

我设法解决了这个问题.实际上这是我的错误，模式只是shapeless而不是com.chuusai.shapeless之类的东西.这可行:

I managed to solve the issue. It was actually my mistake, the pattern is simply shapeless and not anything like com.chuusai.shapeless. This worked:

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.0.0</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <createDependencyReducedPom>false</createDependencyReducedPom>
                    <relocations>
                        <relocation>
                            <pattern>shapeless</pattern>
                            <shadedPattern>com.matek.shaded.shapeless</shadedPattern>
                        </relocation>
                    </relocations>
                </configuration>
            </plugin>

这篇关于使用Pureconfig进行Spark-正确的Maven Shade插件配置的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！