遮蔽第三方类 [英] Shading over third party classes

问题描述

我目前在将 uber-jar 部署到 Spark Streaming 应用程序时遇到问题，其中存在具有不同版本的一致 JAR，这会导致 spark 引发运行时异常.有问题的库是 TypeSafe Config.

I'm currently facing a problem with deploying an uber-jar to a Spark Streaming application, where there are congruent JARs with different versions which are causing spark to throw run-time exceptions. The library in question is TypeSafe Config.

在尝试了很多事情之后，我的解决方案是推迟对提供的依赖项进行着色，这样它就不会与 Spark 在运行时提供的 JAR 发生冲突.

After attempting many things, my solution was to defer to shading the provided dependency so it won't clash with the JAR provided by Spark at run-time.

因此，我查看了 sbt-assembly 的文档在阴影下，我看到了以下示例:

Hence, I went to the documentation for sbt-assembly and under shading, I saw the following example:

assemblyShadeRules in assembly := Seq(
      ShadeRule.rename("org.apache.commons.io.**" -> "shadeio.@1")
      .inLibrary("commons-io" % "commons-io" % "2.4", ...).inProject
)

试图遮蔽 com.typesafe.config，我尝试将以下解决方案应用于我的 build.sbt:

Attempting to shade over com.typesafe.config, I tried applying the following solution to my build.sbt:

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.typesafe.config.**" -> "shadeio.@1").inProject
)

我认为它应该在我的项目中重命名对 TypeSafe Config 的任何引用.但是，这行不通.它匹配我项目中的多个类，并导致它们从 uber jar 中删除.我在尝试运行 sbt assembly 时看到了这一点:

I assumed it was supposed to rename any reference to TypeSafe Config in my project. But, this doesn't work. It matches multiple classes in my project and causing them to be removed from the uber jar. I see this when trying to run sbt assembly:

Fully-qualified classname does not match jar entry:
  jar entry: ***/Identifier.class
  class name: **/Identifier.class
Omitting ***/OtherIdentifier.class.
Fully-qualified classname does not match jar entry:
  jar entry: ***SparkBaseJobRunner$$anonfun$1.class
  class name: ***/SparkBaseJobRunner$$anonfun$1.class

我也尝试过使用:

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.typesafe.config.**" -> "shadeio.@1")
           .inLibrary("com.typesafe" % "config" % "1.3.0")

这确实完成了 uber JAR 的组装过程，但没有达到预期的运行时效果.

This did finish the assemblying process of the uber JAR, but didn't have the desired run time effect.

我不确定我是否完全理解阴影对我使用 sbt 的构建过程的影响.

I'm not sure I fully comprehend the effect shading has on my build process with sbt.

如何在我的项目中遮蔽对 com.typesafe.config 的引用，这样当我在运行时调用库时，Spark 将加载我的遮蔽库并避免由版本控制引起的冲突?

How can I shade over references to com.typesafe.config in my project so when I invoke the library at run-time Spark will load my shaded library and avoid the clash caused by versioning?

我正在运行 sbt-assembly v0.14.1

I'm running sbt-assembly v0.14.1

推荐答案

原来 这个是 sbt-assembly 中的一个错误，它在 Windows 上完全破坏了阴影. 这导致源文件从 uber JAR 中删除，并且由于上述类不可用而导致测试失败.

Turns out this was a bug in sbt-assembly where shading was completely broken on Windows. This caused source files to be removed from the uber JAR, and for tests to fail as the said classes were unavailable.

我创建了一个 拉取请求来解决这个问题.从 SBT 的 0.14.3 版本开始，阴影功能可以正常工作.您需要做的就是更新到 plugins.sbt 中的相关版本:

I created a pull request to fix this. Starting version 0.14.3 of SBT, the shading feature works properly. All you need to do is update to the relevant version in plugins.sbt:

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")

为了在项目中对特定 JAR 进行着色，请执行以下操作:

In order to shade a specific JAR in your project, you do the following:

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.typesafe.config.**" -> "my_conf.@1")
    .inLibrary("com.typesafe" % "config" % "1.3.0")
    .inProject
)

这将重命名 com.typesafe.config 程序集以打包在 my_conf 中.然后，您可以在程序集上使用 jar -tf 来查看它(为简洁起见，省略了不相关的部分):

This will rename the com.typesafe.config assembly to be packaged inside my_conf. You can then view this using jar -tf on your assembly (omitted irrelevant parts for brevity):

***> jar -tf myassembly.jar
my_conf/
my_conf/impl/
my_conf/parser/

编辑

我写了一篇博文为任何有兴趣更深入解释的人描述问题和导致该问题的过程.

I wrote a blog post describing the issue and the process that led to it for anyone interested in a more in-depth explanation.

这篇关于遮蔽第三方类的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！