Spark无法与pureconfig一起使用 [英] Spark not working with pureconfig

问题描述

我正在尝试将pureConfig和configFactory用于我的Spark应用程序配置. 这是我的代码:

I'm trying to use pureConfig and configFactory for my spark application configuration. here is my code:

import pureconfig.{loadConfigOrThrow}
object Source{
  def apply(keyName: String, configArguments: Config): Source = {
    keyName.toLowerCase match {
      case "mysql" =>
          val properties = loadConfigOrThrow[DBConnectionProperties](configArguments)
          new MysqlSource(None, properties)
      case "files" =>
        val properties = loadConfigOrThrow[FilesSourceProperties](configArguments)
        new Files(properties)
      case _ => throw new NoSuchElementException(s"Unknown Source ${keyName.toLowerCase}")
    }

  }
}

import Source
val config = ConfigFactory.parseString(result.mkString("\n"))
    val source = Source("mysql",config.getConfig("source.mysql"))

当我从IDE(intelliJ)或直接从Java运行它时 (即Java jar ...)，效果很好.

when I run it from the IDE (intelliJ) or directly from java (i.e java jar...) it works fine.

但是当我使用spark-submit运行它时，它失败并显示以下错误:

But when I run it with spark-submit it fails with the following error:

Exception in thread "main" java.lang.NoSuchMethodError: shapeless.Witness$.mkWitness(Ljava/lang/Object;)Lshapeless/Witness;

通过快速搜索，我发现了与此问题类似的问题. 这表明其原因是由于spark和pureConfig都依赖Shapeless但具有不同的版本，

From a quick search I found a similar similar to this question. which suggest the reason for this is due to the fact both spark and pureConfig depends on Shapeless but with different versions,

我尝试按照答案中的建议对其进行着色

I tried to shade it as suggested in the answer

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("shapeless.**" -> "shadeshapless.@1")
    .inLibrary("com.github.pureconfig" %% "pureconfig" % "0.7.0").inProject
)

但效果不佳 可能是出于其他原因吗? 知道什么可行吗?

but it didn't work as well can it be from a different reason? any idea what may work?

谢谢

推荐答案

除了pureconfig之外，您还必须在其自己的JAR中隐藏无形的阴影:

You also have to shade shapeless inside its own JAR, in addition to pureconfig:

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("shapeless.**" -> "shadeshapless.@1")
    .inLibrary("com.chuusai" % "shapeless_2.11" % "2.3.2")
    .inLibrary("com.github.pureconfig" %% "pureconfig" % "0.7.0")
    .inProject
)

确保添加正确的无变形版本.

Make sure to add the correct shapeless version.

这篇关于Spark无法与pureconfig一起使用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！