Spark和Influx:OKIO冲突 [英] Spark and Influx: OKIO conflict

问题描述

我在Spark Yarn上运行作业，并尝试向Influx DB发送消息，但是由于okio冲突而崩溃:

I'm running a job on Spark Yarn and trying to emit messages to Influx DB but I'm crashing on an okio conflict:

22:17:54 ERROR ApplicationMaster - User class threw exception: java.lang.NoSuchMethodError: okio.BufferedSource.readUtf8LineStrict(J)Ljava/lang/String;
java.lang.NoSuchMethodError: okio.BufferedSource.readUtf8LineStrict(J)Ljava/lang/String;
    at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:212)
    at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)

这是我的依赖项:

val cdhVersion = "cdh5.12.2"
val sparkVersion = "2.2.0.cloudera2"
val parquetVersion = s"1.5.0-$cdhVersion"
val hadoopVersion = s"2.6.0-$cdhVersion"
val awsVersion = "1.11.295"
val log4jVersion = "1.2.17"
val slf4jVersion = "1.7.5" 

lazy val sparkDependencies = Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-hive" % sparkVersion,
  "org.apache.spark" %% "spark-sql" % sparkVersion,
  "org.apache.spark" %% "spark-streaming" % sparkVersion,
  "org.apache.hadoop" % "hadoop-common" % "2.2.0"
)

lazy val otherDependencies = Seq(
  "org.apache.spark" %% "spark-streaming-kinesis-asl" % "2.2.0",
  "org.clapper" %% "grizzled-slf4j" % "1.3.1",
  "org.apache.logging.log4j" % "log4j-slf4j-impl" % "2.6.2" % "runtime",
  "org.slf4j" % "slf4j-log4j12" % slf4jVersion,
  "com.typesafe" % "config" % "1.3.1",
  "org.rogach" %% "scallop" % "3.0.3",
  "org.influxdb" % "influxdb-java" % "2.9"
)


libraryDependencies ++= sparkDependencies.map(_ % "provided" ) ++ otherDependencies

dependencyOverrides ++= Set("com.squareup.okio" % "okio" % "1.13.0")

使用相同的jar，我可以运行成功的测试以实例化非火花作业中的InfluxDb实例.但是尝试从Spark执行某些操作会抛出上述错误.听起来spark必须具有它自己的OKIO版本，这会在我使用spark-submit时在运行时引起此冲突. ...但是当我转储依赖树时，它并没有显示出来. 关于如何将所需版本的okio 1.13.0引入Spark集群运行路径的任何建议?

Using the same jar I can run a succesful test to instantiate an InfluxDb instance in a non-spark job. But trying to do some from Spark throws the above error. Sounds like spark must have it's own version of OKIO that's causing this conflict at run when I use spark-submit. ... But it doesn't show that when I dump the dependency tree. Any advice on how I can bring my desired version of okio 1.13.0 to the spark cluster run path?

(在输入文字时，我正在考虑尝试着色，现在我会这样做) 谢谢

(as I'm typing I'm thinking to try shading which I will do now) Thanks

推荐答案

在我的案例中，将Apache Spark 1.6.3与Hadoop HDP分发一起使用"

In my case "using Apache Spark 1.6.3 with Hadoop HDP distribution"

1. 我运行spark-shell并在Web UI上查看使用了什么jar
2. 搜索okhttp jar tf /usr/hdp/current/spark-client/lib/spark-assembly-1.6.3.2.6.3.0-235-hadoop2.7.3.2.6.3.0-235.jar | grep okhttp
3. 提取okhttp版本jar xf /usr/hdp/current/spark-client/lib/spark-assembly-1.6.3.2.6.3.0-235-hadoop2.7.3.2.6.3.0-235.jar META-INF/maven/com.squareup.okhttp/okhttp/pom.xml

1. I run spark-shell and see on web UI what jar are used
2. Search okhttp jar tf /usr/hdp/current/spark-client/lib/spark-assembly-1.6.3.2.6.3.0-235-hadoop2.7.3.2.6.3.0-235.jar | grep okhttp
3. Extract okhttp version jar xf /usr/hdp/current/spark-client/lib/spark-assembly-1.6.3.2.6.3.0-235-hadoop2.7.3.2.6.3.0-235.jar META-INF/maven/com.squareup.okhttp/okhttp/pom.xml

=>版本2.4.0

=> version 2.4.0

不知道向谁提供此版本.

No idea who is provided this version.

这篇关于Spark和Influx:OKIO冲突的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！