SBT测试错误:java.lang.NoSuchMethodError:net.jpountz.lz4.LZ4BlockInputStream [英] SBT Test Error: java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream

问题描述

遇到以下异常时，当我尝试使用scalatest在SBT窗口上对我的Spark Streaming代码执行单元测试时.

Getting Below exception , when i tried to perform unit tests for my spark streaming code on SBT windows using scalatest.

sbt testOnly <<ClassName>>

sbt testOnly <<ClassName>>

*
*
*
*
*
*

*
*
*
*
*
*

2018-06-18 02:39:00错误执行程序:91-阶段1.0(TID 11)中的任务1.0发生异常 java.lang.NoSuchMethodError:net.jpountz.lz4.LZ4BlockInputStream.(Ljava/io/InputStream; Z)V 在org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122) 在org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163) 在org.apache.spark.serializer.SerializerManager.wrapStream(SerializerManager.scala:124) 在org.apache.spark.shuffle.BlockStoreShuffleReader $$ anonfun $ 2.apply(BlockStoreShuffleReader.scala:50) 在org.apache.spark.shuffle.BlockStoreShuffleReader $$ anonfun $ 2.apply(BlockStoreShuffleReader.scala:50) 在org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:417) 在org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:61) 在scala.collection.Iterator $$ anon $ 12.nextCur(Iterator.scala:435) 在scala.collection.Iterator $$ anon $ 12.hasNext(Iterator.scala:441) 在scala.collection.Iterator $$ anon $ 11.hasNext(Iterator.scala:409) 在org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32) 在org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37) 在scala.collection.Iterator $$ anon $ 11.hasNext(Iterator.scala:409) 在org.apache.spark.sql.catalyst.expressions.GeneratedClass $ GeneratedIteratorForCodegenStage1.sort_addToSorter $(未知来源)处 在org.apache.spark.sql.catalyst.expressions.GeneratedClass $ GeneratedIteratorForCodegenStage1.processNext(未知来源) 在org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) 在org.apache.spark.sql.execution.WholeStageCodegenExec $$ anonfun $ 10 $$ anon $ 1.hasNext(WholeStageCodegenExec.scala:614) 在org.apache.spark.sql.execution.GroupedIterator $ .apply(GroupedIterator.scala:29) 在org.apache.spark.sql.execution.streaming.FlatMapGroupsWithStateExec $ StateStoreUpdater.updateStateForKeysWithData(FlatMapGroupsWithStateExec.scala:176)**

2018-06-18 02:39:00 ERROR Executor:91 - Exception in task 1.0 in stage 3.0 (TID 11) java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.(Ljava/io/InputStream;Z)V at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122) at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163) at org.apache.spark.serializer.SerializerManager.wrapStream(SerializerManager.scala:124) at org.apache.spark.shuffle.BlockStoreShuffleReader$$anonfun$2.apply(BlockStoreShuffleReader.scala:50) at org.apache.spark.shuffle.BlockStoreShuffleReader$$anonfun$2.apply(BlockStoreShuffleReader.scala:50) at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:417) at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:61) at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.sort_addToSorter$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614) at org.apache.spark.sql.execution.GroupedIterator$.apply(GroupedIterator.scala:29) at org.apache.spark.sql.execution.streaming.FlatMapGroupsWithStateExec$StateStoreUpdater.updateStateForKeysWithData(FlatMapGroupsWithStateExec.scala:176)**

尝试了几件事以排除net.jpountz.lz4 jar(带有其他帖子的建议)，但输出同样出现错误.

Tried couple of things to exclude net.jpountz.lz4 jar( with suggestions from other posts) but again same error in output.

当前使用spark 2.3，scalatest 3.0.5和Scala 2.11版本.我只有在升级到Spark 2.3和Scalatest 3.0.5以后才看到此问题

Currently using spark 2.3 , scalatest 3.0.5, Scala 2.11 version . i see this issue only after upgrade to spark 2.3 and scalatest 3.0.5

有什么建议吗?

推荐答案

Kafka与Spark的依赖项冲突，这就是导致我遇到此问题的原因.

Kafka has a conflicting dependency with Spark and that's what caused this issue for me.

这是您排除依赖项的方法在您的sbt文件中

This is how you can exclude the dependency in you sbt file

lazy val excludeJpountz = ExclusionRule(organization = "net.jpountz.lz4", name = "lz4")

lazy val kafkaClients = "org.apache.kafka" % "kafka-clients" % userKafkaVersionHere excludeAll(excludeJpountz) // add more exclusions here

使用此kafkaClients依赖项时，现在将排除有问题的lz4库.

When you use this kafkaClients dependency it would now exclude the problematic lz4 library.

更新: 这似乎是Kafka 0.11.x.x和更早版本的问题.从1.x.x开始，Kafka似乎已不再使用有问题的net.jpountz.lz4库.因此，将最新的Kafka(1.x)与最新的Spark(2.3.x)一起使用应该不会出现此问题.

Update: This appears to be an issue with Kafka 0.11.x.x and earlier version. As of 1.x.x Kafka seems to have moved away from using the problematic net.jpountz.lz4 library. Therefore, using latest Kafka (1.x) with latest Spark (2.3.x) should not have this issue.

这篇关于SBT测试错误:java.lang.NoSuchMethodError:net.jpountz.lz4.LZ4BlockInputStream的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！