java.lang.NoClassDefFoundError:无法初始化类org.xerial.snappy.Snappy [英] java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy

问题描述

我正在尝试从DSE 5.0.11运行spark-shell.我可以成功创建和RDD，但是尝试查询它会得到:

I am trying to run spark-shell from DSE 5.0.11. I can successfully create and RDD, but trying to query it yields:

<pre>
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
        at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1212)
        at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
        at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
        at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
        at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
        at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
        at org.apache.spark.scheduler.Task.run(Task.scala:89)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
        at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65)
        at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
        at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:167)
        at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1205)
        ... 11 more
Caused by: java.lang.IllegalArgumentException: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
        at org.apache.spark.io.SnappyCompressionCodec$.liftedTree1$1(CompressionCodec.scala:171)
        at org.apache.spark.io.SnappyCompressionCodec$.org$apache$spark$io$SnappyCompressionCodec$$version$lzycompute(CompressionCodec.scala:168)
        at org.apache.spark.io.SnappyCompressionCodec$.org$apache$spark$io$SnappyCompressionCodec$$version(CompressionCodec.scala:168)
        at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:152)
        ... 20 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
        at org.apache.spark.io.SnappyCompressionCodec$.liftedTree1$1(CompressionCodec.scala:169)
        ... 23 more
</pre>

我可以在类路径中看到snappy-java和snappy-0.2.我正在使用:dse spark --conf spark.cassandra.auth.username = --conf spark.cassandra.auth.password =

I can see snappy-java and snappy-0.2 in the classpath. I am running using: dse spark --conf spark.cassandra.auth.username= --conf spark.cassandra.auth.password=

什么可能导致此错误?

推荐答案

在我的情况下，此错误是由/tmp挂载noexec引起的.在我的spark executor选项中设置java.io.tmpdir或org.xerial.snappy.tempdir可以解决此问题.

In my case this error was caused by /tmp being mounted noexec. Setting either java.io.tmpdir or org.xerial.snappy.tempdir in my spark executor options fixed the problem.

这篇关于java.lang.NoClassDefFoundError:无法初始化类org.xerial.snappy.Snappy的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！