找不到Apache Spark方法sun.nio.ch.DirectBuffer.cleaner()Lsun/misc/Cleaner; [英] Apache Spark method not found sun.nio.ch.DirectBuffer.cleaner()Lsun/misc/Cleaner;

查看:1073
本文介绍了找不到Apache Spark方法sun.nio.ch.DirectBuffer.cleaner()Lsun/misc/Cleaner;的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

在spark-shell中运行自动数据处理脚本时遇到此问题.前几次迭代可以正常工作,但总是迟早会遇到此错误.我用谷歌搜索了这个问题,但没有找到完全匹配的结果.其他类似问题不在Spark上下文范围内.我想这可能与JVM版本有关,但我无法弄清楚如何解决该问题.

I encounter this problem while running an automated data processing script in spark-shell. First couple of iterations work fine, but it always sooner or later bumps into this error. I googled this issue but haven't found an exact match. Other similar issues are outside of spark context. I guess it may have something to do with JVM version, but I cannot figure out how to solve the problem.

我在Spark独立集群中使用了2台机器.

I used 2 machines within a spark standalone cluster.

1号机器Java信息:

java 10.0.2 2018-07-17 Java(TM) SE Runtime Environment 18.3 (build 10.0.2+13) Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10.0.2+13, mixed mode)

java 10.0.2 2018-07-17 Java(TM) SE Runtime Environment 18.3 (build 10.0.2+13) Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10.0.2+13, mixed mode)

2号机器Java信息:

openjdk 10.0.2 2018-07-17 OpenJDK Runtime Environment (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.4) OpenJDK 64-Bit Server VM (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.4, mixed mode)

openjdk 10.0.2 2018-07-17 OpenJDK Runtime Environment (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.4) OpenJDK 64-Bit Server VM (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.4, mixed mode)

错误信息: WARN TaskSetManager:66 - Lost task 3.0 in stage 28.0 (TID 1368, 169.254.115.145, executor 1): java.lang.NoSuchMethodError: sun.nio.ch.DirectBuffer.cleaner()Lsun/misc/Cleaner; at org.apache.spark.storage.StorageUtils$.cleanDirectBuffer(StorageUtils.scala:212) at org.apache.spark.storage.StorageUtils$.dispose(StorageUtils.scala:207) at org.apache.spark.storage.StorageUtils.dispose(StorageUtils.scala) at org.apache.spark.io.NioBufferedFileInputStream.close(NioBufferedFileInputStream.java:130) at java.base/java.io.FilterInputStream.close(FilterInputStream.java:180) at org.apache.spark.io.ReadAheadInputStream.close(ReadAheadInputStream.java:400) at org.apache.spark.util.collection.unsafe.sort.UnsafeSorterSpillReader.close(UnsafeSorterSpillReader.java:152) at org.apache.spark.util.collection.unsafe.sort.UnsafeSorterSpillReader.loadNext(UnsafeSorterSpillReader.java:124) at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter$SpillableIterator.loadNext(UnsafeExternalSorter.java:590) at org.apache.spark.sql.execution.UnsafeKVExternalSorter$KVSorterIterator.next(UnsafeKVExternalSorter.java:287) at org.apache.spark.sql.execution.aggregate.SortBasedAggregator$$anon$1.findNextSortedGroup(ObjectAggregationIterator.scala:276) at org.apache.spark.sql.execution.aggregate.SortBasedAggregator$$anon$1.hasNext(ObjectAggregationIterator.scala:247) at org.apache.spark.sql.execution.aggregate.ObjectAggregationIterator.hasNext(ObjectAggregationIterator.scala:81) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:148) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55) at org.apache.spark.scheduler.Task.run(Task.scala:121) at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:844)

Error Information: WARN TaskSetManager:66 - Lost task 3.0 in stage 28.0 (TID 1368, 169.254.115.145, executor 1): java.lang.NoSuchMethodError: sun.nio.ch.DirectBuffer.cleaner()Lsun/misc/Cleaner; at org.apache.spark.storage.StorageUtils$.cleanDirectBuffer(StorageUtils.scala:212) at org.apache.spark.storage.StorageUtils$.dispose(StorageUtils.scala:207) at org.apache.spark.storage.StorageUtils.dispose(StorageUtils.scala) at org.apache.spark.io.NioBufferedFileInputStream.close(NioBufferedFileInputStream.java:130) at java.base/java.io.FilterInputStream.close(FilterInputStream.java:180) at org.apache.spark.io.ReadAheadInputStream.close(ReadAheadInputStream.java:400) at org.apache.spark.util.collection.unsafe.sort.UnsafeSorterSpillReader.close(UnsafeSorterSpillReader.java:152) at org.apache.spark.util.collection.unsafe.sort.UnsafeSorterSpillReader.loadNext(UnsafeSorterSpillReader.java:124) at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter$SpillableIterator.loadNext(UnsafeExternalSorter.java:590) at org.apache.spark.sql.execution.UnsafeKVExternalSorter$KVSorterIterator.next(UnsafeKVExternalSorter.java:287) at org.apache.spark.sql.execution.aggregate.SortBasedAggregator$$anon$1.findNextSortedGroup(ObjectAggregationIterator.scala:276) at org.apache.spark.sql.execution.aggregate.SortBasedAggregator$$anon$1.hasNext(ObjectAggregationIterator.scala:247) at org.apache.spark.sql.execution.aggregate.ObjectAggregationIterator.hasNext(ObjectAggregationIterator.scala:81) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:148) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55) at org.apache.spark.scheduler.Task.run(Task.scala:121) at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:844)

推荐答案

我设法通过将spark的JAVA_HOME设置为java8 JDK来解决了该问题.这是一个相当新的问题,但已被Spark的开发人员发现,请参见此处 https://github.com/apache/spark/pull/22993/files/7f58ae61262d7c2f2d70c24d051c63e8830d5062 .

I managed to solve the problem by setting JAVA_HOME for spark to a java8 JDK. It is a quite new issue but has been spotted by the developers of spark, see here https://github.com/apache/spark/pull/22993/files/7f58ae61262d7c2f2d70c24d051c63e8830d5062.

官方网站提供的最新预编译星火已于11月2日发布,此拉取请求随后发生.希望以后的发行版可以避免使用较新版本的Java出现此问题.

The latest pre-compiled spark provided by the official site was released on Nov 2 and this pull request happened later. Hopefully later release would avoid this issue with java of newer version.

这篇关于找不到Apache Spark方法sun.nio.ch.DirectBuffer.cleaner()Lsun/misc/Cleaner;的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆