会发生什么-NoSuchMethodError:com.datastax.driver.core.ResultSet.fetchMoreResults [英] What happens - NoSuchMethodError: com.datastax.driver.core.ResultSet.fetchMoreResults

问题描述

cassandra-connector-assembly-2.0.0.

与Scala 2.11.8，cassandra-driver-core-3.1.0

sc.cassandraTable("mykeyspace", "mytable").select("something").where("key=?", key).mapPartitions(par => {
    par.map({ row => (row.getString("something"), 1 ) })
})
.reduceByKey(_ + _).collect().foreach(println)

同一工作可以读取较少的海量数据

The same job works fine for reading less mass data

java.lang.NoSuchMethodError: com.datastax.driver.core.ResultSet.fetchMoreResults()Lshade/com/datastax/spark/connector/google/common/util/concurrent/ListenableFuture;
    at com.datastax.spark.connector.rdd.reader.PrefetchingResultSetIterator.maybePrefetch(PrefetchingResultSetIterator.scala:26)
    at com.datastax.spark.connector.rdd.reader.PrefetchingResultSetIterator.next(PrefetchingResultSetIterator.scala:39)
    at com.datastax.spark.connector.rdd.reader.PrefetchingResultSetIterator.next(PrefetchingResultSetIterator.scala:17)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
    at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
    at com.datastax.spark.connector.util.CountingIterator.next(CountingIterator.scala:16)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
    at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:194)
    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
    at org.apache.spark.scheduler.Task.run(Task.scala:85)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

任何人都可以建议或指出该问题以及可能的解决方案吗?

Can any one suggest or point out to the issue, and a possible solution?

推荐答案

与Cassandra驱动程序核心

It is a conflict with the Cassandra driver-core that

libraryDependencies += "com.datastax.spark" % "spark-cassandra-connector_2.11" % "2.0.0-M3"

带来.

如果您进入〜/.ivy2/cache/com.datastax.spark/spark-cassandra-connector_2.11，您将找到一个名为ivy-2.0.0-M3.xml的文件

If you go into the ~/.ivy2/cache/com.datastax.spark/spark-cassandra-connector_2.11 you will find a file called ivy-2.0.0-M3.xml

在该文件中，依赖性为

com.datastax.cassandra" name="cassandra-driver-core" rev="3.0.2" force="true" 

请注意，这是Cassandra驱动程序内核的3.0.2版本，被最新版本的驱动程序所取代.

Note that it is the 3.0.2 version of Cassandra driver core which gets overrun by the more recent one.

碰巧

It just so happens that the latest source on Github does not show a implementation for fetchMoreResults which is inherited from interface PagingIterable

如果您在Github上将Git版本回滚到3.0.x，则会找到

If you roll back the Git version to 3.0.x on Github, you'll find

 public ListenableFuture<ResultSet> fetchMoreResults();

因此，最新的Cassandra核心驱动程序似乎不完整地冲出了大门.否则我可能会丢失一些东西.希望这会有所帮助.

So it looks like the newest Cassandra core drivers were rushed out the door incomplete. Or I might be missing something. Hope this helps.

tl; dr;删除最新的驱动程序，并使用spark cassandra连接器中嵌入的驱动程序.

tl;dr; Remove the latest driver and use the one embedded in the spark cassandra connector.

这篇关于会发生什么-NoSuchMethodError:com.datastax.driver.core.ResultSet.fetchMoreResults的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！