鲜明打电话映射到一起，抛出NPE火花库 [英] call of distinct and map together throws NPE in spark library

问题描述

我不能确定这是否是一个错误，所以如果你做这样的事情。

I am unsure if this is a bug, so if you do something like this

// d:spark.RDD[String]
d.distinct().map(x => d.filter(_.equals(x)))

你会得到一个Java NPE。但是，如果你做了收集后，立即不同的，一切都会好起来的。

you will get a Java NPE. However if you do a collect immediately after distinct, all will be fine.

我使用火花 0.6.1。

推荐答案

星火不支持引用其他RDDS，因此NullPointerException异常嵌套RDDS或用户定义的函数;看到此线程上的火花用户邮件列表。

Spark does not support nested RDDs or user-defined functions that refer to other RDDs, hence the NullPointerException; see this thread on the spark-users mailing list.

它看起来像你的当前code试图组 D 按值的元素;你可以用 GROUPBY（）的 RDD 方式：

It looks like your current code is trying to group the elements of d by value; you can do this efficiently with the groupBy() RDD method:

scala> val d = sc.parallelize(Seq("Hello", "World", "Hello"))
d: spark.RDD[java.lang.String] = spark.ParallelCollection@55c0c66a

scala> d.groupBy(x => x).collect()
res6: Array[(java.lang.String, Seq[java.lang.String])] = Array((World,ArrayBuffer(World)), (Hello,ArrayBuffer(Hello, Hello)))

这篇关于鲜明打电话映射到一起，抛出NPE火花库的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！