值连接不是org.apache.spark.rdd.RDD成员 [英] value join is not a member of org.apache.spark.rdd.RDD

问题描述

我得到这个错误：

value join is not a member of 
    org.apache.spark.rdd.RDD[(Long, (Int, (Long, String, Array[_0])))
        forSome { type _0 <: (String, Double) }]

我发现的唯一建议是进口org.apache.spark.SparkContext ._ 我已经这样做。

我在做什么错了？

编辑：改变code，以消除 forSome （即，当对象的类型 org.apache.spark.rdd.RDD [（龙（智力，（长，字符串数组[（字符串，双人间）））））解决了这个问题。
这是星火的错误吗？

changing the code to eliminate forSome (i.e., when the object has type org.apache.spark.rdd.RDD[(Long, (Int, (Long, String, Array[(String, Double)])))) solved the problem. Is this a bug in Spark?

推荐答案

加入是成员 org.apache.spark.rdd.PairRDDFunctions 。那么，为什么隐类不会触发？

join is a member of org.apache.spark.rdd.PairRDDFunctions. So why does the implicit class not trigger?

scala> val s = Seq[(Long, (Int, (Long, String, Array[_0]))) forSome { type _0 <: (String, Double) }]()
scala> val r = sc.parallelize(s)
scala> r.join(r) // Gives your error message.
scala> val p = new org.apache.spark.rdd.PairRDDFunctions(r)
<console>:25: error: no type parameters for constructor PairRDDFunctions: (self: org.apache.spark.rdd.RDD[(K, V)])(implicit kt: scala.reflect.ClassTag[K], implicit vt: scala.reflect.ClassTag[V], implicit ord: Ordering[K])org.apache.spark.rdd.PairRDDFunctions[K,V] exist so that it can be applied to arguments (org.apache.spark.rdd.RDD[(Long, (Int, (Long, String, Array[_0]))) forSome { type _0 <: (String, Double) }])
 --- because ---
argument expression's type is not compatible with formal parameter type;
 found   : org.apache.spark.rdd.RDD[(Long, (Int, (Long, String, Array[_0]))) forSome { type _0 <: (String, Double) }]
 required: org.apache.spark.rdd.RDD[(?K, ?V)]
Note: (Long, (Int, (Long, String, Array[_0]))) forSome { type _0 <: (String, Double) } >: (?K, ?V), but class RDD is invariant in type T.
You may wish to define T as -T instead. (SLS 4.5)
       val p = new org.apache.spark.rdd.PairRDDFunctions(r)
               ^
<console>:25: error: type mismatch;
 found   : org.apache.spark.rdd.RDD[(Long, (Int, (Long, String, Array[_0]))) forSome { type _0 <: (String, Double) }]
 required: org.apache.spark.rdd.RDD[(K, V)]
       val p = new org.apache.spark.rdd.PairRDDFunctions(r)

我敢肯定，错误消息，大家都很清楚，其他人，只是为我自己慢的自我让我们试着理解它。 PairRDDFunctions 有两个类型参数， K 和 V 。你的 forSome 是整双，所以它不能被分割为独立的 K 和 V 类型。有没有 K 和 V 的 RDD [（K，V）] 就等于您的RDD类型。

I'm sure that error message is clear to everyone else, but just for my own slow self let's try to make sense of it. PairRDDFunctions has two type parameters, K and V. Your forSome is for the whole pair, so it cannot be split into separate K and V types. There are no K and V that RDD[(K, V)] would equal your RDD type.

不过，你可以有 forSome 仅适用于关键，而不是整个对。现在就加入的作品，因为这种类型可以分为 K 和 V 。

However, you could have the forSome only apply to the key, instead of the whole pair. Join works now, because this type can be separated into K and V.

scala> val s2 = Seq[(Long, (Int, (Long, String, Array[_0])) forSome { type _0 <: (String, Double) })]()
scala> val r2 = sc.parallelize(2s)
scala> r2.join(r2)
res0: org.apache.spark.rdd.RDD[(Long, ((Int, (Long, String, Array[_0])) forSome { type _0 <: (String, Double) }, (Int, (Long, String, Array[_0])) forSome { type _0 <: (String, Double) }))] = MapPartitionsRDD[5] at join at <console>:26

这篇关于值连接不是org.apache.spark.rdd.RDD成员的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！