值联接不是org.apache.spark.rdd.RDD [(Long，T)]的成员 [英] Value join is not a member of org.apache.spark.rdd.RDD[(Long, T)]

问题描述

此功能似乎对我的IDE有效:

This function seems valid for my IDE:

def zip[T, U](rdd1:RDD[T], rdd2:RDD[U]) : RDD[(T,U)] = {
    rdd1
      .zipWithIndex
      .map(_.swap)
      .join(
        rdd2
          .zipWithIndex
          .map(_.swap))
      .values
}

但是当我编译时，我得到了:

But when I compile, I get :

值联接不是的成员 org.apache.spark.rdd.RDD [(Long，T)]可能的原因:可能是分号 价值加入"之前缺少? .join(

value join is not a member of org.apache.spark.rdd.RDD[(Long, T)] possible cause: maybe a semicolon is missing before `value join'? .join(

我在Spark 1.6中，我已经尝试导入 org.apache.spark.rdd.RDD ._ 并且在函数定义之外的两个RDD中直接使用该函数时，该函数内部的代码可以很好地工作.

I am in Spark 1.6, I have already tried to import org.apache.spark.rdd.RDD._ and the code inside the function works well when it is directly used on two RDDs outside of a function definition.

有什么主意吗?

推荐答案

如果您更改签名:

def zip[T, U](rdd1:RDD[T], rdd2:RDD[U]) : RDD[(T,U)] = {

进入:

def zip[T : ClassTag, U: ClassTag](rdd1:RDD[T], rdd2:RDD[U]) : RDD[(T,U)] = {

这将编译.

为什么? join是PairRDDFunctions的方法(您的RDD被隐式转换为该类)，具有以下签名:

Why? join is a method of PairRDDFunctions (your RDD is implicitly converted into that class), which has the following signature:

class PairRDDFunctions[K, V](self: RDD[(K, V)])
  (implicit kt: ClassTag[K], vt: ClassTag[V], ord: Ordering[K] = null)

这意味着其构造函数期望类型ClassTag[T]和ClassTag[U]的隐式值，因为它们将用作值类型(PairRDDFunctions定义中的V).您的方法不知道什么是T和U，因此无法提供匹配的隐式值.这意味着隐式转换为PairRDDFunctions失败"(编译器不执行转换)，因此找不到方法join.

This means its constructor expects implicit values of types ClassTag[T] and ClassTag[U], as these will be used as the value types (the V in the PairRDDFunctions definition). Your method has no knowledge of what T and U are, and therefore cannot provide matching implicit values. This means the implicit conversion into PairRDDFunctions "fails" (compiler doesn't perform the conversion) and therefore the method join can't be found.

添加[K : ClassTag]是向该方法添加隐式参数implicit kt: ClassTag[K]的简写，然后由编译器使用并传递给PairRDDFunctions的构造函数.

Adding [K : ClassTag] is shorthand for adding an implicit argument implicit kt: ClassTag[K] to the method, which is then used by compiler and passed to the constructor of PairRDDFunctions.

有关ClassTag及其优势的更多信息，请参见

For more about ClassTags and what they're good for see this good article.

这篇关于值联接不是org.apache.spark.rdd.RDD [(Long，T)]的成员的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！