Spark 2.0 Scala-RDD.toDF() [英] Spark 2.0 Scala - RDD.toDF()

问题描述

我正在使用Spark 2.0 Scala.我能够使用toDF()方法将RDD转换为DataFrame.

I am working with Spark 2.0 Scala. I am able to convert an RDD to a DataFrame using the toDF() method.

val rdd = sc.textFile("/pathtologfile/logfile.txt")
val df = rdd.toDF()

但是对于我来说，我无法在API文档中找到它的位置.它不在RDD下.但这位于DataSet(

But for the life of me I cannot find where this is in the API docs. It is not under RDD. But it is under DataSet (link 1). However I have an RDD not a DataSet.

我也无法在隐式下看到它(链接2 ).

Also I can't see it under implicits (link 2).

所以请帮助我理解为什么可以为我的RDD调用toDF().该方法从何处继承?

So please help me understand why toDF() can be called for my RDD. Where is this method being inherited from?

推荐答案

它来自这里:

说明:如果导入sqlContext.implicits._，则有一个隐式方法将RDD转换为DataSetHolder(rddToDataSetHolder)，然后在DataSetHolder

Explanation: if you import sqlContext.implicits._, you have a implicit method to convert RDD to DataSetHolder (rddToDataSetHolder), then you call toDF on the DataSetHolder

这篇关于Spark 2.0 Scala-RDD.toDF()的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！