如何打印RDD的内容? [英] How to print the contents of RDD?

问题描述

我正在尝试将集合的内容打印到 Spark 控制台.

I'm attempting to print the contents of a collection to the Spark console.

我有一个类型:

linesWithSessionId: org.apache.spark.rdd.RDD[String] = FilteredRDD[3]

我使用命令:

scala> linesWithSessionId.map(line => println(line))

但这是打印出来的:

res1: org.apache.spark.rdd.RDD[Unit] = MappedRDD[4] at map at :19

res1: org.apache.spark.rdd.RDD[Unit] = MappedRDD[4] at map at :19

如何将 RDD 写入控制台或将其保存到磁盘以便查看其内容?

How can I write the RDD to console or save it to disk so I can view its contents?

推荐答案

如果你想查看一个RDD的内容，一种方法是使用collect():

If you want to view the content of a RDD, one way is to use collect():

myRDD.collect().foreach(println)

不过，当 RDD 有数十亿行时，这不是一个好主意.使用 take() 取几个打印出来:

That's not a good idea, though, when the RDD has billions of lines. Use take() to take just a few to print out:

myRDD.take(n).foreach(println)

这篇关于如何打印RDD的内容?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！