如何使用 Scala 将 csv 字符串解析为 Spark 数据帧? [英] How to parse a csv string into a Spark dataframe using scala?

问题描述

我想将包含字符串记录的 RDD 转换为 Spark 数据帧，如下所示.

I would like to convert a RDD containing records of strings, like below, to a Spark dataframe.

"Mike,2222-003330,NY,34"
"Kate,3333-544444,LA,32"
"Abby,4444-234324,MA,56"
....

模式行不在同一个 RDD 中，而是在另一个变量中:

The schema line is not inside the same RDD, but in a another variable:

val header = "name,account,state,age"

所以现在我的问题是，如何使用上述两个，在 Spark 中创建数据帧?我使用的是 Spark 2.2 版.

So now my question is, how do I use the above two, to create a dataframe in Spark? I am using Spark version 2.2.

我搜索了一下，看到一个帖子:我可以吗使用 spark-csv 将表示为字符串的 CSV 读入 Apache Spark.但是，这并不完全是我所需要的，我无法想出一种方法来修改这段代码以在我的情况下工作.

I did search and saw a post: Can I read a CSV represented as a string into Apache Spark using spark-csv . However it's not exactly what I need and I can't figure out a way to modify this piece of code to work in my case.

非常感谢您的帮助.

推荐答案

更简单的方法可能是从 CSV 文件开始并直接将其作为数据帧读取(通过指定架构).你可以在这里看到一个例子:在读取 csv 文件时提供模式作为一个数据框.

The easier way would probably be to start from the CSV file and read it directly as a dataframe (by specifying the schema). You can see an example here: Provide schema while reading csv file as a dataframe.

当数据已经存在于 RDD 中时，您可以使用 toDF() 转换为数据帧.此函数还接受列名作为输入.要使用此功能，首先使用 SparkSession 对象导入 spark 隐式:

When the data already exists in an RDD you can use toDF() to convert to a dataframe. This function also accepts column names as input. To use this functionality, first import the spark implicits using the SparkSession object:

val spark: SparkSession = SparkSession.builder.getOrCreate()
import spark.implicits._

由于 RDD 包含字符串，因此需要首先将其转换为表示数据帧中列的元组.在这种情况下，这将是一个 RDD[(String, String, String, Int)] 因为有四列(最后一个 age 列更改为 int 以说明怎么做).

Since the RDD contains strings it needs to first be converted to tuples representing the columns in the dataframe. In this case, this will be a RDD[(String, String, String, Int)] since there are four columns (the last age column is changed to int to illustrate how it can be done).

假设输入数据在rdd中:

val header = "name,account,state,age"

val df = rdd.map(row => row.split(","))
  .map{ case Array(name, account, state, age) => (name, account, state, age.toInt)}
  .toDF(header.split(","):_*)

结果数据框:

+----+-----------+-----+---+
|name|    account|state|age|
+----+-----------+-----+---+
|Mike|2222-003330|   NY| 34|
|Kate|3333-544444|   LA| 32|
|Abby|4444-234324|   MA| 56|
+----+-----------+-----+---+

这篇关于如何使用 Scala 将 csv 字符串解析为 Spark 数据帧?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！