如何将标题和列添加到数据框火花? [英] How to add header and column to dataframe spark?

问题描述

我有一个数据框，我想在其上添加标题和第一列手动.这是数据框:

I have got a dataframe, on which I want to add a header and a first column manually. Here is the dataframe :

import org.apache.spark.sql.SparkSession 

val spark = SparkSession.builder.master("local").appName("my-spark-app").getOrCreate()
val df = spark.read.option("header",true).option("inferSchema",true).csv("C:\\gg.csv").cache()

数据框的内容

12,13,14
11,10,5
3,2,45

预期输出是

define,col1,col2,col3
c1,12,13,14
c2,11,10,5
c3,3,2,45

推荐答案

这里是一个依赖于 Spark 2.4 的解决方案:

Here is a solution that depends on Spark 2.4:

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.types.{IntegerType, StringType, StructField, StructType}
import org.apache.spark.sql.Row

//First off the dataframe needs to be loaded with the expected schema

val spark = SparkSession.builder().appName().getOrCreate()

val schema = new StructType()
              .add("col1",IntegerType,true)
              .add("col2",IntegerType,true)
              .add("col3",IntegerType,true)

val df = spark.read.format("csv").schema(schema).load("C:\\gg.csv").cache()

val rddWithId = df.rdd.zipWithIndex

// Prepend "define" column of type Long
val newSchema = StructType(Array(StructField("define", StringType, false))         ++ df.schema.fields)

val dfZippedWithId =  spark.createDataFrame(rddWithId.map{ 
                       case (row, index) => 
                       Row.fromSeq(Array("c" + index) ++ row.toSeq)}, newSchema)
// Show results
dfZippedWithId.show

显示:

+------+----+----+----+
|define|col1|col2|col3|
+------+----+----+----+
|    c0|  12|  13|  14|
|    c1|  11|  10|   5|
|    c2|   3|   2|  45|
+------+----+----+----+

这是文档的混合此处 和这个示例.

This is a mix of the documentation here and this example.

这篇关于如何将标题和列添加到数据框火花?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！