更改现有数据框的架构 [英] Change schema of existing dataframe

问题描述

我想更改现有数据框的架构，同时更改遇到错误的架构.是否可以更改数据框的现有架构.

I want to change schema of existing dataframe,while changing the schema I'm experiencing error.Is it possible I can change the existing schema of a dataframe.

val customSchema=StructType(
      Array(
        StructField("data_typ", StringType, nullable=false),
        StructField("data_typ", IntegerType, nullable=false),
        StructField("proc_date", IntegerType, nullable=false),
        StructField("cyc_dt", DateType, nullable=false),
        ));

val readDF=
+------------+--------------------+-----------+--------------------+
|DatatypeCode|         Description|monthColNam|     timeStampColNam|
+------------+--------------------+-----------+--------------------+
|       03099|Volumetric/Expand...|     201867|2018-05-31 18:25:...|
|       03307|  Elapsed Day Factor|     201867|2018-05-31 18:25:...|
+------------+--------------------+-----------+--------------------+

val rows= readDF.rdd
val readDF1 = sparkSession.createDataFrame(rows,customSchema)

预期结果

val newdf=
    +------------+--------------------+-----------+--------------------+
    |data_typ_cd |       data_typ_desc|proc_dt    |     cyc_dt         |
    +------------+--------------------+-----------+--------------------+
    |       03099|Volumetric/Expand...|     201867|2018-05-31 18:25:...|
    |       03307|  Elapsed Day Factor|     201867|2018-05-31 18:25:...|
    +------------+--------------------+-----------+--------------------+

将提供任何帮助

推荐答案

您不能像这样更改架构.传递给createDataFrame的模式对象必须与数据匹配，而不是相反:

You cannot change schema like this. Schema object passed to createDataFrame has to match the data, not the other way around:

• 要解析时间戳数据，请使用相应的功能，例如将字符串字段转换为Spark中时间戳的更好方法
• 要使用cast方法更改其他类型，例如如何在pyspark中将Dataframe列从String类型更改为Double类型

• To parse timestamp data use corresponding functions, for example like Better way to convert a string field into timestamp in Spark
• To change other types use cast method, for example how to change a Dataframe column from String type to Double type in pyspark

这篇关于更改现有数据框的架构的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！