更改现有数据框的架构 [英] Change schema of existing dataframe
本文介绍了更改现有数据框的架构的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我想更改现有数据框的架构,同时更改遇到错误的架构.是否可以更改数据框的现有架构.
I want to change schema of existing dataframe,while changing the schema I'm experiencing error.Is it possible I can change the existing schema of a dataframe.
val customSchema=StructType(
Array(
StructField("data_typ", StringType, nullable=false),
StructField("data_typ", IntegerType, nullable=false),
StructField("proc_date", IntegerType, nullable=false),
StructField("cyc_dt", DateType, nullable=false),
));
val readDF=
+------------+--------------------+-----------+--------------------+
|DatatypeCode| Description|monthColNam| timeStampColNam|
+------------+--------------------+-----------+--------------------+
| 03099|Volumetric/Expand...| 201867|2018-05-31 18:25:...|
| 03307| Elapsed Day Factor| 201867|2018-05-31 18:25:...|
+------------+--------------------+-----------+--------------------+
val rows= readDF.rdd
val readDF1 = sparkSession.createDataFrame(rows,customSchema)
预期结果
val newdf=
+------------+--------------------+-----------+--------------------+
|data_typ_cd | data_typ_desc|proc_dt | cyc_dt |
+------------+--------------------+-----------+--------------------+
| 03099|Volumetric/Expand...| 201867|2018-05-31 18:25:...|
| 03307| Elapsed Day Factor| 201867|2018-05-31 18:25:...|
+------------+--------------------+-----------+--------------------+
将提供任何帮助
推荐答案
您不能像这样更改架构.传递给createDataFrame
的模式对象必须与数据匹配,而不是相反:
You cannot change schema like this. Schema object passed to createDataFrame
has to match the data, not the other way around:
- 要解析时间戳数据,请使用相应的功能,例如将字符串字段转换为Spark中时间戳的更好方法
- 要使用
cast
方法更改其他类型,例如如何在pyspark中将Dataframe列从String类型更改为Double类型
- To parse timestamp data use corresponding functions, for example like Better way to convert a string field into timestamp in Spark
- To change other types use
cast
method, for example how to change a Dataframe column from String type to Double type in pyspark
这篇关于更改现有数据框的架构的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文