创建具有空/空字段值的新数据框 [英] Create new Dataframe with empty/null field values
本文介绍了创建具有空/空字段值的新数据框的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我正在从现有数据帧创建一个新数据帧,但需要在此新 DF 中添加新列(下面代码中的field1").我该怎么做?工作示例代码示例将不胜感激.
I am creating a new Dataframe from an existing dataframe, but need to add new column ("field1" in below code) in this new DF. How do I do so? Working sample code example will be appreciated.
val edwDf = omniDataFrame
.withColumn("field1", callUDF((value: String) => None))
.withColumn("field2",
callUdf("devicetypeUDF", (omniDataFrame.col("some_field_in_old_df"))))
edwDf
.select("field1", "field2")
.save("odsoutdatafldr", "com.databricks.spark.csv");
推荐答案
可以使用lit(null)
:
import org.apache.spark.sql.functions.{lit, udf}
case class Record(foo: Int, bar: String)
val df = Seq(Record(1, "foo"), Record(2, "bar")).toDF
val dfWithFoobar = df.withColumn("foobar", lit(null: String))
这里的一个问题是列类型是null
:
One problem here is that the column type is null
:
scala> dfWithFoobar.printSchema
root
|-- foo: integer (nullable = false)
|-- bar: string (nullable = true)
|-- foobar: null (nullable = true)
并且它不会被 csv
编写器保留.如果这是一个硬性要求,您可以使用 DataType
and it is not retained by the csv
writer. If it is a hard requirement you can cast column to the specific type (lets say String), with either DataType
import org.apache.spark.sql.types.StringType
df.withColumn("foobar", lit(null).cast(StringType))
或字符串描述
df.withColumn("foobar", lit(null).cast("string"))
或者像这样使用 UDF:
or use an UDF like this:
val getNull = udf(() => None: Option[String]) // Or some other type
df.withColumn("foobar", getNull()).printSchema
root
|-- foo: integer (nullable = false)
|-- bar: string (nullable = true)
|-- foobar: string (nullable = true)
可在此处找到等效的 Python:添加空列以触发 DataFrame
A Python equivalent can be found here: Add an empty column to spark DataFrame
这篇关于创建具有空/空字段值的新数据框的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文