在 Spark 中将日期转换为月末 [英] Convert date to end of month in Spark
本文介绍了在 Spark 中将日期转换为月末的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我有一个 Spark DataFrame,如下所示:
I have a Spark DataFrame as shown below:
#Create DataFrame
df <- data.frame(name = c("Thomas", "William", "Bill", "John"),
dates = c('2017-01-05', '2017-02-23', '2017-03-16', '2017-04-08'))
df <- createDataFrame(df)
#Make sure df$dates column is in 'date' format
df <- withColumn(df, 'dates', cast(df$dates, 'date'))
name | dates
--------------------
Thomas |2017-01-05
William |2017-02-23
Bill |2017-03-16
John |2017-04-08
我想将 dates
更改为月末日期,因此它们如下所示.我该怎么做呢?SparkR 或 PySpark 代码都可以.
I want to change dates
to the end of month date, so they would look like shown below. How do I do this? Either SparkR or PySpark code is fine.
name | dates
--------------------
Thomas |2017-01-31
William |2017-02-28
Bill |2017-03-31
John |2017-04-30
推荐答案
您可以使用以下(PySpark):
You may use the following (PySpark):
from pyspark.sql.functions import last_day
df.select('name', last_day(df.dates).alias('dates')).show()
澄清一下,last_day(date)
返回日期所属月份的最后一天.
To clarify, last_day(date)
returns the last day of the month of which date belongs to.
我很确定 sparkR 中也有类似的功能https://spark.apache.org/docs/1.6.2/api/R/last_day.html
I'm pretty sure there is a similar function in sparkR https://spark.apache.org/docs/1.6.2/api/R/last_day.html
这篇关于在 Spark 中将日期转换为月末的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文