在Spark中将日期转换为月末 [英] Convert date to end of month in Spark

问题描述

我有一个Spark DataFrame，如下所示:

I have a Spark DataFrame as shown below:

#Create DataFrame    
df <- data.frame(name = c("Thomas", "William", "Bill", "John"),
      dates = c('2017-01-05', '2017-02-23', '2017-03-16', '2017-04-08'))
df <- createDataFrame(df)

#Make sure df$dates column is in 'date' format    
df <- withColumn(df, 'dates', cast(df$dates, 'date'))

name    | dates
--------------------
Thomas  |2017-01-05
William |2017-02-23
Bill    |2017-03-16
John    |2017-04-08

我想将dates更改为月末日期，因此它们看起来如下所示.我该怎么做呢? SparkR或PySpark代码都可以.

I want to change dates to the end of month date, so they would look like shown below. How do I do this? Either SparkR or PySpark code is fine.

name    | dates
--------------------
Thomas  |2017-01-31
William |2017-02-28
Bill    |2017-03-31
John    |2017-04-30

推荐答案

您可以使用以下(PySpark):

You may use the following (PySpark):

from pyspark.sql.functions import last_day

df.select('name', last_day(df.dates).alias('dates')).show()

为明确起见，last_day(date)返回该日期所属月份的最后一天.

To clarify, last_day(date) returns the last day of the month of which date belongs to.

我很确定sparkR中有类似的功能 https://spark.apache.org/docs/1.6. 2/api/R/last_day.html

I'm pretty sure there is a similar function in sparkR https://spark.apache.org/docs/1.6.2/api/R/last_day.html

这篇关于在Spark中将日期转换为月末的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！