Spark日期解析 [英] Spark date parsing
本文介绍了Spark日期解析的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我正在以这种格式解析一些日期:2009-01-23 18:15:05
使用以下函数
I am parsing some dates in this format: 2009-01-23 18:15:05
using the following function
def loadTransactions (sqlContext: SQLContext, path: String): DataFrame = {
val rowRdd = sqlContext.sparkContext.textFile(path).map { line =>
val tokens = line.split(',')
val dt = new DateTime(tokens(0))
Row(new Timestamp(dt.getMillis))
}
val fields = Seq(
StructField("timestamp", TimestampType, true)
)
val schema = StructType(fields)
sqlContext.createDataFrame(rowRdd, schema)
}
Spark 抛出错误:
Spark is throwing an error:
java.lang.IllegalArgumentException: Invalid format: "2009-01-23 18:15:05" is malformed at " 18:15:05" at org.joda.time.format.DateTimeParserBucket.doParseMillis
我认为这是因为缺少毫秒
I presume that it is due to the fact that the milliseconds are missing
推荐答案
可以使用以下方法代替jodatime
Instead of using jodatime, you can use the following method
def loadTransactions (sqlContext: SQLContext, path: String): DataFrame = {
val rowRdd = sqlContext.sparkContext.textFile(path).map { line =>
val tokens = line.split(',')
Row(getTimestamp(tokens(0)))
}
val fields = Seq(
StructField("timestamp", TimestampType, true)
)
val schema = StructType(fields)
sqlContext.createDataFrame(rowRdd, schema)
}
使用以下函数转换为时间戳.
Use the following function to convert to timestamp.
def getTimestamp(x:String) :java.sql.Timestamp = {
val format = new SimpleDateFormat("yyyy-MM-dd hh:mm:ss")
if (x.toString() == "")
return null
else {
val d = format.parse(x.toString());
val t = new Timestamp(d.getTime());
return t
}
}
这篇关于Spark日期解析的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文