unsupportedOperationException使用Joda时间将字符串转换为DateTime时出错 [英] unsupportedOperationException Error converting string to DateTime using Joda time

问题描述

我正在使用joda.time.Datetime库将字符串转换为datetime字段，但是会引发不受支持的异常 这是主类代码:

I am converting string to datetime field using joda.time.Datetime libraries but it throws unsupported exception Here is main class code:

//create new var with input data without header
var inputDataWithoutHeader: RDD[String] = dropHeader(inputFile)
var inputDF1 = inputDataWithoutHeader.map(_.split(",")).map{p =>
val dateYMD: DateTime = DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss").parseDateTime(p(8))
testData(dateYMD)}.toDF().show()

p(8)是在testData类中定义了数据类型为datetime的columnn，并且该列的CSV数据的值类似于2013-02-17 00:00:00

p(8) is columnn with datatype datetime defined in class testData and CSV data for the column has value like 2013-02-17 00:00:00

这是testData类:

Here is testData Class:

case class testData(StartDate: DateTime) { }

这是我得到的错误:

线程"main"中的异常

Exception in thread "main"

java.lang.UnsupportedOperationException: Schema for type org.joda.time.DateTime is not supported
    at org.apache.spark.sql.catalyst.ScalaReflection$class.schemaFor(ScalaReflection.scala:153)
    at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:29)
    at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:128)
    at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:126)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
    at scala.collection.AbstractTraversable.map(Traversable.scala:105)
    at org.apache.spark.sql.catalyst.ScalaReflection$class.schemaFor(ScalaReflection.scala:126)
    at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:29)
    at org.apache.spark.sql.catalyst.ScalaReflection$class.schemaFor(ScalaReflection.scala:64)
    at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:29)
    at org.apache.spark.sql.SQLContext.createDataFrame(SQLContext.scala:361)
    at org.apache.spark.sql.SQLImplicits.rddToDataFrameHolder(SQLImplicits.scala:47)
    at com.projs.poc.spark.ml.ProcessCSV$delayedInit$body.apply(ProcessCSV.scala:37)

推荐答案

1. 您可以在官方网站中阅读Spark SQL中的文档日期用java.sql.Timestamp表示.如果要使用Joda时间，则必须将输出转换为正确的类型

1. As you can read in the official documentation dates in Spark SQL are represented using java.sql.Timestamp. If you want to use Joda time you have to convert output to the correct type

SparkSQL可以使用类型转换轻松处理标准日期格式:

SparkSQL can easily handle standard date formats using type casting:

sc.parallelize(Seq(Tuple1("2016-01-11 00:01:02")))
  .toDF("dt")
  .select($"dt".cast("timestamp"))

这篇关于unsupportedOperationException使用Joda时间将字符串转换为DateTime时出错的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！