unsupportedOperationException 使用 Joda 时间将字符串转换为 DateTime 时出错 [英] unsupportedOperationException Error converting string to DateTime using Joda time

问题描述

我正在使用 joda.time.Datetime 库将字符串转换为日期时间字段，但它引发了不受支持的异常这是主类代码:

I am converting string to datetime field using joda.time.Datetime libraries but it throws unsupported exception Here is main class code:

//create new var with input data without header
var inputDataWithoutHeader: RDD[String] = dropHeader(inputFile)
var inputDF1 = inputDataWithoutHeader.map(_.split(",")).map{p =>
val dateYMD: DateTime = DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss").parseDateTime(p(8))
testData(dateYMD)}.toDF().show()

p(8) 是具有在类 testData 中定义的数据类型日期时间的 columnn，该列的 CSV 数据具有类似 2013-02-17 00:00:00 的值

p(8) is columnn with datatype datetime defined in class testData and CSV data for the column has value like 2013-02-17 00:00:00

这里是 testData 类:

Here is testData Class:

case class testData(StartDate: DateTime) { }

这是我得到的错误:

线程main"中的异常

Exception in thread "main"

java.lang.UnsupportedOperationException: Schema for type org.joda.time.DateTime is not supported
    at org.apache.spark.sql.catalyst.ScalaReflection$class.schemaFor(ScalaReflection.scala:153)
    at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:29)
    at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:128)
    at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:126)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
    at scala.collection.AbstractTraversable.map(Traversable.scala:105)
    at org.apache.spark.sql.catalyst.ScalaReflection$class.schemaFor(ScalaReflection.scala:126)
    at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:29)
    at org.apache.spark.sql.catalyst.ScalaReflection$class.schemaFor(ScalaReflection.scala:64)
    at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:29)
    at org.apache.spark.sql.SQLContext.createDataFrame(SQLContext.scala:361)
    at org.apache.spark.sql.SQLImplicits.rddToDataFrameHolder(SQLImplicits.scala:47)
    at com.projs.poc.spark.ml.ProcessCSV$delayedInit$body.apply(ProcessCSV.scala:37)

推荐答案

1. 正如您在 官方Spark SQL 中的文档日期使用 java.sql.Timestamp 表示.如果您想使用 Joda 时间，您必须将输出转换为正确的类型

1. As you can read in the official documentation dates in Spark SQL are represented using java.sql.Timestamp. If you want to use Joda time you have to convert output to the correct type

SparkSQL 可以使用类型转换轻松处理标准日期格式:

SparkSQL can easily handle standard date formats using type casting:

sc.parallelize(Seq(Tuple1("2016-01-11 00:01:02")))
  .toDF("dt")
  .select($"dt".cast("timestamp"))

这篇关于unsupportedOperationException 使用 Joda 时间将字符串转换为 DateTime 时出错的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！