由以下原因导致：java.time.format.DateTimeParseException：无法解析文本'2020-05-12 10:23:45'，在索引10处找到了未解析的文本 [英] Caused by: java.time.format.DateTimeParseException: Text '2020-05-12 10:23:45' could not be parsed, unparsed text found at index 10

查看：4804
本文介绍了由以下原因导致：java.time.format.DateTimeParseException：无法解析文本'2020-05-12 10:23:45'，在索引10处找到了未解析的文本的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我正在创建一个UDF，它将为我找到一周的第一天。

I am creating an UDF which will find the first day of the week for me.

该UDF的输入将是存储日期时间在<$ c中的数据帧中的字符串列$ c> yyyy-MM-dd hh：MM：ss 。

The inputs to the UDF will be a String Column from the Dataframe storing datetime in yyyy-MM-dd hh:MM:ss.

我同意可以在没有UDF的情况下建立相同的对象，但是我想探索一下，所有这样做的选择。到目前为止，我对通过UDF的实现感到困惑。

I agree that the same can be established without an UDF but I want to explore , all options of doing this. As of now , I am stuck with the implementation via UDF.

重要注意事项-周开始日是MONDAY。

Important Note - The Week Start Day is MONDAY.

代码-

import org.apache.spark.sql.functions._
import java.time.format.DateTimeFormatter
import java.time.LocalDate
import org.joda.time.DateTimeConstants

val df1 = Seq((1, "2020-05-12 10:23:45", 5000), (2, "2020-11-11 12:12:12", 2000)).toDF("id", "DateTime", "miliseconds")
val findFirstDayOfWeek = udf((x:String) => {
  
  val dateFormat = DateTimeFormatter.ofPattern("yyyy-MM-dd")
  val dayOfWeek = LocalDate.parse(x,dateFormat).getDayOfWeek
  
  if (dayOfWeek != DateTimeConstants.MONDAY )
    {
      val newDate = LocalDate.parse(x).plusDays(DateTimeConstants.MONDAY - dayOfWeek.getValue())
      val firstDateOfTheWeek = newDate.format(dateFormat)
      firstDateOfTheWeek
      
    }
  else
  {
    val newDate = x
    newDate.format(dateFormat)
    
  }
})

val udf_new_df1 = df1.withColumn("week",findFirstDayOfWeek(col("DateTime")))

但是当我运行 display（udf_new_df1）时，出现此错误-（在Databricks上）

But when i am running display(udf_new_df1), I am getting this error - ( On Databricks )

org.apache.spark.SparkException: Failed to execute user defined function($anonfun$1: (string) => string)
    at org.apache.spark.sql.catalyst.expressions.ScalaUDF.eval(ScalaUDF.scala:1066)
    at org.apache.spark.sql.catalyst.expressions.Alias.eval(namedExpressions.scala:152)
    at org.apache.spark.sql.catalyst.expressions.InterpretedMutableProjection.apply(InterpretedMutableProjection.scala:62)
    at org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation$$anonfun$apply$23$$anonfun$applyOrElse$23.apply(Optimizer.scala:1471)
    at org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation$$anonfun$apply$23$$anonfun$applyOrElse$23.apply(Optimizer.scala:1471)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
    at scala.collection.immutable.List.foreach(List.scala:392)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
    at scala.collection.immutable.List.map(List.scala:296)
    at org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation$$anonfun$apply$23.applyOrElse(Optimizer.scala:1471)
    at org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation$$anonfun$apply$23.applyOrElse(Optimizer.scala:1466)
    at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:280)
    at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:280)
    at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:77)
    at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:279)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDown(LogicalPlan.scala:29)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.transformDown(AnalysisHelper.scala:149)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
    at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:285)
    at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:285)
    at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$8.apply(TreeNode.scala:354)
    at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:208)
    at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:352)
    at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:285)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDown(LogicalPlan.scala:29)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.transformDown(AnalysisHelper.scala:149)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
    at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:285)
    at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:285)
    at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$8.apply(TreeNode.scala:354)
    at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:208)
    at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:352)
    at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:285)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDown(LogicalPlan.scala:29)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.transformDown(AnalysisHelper.scala:149)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
    at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:269)
    at org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation$.apply(Optimizer.scala:1466)
    at org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation$.apply(Optimizer.scala:1465)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:112)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:109)
    at scala.collection.IndexedSeqOptimized$class.foldl(IndexedSeqOptimized.scala:57)
    at scala.collection.IndexedSeqOptimized$class.foldLeft(IndexedSeqOptimized.scala:66)
    at scala.collection.mutable.WrappedArray.foldLeft(WrappedArray.scala:35)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:109)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:101)
    at scala.collection.immutable.List.foreach(List.scala:392)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:101)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$executeAndTrack$1.apply(RuleExecutor.scala:80)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$executeAndTrack$1.apply(RuleExecutor.scala:80)
    at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:88)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:79)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$optimizedPlan$1.apply(QueryExecution.scala:94)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$optimizedPlan$1.apply(QueryExecution.scala:94)
    at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
    at org.apache.spark.sql.execution.QueryExecution.optimizedPlan$lzycompute(QueryExecution.scala:93)
    at org.apache.spark.sql.execution.QueryExecution.optimizedPlan(QueryExecution.scala:93)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$toString$2.apply(QueryExecution.scala:263)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$toString$2.apply(QueryExecution.scala:263)
    at org.apache.spark.sql.execution.QueryExecution.stringOrError(QueryExecution.scala:147)
    at org.apache.spark.sql.execution.QueryExecution.toString(QueryExecution.scala:263)
    at org.apache.spark.sql.execution.SQLExecution$$anonfun$withCustomExecutionEnv$1.apply(SQLExecution.scala:102)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:240)
    at org.apache.spark.sql.execution.SQLExecution$.withCustomExecutionEnv(SQLExecution.scala:97)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:170)
    at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$withAction(Dataset.scala:3441)
    at org.apache.spark.sql.Dataset.collectResult(Dataset.scala:2832)
    at com.databricks.backend.daemon.driver.OutputAggregator$.withOutputAggregation0(OutputAggregator.scala:149)
    at com.databricks.backend.daemon.driver.OutputAggregator$.withOutputAggregation(OutputAggregator.scala:54)
    at com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$getResultBufferInternal$1$$anonfun$apply$1.apply(ScalaDriverLocal.scala:318)
    at com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$getResultBufferInternal$1$$anonfun$apply$1.apply(ScalaDriverLocal.scala:303)
    at scala.Option.map(Option.scala:146)
    at com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$getResultBufferInternal$1.apply(ScalaDriverLocal.scala:303)
    at com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$getResultBufferInternal$1.apply(ScalaDriverLocal.scala:267)
    at scala.Option.map(Option.scala:146)
    at com.databricks.backend.daemon.driver.ScalaDriverLocal.getResultBufferInternal(ScalaDriverLocal.scala:267)
    at com.databricks.backend.daemon.driver.DriverLocal.getResultBuffer(DriverLocal.scala:463)
    at com.databricks.backend.daemon.driver.ScalaDriverLocal.repl(ScalaDriverLocal.scala:244)
    at com.databricks.backend.daemon.driver.DriverLocal$$anonfun$execute$8.apply(DriverLocal.scala:373)
    at com.databricks.backend.daemon.driver.DriverLocal$$anonfun$execute$8.apply(DriverLocal.scala:350)
    at com.databricks.logging.UsageLogging$$anonfun$withAttributionContext$1.apply(UsageLogging.scala:238)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
    at com.databricks.logging.UsageLogging$class.withAttributionContext(UsageLogging.scala:233)
    at com.databricks.backend.daemon.driver.DriverLocal.withAttributionContext(DriverLocal.scala:48)
    at com.databricks.logging.UsageLogging$class.withAttributionTags(UsageLogging.scala:271)
    at com.databricks.backend.daemon.driver.DriverLocal.withAttributionTags(DriverLocal.scala:48)
    at com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLocal.scala:350)
    at com.databricks.backend.daemon.driver.DriverWrapper$$anonfun$tryExecutingCommand$2.apply(DriverWrapper.scala:644)
    at com.databricks.backend.daemon.driver.DriverWrapper$$anonfun$tryExecutingCommand$2.apply(DriverWrapper.scala:644)
    at scala.util.Try$.apply(Try.scala:192)
    at com.databricks.backend.daemon.driver.DriverWrapper.tryExecutingCommand(DriverWrapper.scala:639)
    at com.databricks.backend.daemon.driver.DriverWrapper.getCommandOutputAndError(DriverWrapper.scala:485)
    at com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:597)
    at com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:390)
    at com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:337)
    at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:219)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.time.format.DateTimeParseException: Text '2020-05-12 10:23:45' could not be parsed, unparsed text found at index 10
    at java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1952)
    at java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1851)
    at java.time.LocalDate.parse(LocalDate.java:400)
    at linedde9e8e2c7794f68a6e16898b7ed370036.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1.apply(command-14467074:14)
    at linedde9e8e2c7794f68a6e16898b7ed370036.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1.apply(command-14467074:11)
    at org.apache.spark.sql.catalyst.expressions.ScalaUDF$$anonfun$2.apply(ScalaUDF.scala:108)
    at org.apache.spark.sql.catalyst.expressions.ScalaUDF$$anonfun$2.apply(ScalaUDF.scala:107)
    at org.apache.spark.sql.catalyst.expressions.ScalaUDF.eval(ScalaUDF.scala:1063)
    ... 100 more

所以我的问题是为什么在解析日期时间类型为String且格式为yyyy-MM-dd hh：MM：ss的日期时为什么出现问题？

So My Question is Why am I Getting issues while parsing dateTime's of type String and format yyyy-MM-dd hh:MM:ss ?

推荐答案

不确定为什么要使用UDF，但是可以在一周的第一天不使用UDF，如下所示-

Not sure why do you wanted to use UDF, but you can get the first day of week without UDF as below-


一周从星期一

$ b $开始b

使用spark内置函数 date_trunc


Using spark built-in function date_trunc

 val df1 = Seq((1, "2020-05-12 10:23:45", 5000), (2, "2020-11-11 12:12:12", 2000)).toDF("id", "DateTime", "miliseconds")
    df1.withColumn("week", date_trunc("week", $"DateTime"))
      .show(false)

    /**
      * +---+-------------------+-----------+-------------------+
      * |id |DateTime           |miliseconds|week               |
      * +---+-------------------+-----------+-------------------+
      * |1  |2020-05-12 10:23:45|5000       |2020-05-11 00:00:00|
      * |2  |2020-11-11 12:12:12|2000       |2020-11-09 00:00:00|
      * +---+-------------------+-----------+-------------------+
      */


使用UDF 


  // convert dateTime -> date truncated to the first day of week
    val findFirstDayOfWeek = udf((x:String) => {

      val dateFormat = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss")
      val time = LocalDateTime.parse(x,dateFormat)
      val dayOfWeek = time.getDayOfWeek

      if (dayOfWeek.getValue != DateTimeConstants.MONDAY ) {
        val newDateTime = time.plusDays(DateTimeConstants.MONDAY - dayOfWeek.getValue())
        java.sql.Date.valueOf(newDateTime.toLocalDate)
      } else {
        java.sql.Date.valueOf(time.toLocalDate)
      }
    })

    val udf_new_df1 = df1.withColumn("week",findFirstDayOfWeek(col("DateTime")))
    udf_new_df1.show(false)
    udf_new_df1.printSchema()

    /**
      * +---+-------------------+-----------+----------+
      * |id |DateTime           |miliseconds|week      |
      * +---+-------------------+-----------+----------+
      * |1  |2020-05-12 10:23:45|5000       |2020-05-11|
      * |2  |2020-11-11 12:12:12|2000       |2020-11-09|
      * +---+-------------------+-----------+----------+
      *
      * root
      * |-- id: integer (nullable = false)
      * |-- DateTime: string (nullable = true)
      * |-- miliseconds: integer (nullable = false)
      * |-- week: date (nullable = true)
      */

这篇关于由以下原因导致：java.time.format.DateTimeParseException：无法解析文本'2020-05-12 10:23:45'，在索引10处找到了未解析的文本的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
其他开发最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆