spark UDF不接受数组 [英] spark UDF don't accept Array

问题描述

为什么Spark UDF函数不支持scala数组，而使用WrappedArray UDF可以正常工作?在函数定义中，Array [Date]给出了Classcast异常.一旦更改为WrappedArray，它的效果就很好.

Why do Spark UDF functions not support scala arrays, whereas with WrappedArray UDF works fine ? Here in function definition Array[Date] gives Classcast exception. Once changed to WrappedArray its works fine.

def getDate(listOfDate:mutable.WrappedArray[Date], packageSD:Date, durationrange:Int):Date = {
  var nextdate = packageSD.toLocalDate.plusDays(durationrange)
  var billdate:Date = null
  var mindays = durationrange
  var billingdate = listOfDate.map(rec=>
    {
      println("list date"+rec)
      var recdate = rec
      var daysDiff = Math.abs(ChronoUnit.DAYS.between(recdate.toLocalDate,nextdate)).toInt
      if(daysDiff<=mindays) {
        mindays = daysDiff
        billdate = recdate
      }
      println("prefst"+recdate)
      println("nextdate"+nextdate)
      println("billdate"+billdate)
      println("mindays"+mindays)
    }
  )
      return billdate
}

import org.apache.spark.sql.functions.udf
val udffn = udf(getDate _)

推荐答案

UDF需要 Seq ，而 WrappedArray 是 Seq ，而 Array 不是 Seq

UDFs need a Seq and WrappedArray is a Seq, while Array is not Seq

这篇关于spark UDF不接受数组的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！