Spark UDF错误-不支持类型为Any的架构 [英] Spark UDF error - Schema for type Any is not supported

问题描述

我正在尝试创建一个udf，该列将用0替换一列中的负值.

I’m trying to create a udf that will replace negative values in a column with 0.

我的数据框是–称为df，并且包含一列称为avg_x. 这是我创建udf的代码

My dataframe is – called df, and contains one column called avg_x. This is my code for creating a udf

val noNegative = udf {(avg_acc_x: Double) => if(avg_acc_x < 0) 0 else "avg_acc_x"}

我收到此错误

java.lang.UnsupportedOperationException: Schema for type Any is not supported

df.printSchema返回

df.printSchema returns

|-- avg_acc_x: double (nullable = false) 

所以我不明白为什么会发生此错误?

so I don’t understand why this error is occurring?

推荐答案

这是因为else返回String:"avg_acc_x".删除引号:

It's because of the else returning a String: "avg_acc_x". Take away the quotes:

val noNegative = udf {(avg_acc_x: Double) => if(avg_acc_x < 0) 0 else avg_acc_x}

这篇关于Spark UDF错误-不支持类型为Any的架构的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！