将功能应用于Spark Dataframe列 [英] Applying function to Spark Dataframe Column

问题描述

来自R，我习惯于轻松地对列进行操作.有什么简单的方法可以使用我在Scala中编写的此功能

Coming from R, I am used to easily doing operations on columns. Is there any easy way to take this function that I've written in scala

def round_tenths_place( un_rounded:Double ) : Double = {
    val rounded = BigDecimal(un_rounded).setScale(1, BigDecimal.RoundingMode.HALF_UP).toDouble
    return rounded
}

并将其应用于数据框的一列-我希望这样做:

And apply it to a one column of a dataframe - kind of what I hoped this would do:

 bid_results.withColumn("bid_price_bucket", round_tenths_place(bid_results("bid_price")) )

我还没有找到任何简单的方法，并且正在努力寻找方法.有一种比将数据帧转换为RDD和RDD，然后从rdd行中进行选择以获取正确的字段并将函数映射到所有值更简单的方法，是吗?还有更简洁的方法是创建SQL表，然后使用sparkSQL UDF进行此操作?

I haven't found any easy way and am struggling to figure out how to do this. There's got to be an easier way than converting the dataframe to and RDD and then selecting from rdd of rows to get the right field and mapping the function across all of the values, yeah? And also something more succinct creating a SQL table and then doing this with a sparkSQL UDF?

推荐答案

您可以如下定义UDF:

You can define an UDF as follows:

val round_tenths_place_udf = udf(round_tenths_place _)
bid_results.withColumn(
  "bid_price_bucket", val round_tenths_place_udf($"bid_price"))

尽管内置

although built-in Round expression is using exactly the same logic as your function and should be more than enough, not to mention much more efficient:

import org.apache.spark.sql.functions.round

bid_results.withColumn("bid_price_bucket", round($"bid_price", 1))

另请参阅:

• 在Spark中更新数据框列
• 如何将函数应用于Spark DataFrame的列?

• Updating a dataframe column in spark
• How to apply a function to a column of a Spark DataFrame?

这篇关于将功能应用于Spark Dataframe列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！