如何将函数应用于 Spark DataFrame 的列? [英] How to apply a function to a column of a Spark DataFrame?

问题描述

假设我们有一个 Spark DataFrame

Let's assume that we have a Spark DataFrame

df.getClass
Class[_ <: org.apache.spark.sql.DataFrame] = class org.apache.spark.sql.DataFrame

具有以下架构

df.printSchema
root
|-- rawFV: string (nullable = true)
|-- tk: array (nullable = true)
|    |-- element: string (containsNull = true)

假设 tk 列的每一行都是一个字符串数组，那么如何编写一个 Scala 函数来返回每行中元素的数量?

Given that each row of the tk column is an array of strings, how to write a Scala function that will return the number of elements in each row?

推荐答案

您不必编写自定义函数，因为有一个:

You don't have to write a custom function because there is one:

import org.apache.spark.sql.functions.size

df.select(size($"tk"))

如果你真的想要你可以写一个udf:

If you really want you can write an udf:

import org.apache.spark.sql.functions.udf

val size_ = udf((xs: Seq[String]) => xs.size)

甚至创建自定义表达式，但这真的没有意义.

or even create custom a expression but there is really no point in that.

这篇关于如何将函数应用于 Spark DataFrame 的列?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！