如何将Spark中的Dataframe的两列合并为一个2元组? [英] How to merge two columns of a `Dataframe` in Spark into one 2-Tuple?

问题描述

我有一个带有五列的Spark DataFrame df.我想添加另一列，其值为第一和第二列的元组.与withColumn()方法一起使用时，出现不匹配错误，因为输入的不是列类型，而是(Column，Column). 我想知道在这种情况下，除了在行上运行for循环外是否还有解决方案?

I have a Spark DataFrame df with five columns. I want to add another column with its values being the tuple of the first and second columns. When using with withColumn() method, I get the mismatch error, because the input is not Column type, but instead (Column,Column). I wonder if there is a solution beside running for loop over the rows in this case?

var dfCol=(col1:Column,col2:Column)=>(col1,col2)
val vv = df.withColumn( "NewColumn", dfCol( df(df.schema.fieldNames(1)) , df(df.schema.fieldNames(2)) ) )

推荐答案

您可以使用用户定义的函数udf来实现所需的功能.

You can use a User-defined function udf to achieve what you want.

object TupleUDFs {
  import org.apache.spark.sql.functions.udf      
  // type tag is required, as we have a generic udf
  import scala.reflect.runtime.universe.{TypeTag, typeTag}

  def toTuple2[S: TypeTag, T: TypeTag] = 
    udf[(S, T), S, T]((x: S, y: T) => (x, y))
}

用法

df.withColumn(
  "tuple_col", TupleUDFs.toTuple2[Int, Int].apply(df("a"), df("b"))
)

假设"a"和"b"是要放入元组的类型为Int的列.

assuming "a" and "b" are the columns of type Int you want to put in a tuple.

这篇关于如何将Spark中的Dataframe的两列合并为一个2元组?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！