如何将 Spark 中“Dataframe"的两列合并为一个 2-Tuple? [英] How to merge two columns of a `Dataframe` in Spark into one 2-Tuple?

问题描述

我有一个包含五列的 Spark DataFrame df.我想添加另一列，其值是第一列和第二列的元组.使用 withColumn() 方法时，出现不匹配错误，因为输入不是列类型，而是 (Column,Column).在这种情况下，我想知道除了在行上运行 for 循环之外是否还有其他解决方案?

I have a Spark DataFrame df with five columns. I want to add another column with its values being the tuple of the first and second columns. When using with withColumn() method, I get the mismatch error, because the input is not Column type, but instead (Column,Column). I wonder if there is a solution beside running for loop over the rows in this case?

var dfCol=(col1:Column,col2:Column)=>(col1,col2)
val vv = df.withColumn( "NewColumn", dfCol( df(df.schema.fieldNames(1)) , df(df.schema.fieldNames(2)) ) )

推荐答案

您可以使用用户定义的函数 udf 来实现您想要的.

You can use a User-defined function udf to achieve what you want.

object TupleUDFs {
  import org.apache.spark.sql.functions.udf      
  // type tag is required, as we have a generic udf
  import scala.reflect.runtime.universe.{TypeTag, typeTag}

  def toTuple2[S: TypeTag, T: TypeTag] = 
    udf[(S, T), S, T]((x: S, y: T) => (x, y))
}

用法

df.withColumn(
  "tuple_col", TupleUDFs.toTuple2[Int, Int].apply(df("a"), df("b"))
)

假设a"和b"是您想要放入元组的 Int 类型的列.

assuming "a" and "b" are the columns of type Int you want to put in a tuple.

这篇关于如何将 Spark 中“Dataframe"的两列合并为一个 2-Tuple?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！