如何将Spark数据帧中的WrappedArray列转换为Strings? [英] How do I convert a WrappedArray column in spark dataframe to Strings?

问题描述

我正在尝试将包含Array [String]的列转换为String，但始终出现此错误

I am trying to convert a column which contains Array[String] to String, but I consistently get this error

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 78.0 failed 4 times, most recent failure: Lost task 0.3 in stage 78.0 (TID 1691, ip-******): java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to [Ljava.lang.String; 

这是代码段

val mkString = udf((arrayCol:Array[String])=>arrayCol.mkString(","))  
val dfWithString=df.select($"arrayCol").withColumn("arrayString",
      mkString($"arrayCol"))  

推荐答案

WrappedArray不是Array(这是普通的旧Java Array而不是natve Scala集合).您可以将签名更改为:

WrappedArray is not an Array (which is plain old Java Array not a natve Scala collection). You can either change signature to:

import scala.collection.mutable.WrappedArray

(arrayCol: WrappedArray[String]) => arrayCol.mkString(",")

或使用像Seq这样的超类型之一:

or use one of the supertypes like Seq:

(arrayCol: Seq[String]) => arrayCol.mkString(",")

在最新的Spark版本中，您可以改用concat_ws:

In the recent Spark versions you can use concat_ws instead:

import org.apache.spark.sql.functions.concat_ws

df.select(concat_ws(",", $"arrayCol"))

这篇关于如何将Spark数据帧中的WrappedArray列转换为Strings?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！