可以处理Spark中的多字符定界符 [英] Possible to handle multi character delimiter in spark

问题描述

对于正在读取的某些csv文件，我以[~]作为分隔符.

I have [~] as my delimiter for some csv files I am reading.

1[~]a[~]b[~]dd[~][~]ww[~][~]4[~]4[~][~][~][~][~]

我已经尝试过

val rddFile = sc.textFile("file.csv")
val rddTransformed = rddFile.map(eachLine=>eachLine.split("[~]"))
val df = rddTransformed.toDF()
display(df)

但是，与此相关的问题是，它作为单个值数组出现，每个字段中都有[和].因此数组将是

However this issue with this, is that it comes as a single value array with [ and ] in each field. So the array would be

["1[","]a[","]b[",...]

我不能使用

val df = spark.read.option("sep", "[~]").csv("file.csv")

因为不支持多字符分隔符.我还能采取什么其他方法?

Because multi-character seperator is not supported. What other approach can I take?

1[~]a[~]b[~]dd[~][~]ww[~][~]4[~]4[~][~][~][~][~]
2[~]a[~]b[~]dd[~][~]ww[~][~]4[~]4[~][~][~][~][~]
3[~]a[~]b[~]dd[~][~]ww[~][~]4[~]4[~][~][~][~][~]

编辑-这不是重复项，重复的线程涉及多个定界符，这是多个字符单个定界符

Edit - this is not a duplicate, the duplicated thread is about multi delimiters, this is multi-character single delimiter

推荐答案

val df = spark.read.format("csv").load("inputpath")
df.rdd.map(i => i.mkString.split("\\[\\~\\]")).toDF().show(false)

尝试以下

您的另一项要求

val df1 = df.rdd.map(i => i.mkString.split("\\[\\~\\]").mkString(",")).toDF()
val iterationColumnLength = df1.rdd.first.mkString(",").split(",").length
df1.withColumn("value",split(col("value"),",")).select((0 until iterationColumnLength).map(i => col("value").getItem(i).as("col_" + i)): _*).show

这篇关于可以处理Spark中的多字符定界符的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！