可以处理Spark中的多字符定界符 [英] Possible to handle multi character delimiter in spark
本文介绍了可以处理Spark中的多字符定界符的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
对于正在读取的某些csv文件,我以[~]
作为分隔符.
I have [~]
as my delimiter for some csv files I am reading.
1[~]a[~]b[~]dd[~][~]ww[~][~]4[~]4[~][~][~][~][~]
我已经尝试过
val rddFile = sc.textFile("file.csv")
val rddTransformed = rddFile.map(eachLine=>eachLine.split("[~]"))
val df = rddTransformed.toDF()
display(df)
但是,与此相关的问题是,它作为单个值数组出现,每个字段中都有[
和]
.因此数组将是
However this issue with this, is that it comes as a single value array with [
and ]
in each field. So the array would be
["1[","]a[","]b[",...]
我不能使用
val df = spark.read.option("sep", "[~]").csv("file.csv")
因为不支持多字符分隔符.我还能采取什么其他方法?
Because multi-character seperator is not supported. What other approach can I take?
1[~]a[~]b[~]dd[~][~]ww[~][~]4[~]4[~][~][~][~][~]
2[~]a[~]b[~]dd[~][~]ww[~][~]4[~]4[~][~][~][~][~]
3[~]a[~]b[~]dd[~][~]ww[~][~]4[~]4[~][~][~][~][~]
编辑-这不是重复项,重复的线程涉及多个定界符,这是多个字符单个定界符
Edit - this is not a duplicate, the duplicated thread is about multi delimiters, this is multi-character single delimiter
推荐答案
val df = spark.read.format("csv").load("inputpath")
df.rdd.map(i => i.mkString.split("\\[\\~\\]")).toDF().show(false)
尝试以下
您的另一项要求
val df1 = df.rdd.map(i => i.mkString.split("\\[\\~\\]").mkString(",")).toDF()
val iterationColumnLength = df1.rdd.first.mkString(",").split(",").length
df1.withColumn("value",split(col("value"),",")).select((0 until iterationColumnLength).map(i => col("value").getItem(i).as("col_" + i)): _*).show
这篇关于可以处理Spark中的多字符定界符的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文