spark-sql 是否支持输入数据中的多个分隔符? [英] Does spark-sql support multiple delimiters in the input data?

问题描述

我有一个带有多个单字符分隔符的输入数据，如下所示:

I have an input data with multiple single character delimiters as followed :

col1data1"col2data1;col3data1"col4data1
col1data2"col2data2;col3data2"col4data2
col1data3"col2data3;col3data3"col4data3

在上面的数据中 ["] ,[;] 是我的分隔符.

In the above data the ["] ,[;] are my delimiters.

sparkSQL 是否有任何方法可以将输入数据(位于文件中)直接转换为具有列名 col1、col2、col3、col4 的 表

Is there any way in sparkSQL to convert directly the input data( which is in a file) into a table with column names col1,col2,col3,col4

推荐答案

答案是否，spark-sql 不支持多分隔符，但一种方法是尝试读取它将文件转换为 RDD，然后使用常规拆分方法对其进行解析:

The answer is no, spark-sql does not support multi-delimiter but one way to do it is trying to read it your file into an RDD and than parse it using regular splitting methods :

val rdd : RDD[String] = ???
val s = rdd.first()
// res1: String = "This is one example. This is another"

假设您想在空间上分割并点断点.

Let's say that you want to split on space and point break.

所以我们可以考虑将我们的函数应用到我们的 s 值上，如下所示:

so we can consider apply our function on our s value as followed :

s.split(" |\\.")
// res2: Array[String] = Array(This, is, one, example, "", This, is, another)

现在我们可以在整个rdd上应用这个函数:

now we can apply the function on the whole rdd :

rdd.map(_.split(" |\\."))

数据示例:

scala> val s = "col1data1\"col2data1;col3data1\"col4data1"
scala> s.split(";|\"")
res4: Array[String] = Array(col1data1, col2data1, col3data1, col4data1)

更多关于字符串分割的内容:

More on string splitting :

• Scala 拆分字符串示例.
• 如何在 Scala 中拆分字符串但保留与正则表达式匹配的部分?

请记住，您可以应用于常规数据类型的所有内容都可以应用于整个 RDD，然后您要做的就是将 RDD 转换为 DataFrame.

Just remember that everything you can apply on a regular data type you can apply on a whole RDD, then all you have to do is converting your RDD to a DataFrame.

这篇关于spark-sql 是否支持输入数据中的多个分隔符?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！