如何转换json数组< String>到spark sql中的csv [英] How to convert json array<String> to csv in spark sql

问题描述

我已尝试过此查询以获取linkedin数据所需的经验。

I have tried this query to get required experience from linkedin data.

 Dataset<Row> filteredData = spark
                    .sql("select full_name ,experience from (select *, explode(experience['title']) exp from tempTable )"
                            + "  a where lower(exp) like '%developer%'");

但我收到此错误：

最后我尝试但我得到了更多具有相同名称的行。

and finally I tried but I got more rows with the same name .

Dataset<Row> filteredData = spark
                    .sql("select full_name ,explode(experience) from (select *, explode(experience['title']) exp from tempTable )"
                            + "  a where lower(exp) like '%developer%'");

请给我提示，如何在同一列中将字符串数组转换为逗号分隔字符串。

Please give me hint, how to convert array of string to comma separated string in the same column.

推荐答案

您可以应用UDF制作逗号分隔字符串

You can apply UDF for making a comma separate string

创建像这样的UDF

def mkString(value: WrappedArray[String]): String = value.mkString(",")

在sparkSQL上下文中注册UDF

Register UDF in sparkSQL context

sqlContext.udf.register("mkstring", mkString _)

应用它on SparkSQL查询

Apply it on SparkSQL query

sqlContext.sql(select mkstring(columnName) from tableName)

它将返回逗号单独的数组值

it will return comma separate value of array

这篇关于如何转换json数组< String>到spark sql中的csv的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！