如何转换json数组<String>到spark sql中的csv [英] How to convert json array&lt;String&gt; to csv in spark sql

问题描述

我已尝试此查询以从linkedin 数据中获得所需的经验.

I have tried this query to get required experience from linkedin data.

 Dataset<Row> filteredData = spark
                    .sql("select full_name ,experience from (select *, explode(experience['title']) exp from tempTable )"
                            + "  a where lower(exp) like '%developer%'");

但是我收到了这个错误:

But I got this error:

最后我试过了，但我得到了更多同名的行.

and finally I tried but I got more rows with the same name .

Dataset<Row> filteredData = spark
                    .sql("select full_name ,explode(experience) from (select *, explode(experience['title']) exp from tempTable )"
                            + "  a where lower(exp) like '%developer%'");

请给我提示，如何将字符串数组转换为同一列中的逗号分隔字符串.

Please give me hint, how to convert array of string to comma separated string in the same column.

推荐答案

您可以应用 UDF 来制作逗号分隔的字符串

You can apply UDF for making a comma separate string

像这样创建UDF

def mkString(value: WrappedArray[String]): String = value.mkString(",")

在 sparkSQL 上下文中注册 UDF

Register UDF in sparkSQL context

sqlContext.udf.register("mkstring", mkString _)

将其应用于 SparkSQL 查询

Apply it on SparkSQL query

sqlContext.sql(select mkstring(columnName) from tableName)

它将返回数组的逗号分隔值

it will return comma separate value of array

这篇关于如何转换json数组<String>到spark sql中的csv的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！