在csv文件中触发处理json数据 [英] spark process json data inside a csv file

问题描述

如何处理csv文件中的json数据，我尝试使用from_json，但与此同时，我需要指定架构，因为架构会不断变化.

How to process json data inside a csv file I am trying to use from_json but with that I need to specify my schema where as my schema keeps varying.

示例输入:-

userid   type          data
26594    p.v    {}                                                                                                                                                                                                                                                                                                                                                             
26594    s.s    {"sIds":["1173","1342","12345"]}

26594    s.r    {"bp":"sw"}                                                                                                                                                                                                                                                                                                                                
26594    p.v      {}                                                                                                                                                                                                                                                                                                                                                             
26594    s.r     {"c":"tracking","action":"n","label":"ank"}                                                                                                                                                                                                                                                                            
26593    p.v     {}                                                                                                                                                                                                                                                                                                                                                             
26594    p.sr     {"pId":"11234","pName":"sahkas","s":"n","is":"F","totalCount":0,"scount":0}  

我正在寻求将其转换为可查询json的数据框.

I am looking to convert this into a dataframe using which we can query the json.

寻找类似的输出:-

 userid    type    data_sids    data_bp    data_c    data_action    data_label
 26594     p.v      null         null       null     null    null
 26594     s.s      1173         null       null     null    null
 26594     s.s      1173         null       null     null    null  
 26594     s.s      1342         null       null     null    null
 26594     s.s      12345         null       null     null    null  
 26594      s.r     null          sw          null    null     null

这可行吗?

您能帮我吗?

谢谢

安库什·雷迪(Ankush Reddy).

Ankush Reddy.

推荐答案

我的建议是使用RDD来完成此任务.写这样的东西:

My advice is working with RDDs for this task. Write something like this:

rdd = # collection of Rows with the following fields: userid, type, data - your CSV
def flatten_json(userid, type, data_json):
    final_row = {"userid": userid, "type": type, "data_sids": data_json["sids"], ...}
    return Row(**final_row)
rdd = rdd.map(lambda row: flatten_json(row["userid"], row["type"], row["data"]))
df = spark.createDataFrame(rdd)

就是这样:)

这篇关于在csv文件中触发处理json数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！