使用UDF解析来自PySpark Dataframe的嵌套XML字段 [英] Parsing the nested XML fields from PySpark Dataframe using UDF
本文介绍了使用UDF解析来自PySpark Dataframe的嵌套XML字段的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我有一个方案,其中我将XML数据放在DataFrame列中。
性别 | 更新时间 | 访问者 |
---|---|---|
F | 1574264158 | <;?xml版本=&qot;1.0;编码=";utf-8 |
我想使用UDF将访问者列-嵌套的XML字段解析为Dataframe中的列
XML格式
<?xml version="1.0" encoding="utf-8"?> <visitors> <visitor id="9615" age="68" sex="F" /> <visitor id="1882" age="34" sex="M" /> <visitor id="5987" age="23" sex="M" /> </visitors>
推荐答案
无需使用UDF即可使用xpath
查询:
df = spark.createDataFrame([['<?xml version="1.0" encoding="utf-8"?> <visitors> <visitor id="9615" age="68" sex="F" /> <visitor id="1882" age="34" sex="M" /> <visitor id="5987" age="23" sex="M" /> </visitors>']], ['visitors'])
df.show(truncate=False)
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|visitors |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|<?xml version="1.0" encoding="utf-8"?> <visitors> <visitor id="9615" age="68" sex="F" /> <visitor id="1882" age="34" sex="M" /> <visitor id="5987" age="23" sex="M" /> </visitors>|
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
df2 = df.selectExpr(
"xpath(visitors, './visitors/visitor/@id') id",
"xpath(visitors, './visitors/visitor/@age') age",
"xpath(visitors, './visitors/visitor/@sex') sex"
).selectExpr(
"explode(arrays_zip(id, age, sex)) visitors"
).select('visitors.*')
df2.show(truncate=False)
+----+---+---+
|id |age|sex|
+----+---+---+
|9615|68 |F |
|1882|34 |M |
|5987|23 |M |
+----+---+---+
如果您坚持使用自定义项:
import xml.etree.ElementTree as ET
import pyspark.sql.functions as F
@F.udf('array<struct<id:string, age:string, sex:string>>')
def parse_xml(s):
root = ET.fromstring(s)
return list(map(lambda x: x.attrib, root.findall('visitor')))
df2 = df.select(
F.explode(parse_xml('visitors')).alias('visitors')
).select('visitors.*')
df2.show()
+----+---+---+
| id|age|sex|
+----+---+---+
|9615| 68| F|
|1882| 34| M|
|5987| 23| M|
+----+---+---+
这篇关于使用UDF解析来自PySpark Dataframe的嵌套XML字段的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文