嵌套json中的结构化流不同模式 [英] structured streaming different schema in nested json

问题描述

我有一个场景，传入的消息是一个 Json，它有一个标题说 tablename，数据部分有表列数据.现在我想把它写到 parquet 到单独的文件夹说 /emp 和 /dept.我可以通过基于表名聚合行在常规流中实现这一点.但是在结构化流媒体中，我无法拆分它.我如何才能在结构化流媒体中实现这一点.

Hi I have a scenario where the incoming message is a Json which has a header say tablename and the data part has the table column data. Now i want to write this to parquet to separate folders say /emp and /dept. I can achieve this in regular streaming by aggregating rows based on the tablname. But in structured streaming I am unable to split this. How can I achieve this in structured streaming.

{"tableName":"employee","data":{"empid":1","empname":"john","dept":"CS"}{"tableName":"employee","data":{"empid":2","empname":"james","dept":"CS"}{"tableName":"dept","data":{"dept":"1","deptname":"CS","desc":"计算机科学部"}

{"tableName":"employee","data":{"empid":1","empname":"john","dept":"CS"} {"tableName":"employee","data":{"empid":2","empname":"james","dept":"CS"} {"tableName":"dept","data":{"dept":"1","deptname":"CS","desc":"COMPUTER SCIENCE DEPT"}

推荐答案

我通过循环遍历预期表的列表和每个然后从数据框中过滤记录并应用模式 &特定于表的编码器，然后写入 sink .所以读取只发生一次，并且对于每个表 writeStream 将被调用并且它工作正常.谢谢大家的帮助

i got this working by looping through the list of expected tables and for each of then filter the records from the dataframe and apply the schema & encoder specific to the table and then write to sink . So the read happens only once and for each table writeStream will be called and its working fine. Thanks for all the help

这也负责根据表对镶木地板输出文件夹进行动态分区.

This takes care of dynamic partitioning of the parquet output folder based on the tables as well.

这篇关于嵌套json中的结构化流不同模式的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！