nifi ConvertRecord JSON to CSV 只获得单条记录? [英] nifi ConvertRecord JSON to CSV getting only single record?

问题描述

我设置了以下流程以读取 json 数据并使用 convertRecord 处理器将其转换为 csv.但是，输出流文件只填充了单个记录(我假设只有第一条记录)而不是所有记录.

有人可以帮忙提供正确的配置吗?

源 json 数据:

{"creation_Date": "2018-08-19", "Hour_of_day": 7, "log_count": 2136}{creation_Date":2018-08-19"，Hour_of_day":17，log_count":606}{creation_Date":2018-08-19"，Hour_of_day":14，log_count":1328}{creation_Date":2018-08-19"，Hour_of_day":20，log_count":363}

流程:

ConvertRecord 处理器配置:

JsonTreeReader 控制器配置:

CSVrecordsetWriter 控制器配置:

AvroSchemaRegistry 控制器配置:

<代码>{"类型": "记录","name": "demo_schema",领域":[{ "name": "creation_Date", "type": "string"},{ "name": "Hour_of_day", "type": "string"},{ "name": "log_count", "type": "string"}]}

我得到的流文件内容:

creation_Date,Hour_of_day,log_count2018-08-16,0,3278

我需要什么:

creation_Date,Hour_of_day,log_count2018-08-16,0,32782018-08-17,4,2782018-08-18,10,6723

希望我能详细解释这种情况，如果有人可以帮助纠正配置以便我获得完整的数据，我将不胜感激.提前谢谢你！

解决方案

您正面临这个问题

从 NiFi-1.7+ 版本开始，我们不需要在 JsonTreeReader 控制器服务中配置任何新的/额外的东西，因为 NiFi 能够读取 json 每行格式 还有.

更新:

合并内容配置:

如果我们使用 MergeContent 处理器，请按照以下屏幕截图所示的方式配置处理器.

<块引用>

分隔符策略Text

标题[

页脚]

分界符,

此外，我建议使用 MergeRecord 处理器而不是 MergeContent 处理器，它将负责创建有效的 json 消息数组.

I have the below flow set up for reading json data and convert it to csv using the convertRecord processor. However, the output flowfile is only populated with single record (I am assuming only the first record) instead of all the records.

Can someone help provide the correct configuration?

Source json data:

{"creation_Date": "2018-08-19", "Hour_of_day": 7, "log_count": 2136}
{"creation_Date": "2018-08-19", "Hour_of_day": 17, "log_count": 606}
{"creation_Date": "2018-08-19", "Hour_of_day": 14, "log_count": 1328}
{"creation_Date": "2018-08-19", "Hour_of_day": 20, "log_count": 363}

flow:

ConvertRecord processor config:

JsonTreeReader controller config:

CSVrecordsetWriter controller config:

AvroSchemaRegistry controller config:

{
  "type": "record",
  "name": "demo_schema",
  "fields":
  [
    { "name": "creation_Date", "type": "string"},
    { "name": "Hour_of_day", "type": "string"},
    { "name": "log_count", "type": "string"}
  ]
}

Flowfile content that I am getting:

creation_Date,Hour_of_day,log_count
2018-08-16,0,3278

what I need:

creation_Date,Hour_of_day,log_count
2018-08-16,0,3278
2018-08-17,4,278
2018-08-18,10,6723

Hope I was elaborate to explain the situation and would appreciate if someone can help correct the configuration so that I get the complete data. Thank you in advance!

解决方案

You are facing this NIFI-4456 bug and has been fixed starting from NiFi-1.7 version.

To work around this issue:

1.use SplitText processor with split line count =1

2.Then use MergeContent/MergeRecord processors (using Defragments as merge strategy) and make as valid json array of messages

if you are using Merge Record processor then Reader and Writer controller services needs to be in Json format.

3.Then feed the merged relation to ConvertRecord processor.

Flow:

Starting from NiFi-1.7+ version, we don't need to configure any thing new/additional in JsonTreeReader controller service as NiFi is able to read json per line format also.

Update:

MergeContent Configs:

if we are using MergeContent processor please configure the processor like this way as shown in the below screenshot.

Delimiter Strategy Text

Header [

Footer ]

Demarcator ,

In addition i would suggest to use MergeRecord processor instead of MergeContent processor which will take care of creating a valid array of json messages.

这篇关于nifi ConvertRecord JSON to CSV 只获得单条记录?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！