nifi ConvertRecord JSON转换为CSV仅获得单个记录? [英] nifi ConvertRecord JSON to CSV getting only single record?

问题描述

我为读取json数据设置了以下流程，并使用convertRecord处理器将其转换为csv.但是，输出流文件仅填充单个记录(我假设仅是第一条记录)，而不是所有记录.

I have the below flow set up for reading json data and convert it to csv using the convertRecord processor. However, the output flowfile is only populated with single record (I am assuming only the first record) instead of all the records.

有人可以提供正确的配置吗?

Can someone help provide the correct configuration?

源json数据:

Source json data:

{"creation_Date": "2018-08-19", "Hour_of_day": 7, "log_count": 2136}
{"creation_Date": "2018-08-19", "Hour_of_day": 17, "log_count": 606}
{"creation_Date": "2018-08-19", "Hour_of_day": 14, "log_count": 1328}
{"creation_Date": "2018-08-19", "Hour_of_day": 20, "log_count": 363}

流程:

flow:

ConvertRecord处理器配置:

ConvertRecord processor config:

JsonTreeReader控制器配置:

JsonTreeReader controller config:

CSVrecordsetWriter控制器配置:

CSVrecordsetWriter controller config:

AvroSchemaRegistry控制器配置:

AvroSchemaRegistry controller config:

{
  "type": "record",
  "name": "demo_schema",
  "fields":
  [
    { "name": "creation_Date", "type": "string"},
    { "name": "Hour_of_day", "type": "string"},
    { "name": "log_count", "type": "string"}
  ]
}

我得到的

流文件内容:

Flowfile content that I am getting:

creation_Date,Hour_of_day,log_count
2018-08-16,0,3278

我需要什么:

what I need:

creation_Date,Hour_of_day,log_count
2018-08-16,0,3278
2018-08-17,4,278
2018-08-18,10,6723

希望我很详细地解释了这种情况，如果有人可以帮助更正配置以使我获得完整的数据，我将不胜感激.预先谢谢你！

Hope I was elaborate to explain the situation and would appreciate if someone can help correct the configuration so that I get the complete data. Thank you in advance!

推荐答案

您正在面对 NIFI-4456 错误，并已从NiFi-1.7 版本开始修复.

You are facing this NIFI-4456 bug and has been fixed starting from NiFi-1.7 version.

要解决此问题:

1.将SplitText processor与split line count =1

2.然后使用MergeContent/MergeRecord处理器(使用碎片整理"作为合并策略)，并作为有效的json消息数组

2.Then use MergeContent/MergeRecord processors (using Defragments as merge strategy) and make as valid json array of messages

如果使用的是Merge Record处理器，则使用Reader和Writer控制器 服务必须采用Json格式.

if you are using Merge Record processor then Reader and Writer controller services needs to be in Json format.

3.然后将合并关系提供给ConvertRecord处理器.

3.Then feed the merged relation to ConvertRecord processor.

流量:

从 NiFi-1.7 + 版本开始，我们不需要在JsonTreeReader控制器服务中配置任何新的/附加功能，因为 NiFi能够按行格式读取json 也可以.

Starting from NiFi-1.7+ version, we don't need to configure any thing new/additional in JsonTreeReader controller service as NiFi is able to read json per line format also.

更新:

MergeContent配置:

MergeContent Configs:

如果我们使用的是MergeContent处理器，请按照以下屏幕截图所示的方式配置处理器.

if we are using MergeContent processor please configure the processor like this way as shown in the below screenshot.

分隔符策略Text

标题[

页脚]

分界器,

此外，我建议使用 MergeRecord处理器而不是MergeContent处理器，它将负责创建有效的json消息数组.

In addition i would suggest to use MergeRecord processor instead of MergeContent processor which will take care of creating a valid array of json messages.

这篇关于nifi ConvertRecord JSON转换为CSV仅获得单个记录?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！