在 Apache NiFi 中使用 AvroSchemaRegistry [英] Using AvroSchemaRegistry in Apache NiFi

问题描述

我有 5 个不同的 CSVReader 控制器服务.除了 schema text(因为不同的标题)和 1 个 CSVRecordSetWriter 外，它们的配置是相同的.

I have 5 different CSVReader controller services. Their configuration is the same except schema text (because different headers) and 1 CSVRecordSetWriter.

我只想留下一个 CSVReader 并动态设置 schema text.我阅读了 AvroSchemaRegistry，但我不太清楚如何使用它.

I want to leave only one CSVReader and set schema text dynamically. I read about AvroSchemaRegistry but I didn't clearly understand how to use it.

我是否应该创建 5 个不同的具有 2 个属性的 AvroSchemaRegistry 控制器:name 和 value?F.e.我想放置以下架构:

Should i create 5 different AvroSchemaRegistry controllers with 2 attributes: name and value? F.e. i want to put following schema:

{
    "type": "record",
    "name": "campaigns",
    "namespace": "common",
    "fields": [
        {"name": "campaign_name", "type": "string"},
        {"name": "campaign_id", "type": "long"},
        {"name": "date", "type" : {"type": "int", "logicalType" : "date"}}
    ]
}

我应该使用以下命令创建 AvroSchemaRegistry:

I should create AvroSchemaRegistry with:

• name = 广告系列；
• value = 上面的完整代码.

• name = campaigns;
• value = Full Code above.

对于另一个模式，我应该创建另一个具有另一个属性 name 和 value 的 AvroSchemaResgitry 控制器?

For another schema i should create another AvroSchemaResgitry controller with another attributes name and value?

之后，如何配置 CSVReader 和 CSVRecordSetWriter 以使用这些模式?最后，我应该如何处理流文件?添加其他属性?什么样的?

After that, how to configure CSVReader and CSVRecordSetWriter to work with these schemas? And finally, what should i do with flowfiles? Add additional attributes? What kind of?

推荐答案

像这样配置您的流程(根据您的要求进行更改)，

Configure your flow something like this (make changes as per your requirement) ,

1. UpdateAttribute 配置以派生/硬编码流文件特定架构-

1. UpdateAttribute configuration to derive/hard code flowfile specific schema-

1. ValidateRecord 配置以使用通用 csv 读取器并动态传递架构 -

1. ValidateRecord configuration to use generic csv reader and pass schema dynamically -

1. CSVReader 控制器服务使用动态传递的模式并设置模式访问策略 -

1. CSVReader controller service to use dynamically passed schema and set schema access strategy -

如果您希望使用 NiFi 支持的模式注册表，请将所有模式放在注册表中并设置 schema.name 和 access strategy 属性以供记录读取器/写入器访问来自注册表的架构，但首先您需要在控制器服务中添加/配置架构注册表提供程序.

If you wish to use NiFi supported schema registry then put all of your schemas in registry and set schema.name and access strategy properties for record reader/writer to access schema from registry, but first you need to add/configure schema registry provider in controller service.

这篇关于在 Apache NiFi 中使用 AvroSchemaRegistry的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！