kafka 过滤/动态主题创建 [英] kafka filtering/Dynamic topic creation

问题描述

我正在研究 kafka、spark-streaming 并尝试实现 Reconciliation.

I am working on kafka, spark-streaming and trying to achieve Reconciliation.

Producer Message: 大约 5 个字段由 ',' 分隔，其中之一是 UUID(唯一标识符)例如:A,B,producer,UUID_1,E

Producer Message: some 5 fields separated by ',' and one of those would be UUID(unique Identifier) ex: A,B,producer,UUID_1,E

消费者信息:大约 5 个字段由,"分隔，其中之一是 UUID例如:A、B、消费者、UUID_1、E

Consumer Message: some 5 fields separated by ',' and one of those would be UUID ex:A,B,consumer,UUID_1,E

这里的 UUID 是生产者和消费者之间的关系，它是动态的

Here UUID is the relation between producer and consumer and it is dynamic

在 kafka 中，我需要某种过滤，通过这些过滤，所有特定的生产者/消费者消息都将位于一个特定的主题或分区中.

In kafka i need some sort of filtering by which all specific producer/consumer messages would be in one particular topic or partition.

我确实有如下想法但无法实施:

I do had an idea as below but unable to implement it:

根据我的研究动态分区是不允许的，这是正确的吗?

As per my research dynamic partitions are not allowed, is this correct?

所以我看到的唯一选项是，根据收到的 UUID 创建一个动态主题，然后将每个 UUID 特定消息路由到该主题，一旦消息被消费，该主题需要删除.这是正确的方法吗?如果是，如何实现?

so the only option i see is, Create a dynamic topic based on the received UUID, then route each UUID specific messages to that topic and once the messages got consumed the topic needs to be deleted. Is this the correct approach and if yes how to achieve this?

推荐答案

根据我的研究，不允许动态分区，这是正确的吗?

As per my research dynamic partitions are not allowed, is this correct?

分区可以增加，但不能减少.

Partitions can be increase,but can't be decrease.

根据收到的 UUID 创建动态主题

Create a dynamic topic based on the received UUID

不要那样做，它违反了 Topic 语义，Topic 表示同类消息.
只需将'UUID_1'设置为消息'Key'字段，因为Kafka根据'Key'的散列将相同的'Key'消息发送到相同的'Partition'.

Dont't do that,it violate the Topic semantic,Topic means same kind messages.
Just set the 'UUID_1' into message 'Key' field,as Kafka send same 'Key' message to the same 'Partition' accroding to the hash of 'Key'.

这篇关于kafka 过滤/动态主题创建的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！