如何从Kafka中的两个不同集群消费? [英] How to consume from two different clusters in Kafka?

问题描述

我有两个 kafka 集群，比如 A 和 B，B 是 A 的副本.我想仅在 A 关闭时才使用来自集群 B 的消息，反之亦然.然而，使用来自两个集群的消息会导致重复的消息.那么有什么方法可以配置我的 kafka 消费者以仅接收来自一个集群的消息.

I have two kafka clusters say A and B, B is replica of A. I would like to consume messages from cluster B only if A is down and viceversa. Nevertheless consuming messages from both the clusters would result in duplicate messages. So is there any way I can configure my kafka consumer to receive messages from only one cluster.

谢谢——

推荐答案

那么有什么方法可以配置我的 kafka 使用者以仅接收来自一个集群的消息.

So is there any way I can configure my kafka consumer to receive messages from only one cluster.

是的:Kafka 消费者实例将始终仅从一个 Kafka 集群接收消息.也就是说，没有内置选项可以使用相同的使用者实例来读取 2+ 个集群.但我认为您正在寻找不同的东西，见下文.

Yes: a Kafka consumer instance will always receive messages from one Kafka cluster only. That is, there's no built-in option to use the same consumer instance for reading from 2+ clusters. But I think you are looking for something different, see below.

我只想在 A 关闭时使用来自集群 B 的消息，反之亦然.然而，使用来自两个集群的消息会导致重复消息.

I would like to consume messages from cluster B only if A is down and viceversa. Nevertheless consuming messages from both the clusters would result in duplicate messages.

Kafka 的消费者 API 中没有内置的故障转移支持，例如如果集群 A 发生故障，则切换到集群 B".如果您需要此类行为(如您的情况)，则需要在使用 Kafka 消费者 API 的应用程序中执行此操作.

There's no built-in failover support such as "switch to cluster B if cluster A fails" in Kafka's consumer API. If you need such behavior (as in your case), you would need to do so in your application that uses the Kafka consumer API.

例如，您可以创建一个消费者实例以从集群 A 中读取数据，监控该实例和/或该集群以确定是否需要故障转移到集群 B，以及(如果需要)通过创建另一个消费者来执行到 B 的故障转移在 A 失败的情况下从 B 读取的实例.

For example, you could create a consumer instance to read from cluster A, monitor that instance and/or that cluster to determine whether failover to cluster B is required, and (if needed) perform the failover to B by creating another consumer instance to read from B in the event that A fails.

然而，有一些问题使这种故障转移行为比我的简化示例更复杂.一个困难是在切换到 B 时知道来自集群 A 的哪些消息已经被读取:这很棘手，因为通常集群之间的消息偏移量不同，因此确定消息的副本"(在 B 中)是否已经被读取(来自 A)不是微不足道的.

There are a few gotchas however that makes this failover behavior more complex than my simplified example. One difficulty is to know which messages from cluster A have already been read when switching over to B: this is tricky because, typically, the message offsets differ between clusters so determining whether the "copy" of a message (in B) was already read (from A) is not trivial.

注意:有时您可以在以下情况下简化此类应用程序/此类故障转移逻辑:消息处理是幂等的(即重复消息/重复处理消息不会改变处理结果).

Note: Sometimes you can simplify such an application / such a failover logic in situations where e.g. message processing is idempotent (i.e. where duplicate messages / duplicate processing of messages will not alter the processing outcome).

这篇关于如何从Kafka中的两个不同集群消费?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！