如何从Kafka的两个不同集群中消费? [英] How to consume from two different clusters in Kafka?

问题描述

我有两个kafka群集，分别是A和B，B是A的副本.我只想在A断开时使用群集B的消息，反之亦然.但是，从两个群集中消费消息将导致消息重复.因此，有什么方法可以配置我的kafka使用者以仅从一个群集接收消息.

I have two kafka clusters say A and B, B is replica of A. I would like to consume messages from cluster B only if A is down and viceversa. Nevertheless consuming messages from both the clusters would result in duplicate messages. So is there any way I can configure my kafka consumer to receive messages from only one cluster.

谢谢-

推荐答案

因此，有什么方法可以配置我的kafka使用者以仅从一个群集接收消息.

So is there any way I can configure my kafka consumer to receive messages from only one cluster.

是:Kafka使用者实例将始终仅从一个Kafka集群接收消息.也就是说，没有内置选项可以使用相同的使用者实例来读取2个以上的集群.但我认为您正在寻找与众不同的东西，请参见下文.

Yes: a Kafka consumer instance will always receive messages from one Kafka cluster only. That is, there's no built-in option to use the same consumer instance for reading from 2+ clusters. But I think you are looking for something different, see below.

仅当A关闭时，我才想使用来自群集B的消息，反之亦然.但是，如果同时使用两个群集中的消息，则会导致消息重复.

I would like to consume messages from cluster B only if A is down and viceversa. Nevertheless consuming messages from both the clusters would result in duplicate messages.

Kafka的使用者API中没有内置的故障转移支持，例如如果群集A发生故障，则切换到群集B".如果需要这种行为(如您的情况)，则需要在使用Kafka使用者API的应用程序中进行.

There's no built-in failover support such as "switch to cluster B if cluster A fails" in Kafka's consumer API. If you need such behavior (as in your case), you would need to do so in your application that uses the Kafka consumer API.

例如，您可以创建一个消费者实例以从集群A中读取信息，监视该实例和/或该集群以确定是否需要对集群B进行故障转移，并(如果需要)通过创建另一个消费者来执行对B的故障转移A发生故障时从B读取实例.

For example, you could create a consumer instance to read from cluster A, monitor that instance and/or that cluster to determine whether failover to cluster B is required, and (if needed) perform the failover to B by creating another consumer instance to read from B in the event that A fails.

但是，有一些陷阱使此故障转移行为比我的简化示例更为复杂.一个困难是要知道切换到B时已经读取了群集A的哪些消息:这很棘手，因为通常，群集之间的消息偏移量不同，因此确定是否已经读取了消息("B"中的副本") (来自A)并非无关紧要.

There are a few gotchas however that makes this failover behavior more complex than my simplified example. One difficulty is to know which messages from cluster A have already been read when switching over to B: this is tricky because, typically, the message offsets differ between clusters so determining whether the "copy" of a message (in B) was already read (from A) is not trivial.

注意:有时，您可以在以下情况下简化此类应用程序/此类故障转移逻辑:消息处理是幂等的(即重复消息/重复处理消息不会改变处理结果).

Note: Sometimes you can simplify such an application / such a failover logic in situations where e.g. message processing is idempotent (i.e. where duplicate messages / duplicate processing of messages will not alter the processing outcome).

这篇关于如何从Kafka的两个不同集群中消费?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！