如何为Kubernetes中托管的多个应用程序选择Kafka交易ID? [英] How to choose Kafka transaction id for several applications, hosted in Kubernetes?

问题描述

我有一个经典的微服务架构.因此，有不同的应用程序.每个应用程序可能具有1..N实例.系统已部署到Kubernetes.，因此，我们有许多不同的PODs，它们可以随时启动和停止.

I have a classic microservice architecture. So, there are differ applications. Each application may have 1..N instances. The system is deployed to Kubernetes. So, we have many differ PODs, which can start and stop in any time.

我想实现读取过程写入模式，因此我需要卡夫卡交易.

I want to implement read-process-write pattern, so I need Kafka transactions.

要配置事务，我需要为每个Kafka生产者设置一些transaction id. (实际上，我需要transaction-id-prefix，因为我将Spring用于我的应用程序，并且它具有这样的API).重新启动应用程序后，这些IDs必须相同.

To configure transactions, I need to set some transaction id for each Kafka producer. (Actually, I need transaction-id-prefix, because of I use Spring for my applications, and it has such API). These IDs have to be the same, after application is restarted.

那么，如何为Kubernetes中托管的多个应用程序选择Kafka交易ID?

So, how to choose Kafka transaction id for several applications, hosted in Kubernetes?

推荐答案

如果使用者启动事务(读过程写)，则对于同一应用程序的所有实例，事务ID前缀必须相同(这样僵尸重新平衡后，围栏可以正常工作).实际使用的交易ID为<prefix><group>.<topic>.<partition>.

If the consumer starts the transaction (read-process-write) then the transaction id prefix must be the same for all instances of the same app (so that zombie fencing works correctly after a rebalance). The actual transaction id used is <prefix><group>.<topic>.<partition>.

如果您有多个应用程序，则它们应具有唯一的前缀(尽管它们来自不同的主题，但它们仍将是唯一的).

If you have multiple apps, they should have unique prefixes (although if they consume from different topics, they will be unique anyway).

对于仅生产者事务，前缀在每个实例中必须唯一(以防止kafka围栏生产者).

For producer-only transactions, the prefix must be unique in each instance (to prevent kafka fencing the producers).

这篇关于如何为Kubernetes中托管的多个应用程序选择Kafka交易ID?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！