如何为托管在 Kubernetes 中的多个应用程序选择 Kafka 事务 ID? [英] How to choose Kafka transaction id for several applications, hosted in Kubernetes?

问题描述

我有一个经典的微服务架构.所以，有不同的应用.每个应用程序可能有 1..N 个实例.系统部署到Kubernetes. 所以，我们有很多不同的PODs，可以随时启动和停止.

我想实现 read-process-write 模式，所以我需要卡夫卡交易.

要配置事务，我需要为每个 Kafka 生产者设置一些 transaction id.(实际上，我需要 transaction-id-prefix，因为我的应用程序使用 Spring，并且它有这样的 API).这些 IDs 必须相同，在应用程序重新启动后.

那么，如何为托管在 Kubernetes 中的多个应用程序选择 Kafka 事务 ID?

解决方案

如果消费者开始事务(读-进程-写)，那么同一个应用程序的所有实例的事务 id 前缀必须相同(这样僵尸重新平衡后，围栏可以正常工作).实际使用的事务 ID 是

...

如果你有多个应用，它们应该有唯一的前缀(尽管如果它们从不同的主题消费，它们无论如何都是唯一的).

对于仅生产者交易，前缀在每个实例中必须是唯一的(以防止 kafka 屏蔽生产者).

I have a classic microservice architecture. So, there are differ applications. Each application may have 1..N instances. The system is deployed to Kubernetes. So, we have many differ PODs, which can start and stop in any time.

I want to implement read-process-write pattern, so I need Kafka transactions.

To configure transactions, I need to set some transaction id for each Kafka producer. (Actually, I need transaction-id-prefix, because of I use Spring for my applications, and it has such API). These IDs have to be the same, after application is restarted.

So, how to choose Kafka transaction id for several applications, hosted in Kubernetes?

解决方案

If the consumer starts the transaction (read-process-write) then the transaction id prefix must be the same for all instances of the same app (so that zombie fencing works correctly after a rebalance). The actual transaction id used is <prefix><group>.<topic>.<partition>.

If you have multiple apps, they should have unique prefixes (although if they consume from different topics, they will be unique anyway).

For producer-only transactions, the prefix must be unique in each instance (to prevent kafka fencing the producers).

这篇关于如何为托管在 Kubernetes 中的多个应用程序选择 Kafka 事务 ID?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！