春天卡夫卡和交易 [英] Spring Kafka and Transactions
问题描述
我想将Spring Kafka与Transactions一起使用，但我真的不了解应该如何配置它以及如何工作.
I'd like to use Spring Kafka with Transactions but I don't really understand how it is supposed to be configured and how it works.
这是我的配置
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
props.put(ProducerConfig.RETRIES_CONFIG, String.valueOf(Integer.MAX_VALUE));
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 1);
props.put(ProducerConfig.ACKS_CONFIG, "all");
此配置在带有事务ID前缀的DefaultKafkaProducerFactory中使用:
This config is used in a DefaultKafkaProducerFactory with transaction id prefix:
defaultKafkaProducerFactory.setTransactionIdPrefix("my_app.");
问题1:
我应该如何选择此交易ID前缀?如果我理解正确，spring会使用该前缀为每个创建的生产者生成一个事务ID.
How am I supposed to chose this transaction id prefix ? If I understand correctly, this prefix is used by spring to generate a transactional id for each producer created.
为什么我们不能只使用"UUID.randomUUID()?
Why couldn't we just use "UUID.randomUUID() ?
问题2:
如果生产者被销毁，它将生成一个新的交易ID.因此，如果应用程序崩溃，则在重新启动时它将重用旧的事务ID.
If the producer is destroyed, it will generate a new transactional id. Therefor, if the application crash, on restart it will reuse old transactional id.
那正常吗?
问题3:
我正在使用部署在云上的应用程序，该应用程序可以自动按比例放大/缩小.这意味着我的前缀无法固定，因为我在每个实例上的所有生产者都会有冲突的交易ID.
I'm using an application deployed on cloud which can be automatically scaled up/down. This means my prefix can not be fixed since all my producers on each instances will have transactional id in conflict.
我应该在其中添加随机部分吗?实例按比例缩小/放大或崩溃并重新启动时，是否需要恢复相同的前缀?
Should I add a random part in it ? Do I need to recover the same prefix when an instance is scaled down/up or crash and restart ?
问题4:
最后但并非最不重要的一点是，我们正在使用Kafka的凭据.这似乎不起作用:
Last but not least, we are using credentials for our Kafka. This does not seems to work :
Current ACLs for resource `TransactionalId:my_app*`:
User:CN... has Allow permission for operations: All from hosts: *
我应该如何设置知道我的交易ID的ACL?
How should I set my ACLs knowing that my transactional ids are generated ?
编辑1
进一步阅读后，如果我理解正确的话.
After further reading, if I understand correctly.
如果您有一个C0(消费者)从P0(分区)中读取.如果经纪人开始进行消费者再平衡.可以将P0分配给另一个使用者C1.消费者C1应该使用与先前C0相同的交易ID，以防止重复(僵尸隔离)?
If you have a C0(consumer) reading from P0(partition).If the broker start a consumer rebalance. P0 could be assigned to another consumer C1. This consumer C1 should use the same transaction id than then previous C0 to prevent duplicate (Zombies fencing) ?
您如何在spring-kafka中实现这一目标?事务标识似乎与使用者无关，因此分区已读取.
How do you achieve this in spring-kafka ? The transaction id seems to have nothing to do with the consumer and thus the partition read.
谢谢
推荐答案
-
由于僵尸防护，您不能使用随机TID-如果服务器崩溃，您可能会在该主题中进行部分事务，该事务永远不会完成，并且任何写入操作都不会占用任何分区进行该交易.
You can't use a random TID because of the zombie fencing - if the server crashes you could have a partial transaction in the topic which will never be completed and nothing more will be consumed from any partitions with a write for that transaction.
这是出于设计原因-出于上述原因.
That is by design - for the above reason.
同样，您不能随机化；由于上述原因.
Again, you can't randomize; for the above reason.
例如，
Cloud Foundry具有一个指示实例索引的环境变量.如果您使用的云平台不包含类似内容，则必须以某种方式对其进行仿真.然后，在交易ID中使用它:
Cloud Foundry, for example, has an environment variable that indicates the instance index. If you are using a cloud platform that doesn't include something like that, you would have to simulate it somehow. Then, use it in the transaction id:
spring.kafka.producer.transaction-id-prefix=foo-${instance.index}-
-
ACL-我无法回答；我对kafka权限不熟悉；最好单独问一个问题.
ACLs - I can't answer that; I am not familiar with kafka permissions; might be better to ask a separate question for that.
我认为我们需要向Spring添加一些逻辑，以确保特定主题/分区始终使用相同的交易ID.
I think we need to add some logic to Spring to ensure the same transaction id is always used for a particular topic/partition.
https://github.com/spring-projects/spring-kafka/issues/800#issuecomment-419501929
编辑
自此回答以来，一切都发生了变化(KIP-447)；如果您的经纪人是2.5.0或更高版本-请参见. https://docs.spring.io/spring-kafka/docs/2.5.5.RELEASE/reference/html/#exactly-once 和 https://docs.spring.io/spring-kafka/docs/2.6.0-SNAPSHOT/reference/html/#exactly-一次
Things have changed since this answer (KIP-447); if your brokers are 2.5.0 or later - see. https://docs.spring.io/spring-kafka/docs/2.5.5.RELEASE/reference/html/#exactly-once and https://docs.spring.io/spring-kafka/docs/2.6.0-SNAPSHOT/reference/html/#exactly-once
这篇关于春天卡夫卡和交易的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
