保证 Kafka Producers 唯一的全局事务 [英] Guarantee unique global transaction for Kafka Producers

问题描述

在最新版本的 Kafka 0.11.0.0 中，Apache 团队引入了幂等生产者和事务.是否可以保证我们要记录的一整套消息(例如 100 万条)仅在最后提交?我希望，例如，如果生产者与经纪人失去连接并且无法恢复它，消费者将看不到任何消息.是否可以?

With the last version of Kafka 0.11.0.0 the Apache team is introducing idempotent producer and transactions. Is It possible to guarantee that an entire set of messages (for example 1 million) we want to log, will be committed only at the end? I would like that, if for example the Producers loose the connection with the brokers and cannot restabilish it, no messages will be seen by the consumers. Is it possible?

推荐答案

是的，可以在生产者中使用 Transactions.您启动一个事务，发布所有消息，然后提交该事务.所有消息一次写入 Kafka，但新的 READ_COMMITTED 模式下的消费者只会在生产者提交事务并在 Kafka 提交日志中添加一个特殊的事务标记后才能看到这些消息.

Yes this is possible using Transactions in your producer. You start a transaction, publish all your messages, and then commit the transaction. All the messages are written to Kafka one at a time but consumers in the new READ_COMMITTED mode will only see the messages after the transaction is committed by the producer and a special transaction marker is added to the Kafka commit log.

未处于 READ_COMMITTED 模式的消费者可以看到单独写入的消息，即使它们可能尚未(或从未)提交.

Consumers not in READ_COMMITTED mode can see the messages as they are written individually even though they may not yet (or ever) be committed.

一个打开的事务可以保持未提交的时间是有限制的，因此最终如果生产者死亡并且没有明确结束事务，它将超时并回滚并且 READ_COMMITTED 消费者将永远不会看到这些消息.

There is a limit to how long an open transaction can stay uncommitted so eventually if the producer dies and does not explicitly end the transaction it will timeout and rollback and READ_COMMITTED consumers will never see those messages.

这篇关于保证 Kafka Producers 唯一的全局事务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！