使用幂等Kafka Producer时的订购保证 [英] Ordering guarantees when using idempotent Kafka Producer
问题描述
我在应用程序中使用的是Kafka 1.0.1，并且已经开始使用0.11中引入的Idempotent Producer功能，并且在使用Idempontent功能时难以理解订购保证.
I am using Kafka 1.0.1 in my application and I have started using the Idempotent Producer feature that was introduced in 0.11, and I've having trouble understanding the ordering guarantees when using the Idempontent feature.
我的生产者的配置是:
enable.idempotence = true
max.in.flight.requests.per.connection = 5
重试= 50
acks =全部
根据文档:
重试
设置一个大于零的值将导致客户端重新发送其发送失败并带有潜在的瞬时错误的任何记录.请注意，此重试与客户端在收到错误后重新发送记录没有什么不同.在不将max.in.flight.requests.per.connection设置为1的情况下允许重试可能会更改记录的顺序，因为如果将两个批次发送到单个分区，并且第一个失败并被重试，但是第二个成功，则记录在第二批中可能会首先出现.
Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 will potentially change the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch may appear first.
enable.幂等
设置为"true"时，生产者将确保每条消息的确切副本被写入流中.如果为"false"，则生产者由于代理失败等原因而重试，可能会在流中写入重试消息的副本.请注意，启用幂等性要求max.in.flight.requests.per.connection小于或等于5，重试大于0，ack必须为"all".如果用户未明确设置这些值，则将选择合适的值.如果设置了不兼容的值，则将引发ConfigException.
When set to 'true', the producer will ensure that exactly one copy of each message is written in the stream. If 'false', producer retries due to broker failures, etc., may write duplicates of the retried message in the stream. Note that enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0 and acks must be 'all'. If these values are not explicitly set by the user, suitable values will be chosen. If incompatible values are set, a ConfigException will be thrown.
我的配置似乎符合要求，但似乎不符合要求.
My configuration seems to be according to the requirements, but they don't seem to align.
我要处理的另一个问题是 OutOfOrderSequenceException :根据文档，如果我遇到此异常，则意味着生产者有可能出现故障.但是，如果我的生产者配置了 max.in.flight.requests.per.connection = 5 ，并且假设第二个请求出现了乱序异常，那么以下所有请求均会发生什么已经在飞行吗?这是否意味着我确定会出现故障?
Another question I have has to do with the OutOfOrderSequenceException:
According to the documentation, if I get this exception it means that the producer is in risk of becoming out of order. But if my producer is configured with max.in.flight.requests.per.connection = 5 and let's say that the second request got the out of order exception, what happens to all of the following requests that are already in flight? will this mean that I am for sure out of order?
推荐答案
在KafkaProducer中启用幂等时，可以保证顺序.
The ordering is guaranteed when enabling idempotence in the KafkaProducer.
即使您的 max.in.flight.requests.per.connection 大于或等于1，幂等的KafkaProducer仍将确保TopicPartition中的排序.关于重试"的描述在本说明书中不作赘述.与 max.in.flight 的关系仅在禁用幂等性时适用.
Even if you have max.in.flight.requests.per.connection larger or equal to 1 an idempotent KafkaProducer will still ensure the ordering within a TopicPartition. The description on the "retries" with the relation to the max.in.flight is only applicable if idempotence is disabled.
幂等的KafkaProducer使用内部递增序列号来确保对多达5个飞行中的请求进行排序，如Pull-Request
An idempotent KafkaProducer uses an internal incrementing sequence number to ensure the ordering up to 5 in.flight requests as described in the Pull-Request #3743:
"[[...]我们保留5个较早批次的记录元数据."
"[...] we retain the record metadata for 5 older batches."
此外， enable.idempotence 上的文档告知您拥有最多5机上要求.否则，您将获得 ConfigException .
Also, the documentation on the enable.idempotence informs about having at maximum 5 in-flight request. Otherwise you would get a ConfigException.
更多详细信息，请参见 KAFKA-5494 和有关最大化飞行的设计>1启用幂等.其中第5步和第6步将阐明您的问题:
More details are given in KAFKA-5494 and the corresponding documentation on Design for max.in.flight > 1 with idempotence enabled. where the steps 5 and 6 will clarify your question:
有了上述假设，解决方案如下:
With the above assumptions in place, the solution is as follows:
-
我们跟踪发送到分区的最后一个已确认序列.每次成功确认后都会更新一次，因此应该一直在增加.
We keep track of the last acknowledged sequence sent to a partition. This is updated on every successful ack and thus should always be increasing.
我们跟踪给定分区的批处理绑定的下一个序列.
We keep track of the next sequence for a batch bound for a given partition.
当批次排空时，我们分配了nextSequence.我们还将nextSequence增加一批记录的数量.
We assigned the nextSequence when the batch is drained. We also increment the nextSequence by the record count of a batch.
如果生产请求成功，我们将最后确认的序列设置为批次的最后序列.
If a produce request succeeds, we set the last ack’d sequence to be the last sequence of the batch.
如果生产请求失败，则成功进行的飞行批处理也会因OutOfOrderSequenceException而失败.
正因如此，如果批次的序列号不是最后一个确认序列的后继，并且如果失败并出现OutOfOrderSequenceException，则我们认为此序列是可重试的.
重新处理批次时，我们先删除生产者ID和序列号，然后再将其插入队列.
When a batch is requeued, we erase the producer id and sequence number before inserting it in the queue.
当第一个飞行批次失败时(无论出于何种原因)，我们将nextSequence重置为lastAckdSequence + 1.
When the first inflight batch fails (for whatever reason), we reset the nextSequence to be the lastAckdSequence + 1.
因此，如果一个批次致命失败，则后续批次的序列号在重试时将有所不同.很好，因为先前的失败是 OutOfSequence 异常，它明确表示请求被拒绝.
Thus if a batch fails fatally, the sequence numbers of succeeding batches will be different on the retry. This is fine, because the previous failure was an OutOfSequence exception, which categorically means that the request was rejected.
" [[...]如果我的生产者配置了max.in.flight.requests.per.connection = 5，并且让我们说第二个请求出现了乱序异常，则以下所有情况都会发生已在执行中的请求?这是否意味着我确定会出现故障?"
"[...] if my producer is configured with max.in.flight.requests.per.connection = 5 and let's say that the second request got the out of order exception, what happens to all of the following requests that are already in flight? will this mean that I am for sure out of order?"
如第5步和第6步所述，所有后续的进行中请求也将失败，并返回可重试 OutOfOrderSequenceExcpetion .当重试大于 0 时，幂等的KafkaProducer将能够保持顺序.
As mentioned in step 5 and 6 all succeeding in-flight requests will also fail with an retriable OutOfOrderSequenceExcpetion. As retries are greater than 0 the idempotent KafkaProducer will be able to keep the order.
这篇关于使用幂等Kafka Producer时的订购保证的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
