卡夫卡 UNKNOWN_PRODUCER_ID 异常 [英] Kafka UNKNOWN_PRODUCER_ID exception

问题描述

我在使用 kafka 流时有时会发现 UNKNOWN_PRODUCER_ID 异常.

I sometimes find UNKNOWN_PRODUCER_ID exception when using kafka streams.

2018-06-25 10:31:38.329  WARN 1 --- [-1-1_0-producer] o.a.k.clients.producer.internals.Sender  : [Producer clientId=default-groupz-7bd94946-3bc0-4400-8e73-7126b9b9c0d4-StreamThread-1-1_0-producer, transactionalId=default-groupz-1_0] Got error produce response with correlation id 1996 on topic-partition default-groupz-mplat-five-minute-stat-urlCount-counts-store-changelog-0, retrying (2147483646 attempts left). Error: UNKNOWN_PRODUCER_ID

参考官方文档:

如果代理找不到与相关生产者Id相关联的生产者元数据.这个例如，如果生产者的记录被删除，可能会发生因为他们的保留时间已经过去.一旦最后的记录producerId 被删除，生产者的元数据从经纪人，生产者未来的追加将返回此异常.

This exception is raised by the broker if it could not locate the producer metadata associated with the producerId in question. This could happen if, for instance, the producer's records were deleted because their retention time had elapsed. Once the last records of the producerId are removed, the producer's metadata is removed from the broker, and future appends by the producer will return this exception.

它说一种可能性是生产者空闲时间超过保留时间(默认为一周)，因此生产者的元数据将从代理中删除.经纪人无法找到生产者元数据是否还有其他原因?

It says one possibility is that a producer is idle for more than retention time (by default a week) so the producer's metadata will be removed from broker. Are there any other reasons that brokers could not locate producer metadata?

推荐答案

您可能正在遇到 https://issues.apache.org/jira/browse/KAFKA-7190.正如那张票上所说:

You might be experiencing https://issues.apache.org/jira/browse/KAFKA-7190. As it says in that ticket:

当一个流应用程序的流量很小时，消费者清除可能会删除甚至生产者发送的最后一条消息(即所有由生产者发送的消息)该生产者已被消费并提交)，因此，经纪人将删除该生产者的 ID.下次当这个生产者试图发送，它会得到这个 UNKNOWN_PRODUCER_ID 错误代码，但在这种情况下，这个错误是可重试的:生产者只会得到一个新的生产者 ID 和重试，然后这一次它会成功.

When a streams application has little traffic, then it is possible that consumer purging would delete even the last message sent by a producer (i.e., all the messages sent by this producer have been consumed and committed), and as a result, the broker would delete that producer's ID. The next time when this producer tries to send, it will get this UNKNOWN_PRODUCER_ID error code, but in this case, this error is retriable: the producer would just get a new producer id and retries, and then this time it will succeed.

此问题也在 https://cwiki.apache.org/confluence/display/KAFKA/KIP-360%3A+Improve+handling+of+unknown+producer

这篇关于卡夫卡 UNKNOWN_PRODUCER_ID 异常的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！