Kafka UNKNOWN_PRODUCER_ID异常 [英] Kafka UNKNOWN_PRODUCER_ID exception

问题描述

使用kafka流时，有时会发现UNKNOWN_PRODUCER_ID异常.

I sometimes find UNKNOWN_PRODUCER_ID exception when using kafka streams.

2018-06-25 10:31:38.329  WARN 1 --- [-1-1_0-producer] o.a.k.clients.producer.internals.Sender  : [Producer clientId=default-groupz-7bd94946-3bc0-4400-8e73-7126b9b9c0d4-StreamThread-1-1_0-producer, transactionalId=default-groupz-1_0] Got error produce response with correlation id 1996 on topic-partition default-groupz-mplat-five-minute-stat-urlCount-counts-store-changelog-0, retrying (2147483646 attempts left). Error: UNKNOWN_PRODUCER_ID

参考官方文件:

如果经纪人无法找到该异常，则会引发此异常 与相关的producerId相关联的生产者元数据.这 例如，如果删除了生产者的记录，可能会发生 因为他们的保留时间已经过去了.一旦最后的记录 删除producerId，将生产者的元数据从中删除. 经纪人，生产者将来的附加操作将返回此异常.

This exception is raised by the broker if it could not locate the producer metadata associated with the producerId in question. This could happen if, for instance, the producer's records were deleted because their retention time had elapsed. Once the last records of the producerId are removed, the producer's metadata is removed from the broker, and future appends by the producer will return this exception.

它说，一种可能性是生产者闲置的时间超过了保留时间(默认情况下为一周)，因此该生产者的元数据将从代理中删除.经纪人无法找到生产者元数据还有其他原因吗?

It says one possibility is that a producer is idle for more than retention time (by default a week) so the producer's metadata will be removed from broker. Are there any other reasons that brokers could not locate producer metadata?

推荐答案

您可能正在遇到 https://issues.apache.org/jira/browse/KAFKA-7190 .如票证中所述:

You might be experiencing https://issues.apache.org/jira/browse/KAFKA-7190. As it says in that ticket:

当流应用程序的流量很少时，消费者清除可能会删除 甚至是生产者发送的最后一条消息(即， 该生产者已被消耗和承诺)，因此，经纪人 会删除该生产者的ID.下一次该生产者尝试 发送，它将获得此UNKNOWN_PRODUCER_ID错误代码，但在这种情况下， 此错误是可重试的:生产者将只获得一个新的生产者ID，然后 重试，然后这次将成功.

When a streams application has little traffic, then it is possible that consumer purging would delete even the last message sent by a producer (i.e., all the messages sent by this producer have been consumed and committed), and as a result, the broker would delete that producer's ID. The next time when this producer tries to send, it will get this UNKNOWN_PRODUCER_ID error code, but in this case, this error is retriable: the producer would just get a new producer id and retries, and then this time it will succeed.

此问题也在 查看全文