Kafka Mirror Maker 无法复制 __consumer_offset 主题 [英] Kafka Mirror Maker failing to replicate __consumer_offset topic

查看:46
本文介绍了Kafka Mirror Maker 无法复制 __consumer_offset 主题的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我正在尝试利用镜像制造商将 __consumer_offsets 主题与其他主题一起复制.

I am trying to make use of mirror maker to replicate __consumer_offsets topic along with other topics.

它给出了如下所述的错误.

It is giving error as mentioned below.

[2018-10-24 16:16:03,802] ERROR 向主题发送消息时出错__consumer_offsets with key: 16 bytes, value: 445 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)org.apache.kafka.common.errors.InvalidTopicException:请求试图对无效主题执行操作.[2018-10-2416:16:03,828] 向主题发送消息时出错__consumer_offsets with key: 29 bytes, value: 754 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)org.apache.kafka.common.errors.InvalidTopicException:请求试图对无效主题执行操作.

[2018-10-24 16:16:03,802] ERROR Error when sending message to topic __consumer_offsets with key: 16 bytes, value: 445 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.InvalidTopicException: The request attempted to perform an operation on an invalid topic. [2018-10-24 16:16:03,828] ERROR Error when sending message to topic __consumer_offsets with key: 29 bytes, value: 754 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.InvalidTopicException: The request attempted to perform an operation on an invalid topic.

有什么办法可以解决这个问题吗?

Is there any way to resolve this?

SlideShare 在幻灯片 21 上,它提到了在多个数据中心复制偏移主题.有人可以告诉我实现相同目标的可能方法吗?

In one of the confluent presentation in SlideShare on slide21, it mentioned about replicating offset topic in multiple datacenters. Can someone tell me the possible way to achieve the same?

或者有其他更好的方法来为 Kafka 备份和恢复策略.

Or there is any other better way for backup and restore policy for Kafka.

推荐答案

在您的 consumer.config 中添加:

exclude.internal.topics=false

并将其添加到您的 producer.config 中:

And add this in your producer.config:

client.id=__admin_client


原因是,除非您声明 AdminClient.scala 使用的 client.id=__admin_client,否则生产者无法写入内部主题.在这里找到:https://issues.apache.org/jira/browse/KAFKA-6524

Reason being, Producers cannot write to internal topics unless you declare client.id=__admin_client which is used by AdminClient.scala. Found here: https://issues.apache.org/jira/browse/KAFKA-6524

这篇关于Kafka Mirror Maker 无法复制 __consumer_offset 主题的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
相关文章
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆