Kafka 消费者无法使用引导服务器名称消费消息 [英] Kafka consumer not able to consume messages using bootstrap server name
问题描述
我在使用引导服务器(即 Kafka 服务器)使用消息时遇到问题.知道为什么它不能在没有 zookeeper 的情况下使用消息吗?
- Kafka 版本:kafka_2.11-1.0.0
- Zookeeper 版本:kafka_2.11-1.0.0
- Zookeeper 主机和端口:zkp02.mp.com:2181
- Kafka 主机和端口:kfk03.mp.com:9092
产生一些消息:
[kfk03.mp.com ~]$/bnsf/kafka/bin/kafka-console-producer.sh --broker-list kfk03.mp.com:9092 --topic 测试>嗨>嗨
如果我给--bootstrap-server
,消费者就不能消费消息:
[kfk03.mp.com ~]$/bnsf/kafka/bin/kafka-console-consumer.sh --bootstrap-server kfk03.mp.com:9092 --topic test --from-beginning
当给定 --zookeeper
服务器而不是 --bootstrap-server
时,消费者能够消费消息 -:
[kfk03.mp.com ~]$/bnsf/kafka/bin/kafka-console-consumer.sh --zookeeper zkp02.mp.com:2181 --topic test --from-beginning不推荐将 ConsoleConsumer 与旧消费者一起使用,并将在未来的主要版本中删除.考虑通过传递 [bootstrap-server] 而不是 [zookeeper] 来使用新的使用者.{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}你好{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}你好你好乌塔姆你好你好你好你好你好^C共处理了17条消息
在使用 bootstrap-server 参数从 kafka 消费消息时,连接是通过 kafka 服务器而不是 zookeeper 进行的.Kafka 代理将偏移详细信息存储在 __consumer_offsets 主题中.
检查 __consumer_offsets 是否存在于您的主题列表中.如果不存在,请检查 kafka 日志以查找原因.
我们遇到了类似的问题.在我们的例子中,__consumer_offsets 没有被创建,因为以下错误:
<前>错误 [KafkaApi-1001] 活动代理的数量1"不满足偏移主题所需的复制因子3"(通过offsets.topic.replication.factor"配置).I am facing an issue while consuming message using the bootstrap-server i.e. Kafka server. Any idea why is it not able to consume messages without zookeeper?
- Kafka Version: kafka_2.11-1.0.0
- Zookeeper Version: kafka_2.11-1.0.0
- Zookeeper Host and port: zkp02.mp.com:2181
- Kafka Host and port: kfk03.mp.com:9092
Producing some message:
[kfk03.mp.com ~]$ /bnsf/kafka/bin/kafka-console-producer.sh --broker-list kfk03.mp.com:9092 --topic test
>hi
>hi
Consumer not able to consume messages if I give –-bootstrap-server
:
[kfk03.mp.com ~]$
/bnsf/kafka/bin/kafka-console-consumer.sh --bootstrap-server kfk03.mp.com:9092 --topic test --from-beginning
Consumer able to consume messages when --zookeeper
server is given instead of --bootstrap-server
-:
[kfk03.mp.com ~]$ /bnsf/kafka/bin/kafka-console-consumer.sh --zookeeper zkp02.mp.com:2181 --topic test --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}
{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}
{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}
hi
{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}
{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}
{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}
{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}
{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}
hi
hi
uttam
hi
hi
hi
hello
hi
^CProcessed a total of 17 messages
While consuming messages from kafka using bootstrap-server parameter, the connection happens via the kafka server instead of zookeeper. Kafka broker stores offset details in __consumer_offsets topic.
Check if __consumer_offsets is present in your topics list. If it's not there, check kafka logs to find the reason.
We faced a similar issue. In our case the __consumer_offsets was not created because of the following error:
ERROR [KafkaApi-1001] Number of alive brokers '1' does not meet the required replication factor '3' for the offsets topic (configured via 'offsets.topic.replication.factor').
这篇关于Kafka 消费者无法使用引导服务器名称消费消息的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!