使用 Python 流式传输 Kafka :kafka.errors.NoBrokersAvailable: NoBrokersAvailable [英] Streaming Kafka with Python :kafka.errors.NoBrokersAvailable: NoBrokersAvailable

问题描述

我试过 本教程 并已成功使用在我工作场所的服务器上发布的 kafka 主题.我不是生产者，只是纯粹的消费者.但是，该教程中的代码是在终端中进行流式传输.现在我想用 Python 尝试它并将消息记录到文本文件(或类似的文件)中.

I have tried this tutorial and have successfully consumed kafka topics that are published on a server at my work place. I am not the producer, just purely a consumer. However, the code in that tutorial is to stream in a terminal. Now I want to try it with Python and record the messages into text file (or something of that sort).

这是我使用的代码，在阅读了更多线程和教程后(例如 此处):

This is the code I use, after reading a few more threads and tutorials (such as here):

from kafka import KafkaConsumer

bootstrap_servers = ['xxxxxx:xxxx']
topicName: = 'my_topic_name'
consumer = KafkaConsumer(topicName, group_id='group1', bootstrap_servers=bootstrap_servers, consumer_timeout_ms=1000)

for msg in consumer:
    print(msg.value)

这里我想先把消息打印出来.但我在 1000 毫秒超时后收到此错误:

Here I want to first print out the message. But I get this error after 1000ms timeout:

kafka.errors.NoBrokersAvailable: NoBrokersAvailable

这对我来说听起来很合乎逻辑，因为需要经纪人，而上面的代码似乎对经纪人没有任何作用.

which sounds logical to me, since a broker is needed and the code above does not seem to do anything with a broker.

如果我不设置 consumer_timeout_ms=1000ms，Python 控制台就会卡住而不显示任何内容.

If I don't set the consumer_timeout_ms=1000ms, the Python console get just stuck without displaying anything.

我该如何解决这个问题?

How do I resolve this?

更多详情:我正在并行执行以下操作:

More details: I am doing the following in parallel:

1 - 在一个终端中运行 zookeeper

1 - Run zookeeper in one terminal

2 - 在另一个终端运行 kafka 集群

2 - Run kafka cluster in another terminal

3 - 使用命令 kafka-consumer-console

4 - 在另一个终端中，运行此问题中的 Python 代码.

4 - In another terminal, run the Python code in this question.

所有这些终端都是 WLS2 (Windows) 中的 Ubuntu.

All of these terminals are Ubuntu in WLS2 (Windows).

推荐答案

如果您能够将 WSL 终端与 kafka-console-consumer 一起使用，那么在那里运行 Python 代码/em> 应该是一样的.

If you're able to use the WSL terminal with kafka-console-consumer, then running Python code there should work the same.

如果您连接到远程 Kafka 服务器，则 WSL2 网络设置可能根本无法访问该地址.(其他地方的多个问题都在谈论 WSL2 并且没有外部互联网访问).因此，您真的应该考虑在 Windows 主机本身上运行 Python 代码.否则，听起来您需要调整网络配置.

If you're connecting to a remote Kafka server, chances are the WSL2 network settings are simply not able to reach that address. (multiple issues elsewhere talk about WSL2 and not having external internet access) . Therefore, you should really consider running Python code on the Windows host itself. Otherwise, sounds like you'll need to adjust your network configurations.

for 循环将等待新 消息发送到主题，在您向消费者添加另一个参数告诉它之前不会读取现有数据

The for loop will wait for new messages to a topic, not read existing data until you add another parameter to the consumer to tell it to

FWIW，你可以使用 kafka-console-consumer ... >>file.txt 写入文件

FWIW, you can use kafka-console-consumer ... >> file.txt to write out to a file

这篇关于使用 Python 流式传输 Kafka :kafka.errors.NoBrokersAvailable: NoBrokersAvailable的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！