在WSL2中通过IDE连接到Kafka服务器时出错 [英] Error connecting to kafka server via IDE in WSL2
问题描述
我无法通过在Windows上运行的intellij或vscode连接到在ubuntu上运行的kafka服务器(我在WSL2上尝试过的第一台服务器).我什至尝试使用VM的IP，但没有运气.据我了解，按照该文档，我们应该能够使用本地主机"进行连接
我遇到了同样的问题，很难解决.当我发现WSL2中的这个已关闭但显然仍未解决的问题时，突破就来了.基本上，发生的事情是我无法从Windows 10中的IntelliJ访问Ubuntu/WSL2的本地主机.因此，当我在Intellij中编译并运行程序时，它给了我您所发布的错误.
我的设置的一些详细信息:
操作系统:Windows 10，2004版(操作系统内部版本19041.630)
我的build.sbt:
scalaVersion:="2.12.10&";libraryDependencies + ="org.apache.spark"%%"spark-sql"％"3.0.1"libraryDependencies + ="org.apache.spark"%%火花流"％"3.0.1"libraryDependencies + ="org.apache.bahir"%%"spark-streaming-twitter"％"2.4.0&"libraryDependencies + ="org.apache.httpcomponents"％"httpclient"％"4.5.12"；libraryDependencies + ="commons-io";％"commons-io"；％"2.8.0&"libraryDependencies + ="org.apache.spark"％"spark-sql-kafka-0-10_2.12"％"3.0.1" 这是我要运行的Scala代码，它读取一个主题(quickstart-events)并发布到另一个主题(aux-output):
软件包kafka导入org.apache.spark.sql.SparkSession对象kafkaRunner {def main(args:Array [String]):Unit = {val spark = SparkSession.builder().appName("Kafka First App").master("local [*]").getOrCreate()导入spark.implicits._val df =火花.readStream.format("kafka").option("kafka.bootstrap.servers"，"localhost:9092").option("subscribe"，"quickstart-events").加载()df.writeStream.format("kafka").option("kafka.bootstrap.servers"，"localhost:9092").option(主题"，辅助输出").option("checkpointLocation"，"/tmp/kafka-checkpoint").开始().awaitTermination()}} 我已经运行了很多次程序，并且重新开始，我删除了zookeeper和kafkas/tmp文件.我不知道这些文件的价值，因此请谨慎行事.我删除了这三个目录:
- /tmp/kafka-logs
- /tmp/zookeeper
- /tmp/kafka-checkpoint(这是我在程序中设置的目录，您的目录可能有所不同，但是当我未设置此目录时，Spark引发了错误).
接下来，我从ubuntu的Kafka目录中运行了这些命令.每个都在单独的终端窗口中.
[终端窗口1-Zookeeper服务器]bin/zookeeper-server-start.sh config/zookeeper.properties[终端窗口2-kafka服务器，**要等到Zookeeper完成加载后再运行**]bin/kafka-server-start.sh config/server.properties[终端窗口3-创建我们的2个主题，然后对** quickstart **运行生产者](以下三个命令在同一窗口中运行)bin/kaftopics.sh --create.sh --topic quickstart-events --bootstrap-server localhost:9092bin/kaftopics.sh --create.sh --topic aux-output --bootstrap-server本地主机:9092bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092[终端窗口4-为** quickstart **频道创建使用者"bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092[终端窗口5-为** aux-out **频道创建使用者]bin/kafka-console-consumer.sh --topic aux-out --from-beginning --bootstrap-server localhost:9092[终端窗口6-用于运行sbt] 我花时间在生产者中输入一些行(第3窗口)，并在快速入门消费者中寻找输出(第4窗口).辅助输出(第5窗口)中不应显示任何内容，该程序将在sbt中运行时生成.
然后我运行了程序.我没有从Windows中移动项目，而是导航到Ubuntu(/mnt/c/User/me/lots/of/dir/kafkaProject)中的Windows目录.我从 build.sbt 在目录中启动 sbt .加载 sbt 后，我将编译"并运行"
它开始像火花作业一样处理，但是随后文本开始飞走.在这种情况下，您应该在辅助输出中的 quickstart 主题输出中看到您的输入.
在程序运行时，在窗口3的生产者中输入的文本应显示在4和5中.
我没有提到的一件事是，尝试并多次运行该程序失败后，我确实执行了"wsl.exe --shutdown"操作.并重新启动了我所有的窗口以进行干净的启动.如果收到说主题丢失的错误，请尝试更改主题名称并重新开始.我发现有时我以前使用过且无法正常工作的主题已损坏.我相信还有一些我暂时没有发现的临时文件正在缓存主题，但是一旦工作，我便继续前进.
祝你好运！
I'm not able to connect to a kafka server(first server I tried on WSL2) running on ubuntu, via intellij or vscode running on windows. I even tried using the VM's IP, but no luck. As I understand, we should be able to connect using 'localhost' as per this doc https://docs.microsoft.com/en-us/windows/wsl/compare-versions am I missing something?
Here is my code
Properties producerProperties = new Properties();
producerProperties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
producerProperties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
producerProperties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
KafkaProducer<String, String> producer = new KafkaProducer<String, String>(producerProperties);
ProducerRecord<String, String> record = new ProducerRecord<>("topic_1", "hello from java");
producer.send(record);
producer.flush();
producer.close();
And Here's the error
I had the same problem, it was quite difficult to solve. The breakthrough came when I found this closed, but apparently still broken, issue in WSL2. Basically, what was happening is I could not access Ubuntu/WSL2's localhost from IntelliJ in Windows 10. So, when I compiled and ran my program in Intellij, it gave me the error you posted.
Some details of my setup:
OS: Windows 10, version 2004 (OS Build 19041.630)
My build.sbt:
scalaVersion := "2.12.10"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.0.1"
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "3.0.1"
libraryDependencies += "org.apache.bahir" %% "spark-streaming-twitter" % "2.4.0"
libraryDependencies += "org.apache.httpcomponents" % "httpclient" % "4.5.12"
libraryDependencies += "commons-io" % "commons-io" % "2.8.0"
libraryDependencies += "org.apache.spark" % "spark-sql-kafka-0-10_2.12" % "3.0.1"
This is the Scala code I am trying to run, it reads in one topic (quickstart-events) and posts to another (aux-output):
package kafka
import org.apache.spark.sql.SparkSession
object kafkaRunner {
def main(args: Array[String]): Unit = {
val spark = SparkSession.builder()
.appName("Kafka First App")
.master("local[*]")
.getOrCreate()
import spark.implicits._
val df = spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", "quickstart-events")
.load()
df
.writeStream
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("topic", "aux-output")
.option("checkpointLocation", "/tmp/kafka-checkpoint")
.start()
.awaitTermination()
}
}
I had already run the program many times and to have a fresh start I deleted zookeeper and kafkas /tmp files. I don't know how valuable these files are, so please proceed with caution. I deleted these three directories:
- /tmp/kafka-logs
- /tmp/zookeeper
- /tmp/kafka-checkpoint (This was a directory I set in my program, yours may be different, but Spark threw an error when I did not set this).
Next, I ran these commands from my Kafka directory in ubuntu. Each in a separate terminal window.
[terminal window 1 - zookeeper server]
bin/zookeeper-server-start.sh config/zookeeper.properties
[terminal window 2 - kafka server, **wait until zookeeper finishes loading before running**]
bin/kafka-server-start.sh config/server.properties
[terminal window 3 - create our 2 topics, then run the producer for **quickstart**]
(the following three commands were run in the same window)
bin/kaftopics.sh --create.sh --topic quickstart-events --bootstrap-server localhost:9092
bin/kaftopics.sh --create.sh --topic aux-output --bootstrap-server localhost:9092
bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092
[terminal window 4 - create a consumer for **quickstart** channel]
bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092
[terminal window 5 - create a consumer for **aux-out** channel]
bin/kafka-console-consumer.sh --topic aux-out --from-beginning --bootstrap-server localhost:9092
[terminal window 6 - use to run sbt]
I took this time to type some lines into the producer (window 3) and looked for output in the quickstart consumer (window 4). Nothing should show in aux-out (window 5), this will be generated when the program is run in sbt.
Then I ran my program. Rather than moving my project from Windows, I navigated to the windows directory in Ubuntu (/mnt/c/User/me/lots/of/dir/kafkaProject). I started sbt in the directory with build.sbt. Once sbt loaded I 'compile' and 'run'
It starts processing like a spark job, but then text starts to fly by. This is when you should see your input from quickstart topic output in aux-out.
Text entered in window 3's producer should show up in 4 and 5 while the program is running.
One thing I didn't mention is that after trying and failing to run the program a few times, I did do a "wsl.exe --shutdown" and restarted all my windows for a clean start. If you get errors that say the topic is missing, try changing the topic names and starting again. I found that sometimes topics I had used before and that didn't work were corrupted. I believe there is some other temporary file that I haven't discovered yet that is caching the topics, but I moved on once I got it working.
Good luck!
这篇关于在WSL2中通过IDE连接到Kafka服务器时出错的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
