Apache Zeppelin &Spark Streaming:Twitter 示例仅适用于本地 [英] Apache Zeppelin & Spark Streaming: Twitter Example only works local

问题描述

我刚刚从

屏幕截图:集群设置(不起作用！)

作业似乎在集群模式下正确运行:

解决方案

我尝试了 2 天后得到了它！

本地 Zeppelin Spark 解释器和 Spark Cluster 之间的区别似乎是，本地解释器包含了执行 Twitter Streaming 示例所需的 Twitter Utils，而 Spark Cluster 默认没有这个库.

因此，在以 Spark 集群为 master 启动应用程序之前，您必须在 Zeppelin Notebook 中手动添加依赖项.所以 Notebook 的第一段必须是:

%depz.resetz.load("org.apache.spark:spark-streaming-twitter_2.10:1.5.1")

如果在运行这一段时发生错误，只需尝试通过 ./bin/zeppelin-daemon.sh stop (& start) 重新启动 Zeppelin 服务器！

I just added the example project to my Zeppelin Notebook from http://zeppelin-project.org/docs/tutorial/tutorial.html (section "Tutorial with Streaming Data"). The problem I now have is that the application seems only to work local. If I change the Spark interpreter setting "master" from "local[*]" to "spark://master:7077" the application won't bring any result anymore when I'm doing the same SQL statement. Am I doing anything wrong? I already restarted the Zeppelin interpreter, also the whole Zeppelin daemon and the Spark cluster, nothing solved the issue! Can someone help.

I use the following installation:

• Spark 1.5.1 (prebuild for Hadoop 2.6+), Master + 2x Slaves
• Zeppelin 0.5.5 (installed on Spark's master node)

EDIT Also the following installation won't work for me:

• Spark 1.5.0 (prebuild for Hadoop 2.6+), Master + 2x Slaves
• Zeppelin 0.5.5 (installed on Spark's master node)

Screenshot: local setting (works!)

Screenshot: cluster setting (won't work!)

The job seems to run correctly in cluster mode:

解决方案

I got it after 2 days of trying around!

The difference between the local Zeppelin Spark interpreter and the Spark Cluster seems to be, that the local one has included the Twitter Utils which are needed for executing the Twitter Streaming example, and the Spark Cluster doesn't have this library by default.

Therefore you have to add the dependency manually in the Zeppelin Notebook before starting the application with Spark cluster as master. So the first paragraph of the Notebook must be:

%dep
z.reset
z.load("org.apache.spark:spark-streaming-twitter_2.10:1.5.1")

If an error occures on running this paragraph, just try to restart the Zeppelin server via ./bin/zeppelin-daemon.sh stop (& start)!

这篇关于Apache Zeppelin &Spark Streaming:Twitter 示例仅适用于本地的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！