在火花集群模式下运行 zeppelin [英] Running zeppelin on spark cluster mode
问题描述
我正在使用本教程 在 docker 容器中的纱线模式下火花集群 以纱线模式在火花集群中启动齐柏林飞艇.但是我被困在第 4 步.我在我的 docker 容器中找不到 conf/zeppelin-env.sh 来进行进一步的配置.我尝试将 zeppelin 的这些 conf 文件夹放入其中，但现在成功了.除了那个 zeppelin notebook 也没有在 localhost:9001 上运行.
I am using this tutorial spark cluster on yarn mode in docker container to launch zeppelin in spark cluster in yarn mode. However I am stuck at step 4. I can't find conf/zeppelin-env.sh in my docker container to put further configuration. I tried putting these conf folder of zeppelin but yet now successful. Apart from that zeppelin notebook is also not running on localhost:9001.
我对分布式系统很陌生，如果有人能帮我在纱线模式下在火花集群上启动 zeppelin，那就太好了.
I am very new to distributed system, it would be great if someone can help me start zeppelin on spark cluster in yarn mode.
这是我的 docker-compose 文件，用于启用与 Spark 集群的 zeppelin 对话.
Here is my docker-compose file to enable zeppelin talk with spark cluster.
version: '2'
services:
sparkmaster:
build: .
container_name: sparkmaster
ports:
- "8080:8080"
- "7077:7077"
- "8888:8888"
- "8081:8081"
- "8082:8082"
- "5050:5050"
- "5051:5051"
- "4040:4040"
zeppelin:
image: dylanmei/zeppelin
container_name: zeppelin-notebook
env_file:
- ./hadoop.env
environment:
ZEPPELIN_PORT: 9001
CORE_CONF_fs_defaultFS: "hdfs://namenode:8020"
HADOOP_CONF_DIR_fs_defaultFS: "hdfs://namenode:8020"
SPARK_MASTER: "spark://spark-master:7077"
MASTER: "yarn-client"
SPARK_HOME: spark-master
ZEPPELIN_JAVA_OPTS: >-
-Dspark.driver.memory=1g
-Dspark.executor.memory=2g
ports:
- 9001:9001
volumes:
- ./data:/usr/zeppelin/data
- ./notebooks:/usr/zeppelin/notebook
推荐答案
这是您用来启动独立 Spark 集群的 dockerfile.
this is the dockerfile you used to launch the standalone spark cluster.
但是容器内部没有Zeppelin实例，所以你必须在本地机器上使用Zeppelin.
But there is no Zeppelin instance inside the container, so you have to use Zeppelin on your local machine.
请下载并使用它.
这篇关于在火花集群模式下运行 zeppelin的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
