具有Apache Spark的Docker容器在独立集群模式下 [英] Docker Container with Apache Spark in standalone cluster mode
问题描述
我正在尝试构建一个包含Apache Spark的docker映像。 IT建立在openjdk-8-jre官方映像上。
I am trying to construct a docker image containing Apache Spark. IT is built upon the openjdk-8-jre official image.
目标是在群集模式下执行Spark，因此至少有一个主站（通过 sbin / start-master.sh ）和一个或多个从站（ sbin / start-slave.sh ）。有关我的Docker文件和入口脚本，请参阅 spark-standalone-docker 。
The goal is to execute Spark in cluster mode, thus having at least one master (started via sbin/start-master.sh) and one or more slaves (sbin/start-slave.sh). See spark-standalone-docker for my Dockerfile and entrypoint script.
构建本身实际上是通过的，问题是当我想要运行容器时，它会在不久之后启动和停止。原因是Spark master启动脚本以守护进程模式启动主服务器并退出。因此，容器终止，因为没有进程在前台运行了。
The build itself actually goes through, the problem is that when I want to run the container, it starts and stops shortly after. The cause is that Spark master launch script starts the master in daemon mode and exits. Thus the container terminates, as there is no process running in the foreground anymore.
明显的解决方案是在前台运行Spark主进程，但我无法弄清楚怎么（谷歌也没有任何东西）。我的解决方案解决方案是在Spark日志目录上运行 tails -f 。
The obvious solution is to run the Spark master process in foreground, but I could not figure out how (Google did not turn up anything either). My "workaround-solution" is to run tails -f on the Spark log directory.
因此，我的问题是：
- 如何在前台运行Apache Spark Master？
- 如果第一个不是可能/可行/任何，什么是保持容器活着（我真的不想使用无限循环和睡眠命令）的首选（即最佳实践）解决方案？
推荐答案
如何在前台运行Apache Spark Master？
How can you run Apache Spark Master in foreground?
您可以使用 spark-class 与 Master 。 / p>
You can use spark-class with Master.
bin/spark-class org.apache.spark.deploy.master.Master
和同样的工作人员：
bin/spark-class org.apache.spark.deploy.worker.Worker $MASTER_URL
如果你是寻找生产就绪解决方案你应该consi使用适当的主管，如 dumb-init 或 tini 。
If you're looking for production ready solution you should consider using a proper supervisor like dumb-init or tini.
这篇关于具有Apache Spark的Docker容器在独立集群模式下的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
