星火Mesos调度 [英] Spark Mesos Dispatcher

问题描述

我的团队正在部署在Amazon云新大数据架构。我们有Mesos和运行星火作业。

My team is deploying a new Big Data architecture on Amazon Cloud. We have Mesos up and running Spark jobs.

我们是从同一个集群内的堡垒主机星火提交作业（即：罐）。然而这样做，堡垒主机是驱动程序，这就是所谓的的客户端模式（如果我理解正确的）。

We are submitting Spark jobs (i.e.: jars) from a bastion host inside the same cluster. Doing so, however, the bastion host is the driver program and this is called the client mode (if I understood correctly).

我们想尝试群集模式，但我们不知道从哪里开始调度过程。

We would like to try the cluster mode, but we don't understand where to start the dispatcher process.

文档说在集群中启动它，但我很困惑，因为我们的主人没有安装星火做的，我们用动物园管理员的主选。它启动从属节点上不是vailable选择，因为奴隶会失败，我们不希望暴露一个从IP或公共DNS的堡垒主机。

The documentation says to start it in the cluster, but I'm confused since our masters don't have Spark installed and we use Zookeeper for master election. Starting it on a slave node is not a vailable option, since slave can fail and we don't want to expose a slave ip or public DNS to the bastion host.

它是正确的，开始在堡垒主机上的调度员？

Is it correct to start the dispatcher on the bastion host?

非常感谢你。

推荐答案

文档不是很详细。
但是，我们是什么，我们发现相当满意：
根据文档，不支持集群模式Mesos集群（和对于Python应用程序）。

Documentation is not very detailed. However, we are quite happy with what we discovered: according to the documentation, cluster mode is not supported for Mesos clusters (and for Python applications).

但是，我们使用开始调度 - 主mesos：// ZK：// ...

有关递交申请，需要具备以下条件：

For submitting applications, you need the following:

spark-submit --deploy-mode cluster <other options> --master mesos://<dispatcher_ip>:7077 <ClassName> <jar>

如果你从一个堡垒机上运行这个命令，它不会工作，因为Mesos主机将寻找在作为堡垒相同的路径submitable罐子。我们最终露出文件作为一个可下载的URL。

If you run this command from a bastion machine, it won't work, because the Mesos master will look for the submitable jar in the same path as the bastion. We ended exposing the file as a downloadable URL.

希望这有助于

这篇关于星火Mesos调度的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！