我应该选择哪种星火集群类型？ [英] Which cluster type should I choose for Spark?

问题描述

我是新来的Apache星火，我刚刚得知，桅杆支持3种集群的：

I am new to Apache Spark, and I just learned that Spars supports 3 types of cluster:

• 独立 - 这意味着星火将管理自己的集群


• 纱线 - 使用Hadoop的YARN资源管理器


• Mesos - Apache的专用资源经理的项目

由于我是新来的火花，我觉得我应该尝试的独立第一位。但我不知道哪一个是推荐的。再说了，今后我需要建立一个大型集群（数百个实例），我应该去哪个集群类型？

Since I am new to Spark, I think I should try Standalone first. But I wonder which one is the recommended. Say, in the future I need to build a large cluster (hundreds of instances), which cluster type should I go to?

推荐答案

我认为最好的回答是那些谁星火工作。所以，从学习星火

I think the best to answer that are those who work on Spark. So, from Learning Spark

有一个独立的集群开始，如果这是一个新的部署。
  独立模式是最简单的设置和将提供几乎所有
  相同的功能，作为其他群集管理器，如果你只
  运行的火花。

Start with a standalone cluster if this is a new deployment. Standalone mode is the easiest to set up and will provide almost all the same features as the other cluster managers if you are only running Spark.

如果您想与其他应用程序的运行星火，或者使用
  丰富的资源调度能力（例如队列），两纱和
  Mesos提供这些功能。这些中，纱线很可能会
  preinstalled在很多的Hadoop发行版。

If you would like to run Spark alongside other applications, or to use richer resource scheduling capabilities (e.g. queues), both YARN and Mesos provide these features. Of these, YARN will likely be preinstalled in many Hadoop distributions.

在两纱和独立模式Mesos的一个优点是它的
  细粒度共享选项，它可以让互动应用，
  作为星火外壳缩小命令之间的CPU分配。
  这使得在多个用户环境的吸引力
  运行交互式shell。

One advantage of Mesos over both YARN and standalone mode is its fine-grained sharing option, which lets interactive applications such as the Spark shell scale down their CPU allocation between commands. This makes it attractive in environments where multiple users are running interactive shells.

在所有的情况下，最好是作为HDFS同一节点上运行火花
  快速访问存储。您可以安装Mesos或独立
  在同一节点上的群集管理器手动，或大部分的Hadoop
  分布已经安装纱线和HDFS在一起。

In all cases, it is best to run Spark on the same nodes as HDFS for fast access to storage. You can install Mesos or the standalone cluster manager on the same nodes manually, or most Hadoop distributions already install YARN and HDFS together.

这篇关于我应该选择哪种星火集群类型？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！