Hadoop 的 HDFS 与 Spark [英] Hadoop's HDFS with Spark

问题描述

我是集群计算的新手，我正在尝试在 Spark 中设置一个最小的 2 节点集群.我仍然有点困惑:我是否必须先设置完整的 Hadoop 安装，或者 Spark 是否附带包含其中的 Hadoop 版本?

I am new to cluster-computing and I am trying to set up a minimal 2-node cluster in Spark. What I am still a bit confused about: Do I have to set up a full Hadoop installation first oder does Spark ship with an included Hadoop version inside?

我发现的有关 Spark 的内容并没有真正说明这一点.我知道 Spark 是对 Hadoop 的扩展而不是替代它，但我不清楚它是否需要独立运行的 Hadoop 系统.

The stuff I find about Spark does not really make it this clear. I understood that Spark is meant as an extension to Hadoop rather than replacing it, but if it requires an independently running Hadoop system does not get clear to me.

我需要一个 HDFS，所以只使用 Hadoop 的文件系统部分就足够了吗?

I require a HDFS, is it thus enough to just use the file-system part of Hadoop?

有人可以向我指出这个可能很明显的事情吗?

Could someone point this probably obvious thing out to me?

推荐答案

Apache Spark 独立于 Hadoop.Spark 允许您使用不同的数据源(包括 HDFS)，并且能够在独立集群中运行，或使用现有的资源管理框架(例如 YARN、Mesos).

Apache Spark is independent from Hadoop. Spark allows you to use different sources of data (incl. HDFS) and is capable of running either in a standalone cluster, or using an existing resource management framework (eg. YARN, Mesos).

所以如果您只对 Spark 感兴趣，则无需安装 Hadoop.

So if you're only interested in Spark, there is no need to install Hadoop.

这篇关于Hadoop 的 HDFS 与 Spark的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！