具有Docker的多节点Hadoop集群 [英] Multi-node Hadoop cluster with Docker

问题描述

我正在基于 Docker 的环境中的多节点 Hadoop 集群的计划阶段。所以它应该是基于一个轻量级易于使用的虚拟化系统。
当前架构（关于文档）包含1个主节点和3个从节点。该主机使用 HDFS 文件系统和 KVM 进行虚拟化。
整个云由 Cloudera Manager 管理。在这个集群上安装了几个Hadoop模块。还有一个 NodeJS 数据上传服务。
这一次，我应该建立一个基于Docker 的架构。
我已经阅读了几个教程，并提出了一些意见，但也提出了一些问题。

I am in planning phase of a multi-node Hadoop cluster in a Docker based environment. So it should be based on a lightweight easy to use virtualized system. Current architecture (regarding to documentation) contains 1 master and 3 slave nodes. This host machine uses HDFS filesystem and KVM for virtualization. The whole cloud is managed by Cloudera Manager. There are several Hadoop modules installed on this cluster. There is also a NodeJS data upload service. This time I should make architecture Docker based. I have read several tutorials and have some opinions, but also open questions.

A。 a href =https://github.com/Lewuathe/docker-hadoop-cluster =nofollow> https://github.com/Lewuathe/docker-hadoop-cluster 是我的良好基础项目？我还发现了一个官方的图像，但它是单节点。

A. What do you think, is https://github.com/Lewuathe/docker-hadoop-cluster a good base for my project? I have found also an official image, but it is single-node.

B。如果我想在单个容器中进行更改，系统需求如何变化？这将是巨大的，因为这个架构应该在不同的地方工作，所以可以很容易地在这些地点之间转移变化。

B. How will system requirements change if I would like to make this in a single container? It would be great, because this architecture should work in different locations, so changes can be easily transferred between these locations. Synchronization between these so called clones would be important.

C。你有其他一些想法，也许最好的做法？

C. Do you have some other ideas, maybe best practices?

推荐答案

截至2016年9月，没有快速回答。

As of September 2016 there is no quick answer.

https://github.com/Lewuathe/docker-hadoop-cluster 似乎不是一个好的开始，因为对于您的B.选项应该是通用的

https://github.com/Lewuathe/docker-hadoop-cluster does not seem like a good start, as it should be universal for your B. option

关注 https://github.com/sequenceiq/hadoop-docker 和 https： //github.com/kiwenlau/hadoop-cluster-docker

这篇关于具有Docker的多节点Hadoop集群的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！