如何配置多节点Apache Storm集群 [英] How to configure multi-node Apache Storm cluster
问题描述
我正在关注 http://jayatiatblogs.blogspot.com/2011/11/storm-installation.html & http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin .html#sc_zkMulitServerSetup 以便在AWS EC2的Ubuntu 14.04 LTS中设置Apache Storm集群.
I'm following http://jayatiatblogs.blogspot.com/2011/11/storm-installation.html & http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_zkMulitServerSetup to set up Apache Storm cluster in Ubuntu 14.04 LTS at AWS EC2.
我的主节点是10.0.0.185. 我的从属节点是10.0.0.79、10.0.0.124&在其Zookeeper数据中的myid分别为1、2和3的10.0.0.84.我建立了一个由所有3个从属节点组成的Apache Zookeeper集成体.
My master node is 10.0.0.185. My slave nodes are 10.0.0.79, 10.0.0.124 & 10.0.0.84 with myid of 1, 2 and 3 in their zookeeper-data respectively. I set up an ensemble of Apache Zookeeper consists of all the 3 slave nodes.
以下是我的从属节点的zoo.cfg:
Below are my zoo.cfg for my slave nodes:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/home/ubuntu/zookeeper-data
clientPort=2181
server.1=10.0.0.79:2888:3888
server.2=10.0.0.124:2888:3888
server.3=10.0.0.84:2888:3888
autopurge.snapRetainCount=3
autopurge.purgeInterval=1
下面是我的从属节点的storm.yaml:
Below are my storm.yaml for my slave nodes:
########### These MUST be filled in for a storm configuration
storm.zookeeper.server:
- "10.0.0.79"
- "10.0.0.124"
- "10.0.0.84"
# - "localhost"
storm.zookeeper.port: 2181
# nimbus.host: "localhost"
nimbus.host: "10.0.0.185"
storm.local.dir: "/home/ubuntu/storm/data"
java.library.path: "/usr/lib/jvm/java-7-oracle"
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
- 6704
#
# worker.childopts: "-Xmx768m"
# nimbus.childopts: "-Xmx512m"
# supervisor.childopts: "-Xmx256m"
#
# ##### These may optionally be filled in:
#
## List of custom serializations
# topology.kryo.register:
# - org.mycompany.MyType
# - org.mycompany.MyType2: org.mycompany.MyType2Serializer
#
## List of custom kryo decorators
# topology.kryo.decorators:
# - org.mycompany.MyDecorator
#
## Locations of the drpc servers
# drpc.servers:
# - "server1"
# - "server2"
## Metrics Consumers
# topology.metrics.consumer.register:
# - class: "backtype.storm.metric.LoggingMetricsConsumer"
# parallelism.hint: 1
# - class: "org.mycompany.MyMetricsConsumer"
# parallelism.hint: 1
# argument:
# - endpoint: "metrics-collector.mycompany.org"
下面是我的 master 节点的storm.yaml:
Below are the storm.yaml for my master node:
########### These MUST be filled in for a storm configuration
storm.zookeeper.servers:
- "10.0.0.79"
- "10.0.0.124"
- "10.0.0.84"
# - "localhost"
#
storm.zookeeper.port: 2181
nimbus.host: "10.0.0.185"
# nimbus.thrift.port: 6627
# nimbus.task.launch.secs: 240
# supervisor.worker.start.timeout.secs: 240
# supervisor.worker.timeout.secs: 240
ui.port: 8772
# nimbus.childopts: "‐Xmx1024m ‐Djava.net.preferIPv4Stack=true"
# ui.childopts: "‐Xmx768m ‐Djava.net.preferIPv4Stack=true"
# supervisor.childopts: "‐Djava.net.preferIPv4Stack=true"
# worker.childopts: "‐Xmx768m ‐Djava.net.preferIPv4Stack=true"
storm.local.dir: "/home/ubuntu/storm/data"
java.library.path: "/usr/lib/jvm/java-7-oracle"
# supervisor.slots.ports:
# - 6700
# - 6701
# - 6702
# - 6703
# - 6704
# worker.childopts: "-Xmx768m"
# nimbus.childopts: "-Xmx512m"
# supervisor.childopts: "-Xmx256m"
# ##### These may optionally be filled in:
#
## List of custom serializations
# topology.kryo.register:
# - org.mycompany.MyType
# - org.mycompany.MyType2: org.mycompany.MyType2Serializer
#
## List of custom kryo decorators
# topology.kryo.decorators:
# - org.mycompany.MyDecorator
#
## Locations of the drpc servers
# drpc.servers:
# - "server1"
# - "server2"
## Metrics Consumers
# topology.metrics.consumer.register:
# - class: "backtype.storm.metric.LoggingMetricsConsumer"
# parallelism.hint: 1
# - class: "org.mycompany.MyMetricsConsumer"
# parallelism.hint: 1
# argument:
# - endpoint: "metrics-collector.mycompany.org"
我在所有从属节点中启动Zookeeper,然后在主节点中启动Storm Nimbus,然后在所有从属节点中启动Storm Supervisor.但是,当我在自己的Storm UI中查看时,集群摘要&中只有1个主管,共有5个位置主管摘要中只有1个主管信息,为什么这样?
I start my zookeeper in all my slave nodes, then start my storm nimbus in my master node, then start storm supervisor in all my slave nodes. However, when I view in my Storm UI, there is only 1 supervisor with total 5 slots in the cluster summary & only 1 supervisor information in the supervisor summary, why so?
如果在这种情况下提交拓扑,实际上有多少个从属节点正在工作?
How many slave nodes is actually working if I submit a topology in this case?
为什么没有3个主管总共15个席位?
Why it is not 3 supervisors with total 15 slots?
要拥有3名主管该怎么办?
What should I do in order to have 3 supervisors?
当我在从属节点中签入supervisor.log时,原因如下:
When I check in the supervisor.log in the slave nodes, the causes is as below:
2015-05-29T09:21:24.185+0000 b.s.d.supervisor [INFO] 5019754f-cae1-4000-beb4-fa0
16bd1a43d still hasn't started
推荐答案
您所做的一切都很完美,而且效果很好.
What you are doing perfect and its works too.
您唯一需要更改的是storm.dir
.在从属节点和主节点中相同,只是在nimbus&中的storm.dir
路径中更改路径.主管节点(不要使用相同的本地路径).当您使用相同的本地路径时,灵气和主管共享相同的ID.他们开始发挥作用,但您看不到8个位置,它们只是向您显示4个位置作为工作人员.
The only thing you should change is your storm.dir
. It is same in the slave and the master nodes just change the path in the storm.dir
path in nimbus & supervisor nodes (don't use same local path). When you use same local path the nimbus and supervisor share same id. They come into play but you don’t see 8 slots they just show you 4 slots as workers.
更改(storm.local.dir:/home/ubuntu/storm/data
),不要在超级用户和雨云台中使用相同的路径.
Change the (storm.local.dir:/home/ubuntu/storm/data
) and don`t use same path in supervisor and nimbus.
这篇关于如何配置多节点Apache Storm集群的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!