如何设置自动缩放 RabbitMQ 集群 AWS [英] How to set up autoscaling RabbitMQ Cluster AWS

问题描述

我正在尝试从 SQS 转移到 RabbitMQ 以进行消息传递服务.我希望构建一个稳定的高可用性排队服务.现在我要使用集群.

I'm trying to move away from SQS to RabbitMQ for messaging service. I'm looking to build a stable high availability queuing service. For now I'm going with cluster.

当前实施，我有三台带有 RabbitMQ 的 EC2 机器，在 AMI 中安装了管理插件，然后我明确地转到每台机器并添加

Current Implementation , I have three EC2 machines with RabbitMQ with management plugin installed in a AMI , and then I explicitly go to each of the machine and add

sudo rabbitmqctl join_cluster rabbit@<hostnameOfParentMachine>

HA 属性设置为 all 并且同步工作.并且在它上面有一个负载均衡器，并分配了 DNS.到目前为止，这件事奏效了.

With HA property set to all and the synchronization works. And a load balancer on top it with a DNS assigned. So far this thing works.

预期实施:创建一个自动扩展的集群环境，在该环境中，上升/下降的机器必须动态加入/移除集群.实现这一目标的最佳方法是什么?请帮忙.

Expected Implementation: Create an autoscaling clustered environment where the machines that go Up/Down has to join/Remove the cluster dynamically. What is the best way to achieve this? Please help.

推荐答案

2 年前我也有类似的配置.

I had a similar configuration 2 years ago.

我决定使用 amazon VPC，默认情况下我的设计总是有两个 RabbitMQ 实例运行，并在集群中配置(称为主节点).rabbitmq 集群支持 内部亚马逊负载均衡器.

I decided to use amazon VPC, by default my design had two RabbitMQ instances always running, and configured in cluster (called master-nodes). The rabbitmq cluster was behind an internal amazon load balancer.

我创建了一个配置了 RabbitMQ 和管理插件的 AMI(称为master-AMI")，然后我配置了自动缩放规则.

I created an AMI with RabbitMQ and management plug-in configured (called "master-AMI"), and then I configured the autoscaling rules.

如果发出自动缩放警报，则会启动新的主 AMI.此 AMI 第一次执行时执行以下脚本:

if an autoscaling alarm is raised a new master-AMI is launched. This AMI executes the follow script the first time is executed:

#!/usr/bin/env python
import json
import urllib2,base64

if __name__ == '__main__':
    prefix =''
    from subprocess import call
    call(["rabbitmqctl", "stop_app"])
    call(["rabbitmqctl", "reset"])
    try:
        _url = 'http://internal-myloadbalamcer-xxx.com:15672/api/nodes'
        print prefix + 'Get json info from ..' + _url
        request = urllib2.Request(_url)

        base64string = base64.encodestring('%s:%s' % ('guest', 'guest')).replace('
', '')
        request.add_header("Authorization", "Basic %s" % base64string)
        data = json.load(urllib2.urlopen(request))
        ##if the script got an error here you can assume that it's the first machine and then 
        ## exit without controll the error. Remember to add the new machine to the balancer
        print prefix + 'request ok... finding for running node'


        for r in data:
            if r.get('running'):
                print prefix + 'found running node to bind..'
                print prefix + 'node name: '+ r.get('name') +'- running:' + str(r.get('running'))
                from subprocess import call
                call(["rabbitmqctl", "join_cluster",r.get('name')])
                break;
        pass
    except Exception, e:
        print prefix + 'error during add node'
    finally:
        from subprocess import call
        call(["rabbitmqctl", "start_app"])


    pass

脚本使用 HTTP APIhttp://internal-myloadbalamcer-xxx.com:15672/api/nodes"找到节点，然后选择一个并将新的AMI绑定到集群.

The scripts uses the HTTP API "http://internal-myloadbalamcer-xxx.com:15672/api/nodes" to find nodes, then choose one and binds the new AMI to the cluster.

作为 HA 策略，我决定使用它:

As HA policy I decided to use this:

rabbitmqctl set_policy ha-two "^two." ^
   "{""ha-mode"":""exactly"",""ha-params"":2,"ha-sync-mode":"automatic"}"

好吧，加入相当"容易，问题是决定何时可以从集群中删除节点.

Well, the join is "quite" easy, the problem is decide when you can remove the node from the cluster.

您不能根据自动缩放规则删除节点，因为您可以将消息发送到您必须使用的队列.

You can’t remove a node based on autoscaling rule, because you can have messages to the queues that you have to consume.

我决定执行一个定期运行到两个主节点实例的脚本:

I decided to execute a script periodically running to the two master-node instances that:

• 通过 API http://node:15672/api/queues
• 如果所有队列的消息计数为零，我可以从负载均衡器中删除该实例，然后从 rabbitmq 集群中删除.

我大致就是这么干的，希望能帮到你.

This is broadly what I did, hope it helps.

我编辑了答案，因为有这个插件可以提供帮助:

I edited the answer, since there is this plugin that can help:

我建议看这个:https://github.com/rabbitmq/rabbitmq-autocluster

插件已移至RabbitMQ官方仓库，可以轻松解决此类问题

The plugin has been moved to the official RabbitMQ repository, and can easly solve this kind of the problems

这篇关于如何设置自动缩放 RabbitMQ 集群 AWS的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！