如何设置自动缩放RabbitMQ Cluster AWS [英] How to set up autoscaling RabbitMQ Cluster AWS

问题描述

我正在尝试从SQS转移到RabbitMQ进行消息传递服务。我正在寻找一个稳定的高可用性排队服务。现在我要去集群了。

当前实现，
我有三个EC2机器，RabbitMQ和管理插件安装在AMI中，然后我明确地每个机器并添加

  sudo rabbitmqctl join_cluster rabbit @< hostnameOfParentMachine> 
  

将HA属性设置为全部，同步工作。并在其上分配一个负载均衡器。到目前为止这件事情是有效的。

预期实施：创建自动缩放群集环境，其中上/下的计算机必须动态加入/删除群集。实现这一目标的最好方法是什么？请帮助。

解决方案

2年前我有一个类似的配置。

我决定使用亚马逊VPC ，默认情况下我的设计有两个RabbitMQ实例总是运行，并在集群中配置（称为主节点）。
该兔子群集位于内部亚马逊负载平衡器。

我用RabbitMQ和管理插件创建了一个AMI（称为master-AMI），然后配置了自动缩放规则。

如果出现自动缩放警报，则会启动新的主AMI。
此AMI执行第一次执行以下脚本：

 ＃！ usr / bin / env python 
 import json 
 import urllib2，base64 
 
 if __name__ =='__main__'：
 prefix =''
 from subprocess导入调用
调用（[rabbitmqctl，stop_app]）
调用（[rabbitmqctl，reset]）
 try：
 _url ='http： /internal-myloadbalamcer-xxx.com:15672/api/nodes'
打印前缀+'从..获取json信息'+ _url 
 request = urllib2.Request（_url）
 
 base64string = base64.encodestring（'％s：％s'％（'guest'，'guest'））。replace（'\\\
'，''）
 request.add_header（Authorization Basic％s％base64string）
 data = json.load（urllib2.urlopen（request））
 ##如果脚本在这里有错误，可以假定它是第一个机器，然后
 ##退出而不控制错误。记住将新机器添加到平衡器
打印前缀+'请求确定...查找运行节点'
 
 
在数据中的r：
如果r 
打印前缀+'发现运行节点绑定'
打印前缀+'节点名：'+ r.get（'name'）+' - 运行： '+ str（r.get（'running'））
从子进程导入调用
调用（[rabbitmqctl，join_cluster，r.get（'name'）]）
打破; 
 pass 
除了异常，e：
打印前缀+'添加节点期间的错误'
 finally：
从子进程导入调用
调用（[rabbitmqctl ，start_app]）
 
 
通过
  

脚本使用HTTP API http://internal-myloadbalamcer-xxx.com:15672/api / nodes 来查找节点，然后选择一个并将新的AMI绑定到集群。

作为HA政策，我决定使用这个：

  rabbitmqctl set_policy ha-two^ two\。 ^ 
{ha-mode：完全，ha-params：2，ha-sync-mode：自动}
  

嗯，加入很简单，问题是决定何时可以从集群中删除节点。 >



您不能根据自动缩放规则删除节点，因为您可以向您必须使用的队列发送消息。

我决定执行一个定期运行到两个主节点实例的脚本：

• 通过API < a href =http：// node：15672 / api / queues =noreferrer> http：// node：15672 / api / queues


• 所有队列的消息计数为零，我可以从负载均衡器中删除实例，然后从rabbitmq集群中删除该实例。

我做了什么，希望它有帮助。

I'm trying to move away from SQS to RabbitMQ for messaging service. I'm looking to build a stable high availability queuing service. For now I'm going with cluster.

Current Implementation , I have three EC2 machines with RabbitMQ with management plugin installed in a AMI , and then I explicitly go to each of the machine and add

sudo rabbitmqctl join_cluster rabbit@<hostnameOfParentMachine>

With HA property set to all and the synchronization works. And a load balancer on top it with a DNS assigned. So far this thing works.

Expected Implementation: Create an autoscaling clustered environment where the machines that go Up/Down has to join/Remove the cluster dynamically. What is the best way to achieve this? Please help.

解决方案

I had a similar configuration 2 years ago.

I decided to use amazon VPC, by default my design had two RabbitMQ instances always running, and configured in cluster (called master-nodes). The rabbitmq cluster was behind an internal amazon load balancer.

I created an AMI with RabbitMQ and management plug-in configured (called "master-AMI"), and then I configured the autoscaling rules.

if an autoscaling alarm is raised a new master-AMI is launched. This AMI executes the follow script the first time is executed:

#!/usr/bin/env python
import json
import urllib2,base64

if __name__ == '__main__':
    prefix =''
    from subprocess import call
    call(["rabbitmqctl", "stop_app"])
    call(["rabbitmqctl", "reset"])
    try:
        _url = 'http://internal-myloadbalamcer-xxx.com:15672/api/nodes'
        print prefix + 'Get json info from ..' + _url
        request = urllib2.Request(_url)

        base64string = base64.encodestring('%s:%s' % ('guest', 'guest')).replace('\n', '')
        request.add_header("Authorization", "Basic %s" % base64string)
        data = json.load(urllib2.urlopen(request))
        ##if the script got an error here you can assume that it's the first machine and then 
        ## exit without controll the error. Remember to add the new machine to the balancer
        print prefix + 'request ok... finding for running node'


        for r in data:
            if r.get('running'):
                print prefix + 'found running node to bind..'
                print prefix + 'node name: '+ r.get('name') +'- running:' + str(r.get('running'))
                from subprocess import call
                call(["rabbitmqctl", "join_cluster",r.get('name')])
                break;
        pass
    except Exception, e:
        print prefix + 'error during add node'
    finally:
        from subprocess import call
        call(["rabbitmqctl", "start_app"])


    pass

The scripts uses the HTTP API "http://internal-myloadbalamcer-xxx.com:15672/api/nodes" to find nodes, then choose one and binds the new AMI to the cluster.

As HA policy I decided to use this:

rabbitmqctl set_policy ha-two "^two\." ^
   "{""ha-mode"":""exactly"",""ha-params"":2,"ha-sync-mode":"automatic"}"

Well, the join is "quite" easy, the problem is decide when you can remove the node from the cluster.

You can’t remove a node based on autoscaling rule, because you can have messages to the queues that you have to consume.

I decided to execute a script periodically running to the two master-node instances that:

• checks the messages count through the API http://node:15672/api/queues
• if the messages count for all queue is zero, I can remove the instance from the load balancer and then from the rabbitmq cluster.

This is broadly what I did, hope it helps.

这篇关于如何设置自动缩放RabbitMQ Cluster AWS的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！