消息队列的有效架构PHP中的辅助系统? [英] Valid Architecture for a Message Queue & Worker System in PHP?

问题描述

我想把我想在PHP应用程序中实现的消息队列模型和作业包起来:

I'm trying to wrap my head around the message queue model and jobs that I want to implement in a PHP app:

• 我的目标是卸载需要发送到多个第三方API的消息/数据，因此访问它们不会降低客户端的速度.因此，将数据发送到消息队列是理想的.

• My goal is to offload messages / data that needs to be sent to multiple third party APIs, so accessing them doesnt slow down the client. So sending the data to a message queue is ideal.

我考虑只使用Gearman来保存MQ/作业，但我想使用诸如SQS或Rackspace Cloud Queues这样的Cloud Queue服务，这样我就不必管理消息.

I considered using just Gearman to hold the MQ/Jobs, but I wanted to use a Cloud Queue service like SQS or Rackspace Cloud Queues so i wouldnt have to manage the messages.

以下是我认为应该做的事情的图示:

Here's a diagram of what I think I should do:

问题:

• 我的工作者，将用PHP编写，他们都必须轮询云队列服务吗?这可能会变得昂贵，尤其是当您有很多工人时.

• My workers, would be written in PHP they all have to be polling the cloud queue service? that could get expensive especially when you have a lot of workers.

我当时想也许有1个工作人员只是为了轮询队列，如果有消息，通知其他工作人员他们有工作，我只需要使用supervisord保持这1个工作人员在线?这种轮询方法比使用可以通知的MQ更好吗?我应该如何每秒轮询一次或尽快轮询MQ?然后如果我发现投票速度放慢了，就增加投票员吗?

I was thinking maybe have 1 worker just for polling the queue, and if there are messages, notify the other workers that they have jobs, i just have to keep this 1 worker online using supervisord perhaps? is this polling method better than using a MQ that can notify? How should I poll the MQ, once every second or as fast as it can poll? and then increase the polling workers if I see it slowing down?

我还考虑对所有消息使用一个队列，然后进行工作人员监视，根据需要在何处处理消息，将消息分发到其他云MQ，因为可能需要处理1条消息2个差异工人.

I was also thinking of having a single queue for all the messages, then the worker monitoring that distributes the messages to other cloud MQs depending on where they need to be processed, since 1 message might need to be processed by 2 diff workers.

我是否仍需要gearman来管理我的工人，还是可以只使用supervisord上下旋转工人?

Would I still need gearman to manage my workers or can I just use supervisord to spin workers up and down?

每发送一条消息与轮询MQ相比，向主工作程序发送一条通知不是更有效，更快捷吗?我假设我需要使用gearman通知我的主要工作人员该MQ有一条消息，因此它可以开始检查它.还是如果我每秒有300条消息，这将产生300个作业来检查MQ?

Isn't it more effective and faster to also send a notification to the main worker whenever a message is sent vs polling the MQ? I assume I would the need to use gearman to notify my main worker that the MQ has a message, so it can start checking it. or if I have 300 messages per second, this would generate 300 jobs to check the MQ?

基本上我该如何尽可能高效地检查MQ?

Basically how could I check the MQ as efficiently and as effectively as possible?

对我的体系结构的建议或更正?

Suggestions or corrections to my architecture?

推荐答案

我的建议基本上可以归结为:保持简单！

My suggestions basically boil down to: Keep it simple!

考虑到这一点，我的第一个建议是删除DispatcherWorker.根据我目前的理解，工作人员的唯一目的是听MAIN队列并将消息转发到不同的任务队列.您的应用程序应注意将正确的消息排入正确的队列(或主题).

With that in mind my first suggestion is to drop the DispatcherWorker. From my current understanding, the sole purpose of the worker is to listen to the MAIN queue and forward messages to the different task queues. Your application should take care of enqueuing the right message onto the right queue (or topic).

我的员工，将用PHP编写，他们都必须轮询云队列服务吗?这可能会变得昂贵，特别是当您有很多工人时.

My workers, would be written in PHP they all have to be polling the cloud queue service? that could get expensive especially when you have a lot of workers.

是的，没有免费的午餐.当然，您可以按天/周时间(如果您的用户在特定时间处于活动状态)，通过应用程序使用情况(当收到更多消息时，将提高轮询率)来适应和优化您的工作者轮询率.请记住，工程成本可能很快会比未优化的轮询更高.

Yes, there is no free lunch. Of course you could adapt and optimize your worker poll rate by application usage (when more messages arrive increase poll rate) by day/week time (if your users are active at specific times), and so on. Keep in mind that engineering costs might soon be higher than unoptimized polling.

相反，您可以考虑推送队列(请参见下文).

Instead, you might consider push queues (see below).

我当时想也许只有1名工人只是为了轮询队列，如果有消息，通知其他工人他们有工作，我只需要让这1名工人在主管的监督下在线?这种轮询方法比使用可以通知的MQ更好吗?我应该如何每秒轮询一次或以最快的速度轮询MQ?然后如果我发现投票速度放慢了，就增加投票员吗?

I was thinking maybe have 1 worker just for polling the queue, and if there are messages, notify the other workers that they have jobs, i just have to keep this 1 worker online using supervisord perhaps? is this polling method better than using a MQ that can notify? How should I poll the MQ, once every second or as fast as it can poll? and then increase the polling workers if I see it slowing down?

这听起来太复杂了.通信不可靠，但是有可靠的消息队列.如果您不想丢失数据，请坚持使用消息队列，不要发明自定义协议.

This sounds too complicated. Communication is unreliable, there are reliable message queues however. If you don't want to loose data, stick to the message queues and don't invent custom protocols.

我还考虑对所有消息使用一个队列，然后进行工作程序监视，根据需要在何处处理消息，将消息分发到其他云MQ，因为1条消息可能需要2个差异工作程序进行处理

I was also thinking of having a single queue for all the messages, then the worker monitoring that distributes the messages to other cloud MQs depending on where they need to be processed, since 1 message might need to be processed by 2 diff workers.

如上所述，应用程序应根据需要将您的消息排队到多个队列中.这样可以使事情简单明了.

As already mentioned, the application should enqueue your message to multiple queues as needed. This keeps things simple and in place.

我是否仍需要装备工来管理我的工人，还是可以只使用主管来上下旋转工人?

Would I still need gearman to manage my workers or can I just use supervisord to spin workers up and down?

消息队列太多，使用它们的方式也更多.通常，如果您使用的是民意测验队列，则需要让自己的工人活着.但是，如果您使用的是推送队列，则队列服务将调用您指定的端点.因此，您只需要确保您的工作人员有空即可.

There are so many message queues and even more ways to use them. In general, if you are using poll queues you'll need to keep your workers alive by yourself. If however you are using push queues, the queue service will call an endpoint specified by you. Thus you'll just need to make sure your workers are available.

基本上我该如何尽可能高效地检查MQ?

Basically how could I check the MQ as efficiently and as effectively as possible?

这取决于您的业务需求和您的员工所做的工作.哪个时间跨度很关键?秒，分，小时，天?如果您使用工作人员发送电子邮件，则无需花费数小时，最好是几秒钟.每3秒或每15秒轮询一次(对于用户)有什么区别吗?

This depends on your business requirements and the job your workers do. What time spans are critical? Seconds, Minutes, Hours, Days? If you use workers to send emails, it shouldn't take hours, ideally a couple of seconds. Is there a difference (for the user) between polling every 3 seconds or every 15 seconds?

我的目标是卸载需要发送到多个第三方API的消息/数据，因此访问它们不会降低客户端的速度.因此，将数据发送到消息队列是理想的.我考虑只使用Gearman来保存MQ/作业，但是我想使用诸如SQS或Rackspace Cloud Queues这样的Cloud Queue服务，这样我就不必管理消息.

My goal is to offload messages / data that needs to be sent to multiple third party APIs, so accessing them doesnt slow down the client. So sending the data to a message queue is ideal. I considered using just Gearman to hold the MQ/Jobs, but I wanted to use a Cloud Queue service like SQS or Rackspace Cloud Queues so i wouldnt have to manage the messages.

实际上，您描述的场景非常适合消息队列. 正如您提到的，您不想管理消息队列本身，也许您也不想管理工作人员?这是推送队列的地方.

Indeed the scenario you describe is a good fit for message queues. As you mentioned you don't want to manage the message queue itself, maybe you do not want to manage the workers either? This is where push queues pop in.

推送队列基本上是呼叫您的工作人员.例如，Amazon ElasticBeanstalk Worker Environments在后台执行繁重的工作(轮询)，只需使用包含队列消息的HTTP请求( Iron.io .

Push queues basically call your worker. For example, Amazon ElasticBeanstalk Worker Environments do the heavy lifting (polling) in the background and simply call your application with an HTTP request containing the queue message (refer to the docs for details). I have personally used the AWS push queues and have been happy with how easy they are. Note, that there are other push queue providers like Iron.io.

正如您所提到的，您正在使用PHP，Symfony有一个 QPush捆绑包，它可以处理传入消息请求.您可能会看一下推出自己的解决方案的代码.

As you mentioned you are using PHP, there is the QPush Bundle for Symfony, which handles incoming message requests. You may have a look at the code to roll your own solution.

这篇关于消息队列的有效架构PHP中的辅助系统?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！