如何通过PHP将AWS SQS/SNS用作推送通知队列来执行繁重的处理任务? [英] How to use AWS SQS/SNS as a push notification queue for heavy processing tasks via PHP?

问题描述

我有一个在机架空间上运行的服务器，该服务器托管一个PHP Web应用程序.

I have a single server running on rackspace which is hosting a single PHP web app.

PHP Web应用程序将接受表单提交，然后需要根据表单字段条目执行任务.

The PHP web app will accept a form submission that then needs to perform a task based on the form field entries.

任务(将其称为生成元数据"任务)需要大量的处理时间.我想知道如何允许表单提交直接保存到数据库中，并在后台运行generate metadata任务时立即向用户显示成功页面.

The task (let's called it the generate metadata task) requires quite a lot of processing time. I was wondering how to allow the form submission to be a straightforward save to database and immediately show success page to user while running the generate metadata task in the background.

我已经使用composer将"aws/aws-sdk-php": "~3.11"安装到了同一Web应用程序中.

I have installed "aws/aws-sdk-php": "~3.11" using composer into the same web app.

我的计划最初是这样的:

My plan is initially this:

用于处理表单提交的代码

code that handles the form submission

$result = $model->save($_POST);
// this code will send the information to either SQS or SNS
$awsClient->sendsMessage($_POST);
if ($result) {
  $this->redirect('success.html');
}

我已阅读有关AWS声明的扇出架构的信息

I have read about the fanout architecture stated by AWS.

我对扇出体系结构示例的问题(据我了解)是

My issues with the fanout architecture example (as I understand it) are this:

1. 将消息发送到SQS或SNS的服务器也将是处理生成元数据任务的服务器.实际上，它是同一个Web应用程序.
2. SQS完成了队列部分(因为我想在FIFO中执行任务，并且完成任务确实需要很长时间).但是，这需要我的Web应用程序才能连续轮询SQS.我想要一个推送通知(从AWS到我的Web应用程序)，而不是我的Web应用程序连续轮询AWS以检查要执行的任务.

我发现了建议的可能解决方案

I found a possible solution suggested here

建议的解决方案是:

1. 将消息发送到SNS主题.

1. send the message to a SNS topic.

SNS主题将同时向SQS队列和我的Web应用程序发送消息.

The SNS topic will message both a SQS queue and my web app.

我的Web应用程序在被触发后，将轮询相同的SQS队列，该队列现在已连续将消息排队，直到队列为空

My web app, after being triggered, will poll the same SQS queue that has now queued the message continuously until the queue is empty

我从中看到的缺点是，我的Web应用程序将在队列本身收到消息之前轮询队列.

使用AWS服务实现推送队列的最佳方法是什么?

What is the best way to implement push queues using AWS services?

推荐答案

我的Web应用程序将在队列本身收到消息之前轮询队列

my web app will poll the queue before the queue itself has the message

那么您还没有尝试过，对吧? :)恐怕您想得太多了. SQS具有 long polling ，这会导致轮询请求在SQS端被挂起，直到至少有一条消息可用为止，这时将返回该消息(达到您请求的最大数目).您可以将较长的轮询等待时间设置为1到20秒.如果在此时间段内没有可用消息，则返回没有消息的响应.

You haven't tried it, then, right? :) I'm afraid you're overthinking this. SQS has long polling, which causes the poll request to be suspended at the SQS side until at least one message is available, at which point that message (up to the max number you requested) will be returned. You can set the long poll wait time from 1 to 20 seconds. If no messages are available within this time frame, the response is returned with no messages.

如果您响应SNS的通知而轮询队列，则 如果您使用长时间轮询，则会在此处查找消息.邮件可能会延迟，但高度不太可能.

If you poll the queue in response to the notification from SNS, you will find messages there if you use long polling. It's possible for messages to be delayed, but highly unlikely.

但是，另一个问题是您断言您不希望该应用程序不断轮询SQS.我经常遇到这个异议，而且它经常放错地方.使用SQS长轮询，恒定"轮询一个空队列意味着每20秒发出一个请求.那是3要求/分钟，180要求/小时，4320要求/天，129600要求/月...事实证明，这少于每月允许的100万免费请求.

The other problem, though, is your assertion that you don't want the app to be constantly polling SQS. I encounter this objection fairly often, and it's often misplaced. With SQS long polling, "constantly" polling an empty queue means one request every 20 seconds. That's 3 req/minute, 180 req/hour, 4320 req/day, 129600 req/month... which turns out to be less than the 1 million free requests allowed each month.

服务器对通知作出反应而不是在后台用适当数量的工作人员轮询队列的问题是，您可能很容易被大约同时到达的大量作业所淹没.如果您收到10个并发请求，您可以处理吗? 100? 1000?通常，对于像这样的异步作业，请求作业的成本(在资源方面)比执行作业要少(例如，上载映像所需的CPU数量要比调整映像大小所需的CPU少得多).除非协调您的反应响应，否则您可能会不堪重负.

The problem with your server reacting to notifications rather than polling a queue in the background with an appropriate number of workers is that you'll potentially be easily overwhelmed by a large batch of jobs arriving at about the same time. If you get 10 concurrent requests, can you handle it? 100? 1000? Often, for jobs that are asynchronous like this, it costs less (in terms of resources) to request the job than it does to perform the job (e.g., uploading an image should require much less CPU than resizing that image would require). Unless you coordinate your reaction response, you could overwhelm your system.

在不适用的情况下，不要陷入轮询不好，推入是好的"的概念陷阱.在绝大多数情况下，这种观点是绝对正确的……轮询几乎总是错误的解决方案……但是对于SQS长轮询，您真正拥有的是一种推送机制，其包装方式使其与HTTP兼容，而轮询的许多固有弊端……消失了.如果您正在进行长时间轮询，则队列为空，并且收到一条消息，您的长时间轮询将几乎立即随该消息一起返回.它不会等待超时发生.毕竟，后台队列监视队列可能是一个好方法.

Don't fall into a conceptual trap of "polling is bad, push is good" where it doesn't apply. The vast majority of the time, that sentiment is absolutely correct... polling is almost always the wrong solution... but with SQS long polling, what you really have is a push mechanism wrapped up in a way that makes it compatible with HTTP, and much of the inherent evil of polling... disappears. If you're in the middle of a long poll, the queue is empty, and a message arrives, your long poll will return with that message almost immediately. It doesn't sit around waiting the timeout to happen. A background process watching the queue may be a good way to go after all.

这篇关于如何通过PHP将AWS SQS/SNS用作推送通知队列来执行繁重的处理任务?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！