AWS SQS异步排队模式(请求/响应) [英] AWS SQS Asynchronous Queuing Pattern (Request/Response)

问题描述

我正在寻找有关产品的建筑设计决策的帮助.

I'm looking for help with an architectural design decision I'm making with a product.

我们有多个生产者(通过对Lambda的API网关调用发起)将消息放置在SQS队列(请求队列)中.可以有多个同时调用，因此将有多个Lambda实例并行运行.

We've got multiple producers (initiated by API Gateway calls into Lambda) that put messages on a SQS queue (the request queue). There can be multiple simultaneous calls, so there would be multiple Lambda instances running in parallel.

然后，我们有一些消费者(比如说二十个EC2实例)对SQS进行长时间轮询以获取处理它们的消息.每个处理一条消息大约需要30-45秒.

Then we have consumers (lets say twenty EC2 instances) who long-poll on the SQS for the message to process them. They take about 30-45 seconds to process a message each.

然后，理想情况下，我想将响应发送回发出请求的生产者-这是我在SQS上苦苦挣扎的部分.从理论上讲，我将有一个单独的响应队列，然后最初的Lambda生产者将使用它们，但是似乎没有办法选择特定的相关响应.也就是说，每个Lambda函数可能会拾取另一个函数的响应.我正在寻找类似于此设计模式的内容: http://soapatterns.org/design_patterns/asynchronous_queuing

I would then ideally like to send the response back to the producer that issued the request - and this is the part I'm struggling with with SQS. I would in theory have a separate response queue that the initial Lambda producers would then be consuming, but there doesn't seem to be a way to cherry pick the specific correlated response. That is, each Lambda function might pick up another function's response. I'm looking for something similar to this design pattern: http://soapatterns.org/design_patterns/asynchronous_queuing

我唯一看到的选择是为每个Lambda API调用创建一个新的SQS响应队列，并在消息中传递其ARN供消费者使用，以进行响应，但是我无法想象这非常有效-特别是每分钟可能有数百条消息时?我缺少明显的东西吗?

The only option that I can see is to create a new SQS Response queue for each Lambda API call, passing in its ARN in the message for the consumers to put the response on, but I can't imagine that's very efficient - especially when there's potentially hundreds of messages a minute? Am I missing something obvious?

我想唯一的另一种选择是建立一个更大的消息代理(例如RabbitMQ/ApacheMQ)环境，但我想尽可能避免这种情况.

I suppose the only other alternative would be setting up a bigger message broker (e.g. RabbitMQ/ApacheMQ) environment, but I'd like to avoid that if possible.

谢谢！

推荐答案

是的，您可以将RabbitMQ用于更多的"rpc"队列模式.

Yes, you could use RabbitMQ for a more "rpc" queue pattern.

但是，如果您想留在AWS内，请尝试使用SQS以外的其他方式进行响应.

But if you want to stay within AWS, try using something other than SQS for the response.

相反，您可以使用S3进行响应.当您的生产者将物料放入SQS中时，请在消息中包括用于响应的S3目标.当您的使用者完成任务时，将响应放在所需的S3位置.

Instead, you could use S3 for the response. When your producer puts the item into SQS, include in the message an S3 destination for the response. When your consumer completes the tasks, put the response in the desired S3 location.

然后您可以检查S3的响应.

Then you can check S3 for the response.

更新

您可以使用Redis完成类似于RPC的消息队列.

You may be able to accomplish an RPC-like message queue using Redis.

https://github.com/ServiceStack/ServiceStack/wiki/Messaging -and-redis

然后，您可以将AWS ElastiCache用于Redis集群.这将完全取代SQS的使用.

Then, you can use AWS ElastiCache for your Redis cluster. This would completely replace the use of SQS.

这篇关于AWS SQS异步排队模式(请求/响应)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！