AWS SQS 异步队列模式(请求/响应) [英] AWS SQS Asynchronous Queuing Pattern (Request/Response)
问题描述
我正在寻求有关产品的架构设计决策的帮助.
I'm looking for help with an architectural design decision I'm making with a product.
我们有多个生产者(由 API 网关调用 Lambda 启动)将消息放在 SQS 队列(请求队列)上.可以有多个同时调用，因此会有多个并行运行的 Lambda 实例.
We've got multiple producers (initiated by API Gateway calls into Lambda) that put messages on a SQS queue (the request queue). There can be multiple simultaneous calls, so there would be multiple Lambda instances running in parallel.
然后我们有消费者(假设有 20 个 EC2 实例)，他们在 SQS 上进行长轮询以获取消息以处理它们.它们每个需要大约 30-45 秒来处理一条消息.
Then we have consumers (lets say twenty EC2 instances) who long-poll on the SQS for the message to process them. They take about 30-45 seconds to process a message each.
理想情况下，我希望将响应发送回发出请求的生产者 - 这是我在 SQS 中遇到的问题.理论上，我会有一个单独的响应队列，最初的 Lambda 生产者将使用它，但似乎没有办法挑选特定的相关响应.也就是说，每个 Lambda 函数可能会获取另一个函数的响应.我正在寻找类似于这种设计模式的东西:http://soapatterns.org/design_patterns/asynchronous_queuing
I would then ideally like to send the response back to the producer that issued the request - and this is the part I'm struggling with with SQS. I would in theory have a separate response queue that the initial Lambda producers would then be consuming, but there doesn't seem to be a way to cherry pick the specific correlated response. That is, each Lambda function might pick up another function's response. I'm looking for something similar to this design pattern: http://soapatterns.org/design_patterns/asynchronous_queuing
我能看到的唯一选择是为每个 Lambda API 调用创建一个新的 SQS 响应队列，在消息中传递其 ARN 以供消费者放置响应，但我无法想象这是非常有效的 -特别是当每分钟可能有数百条消息时?我错过了一些明显的东西吗?
The only option that I can see is to create a new SQS Response queue for each Lambda API call, passing in its ARN in the message for the consumers to put the response on, but I can't imagine that's very efficient - especially when there's potentially hundreds of messages a minute? Am I missing something obvious?
我认为唯一的其他选择是设置更大的消息代理(例如 RabbitMQ/ApacheMQ)环境，但我想尽可能避免这种情况.
I suppose the only other alternative would be setting up a bigger message broker (e.g. RabbitMQ/ApacheMQ) environment, but I'd like to avoid that if possible.
谢谢！
推荐答案
是的，您可以使用 RabbitMQ 来实现更rpc"的队列模式.
Yes, you could use RabbitMQ for a more "rpc" queue pattern.
但如果您想留在 AWS 中，请尝试使用 SQS 以外的其他方法进行响应.
But if you want to stay within AWS, try using something other than SQS for the response.
相反，您可以使用 S3 进行响应.当您的生产者将项目放入 SQS 时，在消息中包含响应的 S3 目标.当您的消费者完成任务后，将响应放在所需的 S3 位置.
Instead, you could use S3 for the response. When your producer puts the item into SQS, include in the message an S3 destination for the response. When your consumer completes the tasks, put the response in the desired S3 location.
然后您可以检查 S3 以获取响应.
Then you can check S3 for the response.
更新
您或许可以使用 Redis 完成类似 RPC 的消息队列.
You may be able to accomplish an RPC-like message queue using Redis.
https://github.com/ServiceStack/ServiceStack/wiki/Messaging-and-redis
然后，您可以将 AWS ElastiCache 用于您的 Redis 集群.这将完全取代 SQS 的使用.
Then, you can use AWS ElastiCache for your Redis cluster. This would completely replace the use of SQS.
这篇关于AWS SQS 异步队列模式(请求/响应)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
