与多个使用者一起使用Amazon SQS [英] Using Amazon SQS with multiple consumers

问题描述

我有一个基于服务的应用程序，该应用程序使用具有多个队列和多个使用者的Amazon SQS.我这样做是为了实现一个基于事件的体系结构并解耦所有服务，其中不同的服务对其他系统的状态变化做出反应.例如:

I have a service-based application that uses Amazon SQS with multiple queues and multiple consumers. I am doing this so that I can implement an event-based architecture and decouple all the services, where the different services react to changes in state of other systems. For example:

• 注册服务:
  • 在新用户注册时发出事件"registration-new".
  • 在更新用户时发出事件"user-updated".
  • 从队列新注册"中读取并在搜索中索引用户.
  • 从用户更新"队列中读取并更新搜索中的用户.
  • 从新注册"队列中读取并发送到Mixpanel.
  • 从用户更新"队列中读取并发送到Mixpanel.

  我遇到了许多问题:

  • 进行轮询时，可以多次接收到一条消息.我可以将许多系统设计为幂等的，但是对于某些服务(例如度量标准服务)而言，难度要大得多.
  • 需要从SQS中的队列中手动删除消息.我已经考虑过要实现一个消息处理服务"，当所有服务都收到消息后处理该消息的删除(每个服务在处理一条消息后都会发出消息已确认"事件).

  我想我的问题是:我应该使用什么模式来确保SQS中的单个队列可以有多个使用方，同时确保消息也可以可靠地传递和删除.谢谢您的帮助.

  I guess my question is this: what patterns should I use to ensure that I can have multiple consumers for a single queue in SQS, while ensuring that the messages also get delivered and deleted reliably. Thank you for your help.

  推荐答案

  我认为您做错了.

  在我看来，您正在使用同一队列来执行多项不同的操作.最好将单个队列用于单个目的.

  It looks to me like you are using the same queue to do multiple different things. You are better of using a single queue for a single purpose.

  与其将事件放入新注册"队列中，然后让两个不同的服务轮询该队列，不但需要读取该消息并对其执行不同的操作(然后需要一个第三个进程，该进程应该是删除其他2条消息后删除该消息.

  Instead of putting an event into the 'registration-new' queue and then having two different services poll that queue, and BOTH needing to read that message and both doing something different with it (and then needing a 3rd process that is supposed to delete that message after the other 2 have processed it).

  一个队列应该用于一个目的.

  One queue should be used for one purpose.

  • 创建一个索引用户搜索"队列和一个发送到混合面板"队列， 因此搜索服务从搜索队列中读取内容，为用户建立索引 并立即删除该邮件.

  • Create a 'index-user-search' queue and a 'send to mixpanels' queue, so the search service reads from the search queues, indexes the user and immediately deletes the message.

  mixpanel服务从mix-panels队列中读取，处理
  邮件并删除该邮件.

  The mixpanel-service reads from the mix-panels queue, processes the
  message and deletes the message.

  注册服务现在不再将新注册"发送到单个队列，而是将其发送到两个队列.

  The registration service, instead of emiting a 'registration-new' to a single queue, now emits it to two queues.

  要更好地迈出一步，请在此处将SNS添加到组合中，并使注册服务向新注册"主题(不是队列)发出SNS消息，然后将我上面提到的两个队列都订阅到以扇出"模式显示该主题.

  To take it one step better, add SNS into the mix here and have the registration service emit an SNS message to the 'registration-new' topic (not queue), and then subscribe both of the queues I mentioned above, to that topic in a 'fan-out' pattern.

  https://aws.amazon.com/blogs/aws/队列和通知现在最好的朋友/

  两个队列都将接收到该消息，但是您只需将其加载到SNS中一次-如果将来第三个不相关的服务还需要处理新注册"事件，则可以创建另一个队列并将其也订阅该主题-它可以在没有依赖关系或不了解其他服务正在做什么的情况下运行-这就是目标.

  Both queues will receive the message, but you only load it into SNS once - if down the road a 3rd unrelated service needs to also process 'registration-new' events, you create another queue and subscribe it to the topic as well - it can run with no dependencies or knowledge of what the other services are doing - that is the goal.

  这篇关于与多个使用者一起使用Amazon SQS的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！