将 Amazon SQS 与多个使用者一起使用 [英] Using Amazon SQS with multiple consumers

问题描述

我有一个基于服务的应用程序，它使用具有多个队列和多个使用者的 Amazon SQS.我这样做是为了我可以实现基于事件的架构并解耦所有服务，其中不同的服务对其他系统的状态变化做出反应.例如:

I have a service-based application that uses Amazon SQS with multiple queues and multiple consumers. I am doing this so that I can implement an event-based architecture and decouple all the services, where the different services react to changes in state of other systems. For example:

• 注册服务:
  • 当新用户注册时发出事件registration-new".
  • 在用户更新时发出事件user-updated".
  • 从队列registration-new"中读取并为搜索中的用户编制索引.
  • 从队列user-updated"中读取并更新搜索中的用户.
  • 从registration-new"队列中读取并发送到 Mixpanel.
  • 从队列user-updated"读取并发送到 Mixpanel.

  我遇到了很多问题:

  • 进行轮询时可以多次收到一条消息.我可以将很多系统设计为幂等的，但对于某些服务(例如指标服务)而言，这会困难得多.
  • 需要从 SQS 中的队列中手动删除消息.我想过实现一个消息处理服务"，当所有服务都收到消息时处理消息的删除(每个服务在处理消息后都会发出消息确认"事件).

  我想我的问题是:我应该使用什么模式来确保 SQS 中的单个队列可以有多个使用者，同时确保消息也能可靠地传送和删除.感谢您的帮助.

  I guess my question is this: what patterns should I use to ensure that I can have multiple consumers for a single queue in SQS, while ensuring that the messages also get delivered and deleted reliably. Thank you for your help.

  推荐答案

  我认为你做错了.

  在我看来，您正在使用同一个队列来做多种不同的事情.您最好将单个队列用于单一目的.

  It looks to me like you are using the same queue to do multiple different things. You are better of using a single queue for a single purpose.

  而不是将事件放入注册新"队列，然后让两个不同的服务轮询该队列，并且两者都需要读取该消息并对其进行不同的处理(然后需要一个假设的第三个进程在其他 2 个处理后删除该消息).

  Instead of putting an event into the 'registration-new' queue and then having two different services poll that queue, and BOTH needing to read that message and both doing something different with it (and then needing a 3rd process that is supposed to delete that message after the other 2 have processed it).

  一个队列应该用于一个目的.

  One queue should be used for one purpose.

  • 创建一个'index-user-search'队列和一个'send to mixpanels'队列，所以搜索服务从搜索队列中读取，索引用户并立即删除消息.

  • Create a 'index-user-search' queue and a 'send to mixpanels' queue, so the search service reads from the search queues, indexes the user and immediately deletes the message.

  mixpanel-service 从 mix-panels 队列中读取数据，处理
  消息并删除该消息.

  The mixpanel-service reads from the mix-panels queue, processes the
  message and deletes the message.

  注册服务不再向单个队列发出registration-new"，现在将其发送到两个队列.

  The registration service, instead of emiting a 'registration-new' to a single queue, now emits it to two queues.

  为了更进一步，在此处添加 SNS 并让注册服务向registration-new"主题(不是队列)发送 SNS 消息，然后订阅我上面提到的两个队列，以该主题以扇出"模式进行.

  To take it one step better, add SNS into the mix here and have the registration service emit an SNS message to the 'registration-new' topic (not queue), and then subscribe both of the queues I mentioned above, to that topic in a 'fan-out' pattern.

  https://aws.amazon.com/blogs/aws/队列和通知现在最好的朋友/

  两个队列都会收到消息，但您只能将其加载到 SNS 中一次 - 如果接下来有第三个不相关的服务也需要处理注册新"事件，您可以创建另一个队列并将其订阅到主题- 它可以在不依赖或不了解其他服务正在做什么的情况下运行 - 这就是目标.

  Both queues will receive the message, but you only load it into SNS once - if down the road a 3rd unrelated service needs to also process 'registration-new' events, you create another queue and subscribe it to the topic as well - it can run with no dependencies or knowledge of what the other services are doing - that is the goal.

  这篇关于将 Amazon SQS 与多个使用者一起使用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！