具有“消息屏障"功能的消息队列? [英] Message Queue with 'Message Barrier' Feature?

问题描述

是否有消息队列实现允许通过在消息流中插入消息屏障"将工作分解为批处理"?让我澄清一下.在将消息屏障之前的所有消息都消耗完之前，不得将消息屏障的所有消息传递到队列的任何使用者.有点像同步点.我还希望 all 个消费者在遇到障碍时都收到通知.

Is there a message queue implementation that allows breaking up work into 'batches' by inserting 'message barriers' into the message stream? Let me clarify. No messages after a message barrier should be delivered to any consumers of the queue, until all messages before the barrier are consumed. Sort of like a synchronization point. I'd also prefer if all consumers received notification when they reached a barrier.

外面有这样的东西吗?

推荐答案

我不知道现有的，广泛使用的实现，但是如果允许的话，我会提出一个非常简单，通用的实现代理，其中:

I am not aware of existing, widely-available implementations, but if you'll allow me I'd propose a very simple, generic implementation using a proxy, where:

• 生产者写到代理队列/主题
• 代理将其转发到原始队列/主题直到屏障消息被代理读取，此时:
  • 代理可以通过将障碍消息转发到原始主题来通知障碍的主题订阅者，或者
  • 代理可以通过以下方式通知壁垒:
    • 定期发布障碍消息，直到障碍被清除；这并不能保证所有使用者都会收到一个准确的通知，尽管所有使用者最终都会清除障碍(有些人可能会收到0条通知，其他人会收到1条以上的通知-所有这些都取决于用于向消费者分发消息的调度程序的类型，例如是否-roundrobin)
    • 使用专门的主题为每个障碍确切地通知每个消费者
    • producers write to the proxy queue/topic
    • the proxy forwards to the original queue/topic until a barrier message is read by the proxy, at which point:
      • the proxy may notify topic subscribers of the barrier by forwarding the barrier message to the original topic, or
      • the proxy may notify queue subscribers of the barrier by:
        • periodically publishing barrier messages until the barrier has been cleared; this does not guarantee that all consumers will receive exactly one notification, although all will eventually clear the barrier (some may receive 0 notifications, others >1 notifications -- all depending on the type of scheduler used to distribute messages to consumers e.g. if non-roundrobin)
        • using a dedicated topic to notify each consumer exactly once per barrier

        感谢Miklos指出，在JMS下，该框架未提供异步交付的确认(在JMS中，所谓的确认"纯粹是消费者方面的概念，不能以此为替代).

        Thanking Miklos for pointing out that under JMS the framework does not provide acknowledgements for asynchronous deliveries (what is referred to as "acknowledgements" in JMS are purely a consumer side concept and are not proxiable as-such.)

        因此，在JMS下，现有的实现(适用于障碍)可能已经通过确认队列"提供了应用程序级别的确认. (与原始队列相反，原始队列将是请求队列".)使用者必须通过向代理确认队列发送确认消息来确认请求的执行；在将确认消息转发给生产者之后，代理将使用确认消息来跟踪清除障碍的时间.

        So, under JMS, the existing implementation (to be adapted for barriers) may already provide application-level acknowledgements via an "acknowledgement queue" (as opposed to the original queue -- which would be a "request queue".) The consumers would have to acknowledge execution of requests by sending acknowledgement messages to the proxy acknowledgement queue; the proxy would use the acknowledgement messages to track when the barrier has been cleared, after having also forwarded the acknowledgement messages to the producer.

        如果现有实现(要适应障碍)尚未已经通过确认队列"提供了应用程序级别的确认，那么您可以:

        If the existing implementation (to be adapted for barriers) does not already provide application-level acknowledgements via an "acknowledgement queue", then you could either:

        • 让代理使用QueueBrowser(已提供)来执行以下操作:
        • have the proxy use the QueueBrowser, provided that:
        1. 您要处理的不是事件，而是
        2. 您要在交货时同步而不是确认执行，并且
        3. 即使在请求实际上已中止并且必须重新交付(即使清除障碍之后)之后，也可以在第一交付上进行同步.我认为Miklos已经指出了这个问题退出IIRC.
        1. you are dealing with queueus not events, that
        2. you want to synchronize on delivery not acknowledgement of execution, and
        3. it is OK to synchronize on first delivery, even if the request was actually aborted and has to be re-delivered (even after the barrier has been cleared.) I think Miklos already pointed this problem out IIRC.

      • 否则，添加代理使用的确认队列，并使使用者适应向其写入确认(基本上是上述JMS场景，除了代理不必将确认消息转发到生产者，除非您的生产者需要功能.)
      • otherwise, add an acknowledgment queue consumed by the proxy, and adapt the consumers to write acknowledgements to it (essentially the JMS scenario above, except it is not necessary for the proxy to forward acknowledgement messages to the producer unless your producer needs the functionality.)

      这篇关于具有“消息屏障"功能的消息队列?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！