使用 RabbitMQ 的工作池和多租户队列 [英] Worker pools and multi-tenant queues with RabbitMQ

问题描述

我正在开发一个基于多租户云的 Web 应用程序(许多客户端，每个客户端都有自己独立的环境"，但都在共享的硬件集上)，我们正在为用户引入该功能批处理工作以供以后处理.批处理工作的类型真的不重要，只是数量足够，没有工作队列就不太实际.我们选择了 RabbitMQ 作为我们的底层队列框架.

I work on a web application that is a multi-tenant cloud based application (lots of clients, each with their own separate "environment", but all on shared sets of hardware) and we're introducing the ability for a user to batch up work for later processing. The types of batched work really isn't important, it's just of sufficient quantity that doing it without a work queue isn't really practical. We've selected RabbitMQ as our underlying queue framework.

因为我们是一个多租户应用程序，我们不一定希望客户端能够为另一个客户端造成冗长的队列处理时间，所以我们提出的一个想法是在每个客户端上创建一个队列基础并在我们所有的客户端队列中都有一个共享的工作池.问题是，据我所知，工作人员直接绑定到特定队列，而不是交换.在我们的理想世界中，我们的客户端队列仍将在没有一个客户端阻塞另一个客户端的情况下从共享工作池中处理，我们可以通过启动更多工作人员或关闭空闲工作人员来根据需要扩大或缩小该工作池.实际上，将工作人员绑定到特定队列可以防止我们这样做，因为我们经常有很多工作人员只是在没有活动的队列上闲置.

Because we're a multi-tenant app, we don't necessarily want clients to be able to cause lengthy queue process times for another client, so one idea that we've floated up is creating a queue on a per client basis and having a shared worker pool pointed across ALL our client queues. The problem is that, to the best that I can figure, workers are directly bound to a specific queue, not an exchange. In our ideal world, our client queues will still be processed, without one client blocking another, from a shared worker pool that we can grow or shrink as necessary by launching more workers or closing down idle ones. Having workers tied to a specific queue prevents us from this in a practical sense, as we'd frequently have lots of workers just idling on a queue with no activity.

是否有相对简单的方法来实现这一点?我对 RabbitMQ 还很陌生，还没有真正能够完成我们所追求的目标.我们也不想编写一个非常复杂的多线程消费者应用程序，这是我们可能负担不起的开发和测试时间.我们的堆栈是基于 Windows/.Net/C# 的，如果这很重要的话，但我认为这不会对手头的问题产生重大影响.

Is there a relatively straight forward to accomplish this? I'm fairly new to RabbitMQ and haven't really been able to accomplish what we're after. We also don't want to have to write a very complex multithreaded consumer application either, that's a time sink in dev and test time that we likely can't afford. Our stack is Windows/.Net/C# based if that's germaine, but I don't think that should have a major bearing in the question at hand.

推荐答案

你可以看一下优先队列的实现(最初问这个问题时没有实现):https://www.rabbitmq.com/priority.html

You could look at the priority queue implementation (which wasn't implemented when this question was originally asked): https://www.rabbitmq.com/priority.html

如果这对您不起作用，您可以尝试其他一些技巧来实现您想要的(应该适用于旧版本的 RabbitMQ):

If that doesn't work for you, you could try some other hacks to achieve what you want (which should work with older versions of RabbitMQ):

您可以将 100 个队列绑定到主题交换并将路由键设置为用户 ID % 100 的哈希，即每个任务将有一个介于 1 和 100 之间的键，并且同一用户的任务将具有相同的键.每个队列都绑定了 1 到 100 之间的唯一模式.现在您有一组工作人员，它们以随机队列编号开始，然后在每个作业之后增加该队列编号，再次 %100 以在队列 100 之后循环回到队列 1.

You could have 100 queues bound to a topic exchange and set the routing key to a hash of the user ID % 100, i.e. each task will have a key between 1 and 100 and tasks for the same user will have the same key. Each queue is bound with a unique pattern between 1 and 100. Now you have a fleet of workers which start with a random queue number and then increment that queue number after each job, again % 100 to cycle back to queue 1 after queue 100.

现在，您的工作人员队列可以并行处理多达 100 个唯一用户，或者如果没有其他工作要做，所有工作人员都可以专注于单个用户.如果工作人员需要在每个作业之间循环遍历所有 100 个队列，那么在只有一个用户在单个队列上有很多作业的情况下，每个作业之间自然会有一些开销.较少数量的队列是解决此问题的一种方法.您还可以让每个工作人员保持与每个队列的连接，并从每个队列中消费最多一条未确认的消息.如果未确认消息超时设置得足够高，worker 可以更快地循环处理内存中的待处理消息.

Now your worker fleet can process up to 100 unique users in parallel, or all the workers can focus on a single user if there is no other work to do. If the workers need to cycle through all 100 queues between each job, in the scenario that only a single user has lot of jobs on a single queue, you're naturally going to have some overhead between each job. A smaller number of queues is one way to deal with this. You could also have each worker hold a connection to each of the queues and consume up to one un-acknowledged message from each. The worker can then cycle through the pending messages in memory much faster, provided the un-acknowledged message timeout is set sufficiently high.

或者，您可以创建两个交换器，每个交换器都有一个绑定队列.所有工作都进入第一个交换和队列，由一组工作人员使用.如果一个工作单元花费的时间太长，工人可以取消它并将其推送到第二个队列.当第一个队列上没有任何内容时，工作人员只会处理第二个队列.您可能还需要几个具有相反队列优先级的工作人员，以确保在有永无止境的短任务流到达时仍然处理长时间运行的任务，以便最终始终处理用户批次.这不会真正将您的工作人员队列分配给所有任务，但它会阻止一个用户长时间运行的任务阻止您的工作人员为同一用户或另一个用户执行短期运行的任务.它还假设您可以取消作业并稍后重新运行它而不会出现任何问题.这也意味着超时并需要以低优先级重新运行的任务会浪费资源.除非你能提前识别快慢任务

Alternatively you could create two exchanges, each with a bound queue. All work goes to the first exchange and queue, which a pool of workers consume. If a unit of work takes too long the worker can cancel it and push it to the second queue. Workers only process the second queue when there's nothing on the first queue. You might also want a couple of workers with the opposite queue prioritization to make sure long running tasks are still processed when there's a never ending stream of short tasks arriving, so that a users batch will always be processed eventually. This won't truly distribute your worker fleet across all tasks, but it will stop long running tasks from one user holding up your workers from executing short running tasks for that same user or another. It also assumes you can cancel a job and re-run it later without any problems. It also means there will be wasted resources from tasks that timeout and need to be re-run as low priority. Unless you can identify fast and slow tasks in advance

如果单个用户有 100 个慢速任务，那么包含 100 个队列的第一个建议也可能有问题，然后另一个用户发布了一批任务.在完成其中一项慢速任务之前，不会查看这些任务.如果事实证明这是一个合理的问题，您可以将这两种解决方案结合起来.

The first suggestion with the 100 queues could also have a problem if there are 100 slow tasks for a single user, then another user posts a batch of tasks. Those tasks won't get looked at until one of the slow tasks is finished. If this turns out to be a legitimate problem you could potentially combine the two solutions.

这篇关于使用 RabbitMQ 的工作池和多租户队列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！