Grails非基于时间的排队 [英] Grails non time based queuing
问题描述
我需要处理上传的文件，最多可能需要1秒钟或10分钟。目前我的解决方案是用30秒的定时器做一个石英工作，然后在任何时候处理和任意工作。这有几个问题。
其中一项：如果工作时间少于几秒，则对于工作队列等待30秒是浪费的。 p>
二：如果队列中只有一个长工作，它可以尝试做两次。
什么我想要的是一个永恒的队列。当添加事物时，如果有空闲的工作人员，则立即开始。有没有解决方案？我正在看jesque，但我不知道它是否可以做到这一点。
您正在寻找的是基本消息队列。有很多选项，但我最喜欢的Grails是RabbitMQ。 Grails插件相当不错，根据我的经验表现良好。
通常，消息队列允许您有N个生产者（创建作业的东西）添加工作消息传递给队列，然后M消费者将作业从队列中取出并处理它们，当工作人员完成作业时，它只是要求队列处理下一个作业，如果没有，它只是等待队列它需要做的事情，队列还会跟踪消息处理的成功/失败（你可以控制这个），这样你就不会把同样的消息传递给多个工作人员。
这样做的优点是不依赖于轮询（因此您可以在事情发生时立即开始处理），并且它还具有更大的可扩展性。您可以根据需要调整生产者和消费者的比例，将输入分离从输出中获取信息，这样您就可以获得流量高峰，然后在您拥有可用的资源（工作人员）的情况下顺利完成工作。
I need to process files which get uploaded and it can take as little as 1 second or as much as 10 minutes. Currently my solution is to make a quartz job with a timer of 30 seconds and then process and arbitrary job whenever it hits. There are several problems with this.
One: if the job will take less than a few seconds it is wasteful to make things wait 30 seconds for the job queue.
Two: if there is only one long job in the queue it could feasibly try to do it twice.
What I want is a timeless queue. When things are added the are started immediately if there is a free worker. Is there a solution for this? I was looking at jesque, but I couldn't tell if it can do this.
What you are looking for is a basic message queue. There are lots of options out there, but my favorite for Grails is RabbitMQ. The Grails plugin for it is quite good and it performs well in my experience.
In general, message queues allow you to have N producers (things creating jobs") adding work messages to a queue and then M consumers pulling jobs off of the queue and processing them. When a worker completes it's job, it simply asks the queue for the next job to process and if there is none, it just waits for the queue to give it something to do. The queue also keeps track of success / failure of message processing (you can control this) so that you don't give the same message to more than one worker.
This has the advantage of not relying on polling (so you can start processing as soon as things come in) and it's also much more scaleable. You can scale both your producers and consumers up or down as needed, decoupling the inputs from the outputs so that you can take a traffic spike and then work your way through it as you have the resources (workers) available.
这篇关于Grails非基于时间的排队的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
