使用Supervisor运行多个Laravel队列工作器 [英] Running multiple Laravel queue workers using Supervisor

问题描述

我通过数据库驱动程序和管理程序使用Laravel队列来保持队列工作器始终运行:

[program:laravel_queue]
command=php artisan queue:listen --timeout=1800 --tries=5
directory=/var/app/current
stdout_logfile=/var/app/support/logs/laravel-queue.log
logfile_maxbytes=0
logfile_backups=0
redirect_stderr=true
autostart=true
autorestart=true
startretries=86400
EOB

某些队列任务可能需要大约10分钟才能完成.

我有两个问题:

1)我如何编辑以上脚本以在同一队列上运行多个(例如3个)队列工作器.

2)是否有一种方法可以根据等待处理的作业数来扩展正在运行的队列工作器的数量?

问题2的原因是，我们有很多繁忙的时间，然后有很多安静的时间，所以我真的不想浪费时间让3个侦听器一直运行.

解决方案

在主管中，您可以使用参数numprocs指定进程数量，因此可以在脚本中添加一行，内容为:

numprocs=5

现在，您可以做一些聪明的事情，例如，如果仅在队列上运行的某些进程花费的时间太长，则可以创建一组不同的队列进程来为轻型进程工作. 要实现此目的，您可以使用一个队列名称(例如--queue=longprocess)和另一个使用--queue=lightprocess的队列名称来创建管理程序配置，并在程序中将作业分派到相应的队列中，这样，长进程不会延迟短进程. /p>

您还可以在一个主管配置文件中指定队列优先级，例如--queue=lightprocess,longprocess.这样，您的工作人员将在运行longprocess之前先查找lightprocess.

要回答第二个问题，不，就主管而言，所有进程都在运行，它不知道队列是忙还是闲，因此它无法杀死进程并根据其使用来创建它们，所以没有，您无法进行动态配置，仅当您有繁忙的进程时才创建更多进程.

请注意，如果分配的进程数超过1，则必须在名称中添加进程数.根据主管配置文档:

Supervisor将启动与numprocs命名的程序一样多的实例.请注意，如果numprocs> 1，则process_name表达式中必须包含％(process_num)s(或任何其他包含process_num的有效Python字符串表达式).

Supervisor配置文档: http://supervisord.org/configuration.html

I using Laravel queues using a database driver and supervisor to keep a queue worker running all the time:

[program:laravel_queue]
command=php artisan queue:listen --timeout=1800 --tries=5
directory=/var/app/current
stdout_logfile=/var/app/support/logs/laravel-queue.log
logfile_maxbytes=0
logfile_backups=0
redirect_stderr=true
autostart=true
autorestart=true
startretries=86400
EOB

Some of the queue tasks can take around 10 minutes to complete.

I have 2 parts to the question:

1) How can i edit the above script to run multiple (e.g. 3) queue workers on the same queue.

2) Is there a way of scaling the number of queue workers running based on the number of jobs waiting to be processed?

The reason for question 2 is that we have batches of busy times and then lots of quiet times, so i don't really want to be wasting resources with 3 listeners running the whole time.

解决方案

In supervisor you specify the amount of processes with the parameter numprocs, so you can add a line to your script that says:

numprocs=5

Now, you can do some clever things, for example if only some of the processes that runs on the queues takes too long, you can create a different set of queues processes to work those and other set for the light processes. To achieve that you can create a supervisor configuration with one queue name like --queue=longprocess and another one with --queue=lightprocess and in your program you dispatch the job in the corresponding queue, that way, long processes won't delay short processes.

You can also specify queue priorities within one supervisor configuration file such as --queue=lightprocess,longprocess. That way, your worker(s) will first look for lightprocess before running longprocess.

To answer your second question, no, in terms of supervisor all processes are running, it doesn't knows if the queue is busy or idle, therefore it can't kill processes and create them based on their use, so no, you can't have a dynamic configuration of creating more processes only when the ones you have are busy.

A note, if you assign more than 1 numprocs, you must add the number of process to the name. According to supervisor configuration docs:

Supervisor will start as many instances of this program as named by numprocs. Note that if numprocs > 1, the process_name expression must include %(process_num)s (or any other valid Python string expression that includes process_num) within it.

Supervisor configuration docs: http://supervisord.org/configuration.html

这篇关于使用Supervisor运行多个Laravel队列工作器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！