Laravel运行多个计划任务 [英] Laravel run multiple scheduled tasks

问题描述

我目前有一个计划的控制台命令，该命令每5分钟运行一次，而不会像这样重叠:

I currently have a scheduled console command that runs every 5 minutes without overlap like this:

 $schedule->command('crawler')
             ->everyFiveMinutes()
             ->withoutOverlapping()
             ->sendOutputTo('../_laravel/storage/logs/scheduler-log.txt');

所以效果很好，但是我目前大约有220页，大约需要3个小时才能完成，以5分钟为增量，这是因为我只是强迫它每间隔抓取10页，因为每页需要20-30秒的时间才能抓取由于各种因素.每个页面都是数据库中的一条记录.如果我最终要爬网10,000页，则此方法将不起作用，因为它将花费24个小时以上的时间，而且每个页面应该每天重新爬网一次.

So it works great, but I currently have about 220 pages that takes about 3 hours to finish in increments of 5 minutes because I just force it to crawl 10 pages at each interval since each page takes like 20-30 seconds to crawl due to various factors. Each page is a record in the database. If I end up having 10,000 pages to crawl, this method would not work because it would take more than 24 hours and each page is supposed to be re-crawled once a day.

因此，我的供应商最多允许10个并发请求(或更高计划的并发请求)，那么并发运行的最佳方法是什么?如果我只是复制调度程序代码，它是否会运行同一命令两次或如果我重复执行10次，是否会运行10次?有什么问题会引起吗?

So my vendor allows up to 10 concurrent requests (or more with higher plans), so what's the best way to run it concurrently? If I just duplicate the scheduler code, does it run the same command twice or like 10 times if I duplicated it 10 times? Any issues that would cause?

然后我需要将参数传递给控制台，例如1、2、3等，在其中可以用来确定要爬网的页面?即1将是1-10个记录，2将是下一个11-20记录，依此类推.

And then I need to pass on parameters to the console such as 1, 2, 3, etc... in which I could use to determine which pages to crawl? i.e. 1 would be 1-10 records, 2 would be next 11-20 records, and so on.

使用此 StackOverfow 答案，我认为我知道如何传递它，像这样:

Using this StackOverfow answer, I think I know how to pass it along, like this:

 $schedule->command('crawler --sequence=1')

但是如何在Command类中读取该参数?它是否只是成为常规的PHP变量，即$sequence?

But how do I read that parameter within my Command class? Does it just become a regular PHP variable, i.e. $sequence?

推荐答案

1. 更好地使用队列进行作业处理
2. 在cron上，将所有作业添加到队列中
3. 运行多个队列工作器，它将并行处理作业

提示:发生在我们身上. 可能是先前添加的作业未完成，但cron再次在队列中添加了相同的任务.由于队列是按顺序工作的.为了避免这种情况的发生，您应该在数据库中标记上次任务何时完成，以便知道何时执行任务(如果任务严重延迟)

Tip: It happened with us. It might happen that job added previously is not complete, yet cron adds the same task in queue again. As queues works sequentially. To save yourself from the situation, you should in database mark when a task is completed last time, so you know when to execute the job (if it was seriously delayed)

这篇关于Laravel运行多个计划任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！