使用大型ETA安排芹菜任务 [英] Scheduling celery tasks with large ETA

问题描述

我目前正在使用

I am currently experimenting with future tasks in celery using the ETA feature and a redis broker. One of the known issues with using a redis broker has to do with the visibility timeout:

如果在可见性超时"中未确认任务，则该任务将重新交付给其他工作人员并执行.

If a task isn’t acknowledged within the Visibility Timeout the task will be redelivered to another worker and executed.

这会导致ETA/倒数/重试任务出现问题，其中执行时间超过了可见性超时；实际上，如果发生这种情况，它将再次循环执行.

This causes problems with ETA/countdown/retry tasks where the time to execute exceeds the visibility timeout; in fact if that happens it will be executed again, and again in a loop.

我可以设想的一些任务将具有数周/月的时间表的预计到达时间.将可见性超时设置得足够大以容纳这些任务可能是不明智的.

Some tasks that I can envision will have an ETA on the timescale of weeks/months. Setting the visibility timeout large enough to encompass these tasks is probably unwise.

是否有前进的路径来使用Redis代理处理这些任务?我知道这个问题.更换经纪人是唯一的选择吗?

Are there any paths forward for processing these tasks with a redis broker? I am aware of this question. Is changing brokers the only option?

推荐答案

我正在通过以下方式使用redis:

I am doing this with redis in the following way:

我们有可以安排发布其某些内容的客户.我们将该版本及其执行时间存储在数据库中.

We have customers that can schedule a release of some of their content. We store the release in our database with the time it should be executed at.

然后，我们使用芹菜拍子执行一项定期任务(每小时一次或适合您的时间)，该任务检查我们的发布表以查找下一个时期内(预定的小时数或适合您的时间)安排的发布.如果发现任何东西，我们将为他们安排芹菜任务.这使我们的ETA较短.

Then we use celery beat to perform a periodic task (hourly or what suits you) that checks our releases table for releases that are scheduled within the next period (again hour or what suits you). if any are found we then schedule a task for them with celery. This allows us to have a short ETA.

这篇关于使用大型ETA安排芹菜任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！