Celery如何以公平的方式分配用户的任务? [英] How can Celery distribute users' tasks in a fair way?

问题描述

我正在执行的任务与抓取有关URL的一些基本信息有关，例如标题，描述和OGP元数据.如果用户A请求抓取200个URL，并且在用户B请求10个URL之后，用户B的等待时间可能会比他/他的预期长得多.

The task I'm implementing is related to scrape some basic info about a URL, such as title, description and OGP metadata. If User A requests 200 URLs to scrape, and after User B requests for 10 URLs, User B may wait much more than s/he expect.

我要实现的目标是基于每个用户或至少在用户之间公平地限制特定任务的速率.

What I'm trying to achieve is to rate limit a specific task on a per user basis or, at least, to be fair between users.

用于速率限制的Celery实现过于广泛，因为

The Celery implementation for rate limiting is too broad, since it uses the task name only

您对实现这种公平有什么建议吗?

Do you have any suggestion to achieve this kind of fairness?

相关 Celery(Django)速率限制

推荐答案

另一种方法是使用锁定来限制单个用户的速率.使用用户ID作为锁名.如果已经持有该锁，则在依赖于任务的某些延迟后重试.

Another way would be to rate limit individual users using a lock. Use the user id as the lock name. If the lock is already held retry after some task dependent delay.

基本上，请执行以下操作:

Basically, do this:

确保一次只能执行一项任务

锁定用户ID，如果无法获取该锁定，则重试而不执行任何操作.另外，最好使用Redis代替Django缓存，但是无论哪种方式都可以.

Lock on the user id and retry instead of doing nothing if the lock can't be acquired. Also, it would be better to use Redis instead of the the Django cache, but either way will work.

这篇关于Celery如何以公平的方式分配用户的任务?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！