我们可以限制luigi Task的吞吐量吗? [英] Can we limit the throughput of a luigi Task?

问题描述

我们有一个Luigi任务，该任务从第三方服务请求一条信息.我们限制了每分钟对该API调用可以执行的调用请求的数量.

We have a Luigi Task that request a piece of information from a 3rd party service. We are limited on the number of call requests we can perform per minute to that API call.

是否可以根据每个任务指定调度程序每单位时间必须运行多少个此类任务?

Is there a way to specify on a per-Task basis how many tasks of this kind must the scheduler run per unit of time?

推荐答案

我们在任务中实现了自己的速率限制.我们的API限制足够低，我们可以用一个线程将其饱和.当我们收到速率限制响应时，我们退出并重试.

We implemented our own rate limiting in the task. Our API limit was low enough that we could saturate it with a single thread. When we received a rate limit response, we just back off and retry.

您可以做的一件事情是将API调用声明为资源.您可以在配置中设置多少可用资源，然后任务以

One thing you can do is to declare the API call as a resource. You can set how many of the resource is available in the config, and then how many of the resource the task consumes as a property on the task. This will then limit you to running n of that task at a time.

在配置中:

[resources]
api=1

在任务代码中:

resources = {"api": 1}

这篇关于我们可以限制luigi Task的吞吐量吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！