获取芹菜的工人ID [英] Get worker ID in Celery

问题描述

我想使用Celery在具有四张Tesla卡的GPU服务器上运行作业.我用四个工人组成的库来经营芹菜工人，这样每张卡片总是可以完成一项工作.

I want to use Celery to run jobs on a GPU server with four Tesla cards. I run the Celery worker with a pool of four workers such that each card always runs one job.

我的问题是如何指示工人每个要求一个GPU.目前，我假设工作进程都应具有连续的进程ID:

My problem is how to instruct the workers to each claim one GPU. Currently I rely on the assumption that the worker processes should all have contiguous process IDs:

device_id = os.getpid() % self.ndevices

但是，我不能保证它总是能正常工作，即当工作进程随时间重新启动时.因此，理想情况下，我想直接获取每个工人的ID.有人可以告诉我是否可以从任务中检查工作人员，或者可以提出其他解决方案来在GPU上分配工作吗?

However, I this is not guaranteed to always work, i.e. when worker processes get restarted over time. So ideally, I would like to get the ID of each worker directly. Can someone tell me if it is possible to inspect the worker from within a task or can suggest a different solution to distribute the jobs across the GPUs?

推荐答案

如果使用的是 CELERYD_POOL ='processes'，则工作池由 billiard 处理.确实暴露了其基于0的进程索引:

If you are using CELERYD_POOL = 'processes', the worker pool is handled by billiard, which does happen to expose its 0-based process index:

from billiard import current_process
from celery import task

@task
def print_info():
    # This will print an int in [0..concurrency[
    print current_process().index

index 是基于0的，并且如果某个工作程序刚好重新启动，它将保留其索引.

The index is 0-based, and if a worker happens to be restarted it will keep its index.

我找不到有关 index 值的任何文档:/

I couldn't find any documentation regarding the index value though :/

这篇关于获取芹菜的工人ID的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！