芹菜-最大限度地减少内存消耗 [英] Celery - minimize memory consumption

问题描述

我们有约300个celeryd进程在64位Ubuntu 10.4下运行，在空闲状态下，每个进程需要约19mb RES，约174mb VIRT，因此-所有进程在空闲状态下大约有6GB RAM.在活动状态下-进程最多需要100mb的RES和约300mb的VIRT

We have ~300 celeryd processes running under Ubuntu 10.4 64-bit , in idle every process takes ~19mb RES, ~174mb VIRT, thus - it's around 6GB of RAM in idle for all processes. In active state - process takes up to 100mb of RES and ~300mb VIRT

每个进程都使用minidom(xml文件<500kb，结构简单)和urllib.

Every process uses minidom(xml files are < 500kb, simple structure) and urllib.

问题是-我们如何减少RAM消耗-至少对于闲置的工人来说，也许一些芹菜或python选项可能会有所帮助?如何确定哪一部分占用了大部分内存?

Quetions is - how can we decrease RAM consuption - at least for idle workers, probably some celery or python options may help? How to determine which part takes most of memory?

UPD :多数民众赞成在航班搜索代理商，一名工人在一个代理商/日期.我们有10个代理商，一个用户搜索== 9个日期，因此每个用户搜索有10 * 9个代理商.

UPD: thats flight search agents, one worker for one agency/date. We have 10 agencies, one user search == 9 dates, thus we have 10*9 agents per one user search.

是否可以按需启动celeryd进程以避免空闲的工作程序(例如Apache上的MaxSpareServers)?

Is it possible start celeryd processes on demand to avoid idle workers(something like MaxSpareServers on apache)?

UPD2:代理生命周期为-发送HTTP请求，等待响应〜10-20秒，解析xml(不到0.02秒)，然后将结果保存到MySQL

UPD2: Agent lifecycle is - send HTTP request, wait for response ~10-20 sec, parse xml( takes less then 0.02s), save result to MySQL

推荐答案

阅读:

http://docs.celeryproject.org/en/latest/userguide/workers.html#concurrency

听起来每个芹菜有一个工人.好像错了每个芹菜你应该有几十个工人.不断增加工作人员的数量(并减少celeryd的数量)，直到系统非常繁忙和缓慢.

It sounds like you have one worker per celeryd. That seems wrong. You should have dozens of workers per celeryd. Keep raising the number of workers (and lowering the number of celeryd's) until your system is very busy and very slow.

这篇关于芹菜-最大限度地减少内存消耗的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！