RabbitMQ/Celery/Django 内存泄漏? [英] RabbitMQ/Celery/Django Memory Leak?

问题描述

我最近接手了我公司正在进行的项目的另一部分，并发现我们的 RabbitMQ/Celery 设置中似乎存在内存泄漏.

I recently took over another part of the project that my company is working on and have discovered what seems to be a memory leak in our RabbitMQ/Celery setup.

我们的系统有 2Gb 的内存，在任何给定时间大约有 1.8Gb 可用.我们有多个任务处理大量数据并将它们添加到我们的数据库中.

Our system has 2Gb of memory, with roughly 1.8Gb free at any given time. We have multiple tasks that crunch large amounts of data and add them to our database.

当这些任务运行时，它们会消耗相当多的内存，很快我们的可用内存就会下降到 16Mb 到 300Mb 之间的任何地方.问题是，这些任务完成后，内存并没有回来.

When these tasks run, they consume a rather large amount of memory, quickly plummeting our available memory to anywhere between 16Mb and 300Mb. The problem is, after these tasks finish, the memory does not come back.

我们正在使用:

• RabbitMQ v2.7.1
• AMQP 0-9-1/0-9/0-8(从RabbitMQ 启动日志)
• 芹菜 2.4.6
• Django 1.3.1
• amqplib 1.0.2
• django-celery 2.4.2
• kombu 2.1.0
• Python 2.6.6
• erlang 5.8

我们的服务器正在运行 Debian 6.0.4.

Our server is running Debian 6.0.4.

我是此设置的新手，因此如果您需要任何其他信息来帮助我确定此问题的根源，请告诉我.

I am new to this setup, so if there is any other information you need that could help me determine where this problem is coming from, please let me know.

所有任务都有返回值，所有任务都有ignore_result=True，CELERY_IGNORE_RESULT设置为True.

All tasks have return values, all tasks have ignore_result=True, CELERY_IGNORE_RESULT is set to True.

非常感谢您的宝贵时间.

Thank you very much for your time.

我当前的配置文件是:

CELERY_TASK_RESULT_EXPIRES = 30
CELERY_MAX_CACHED_RESULTS = 1
CELERY_RESULT_BACKEND = False
CELERY_IGNORE_RESULT = True
BROKER_HOST = 'localhost'
BROKER_PORT = 5672
BROKER_USER = c.celery.u
BROKER_PASSWORD = c.celery.p
BROKER_VHOST = c.celery.vhost

推荐答案

我几乎可以肯定您正在使用 DEBUG=True 运行此设置，这会导致内存泄漏.

I am almost certain you are running this setup with DEBUG=True wich leads to a memory leak.

查看这篇文章:禁用 Celery 的 Django 调试.

如果有帮助，我会发布我的配置.

I'll post my configuration in case it helps.

djcelery.setup_loader()
BROKER_HOST = "localhost"
BROKER_PORT = 5672
BROKER_VHOST = "rabbit"
BROKER_USER = "YYYYYY"
BROKER_PASSWORD = "XXXXXXX"

CELERY_IGNORE_RESULT = True
CELERY_DISABLE_RATE_LIMITS = True
CELERY_ACKS_LATE = True
CELERYD_PREFETCH_MULTIPLIER = 1
CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler"
CELERY_ROUTES = ('FILE_WITH_ROUTES',)

这篇关于RabbitMQ/Celery/Django 内存泄漏?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！