长时间不活动后，芹菜意外关闭 [英] Celery Closes Unexpectedly After Longer Inactivity

问题描述

所以我正在使用RabbitMQ + Celery创建一个简单的RPC体系结构。我有一个RabbitMQ消息代理和一个运行Celery deamon的远程工作者。

第三台服务器公开了一个瘦的RESTful API。收到HTTP请求后，它将任务发送给远程工作程序，等待响应并返回响应。但是我注意到，长时间不活动（例如5分钟内没有传入请求）后，芹菜工人的行为异常。长时间不活动后收到的前3个任务返回此错误：

  exchange.declare：连接意外关闭
  

在执行了三个错误的任务后，它又可以工作了。如果长时间没有任务，则会发生相同的情况。知道吗？

我为芹菜工人准备的初始化脚本：

  ＃描述使用同步代理的芹菜工人 
 
控制台日志
 
在运行级别[2345]上启动
在运行级别[！2345]上停止$ b setuid理查德
 setgid理查德
 
脚本
 chdir / usr / local / myproject / myproject 
 exec / usr / local / myproject / venv / bin / celery worker -n celery_worker_deamon。％h -A proj.sync_celery -Q sync_queue -l info --autoscale = 10,3 --autoreload --purge 
结束脚本
 
重生
  

我的芹菜配置：

 ＃同步阻止任务
 BROKER_URL_SYNC ='amqp：// guest：guest @ localhost：5672 //'
＃异步非阻止任务
 BROKER_URL_ASYNC ='amqp：// guest：guest @localhost：5672 //'
 
＃：仅在您的经纪人受到保护的情况下，将泡菜添加到此列表中。
＃：防止不必要的访问（请参阅userguide / security.html）
 = ['json'] 
 CELERY_TASK_SERIALIZER ='j儿子'
 CELERY_RESULT_SERIALIZER ='json'
 CELERY_TIMEZONE ='UTC'
 CELERY_ENABLE_UTC =真
 CELERY_BACKEND ='amqp'
 
＃http：// docs .celeryproject.org / en / latest / userguide / tasks.html＃disable-rate-limits-if-they-re-not-used 
 CELERY_DISABLE_RATE_LIMITS =真
 
＃http：// docs.celeryproject.org/en/latest/userguide/routing.html 
 CELERY_DEFAULT_QUEUE ='sync_queue'
 CELERY_DEFAULT_EXCHANGE =任务 
 CELERY_DEFAULT_EXCHANGE_TYPE =主题 
 CELERY_DEFAULT_ROUT .default 
 CELERY_QUEUES = {
'sync_queue'：{
'binding_key'：'sync_task。＃'，
}，
'async_queue'：{
'binding_key'：'async_task。＃'，
}，
} 
  

有什么想法吗？

编辑：

好，现在看来是随机发生的。我在RabbitMQ日志中注意到了这一点：

  =警告报告==== 2014年1月6日：： 17：31：54 === 
关闭AMQP连接< 0.295.0> （some_ip_address：36842-> some_ip_address：5672）：
 connection_closed_abruptly 
  

解决方案

您的RabbitMQ服务器或Celery工人是否在负载均衡器后面？如果是，则在一段时间不活动后，负载平衡器将关闭TCP连接。在这种情况下，您将必须从客户端（工作人员）端启用心跳。如果这样做，我不建议为此使用纯Python amqp库。而是将其替换为 librabbitmq 。

So I am using a RabbitMQ + Celery to create a simple RPC architecture. I have one RabbitMQ message broker and one remote worker which runs Celery deamon.

There is a third server which exposes a thin RESTful API. When it receives HTTP request, it sends a task to the remote worker, waits for response and returns a response.

This works great most of the time. However I have notices that after a longer inactivity (say 5 minutes of no incoming requests), the Celery worker behaves strangely. First 3 tasks received after a longer inactivity return this error:

exchange.declare: connection closed unexpectedly

After three erroneous tasks it works again. If there are not tasks for longer period of time, the same thing happens. Any idea?

My init script for the Celery worker:

# description "Celery worker using sync broker"

console log

start on runlevel [2345]
stop on runlevel [!2345]

setuid richard
setgid richard

script
chdir /usr/local/myproject/myproject
exec /usr/local/myproject/venv/bin/celery worker -n celery_worker_deamon.%h -A proj.sync_celery -Q sync_queue -l info --autoscale=10,3 --autoreload --purge
end script

respawn

My celery config:

# Synchronous blocking tasks
BROKER_URL_SYNC = 'amqp://guest:guest@localhost:5672//'
# Asynchronous non blocking tasks
BROKER_URL_ASYNC = 'amqp://guest:guest@localhost:5672//'

#: Only add pickle to this list if your broker is secured
#: from unwanted access (see userguide/security.html)
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'UTC'
CELERY_ENABLE_UTC = True
CELERY_BACKEND = 'amqp'

# http://docs.celeryproject.org/en/latest/userguide/tasks.html#disable-rate-limits-if-they-re-not-used
CELERY_DISABLE_RATE_LIMITS = True

# http://docs.celeryproject.org/en/latest/userguide/routing.html
CELERY_DEFAULT_QUEUE = 'sync_queue'
CELERY_DEFAULT_EXCHANGE = "tasks"
CELERY_DEFAULT_EXCHANGE_TYPE = "topic"
CELERY_DEFAULT_ROUTING_KEY = "sync_task.default"
CELERY_QUEUES = {
    'sync_queue': {
        'binding_key':'sync_task.#',
    },
    'async_queue': {
        'binding_key':'async_task.#',
    },
}

Any ideas?

EDIT:

Ok, now it appears to happen randomly. I noticed this in RabbitMQ logs:

=WARNING REPORT==== 6-Jan-2014::17:31:54 ===
closing AMQP connection <0.295.0> (some_ip_address:36842 -> some_ip_address:5672):
connection_closed_abruptly

解决方案

Is your RabbitMQ server or your Celery worker behind a load balancer by any chance? If yes, then the load balancer is closing the TCP connection after some period of inactivity. In which case, you will have to enable heartbeat from the client (worker) side. If you do, I would not recommend using the pure Python amqp lib for this. Instead, replace it with librabbitmq.

这篇关于长时间不活动后，芹菜意外关闭的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！