您如何处理芹菜引发的异常(不是您的代码)? [英] How do you deal with an exception raised by celery (not your code)?

问题描述

因此，现在在我的烧瓶应用程序中，我正在使用Celery在远程计算机上部署服务器.现在，我有一个枚举状态，它指示部署过程的生命周期:

So in my flask app right now I am using Celery to deploy servers on remote machines. Right now, I have an enum, status, which indicates the lifecycle of my deployment process:

@celery.task(bind=True)
def deploy_server(self, server_id):
  server = Server.query.get(server_id)
  if not server.can_launch():
    return

  try:
    server.status = RemoteStatus.LAUNCHING
    db.session.commit()

    verify_DNS(server)

    host = server.server.ssh_user + '@' + server.server.ip
    execute(fabric_deploy_server, self, server, hosts=host)

    server.status = RemoteStatus.LAUNCHED
    db.session.commit()
  except Exception as e:
    server.status = RemoteStatus.ERROR
    db.session.commit()
    traceback.print_exc()
    raise e

如您所见，在部署服务器时，其状态将更改为正在启动".如果有异常，它将更改为ERROR.

As you can see, when a server is being deployed, its status is changed to "Launching". If there is an exception, it will be changed to ERROR.

我发现了一个异常，该异常完全绕过了这段代码:当我的芹菜服务器上有太多请求时，我得到了这个异常:

I found one exception which completely bypasses this bloc of code: when I overloaded my celery server with too many requests, I get this exception:

[2017-07-09 18:00:03,127: WARNING/PoolWorker-3] /app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py:542: RuntimeWarning: Exception raised outside body: ConnectionError('max number of clients reached',): 
Traceback (most recent call last): 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 427, in trace_task 
    uuid, retval, task_request, publish_result, 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 152, in mark_as_done 
    self.store_result(task_id, result, state, request=request) 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 309, in store_result 
    request=request, **kwargs) 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 652, in _store_result 
    self.set(self.get_key_for_task(task_id), self.encode(meta)) 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 204, in set 
    return self.ensure(self._set, (key, value), **retry_policy) 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 194, in ensure 
    **retry_policy) 
  File "/app/.heroku/python/lib/python2.7/site-packages/kombu/utils/functional.py", line 333, in retry_over_time 
    return fun(*args, **kwargs) 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 213, in _set 
    pipe.execute() 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 2641, in execute 
    return execute(conn, stack, raise_on_error) 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 2495, in _execute_transaction 
    connection.send_packed_command(all_cmds) 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 538, in send_packed_command 
    self.connect() 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 446, in connect 
    self.on_connect() 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 514, in on_connect 
    if nativestr(self.read_response()) != 'OK': 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 577, in read_response 
    response = self._parser.read_response() 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 255, in read_response 
    raise error 
ConnectionError: max number of clients reached 
  exc, exc_info.traceback))) 

与此有关的最大问题是，此错误在我的尝试/捕获"组之外的某个地方引发.因此，当发生该异常时，我所有的服务器都保留在启动"服务器中.模式而不是错误".

My biggest problem with this is that this error is raised somewhere outside of my Try/Catch bloc. Hence, when this exception occurs, all my servers remain in the "Launching" mode rather than "Error".

推荐答案

在Redis 2.4中，硬编码的最大连接数限制为10,000.在redis 2.6+中，您可以在redis.conf中指定最大客户端数.这也不是问题所在，因为经纪人redis拒绝接受连接是问题所在.

In redis 2.4 there is a hard coded limit of max number of connections which is 10,000. In redis 2.6+ you can specify the max number of clients in redis.conf. also this is not a problem of celery your broker redis refused to accept connections that's the problem.

使用redis CLI设置可以由redis同时处理的最大客户端数.查看 redis客户

Set the max number of clients that can be handled by redis simultaneously using redis CLI. Check out redis clients

这篇关于您如何处理芹菜引发的异常(不是您的代码)?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！