使用 django celery 在 redis 上丢失连接时仅执行部分任务 [英] only part of task executed when connection lost on redis using django celery

查看：95
本文介绍了使用 django celery 在 redis 上丢失连接时仅执行部分任务的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我有一个仅部分执行的 django celery 任务.

I have a django celery task that is only partly executing.

我启动了应用程序并且连接看起来不错:

I start up the app and the connection looks good:

INFO/MainProcess] 连接到redis://elasticache.cache.amazonaws.com:6379/0 [2018-02-1723:27:24,314:INFO/MainProcess] 混合:搜索邻居[2018-02-17 23:27:25,339: INFO/MainProcess] 混合:独自一人[2018-02-17 23:27:25,604: INFO/MainProcess]worker1@test_vmstracker_com 准备好了.

INFO/MainProcess] Connected to redis://elasticache.cache.amazonaws.com:6379/0 [2018-02-17 23:27:24,314: INFO/MainProcess] mingle: searching for neighbors [2018-02-17 23:27:25,339: INFO/MainProcess] mingle: all alone [2018-02-17 23:27:25,604: INFO/MainProcess] worker1@test_vmstracker_com ready.

我启动进程并接收到任务并执行:

I initiate the process and the task is received an executed:

[2018-02-17 23:27:49,810: INFO/MainProcess] 收到的任务:tracking.tasks.escalate[92f54d48202]预计到达时间:[2018-02-1807:27:59.797380+00:00] [2018-02-17 23:27:49,830: INFO/MainProcess]收到任务:tracking.tasks.escalate[09a0aebef72b]预计到达时间:[2018-02-1807:28:19.809712+00:00] [2018-02-17 23:28:00,205:警告/ForkPoolWorker-7] -我的应用程序正在运行-

[2018-02-17 23:27:49,810: INFO/MainProcess] Received task: tracking.tasks.escalate[92f54d48202] ETA:[2018-02-18 07:27:59.797380+00:00] [2018-02-17 23:27:49,830: INFO/MainProcess] Received task: tracking.tasks.escalate[09a0aebef72b] ETA:[2018-02-18 07:28:19.809712+00:00] [2018-02-17 23:28:00,205: WARNING/ForkPoolWorker-7] -my app is working-

然后我开始收到错误，它没有完成我的应用程序发送电子邮件的任务

Then I start getting errors and it doesn't finish the task where my app sends an email

[2018-02-17 23:28:00,214: ERROR/ForkPoolWorker-7] 连接到 Redis丢失:现在重试 (0/20).[2018-02-17 23:28:00,220:ERROR/ForkPoolWorker-7] 与 Redis 的连接丢失:重试 (1/2

[2018-02-17 23:28:00,214: ERROR/ForkPoolWorker-7] Connection to Redis lost: Retry (0/20) now. [2018-02-17 23:28:00,220: ERROR/ForkPoolWorker-7] Connection to Redis lost: Retry (1/2

有谁知道为什么只有执行然后连接丢失?

这是完整的堆栈跟踪:

[2018-02-17 23:28:19,382: WARNING/ForkPoolWorker-7] /usr/local/lib/python3.6/site-packages/celery/app/trace.py:549: RuntimeWarning: Exception raised outside body: ConnectionError("Error while reading from socket: ('Connection closed by server.',)",):
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 177, in _read_from_socket
    raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)
OSError: Connection closed by server.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2879, in execute
    return execute(conn, stack, raise_on_error)
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2764, in _execute_transaction
    self.parse_response(connection, '_')
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2838, in parse_response
    self, connection, command_name, **options)
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 680, in parse_response
    response = connection.read_response()
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 624, in read_response
    response = self._parser.read_response()
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 284, in read_response
    response = self._buffer.readline()
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 216, in readline
    self._read_from_socket()
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 191, in _read_from_socket
    (e.args,))
redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 177, in _read_from_socket
    raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)
OSError: Connection closed by server.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 434, in trace_task
    uuid, retval, task_request, publish_result,
  File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 152, in mark_as_done
    self.store_result(task_id, result, state, request=request)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 309, in store_result
    request=request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 652, in _store_result
    self.set(self.get_key_for_task(task_id), self.encode(meta))
  File "/usr/local/lib/python3.6/site-packages/celery/backends/redis.py", line 213, in set
    return self.ensure(self._set, (key, value), **retry_policy)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/redis.py", line 203, in ensure
    **retry_policy)
  File "/usr/local/lib/python3.6/site-packages/kombu/utils/functional.py", line 333, in retry_over_time
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/redis.py", line 222, in _set
    pipe.execute()
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2894, in execute
    return execute(conn, stack, raise_on_error)
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2764, in _execute_transaction
    self.parse_response(connection, '_')
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2838, in parse_response
    self, connection, command_name, **options)
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 680, in parse_response
    response = connection.read_response()
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 624, in read_response
    response = self._parser.read_response()
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 284, in read_response
    response = self._buffer.readline()
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 216, in readline
    self._read_from_socket()
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 191, in _read_from_socket
    (e.args,))
redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',)

  exc, exc_info.traceback)))

推荐答案

问题是我的应用服务器和代理实例太小.我正在使用 ec2.一旦我升级到大型硬件，问题就消失了.无论是ec2实例还是elasticache实例，CPU、内存、网络都太小了.

The problem was that my app server and broker instances were too small. I'm using ec2. As soon as I upgraded to the large hardware, the problem went away. Either the ec2 instance or elasticache instance was too small in terms of CPU, memory, and network.

这篇关于使用 django celery 在 redis 上丢失连接时仅执行部分任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
其他开发最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆