python-rq worker 自动关闭 [英] python-rq worker closes automatically

问题描述

我正在实现 python-rq 以在队列中传递域并使用 Beautiful Soup 抓取它.所以我正在运行多个工人来完成工作.到目前为止，我开始了 22 个工人，所有 22 个工人都在 rq 仪表板中注册.但是一段时间后，工作人员自行停止并且没有显示在仪表板中.但是在 webmin 中，它将所有工作人员显示为正在运行.爬行速度也降低了，即工人没有跑步.我尝试使用 supervisor 和 nohup 运行工人.在这两种情况下，工人都会自行停止.

I am implementing python-rq to pass domains in a queue and scrape it using Beautiful Soup. So i am running multiple workers to get the job done. I started 22 workers as of now, and all the 22 workers is registered in the rq dashboard. But after some time the worker stops by itself and is not getting displayed in dashboard. But in webmin, it displays all workers as running. The speed of crawling has also decreased i.e. the workers are not running. I tried running the worker using supervisor and nohup. In both the cases the workers stops by itself.

这是什么原因?为什么工人会自己停下来?我们可以在单个服务器中启动多少个工作器?

What is the reason for this? Why does workers stops by itself? And how many workers can we start in a single server?

与此同时，每当工作人员从 rq 仪表板中取消注册时，失败计数就会增加.我不明白为什么?

Along with that, whenever a worker is unregistered from the rq dashboard, the failed count increases. I don't understand why?

请帮我解决这个问题.谢谢

Please help me with this. Thank You

推荐答案

好的，我找到了问题所在.这是因为工人超时.

Okay I figured out the problem. It was because of worker timeout.

try:
  --my code goes here--
except Exception, ex:
  self.error += 1
  with open("error.txt", "a") as myfile:
     myfile.write('\n%s' % sys.exc_info()[0] + "{}".format(self.url))
  pass

因此，根据我的代码，如果从每个域中获取 200 个 url(s)，则下一个域将出列.但是对于某些域，条件终止的 url 数量不足(例如只有 1 或 2 个 url).

So according to my code, the next domain is dequeued if 200 url(s) is fetched from each domain. But for some domains there were insufficient number of urls for the condition to terminate (like only 1 or 2 urls).

由于代码捕获了所有异常并附加到 error.txt 文件中.甚至 rq 超时异常 rq.timeouts.JobTimeoutException 也被捕获并附加到文件中.从而使 worker 等待 x 时间，从而导致 worker 终止.

Since the code catches all the exception and appends to error.txt file. Even the rq timeout exception rq.timeouts.JobTimeoutException was caught and was appended to the file. Thus making the worker to wait for x amount of time, which leads to termination of the worker.

这篇关于python-rq worker 自动关闭的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！