芹菜任务分析 [英] Celery tasks profiling

问题描述

正如我在 top 实用程序 celery 中看到的那样，过程会消耗大量的CPU时间.所以我想对其进行分析.

As I can see in top utility celery procecess consume a lot of CPU time. So I want to profile it.

我可以像这样在开发人员机器上手动进行操作:

I can do it manually on developer machine like so:

python -m cProfile -o test-`date +%Y-%m-%d-%T`.prof ./manage.py celeryd -B

但是要获得准确的时间，我需要在生产机器上对其进行分析.在该计算机(Fedora 14)上，celery由初始化脚本启动.例如

But to have accurate timings I need to profile it on production machine. On that machine (Fedora 14) celery is launched by init scripts. E.g.

service celeryd start

我发现这些脚本最终将最终称为 manage.py celeryd_multi .因此，我的问题是如何告诉 celeryd_multi 在启用剖析的情况下启动celery?在我的情况下，这意味着将 -m cProfile -o out.prof 选项添加到 python .

I have figured out these scripts eventually call manage.py celeryd_multi eventually. So my question is how can I tell celeryd_multi to start celery with profiling enabled? In my case this means add -m cProfile -o out.prof options to python.

非常感谢您的帮助.

推荐答案

我认为您正在混淆两个独立的问题.您可能要处理太多单独的任务，否则单独的任务可能效率低下.

I think you're confusing two separate issues. You could be processing too many individual tasks or an individual task could be inefficient.

您可能知道其中的哪一个是问题，但不清楚是哪个问题.

You may know which of these is the problem, but it's not clear from your question which it is.

要跟踪正在处理的任务数量，建议您查看 celerymon .如果您期望某个特定任务出现的频率更高，那么您可以调查从何处调用该任务.

To track how many tasks are being processed I suggest you look at celerymon. If a particular task appears more often that you would expect then you can investigate where it is getting called from.

对整个celery进行概要分析可能无济于事，因为您将获得很多无法控制的代码.正如您所说的，这也意味着您在生产中运行它时遇到了问题.建议您考虑将分析代码直接添加到任务定义中.

Profiling the whole of celery is probably not helpful as you'll get lots of code that you have no control over. As you say it also means you have a problem running it in production. I suggest you look at adding the profiling code directly into your task definition.

您可以将 cProfile.run('func()')用作celery和您的代码之间的间接层，以便对任务的每次运行进行配置.如果您生成一个唯一的文件名并将其作为第二个参数传递给 run ，则您将拥有一个目录，其中充满了概要文件数据，您可以逐个任务检查该文件，或使用 pstats.add 将多个任务运行组合在一起.

You can use cProfile.run('func()') as a layer of indirection between celery and your code so each run of the task is profiled. If you generate a unique filename and pass it as the second parameter to run you'll have a directory full of profile data that you can inspect on a task-by-task basis, or use pstats.add to combine multiple task runs together.

最后，每任务分析意味着您还可以全局或按任务使用项目代码中的设置来打开或关闭分析，而无需修改服务器上的初始化脚本.

Finally, per-task profiling means you can also turn profiling on or off using a setting in your project code either globally or by task, rather than needing to modify the init scripts on your server.

这篇关于芹菜任务分析的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！