Celery在本地系统上是否像python多处理一样高效? [英] Is Celery as efficient on a local system as python multiprocessing is?

问题描述

在为我的应用程序决定使用python多处理，芹菜或pp时遇到了一些麻烦.

I'm having a bit of trouble deciding whatever to use python multiprocessing or celery or pp for my application.

我的应用程序占用大量CPU，但目前仅使用一个cpu，因此，我需要将其分布在所有可用的cpus上(这使我不得不看一下python的多处理库)，但我读到该库无法扩展到其他计算机如果需要的话.现在，我不确定是否需要一台以上的服务器来运行我的代码，但是我正在考虑在本地运行celery，然后进行扩展仅需要添加新的服务器，而不是重构代码(就像我使用的那样)多处理).

My app is very CPU heavy but currently uses only one cpu so, I need to spread it across all available cpus(which caused me to look at python's multiprocessing library) but I read that this library doesn't scale to other machines if required. Right now I'm not sure if I'll need more than one server to run my code but I'm thinking of running celery locally and then scaling would only require adding new servers instead of refactoring the code(as it would if I used multiprocessing).

我的问题:这种逻辑正确吗?并且在本地使用celery是否有任何负面(性能)(如果事实证明具有多个核心的单个服务器可以完成我的任务)?还是建议使用多处理，然后再将其扩展为其他内容?

My question: is this logic correct? and is there any negative(performance) with using celery locally(if it turns out a single server with multiple cores can complete my task)? or is it more advised to use multiprocessing and grow out of it into something else later?

谢谢！

p.s.这是一个个人学习项目，但也许我有一天想在一家公司担任开发人员，并想学习专业人士的工作方式.

p.s. this is for a personal learning project but I would maybe one day like to work as a developer in a firm and want to learn how professionals do it.

推荐答案

我实际上从未使用过Celery，但是我使用过多处理.

I have actually never used Celery, but I have used multiprocessing.

Celery似乎有几种传递消息(任务)的方法，包括您应该能够在不同计算机上运行worker的方法.因此，不利的一面是消息传递的速度可能比多处理的速度慢，但另一方面，您可能会将负载分散到其他计算机上.

Celery seems to have several ways to pass messages (tasks) around, including ways that you should be able to run workers on different machines. So a downside might be that message passing could be slower than with multiprocessing, but on the other hand you could spread the load to other machines.

很正确，多处理只能在一台计算机上运行.但是另一方面，进程之间的通信可以非常快，例如通过使用共享内存.另外，如果您需要处理大量数据，则可以轻松地在本地磁盘上读写数据，而只需在进程之间传递文件名即可.

You are right that multiprocessing can only run on one machine. But on the other hand, communication between the processes can be very fast, for example by using shared memory. Also if you need to process very large amounts of data, you could easily read and write data from and to the local disk, and just pass filenames between the processes.

我不知道Celery如何处理任务失败.例如，任务可能永远无法完成运行，或者可能崩溃，或者如果任务在特定时限内未完成，则您可能希望具有杀死该任务的能力.我不知道如果没有支持，添加支持将有多困难.

I don't know how well Celery would deal with task failures. For example, task might never finish running, or might crash, or you might want to have the ability to kill a task if it did not finish in certain time limit. I don't know how hard it would be to add support for that if it is not there.

多重处理并没有开箱即用的容错能力，但是您可以自己构建它而不会带来太多麻烦.

multiprocessing does not come with fault tolerance out of the box, but you can build that yourself without too much trouble.

这篇关于Celery在本地系统上是否像python多处理一样高效?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！