Python Celery与线程库用于运行异步请求 [英] Python Celery versus Threading Library for running async requests

问题描述

我正在运行一个解析大量数据的python方法。由于它非常耗时，因此我想在一个单独的线程上异步运行它，以便用户仍然可以访问网站/ UI。

I am running a python method that parses a lot of data. Since it is time intensive, I would like to run it asynchronously on a separate thread so the user can still access the website/UI.

使用 from线程执行线程如果用户退出站点或他们继续在服务器上运行，导入线程模块将终止？

Do threads using the "from threading import thread" module terminate if a user exits the site or do they continue to run on the server?

使用Celery与仅使用线程模块相比有什么好处？

What would be the advantages of using Celery versus simply using the threading module for something like this?

推荐答案

Python解释器由于（a）著名的全局解释器锁 aka（GIL）。因此，Python中的线程仅在计算和IO可以同时发生时才提供并行性。计算绑定的任务至少在CPython 2或3下不会从Python线程模型中受益。

The Python interpreter is resolutely single threaded due to the (in)famous global interpreter lock aka (GIL). Thus threading in Python only provides parallelism when computation and IO can happen simultaneously. Compute bound tasks will see little benefit from threading in the Python threading model, at least under CPython 2 or 3.

另一方面，这些限制不适用于多处理，这就是您要使用诸如celery之类的排队系统进行的处理。您可以运行多个可以在多核计算机上同时执行的Python辅助实例。

On the other hand, those limitations don't apply to multiprocessing, which is what you'll be doing with a queuing system like celery. You can run several worker instances of Python which can execute simultaneously on multicore machines, or multiple machines for that matter.

如果我理解您的情况，那么您可以在网站上进行交互开始一项长期工作-排队几乎肯定是要走的路。您将获得真正的并行性，并且可以轻松地将处理转移到其他计算机。

If I understand your scenario - an interaction on a web site kicks off a long-running job - queuing is almost certainly the way to go. You get true parallelism and the easy option to move processing to other machines.

这篇关于Python Celery与线程库用于运行异步请求的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！