在Django Web应用程序中处理计算密集型任务 [英] Handling computationally intensive tasks in a Django webapp

问题描述

我有一个桌面应用程序，正在移植到Django Webapp.该应用程序包含一些计算量很大的部分(使用numpy，scipy和pandas等库).显然，将计算密集型代码导入webapp并运行它不是一个好主意，因为这将迫使客户端等待响应.

I have a desktop app that I'm in the process of porting to a Django webapp. The app has some quite computationally intensive parts (using numpy, scipy and pandas, among other libraries). Obviously importing the computationally intensive code into the webapp and running it isn't a great idea, as this will force the client to wait for a response.

因此，您必须将这些任务放到后台进程中，该进程会通知客户端(我想是通过AJAX)和/或在完成后将结果存储在数据库中.

Therefore, you'd have to farm these tasks out to a background process that notifies the client (via AJAX, I guess) and/or stores the results in the database when it's complete.

在多个并发用户的情况下，您也不希望所有这些任务同时运行，因为即使有少量并发请求，这也是使服务器瘫痪的好方法.理想情况下，您希望Web应用程序的每个实例将其任务放入作业队列中，然后以最佳方式自动运行它们(基于内核数，可用内存等).

You also don't want all these tasks running in simultaneously in the case of multiple concurrent users, since that is a great way to bring your server to its knees even with a small number of concurrent requests. Ideally, you want each instance of your webapp to put its tasks into a job queue, that then automagically runs them in an optimal way (based on number of cores, available memory, etc.).

有没有好的Python库可以帮助解决此类问题?人们在这种情况下是否使用一般策略?还是只是选择一个好的批处理调度程序并为每个进程生成一个新的Python解释器?

Are there any good Python libraries to help resolve this sort of an issue? Are there general strategies that people use in these kinds of situations? Or is this just a matter of choosing a good batch scheduler and spawning a new Python interpreter for each process?

推荐答案

我们开发了一个Django Web应用程序，该应用程序执行大量计算(在高端服务器上，每个过程需要11到88个小时才能完成).

We developed a Django web app which does heavy computation(Each process will take 11 to 88 hours to complete on high end servers).

Celery: Celery是基于分布式的异步任务队列/作业队列消息传递.它专注于实时操作，但也支持调度.

Celery: Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

芹菜优惠

• 异步运行任务.
• 分布式执行昂贵的进程.
• 定期和/或计划的任务.
• 如果出现问题，请重试任务.

这只是冰山一角.芹菜提供了很多功能.看看文档&常见问题解答.

This is just the tip of iceberg. There are a hell lot of features celery offers. Take a look at documentation & FAQ.

您还需要设计一个非常好的用于工作流程的画布.例如，在多个并发用户的情况下，您不希望所有任务同时运行，因为这是一种资源消耗.另外，您可能想根据当前在线的用户安排任务.

You also need to design a very good canvas for workflow. For example, you don't want all tasks running simultaneously in the case of multiple concurrent users, since it is a resource consumption. Also you might want to schedule tasks based on users who are currently online.

您还需要非常好的数据库设计，高效的算法等.

Also you need very good database design, efficient algorithms and so on.

这篇关于在Django Web应用程序中处理计算密集型任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！