如何在Django视图中使用python multiprocessing模块 [英] How to use python multiprocessing module in django view

问题描述

我有一个简单的功能，可以遍历URL列表，使用GET检索一些信息并相应地更新数据库(PostgresSQL).该功能运行完美.但是，一次遍历每个URL会花费太多时间.

I have a simple function that go over a list of URLs, using GET to retrieve some information and update the DB (PostgresSQL) accordingly. The function works perfect. However, going over each URL one at a time talking too much time.

使用python，我可以按照以下步骤完成这些任务:

Using python, I'm able to do to following to parallel these tasks:

from multiprocessing import Pool

def updateDB(ip):
     code goes here...

if __name__ == '__main__':
    pool = Pool(processes=4)              # process per core
    pool.map(updateDB, ip)

这工作得很好.但是，我试图找到如何在Django项目上做同样的事情.目前，我有一个函数(视图)可以遍历每个URL以获得信息并更新数据库.

This is working pretty well. However, I'm trying to find how do the same on django project. Currently I have a function (view) that go over each URL to get the information, and update the DB.

我唯一能找到的就是使用Celery，但这似乎对我想执行的简单任务有些强大.

The only thing I could find is using Celery, but this seems to be a bit overpower for the simple task I want to perform.

我可以做些简单的事情还是必须使用芹菜吗?

Is there anything simple that i can do or do I have to use Celery?

推荐答案

目前，我有一个遍历每个URL的函数(视图)来获取 信息，并更新数据库.

Currently I have a function (view) that go over each URL to get the information, and update the DB.

这意味着响应时间对您而言并不重要，如果您的响应时间减少了4(使用4个子进程/线程)，那么您可以在前台(异步地)执行响应时间.如果是这种情况，您可以简单地将示例代码放入视图中.喜欢

It means response time does not matter for you and instead of doing it in the background (asynchronously), you are OK with doing it in the foreground if your response time is cut by 4 (using 4 sub-processes/threads). If that is the case you can simply put your sample code in your view. Like

from multiprocessing import Pool

def updateDB(ip):
     code goes here...

def my_view(request):
    pool = Pool(processes=4)              # process per core
    pool.map(updateDB, ip)
    return HttpResponse("SUCCESS")

但是，如果要在后台异步进行操作，则应使用Celery或遵循@BasicWolf的建议之一.

But, if you want to do it asynchronously in the background then you should use Celery or follow one of @BasicWolf's suggestions.

这篇关于如何在Django视图中使用python multiprocessing模块的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！