Python CGI队列 [英] Python CGI queue

问题描述

我正在使用Python编写一个相当简单的CGI。我即将把它放到Django等等。总体设置是非常标准的服务器端（ ie 计算在服务器上完成）：

1. 用户上传数据文件并单击运行按钮
   
2. 服务器在后台并行并行作业，使用大量的RAM和处理器电源。 〜5-10分钟后（平均使用情况），程序终止，创建了一个文件的输出和一些.png图文件。
   
3. 服务器显示带有数字和一些摘要文本的网页
   

我不认为有数以百计的人们立即使用这个;然而，由于计算过程需要大量的RAM和处理器功能（每个实例都使用Python的池）来解决大部分CPU密集型任务。

我想知道您是否知道使用排队系统是否值得一试。我遇到了一个名为 beanstalkc 的Python模块，但在页面上它表示它是一个内存中排队系统。

在这种情况下，内存是什么意思？我担心内存，而不仅仅是CPU时间，所以我想确保一次只运行一个作业（或者被保存在RAM中，不管是否收到CPU时间）。

另外，我试图决定是否

• 结果页面（由CGI提供）应该告诉你它的位置排队（直到运行，然后显示实际结果页）
  
  
  
  或
  
  
  
• 用户应将其电子邮件地址提交给CGI将在完成后通过电子邮件发送给结果页面的链接。
  

您认为适用于轻型交通CGI的此类问题的设计方法如何？建议非常感激。

解决方案

绝对使用芹菜。您可以运行一个amqp服务器，或者我认为您可以将该数据库作为邮件的队列。它允许您在后台运行任务，如果需要，可以使用多个工作机器进行处理。如果您使用 django-celery

，也可以执行基于数据库的cron作业

在后台运行任务一样简单：

  @task 
 def add（x，y）：
 return x + y 
  

一个项目我把它分配到4台机器上，它的工作原理很好。

I'm working on a fairly simple CGI with Python. I'm about to put it into Django, etc. The overall setup is pretty standard server side (i.e. computation is done on the server):

1. User uploads data files and clicks "Run" button
2. Server forks jobs in parallel behind the scenes, using lots of RAM and processor power. ~5-10 minutes later (average use case), the program terminates, having created a file of its output and some .png figure files.
3. Server displays web page with figures and some summary text

I don't think there are going to be hundreds or thousands of people using this at once; however, because the computation going on takes a fair amount of RAM and processor power (each instance forks the most CPU-intensive task using Python's Pool).

I wondered if you know whether it would be worth the trouble to use a queueing system. I came across a Python module called beanstalkc, but on the page it said it was an "in-memory" queueing system.

What does "in-memory" mean in this context? I worry about memory, not just CPU time, and so I want to ensure that only one job runs (or is held in RAM, whether it receives CPU time or not) at a time.

Also, I was trying to decide whether

• the result page (served by the CGI) should tell you it's position in the queue (until it runs and then displays the actual results page)
  
  OR
  
  
• the user should submit their email address to the CGI, which will email them the link to the results page when it is complete.

What do you think is the appropriate design methodology for a light traffic CGI for a problem of this sort? Advice is much appreciated.

解决方案

Definitely use celery. You can run an amqp server or I think you can sue the database as a queue for the messages. It allows you to run tasks in the background and it can use multiple worker machines to do the processing if you want. It can also do cron jobs that are database based if you use django-celery

It's as simple as this to run a task in the background:

@task
def add(x, y):
    return x + y

In a project I have it's distributing the work over 4 machines and it works great.

这篇关于Python CGI队列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！