Pyramid Web应用程序中长期运行的任务 [英] Long running tasks in Pyramid web app

问题描述

我需要在Web应用程序的后台运行一些任务(检查代码等)，而不会阻塞视图.

I need to run some tasks in background of web app (checking the code out, etc) without blocking the views.

典型的Queue/Celery场景中的问题是，无论最终结果如何，我都必须确保任务能够完成，甚至在Web应用崩溃或重新启动之前都可以幸免，直到任务完成.

The twist in typical Queue/Celery scenario is that I have to ensure that the tasks will complete, surviving even web app crash or restart until those tasks complete, whatever their final result.

我正在考虑在数据库中记录multiprocessing.Pool的参数，并在webapp重新启动时启动所有未完成的任务.这是可行的，但我想知道是否有更简单或更划算的方法?

I was thinking about recording parameters for multiprocessing.Pool in a database and starting all the incomplete tasks at webapp restart. It's doable, but I'm wondering if there's a simpler or more cost-effective aproach?

更新:为什么芹菜本身不干?好吧，我在某些项目中使用了Celery，这确实是一个很好的解决方案，但是对于此任务而言，它是一个很大的方面:它需要一个单独的服务器，通信等，而我所需要的只是产生一些进程/线程，执行一些在它们中工作(git clone ...，svn co ...)并检查它们是否成功.另一个问题是，我需要使解决方案尽可能小，因为我必须使其遵循精心制定的公司准则，程序等，而为使Celery入职而必须经历的人事管理和官僚开销如果可以的话，我宁愿避免.

UPDATE: Why not Celery itself? Well, I used Celery in some projects and it's really a great solution, but for this task it's on the big side: it requires a separate server, communication, etc., while all I need is spawning a few processes/threads, doing some work in them (git clone ..., svn co ...) and checking whether they succeeded or failed. Another issue is that I need the solution to be as small as possible since I have to make it follow elaborate corporate guidelines, procedures, etc., and the human administrative and bureaucratic overhead I'd have to go through to get Celery onboard is something I'd prefer to avoid if I can.

推荐答案

我建议您使用Celery.

I would suggest you to use Celery.

Celery不需要自己的服务器，您可以让一个工作程序在同一台计算机上运行.您也可以使用SQL数据库使用穷人队列"，而不使用真实"队列/消息服务器(例如RabbitMQ)-此设置看起来很像您要描述的内容，只有一个单独的过程来完成运行任务.

Celery does not require its own server, you can have a worker running on the same machine. You can also have a "poor man's queue" using an SQL database instead of a "real" queue/messaging server such as RabbitMQ - this setup would look very much like what you're describing, only with a separate process doing the long-running tasks.

从Web服务器进程中启动长期运行的任务的问题是，在生产环境中，Web工人"通常由Web服务器管理-可以随时派生或杀死多个工人.您的方法的可行性在很大程度上取决于您所使用的Web服务器及其配置.另外，如果有多个工作人员尝试执行某项任务，那么您可能会遇到一些并发问题.

The problem with starting long-running tasks from the webserver process is that in the production environment the web "workers" are normally managed by the webserver - multiple workers can be spawned or killed at any time. The viability of your approach would highly depend on the web server you're using and its configuration. Also, with multiple workers each trying to do a task you may have some concurrency issues.

除了Celery，另一种选择是查看 UWSGI的后台打印程序子系统，特别是如果您已经在使用UWSGI.

Apart from Celery, another option is to look at UWSGI's spooler subsystem, especially if you're already using UWSGI.

这篇关于Pyramid Web应用程序中长期运行的任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！