基于Python的异步工作流模块:celery工作流和luigi工作流之间有什么区别? [英] Python based asynchronous workflow modules : What is difference between celery workflow and luigi workflow?

问题描述

我正在使用django作为网络框架.我需要一个工作流引擎，该引擎可以执行同步以及异步(批处理任务)任务链.我找到了芹菜和路易吉作为批处理工作流程.我的第一个问题是这两个模块之间有什么区别.

I am using django as a web framework. I need a workflow engine that can do synchronous as well as asynchronous(batch tasks) chain of tasks. I found celery and luigi as batch processing workflow. My first question is what is the difference between these two modules.

Luigi允许我们重新运行失败的任务链，并且仅重新执行失败的子任务.芹菜怎么样:如果我们重新运行链(修复失败的子任务代码后)，它将重新运行已经成功的子任务吗?

Luigi allows us to rerun failed chain of task and only failed sub-tasks get re-executed. What about celery: if we rerun the chain (after fixing failed sub-task code), will it rerun the already succeed sub-tasks?

假设我有两个子任务.第一个创建一些文件，第二个读取这些文件.当我将这些放入芹菜链中时，由于第二项任务中的错误代码，整个链条都失败了.在第二个任务中修复代码后重新运行链时会发生什么?第一个任务会尝试重新创建那些文件吗?

Suppose I have two sub-tasks. The first one creates some files and the second one reads those files. When I put these into chain in celery, the whole chain fails due to buggy code in second task. What happens when I rerun the chain after fixing the code in second task? Will the first task try to recreate those files?

推荐答案

更新:正如Erik指出的那样，对于这种情况，Celery是更好的选择.

Update: As Erik pointed, Celery is better choice for this case.

芹菜:

什么是芹菜?

Celery是一个简单，灵活且可靠的分布式系统，可以处理大量消息，同时为操作提供维护该系统所需的工具.

Celery is a simple, flexible and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system.

为什么使用芹菜?

• 易于使用&具有很多功能.
• django-celery:提供与Django的良好集成.
• flower:Celery分布式任务队列的实时监视器和Web管理员.
• 活动&大型社区(基于Stackoverflow活动，Pyvideos，教程，博客文章).

路易吉

什么是Luigi?

Luigi(Spotify最近开放源代码的Python框架)是一个Python软件包，可帮助您构建复杂的批处理作业管道.它处理依赖关系解析，工作流管理，可视化，处理故障，命令行集成等.

Luigi(Spotify's recently open sourced Python framework) is a Python package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.

为什么要使用Luigi?

• 内置对Hadoop的支持.
• 足够通用，可用于从本地工作站上的简单任务执行和监视，到启动庞大的处理任务链，这些任务可在几天之内在多台计算机之间同步运行.
• Lugi的可视化工具:很好地直观地概述了工作流程的依赖关系图.

结论:如果您只需要简单地安排任务和工具的工具，运行它们，您可以使用Celery.如果您要处理大数据&您可以为Luigi进行大量处理.

Conclusion: If you need a tool just to simply schedule tasks & run them you can use Celery. If you are dealing with big data & huge processing you can go for Luigi.

这篇关于基于Python的异步工作流模块:celery工作流和luigi工作流之间有什么区别?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！