并行任务并发与依赖于Python像GNU Make [英] Parallel Tasking Concurrency with Dependencies on Python like GNU Make

问题描述

我正在寻找一个方法或可能的哲学方法，如何做一些像GNU Make在python。目前，我们使用makefile来执行处理，因为makefile在改变单个选项-j x的并行运行中非常好。此外，gnu make已经有内置的依赖关系堆栈，因此添加辅助处理器或处理更多线程的能力只意味着更新单个选项。

例如：

pre> all：dependency_a dependency_b dependency_c

dependency_a：dependency_d
stuff

dependency_b：dependency_d
stuff

dependency_c：dependency_e
stuff

dependency_d：dependency_f
stuff

dependency_e：
stuff
$如果我们做一个标准的单线程操作（-j），那么我们可以使用一个标准的单线程操作（-j 1），操作顺序可以是：

  dependency_f  - > dependency_d  - > dependency_a  - > dependency_b  - > dependency_e \ 
  - >对于两个线程（-j 2），我们可能会看到：
 
 
 
 
 $ b 
  1：dependency_f  - > dependency_d  - > dependency_a  - > dependency_b 
 
 2：dependency_e  - > dependency_c 
  
有没有人对已经构建的包或方法有任何建议？ 
 
 
 
请提前感谢！
解决方案
您可能需要查看水罐。它是一个基于任务的并行化框架，包括依赖关系跟踪。
 
I'm looking for a method or possibly a philosophical approach for how to do something like GNU Make within python.  Currently, we utilize makefiles to execute processing because the makefiles are extremely good at parallel runs with changing single option: -j x.  In addition, gnu make already has the dependency stacks built into it, so adding a secondary processor or the ability to process more threads just means updating that single option.  I want that same power and flexibility in python, but I don't see it.

As an example:
all:  dependency_a dependency_b dependency_c

dependency_a:  dependency_d
    stuff

dependency_b:  dependency_d
    stuff

dependency_c:  dependency_e
    stuff

dependency_d:  dependency_f
    stuff

dependency_e:
    stuff

dependency_f:
    stuff
If we do a standard single thread operation (-j 1), the order of operation might be:
dependency_f -> dependency_d -> dependency_a -> dependency_b -> dependency_e \
             -> dependency_c
For two threads (-j 2), we might see:
1: dependency_f -> dependency_d -> dependency_a -> dependency_b

2: dependency_e -> dependency_c
Does anyone have any suggestions on either a package already built or an approach?  I'm totally open, provided it's a pythonic solution/approach.

Please and Thanks in advance!
解决方案
You might want to have a look a jug.  It's a task-based parallelisation framework that includes dependency tracking.

                        
这篇关于并行任务并发与依赖于Python像GNU Make的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！