使用Python将巨大的嵌套循环划分为8个(或更多)进程的巧妙方法是什么? [英] what is the neat way to divide huge nested loops to 8(or more) processes using Python?

问题描述

这一次，我面临一个设计"问题.使用Python，我有一个使用5个参数的数学算法实现.为了找到这5个参数的最佳组合，我使用了5层嵌套循环来枚举给定范围内的所有可能组合.完成所需的时间似乎超出了我的预期.所以我认为是时候使用多线程了...

this time i'm facing a "design" problem. Using Python, I have a implement a mathematical algorithm which uses 5 parameters. To find the best combination of these 5 parameters, i used 5-layer nested loop to enumerate all possible combinations in a given range. The time it takes to finish appeared to be beyond my expectation. So I think it's the time to use multithreading...

嵌套循环的核心任务是计算和保存.在当前代码中，每次计算的结果都将附加到列表中，并且该列表将在程序末尾写入文件中.

The task in the core of nested loops are calculation and saving. In current code, result from every calculation is appended to a list and the list will be written to a file at the end of program.

由于我对任何一种语言都没有太多的多线程处理经验，更不用说Python了，所以我想问一些有关该问题的结构应该如何的提示.即，应如何将计算动态分配给线程，以及线程应如何保存结果，然后将所有结果合并到一个文件中.我希望线程数可以调整.

since I don't have too much experience of multithreading in any language, not to mention Python, I would like to ask for some hints on what should the structure be for this problem. Namely, how should the calculations be assigned to the threads dynamically and how should the threads save results and later combine all results into one file. I hope the number of threads can be adjustable.

任何带有代码的插图将非常有帮助.

Any illustration with code will be very helpful.

非常感谢您抽出宝贵的时间，

thank you very much for your time, I appreciate it.

第二天的更新: 感谢所有有用的答案，现在我知道这是多处理而不是多线程.我总是对这两个概念感到困惑，因为我认为如果它是多线程的，那么操作系统将在可用时自动使用多个处理器来运行它. 今晚我会抽出一些时间来进行多处理.

update of 2nd Day: thanks for all helpful answers, now I know that it is multiprocessing instead of multithreading. I always confuse with these two concepts because I think if it is multithreaded then the OS will automatically use multiple processor to run it when available. I will find time to have some hands-on with multiprocessing tonight.

推荐答案

您可以尝试使用水罐，我为类似问题编写的一个库.您的代码将类似于

You can try using jug, a library I wrote for very similar problems. Your code would then look something like

from jug import TaskGenerator
evaluate = TaskGenerator(evaluate)

for p0 in [1,2,3]:
    for p1 in xrange(10):
        for p2 in xrange(10,20):
             for p3 in [True, False]:
                for p4 in xrange(100):
                    results.append(evaluate(p0,p1,p2,p3,p4))

现在，您可以运行任意数量的进程(如果可以访问计算机集群，甚至可以通过网络运行).

Now you could run as many processes as you'd like (even across a network if you have access to a computer cluster).

这篇关于使用Python将巨大的嵌套循环划分为8个(或更多)进程的巧妙方法是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！