python在多个CPU内核上散布subprocess.call [英] python spreading subprocess.call on multiple CPU cores

问题描述

我有一个使用subprocess软件包在shell中运行的python代码：

I have a python code that uses the subprocess package to run in shell:

subprocess.call(mycode.py, shell=inshell)

当我执行top命令时，我看到我只使用了〜30％或更少的CPU。
我意识到某些命令可能正在使用磁盘而不是cpu，因此我在确定速度。
在linux系统上运行它的速度似乎比mac 2核心系统慢。

When I execute the top command I see that I am only using ~30% or less of CPU. I realize some commands may be using disk and not cpu therefore I was timing the speed. The speed running this on a linux system seems slower than a mac 2 core system.

如何将其与线程或多处理程序包并行化，以便可以在所述linux系统上使用多个CPU内核？

How do I parallelize this with threading or multiprocessing package so that I can use multiple CPU cores on said linux system?

推荐答案

对FMc的答案稍作改动，

A little change to FMc's answer,

work_items = [(1, 'A', True), (2, 'X', False), (3, 'B', False)]
def worker(tup):
 for i in range(5000000):
     print(work_items)
 return

pool = Pool(processes = 8)
start = time.time()
work_results = pool.map(worker, work_items)
end = time.time()
print(end-start)
pool.close()
pool.join()

上面的代码需要53.60秒。但是，下面的技巧需要27.34秒。

The code above takes 53.60 seconds. The trick below however, takes 27.34 seconds.

from multiprocessing import Pool
import functools
import time

work_items = [(1, 'A', True), (2, 'X', False), (3, 'B', False)]

def worker(tup):
    for i in range(5000000):
        print(work_items)
    return

def parallel_attribute(worker):
    def easy_parallelize(worker, work_items):
        pool = Pool(processes = 8)
        work_results = pool.map(worker, work_items)
        pool.close()
        pool.join()
    from functools import partial 
    return partial(easy_parallelize, worker)

start = time.time()
worker.parallel = parallel_attribute(worker(work_items))
end = time.time()
print(end - start)

两个注释：
1）使用多处理虚拟
并没有太大的区别2）使用Python的部分函数（带嵌套的作用域）的工作原理像aw强大的包装器，可将计算时间减少1/2。参考： https://www.binpress.com/tutorial/simple-python- parallelism / 121

另外，谢谢FMc！

这篇关于python在多个CPU内核上散布subprocess.call的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！