ThreadPool和Pool用于并行处理 [英] ThreadPool and Pool for parallel processing

问题描述

是否可以通过指定要使用的CPU和内核数，在python中同时使用ThreadPool和Pool来并行化循环?

Is there a way to use both ThreadPool and Pool in python to parallelise a loop by specifying the number of CPUs and cores you wish to use?

例如，我有一个循环执行为:

For example I would have a loop execute as:

from multiprocessing.dummy import Pool as ThreadPool 
from tqdm import tqdm
import numpy as np

def my_function(x):
    return x + 1

pool = ThreadPool(4)
my_array = np.arange(0,1e6,1)


results = list(tqdm(pool.imap(my_function, my_array),total=len(my_array)))

对于4核，但我也想将其分散到多个CPU上，是否有一种简单的方法来适应代码?

For 4 cores but it I wanted to spread these out on multiple CPUs as well, is there a simple way to adapt the code?

推荐答案

您在内核和CPU之间感到困惑.通常，出于所有目的，两者都被认为是相同的(从现在起，我们称它们为处理器).

You are confusing between a core and a CPU. Generally, for all purposes both are considered to be the same(let's call them processor from now on).

在python中创建线程池时，由于python中的全局解释器锁(GIL)，这些线程是用户级线程，并且在同一处理器上运行.由于一次只有一个线程可以控制python解释器.因此，使用(python)线程在数据密集型任务中没有任何真正的并发性.

When creating a thread pool in python, the threads are user level threads and are run on the same processor, due to Global Interpreter Lock(GIL) in python. As only one thread can control the python interpreter at a time. So, using (python)threads we don't get any real concurrency in data-intensive tasks.

该如何解决?简单.生成在不同处理器上运行的多个python进程(每个进程都有自己的解释器).这是使用multi processing(mp)模块的地方，从调用它的父python进程中产生多个进程.

How to solve this? Easy. Spawn multiple python processes running on different processors(each with its own interpreter). This is where the multi processing(mp) module is used, to spawn multiple processes from the parent python process in which it is called.

您可以通过运行htop(在linux，mac上)并分析python进程数来验证这一点.对于mp模块，它们都将与调用pool.map函数的父脚本具有相同的名称.

You can verify this by running htop(on linux, mac) and analysing the number of python processes. In case of mp module, they all will have the same name as the parent script where the pool.map function is called.

• 在8核mac机上代码的时间:39.7秒
• 在同一台机器上为此代码计时:2.9秒(请注意，我最多可以使用8个内核，但出于比较目的，仅使用4个内核)

下面是修改后的代码:

from multiprocessing.dummy import Pool as ThreadPool 
from tqdm import tqdm
import numpy as np
import time
import multiprocessing as mp

def my_function(x):
    return x + 1

pool = ThreadPool(4)
my_array = np.arange(0,1e6,1)


t1 = time.time()
# results = list(tqdm(pool.imap(my_function, my_array),total=len(my_array)))
pool = mp.Pool(processes=4) # Generally, set to 2*num_cores you have
res = pool.map(my_function, my_array)
print("Time taken = ", time.time() - t1)

这篇关于ThreadPool和Pool用于并行处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！