Python 中的 Multiprocessing 中的多线程 [英] Multithreading inside Multiprocessing in Python

问题描述

我正在使用 concurrent.futures 模块进行多处理和多线程.我在配备 16GB RAM、intel i7 第 8 代处理器的 8 核机器上运行它.我在 Python 3.7.2 甚至 Python 3.8.2 上试过这个

I am using concurrent.futures module to do multiprocessing and multithreading. I am running it on a 8 core machine with 16GB RAM, intel i7 8th Gen processor. I tried this on Python 3.7.2 and even on Python 3.8.2

import concurrent.futures
import time

获取列表并将每个元素乘以 2

takes list and multiply each elem by 2

def double_value(x):
  y = []
  for elem in x:
    y.append(2 *elem)
  return y

将元素乘以 2

def double_single_value(x):
  return 2* x

定义一个

import numpy as np
a = np.arange(100000000).reshape(100, 1000000)

运行多个线程并将每个元素乘以 2 的函数

function to run multiple thread and multiple each elem by 2

 def get_double_value(x):
  with concurrent.futures.ThreadPoolExecutor() as executor:
    results = executor.map(double_single_value, x)
  return list(results)

下面显示的代码运行了 115 秒.这仅使用多处理.这段代码的 CPU 利用率为 100%

t = time.time()

with concurrent.futures.ProcessPoolExecutor() as executor:
  my_results = executor.map(double_value, a)
print(time.time()-t)

下面的函数花费了超过 9 分钟并消耗了系统的所有 RAM，然后系统杀死了所有进程.这段代码中的 CPU 利用率也没有达到 100% (~85%)

t = time.time()
with concurrent.futures.ProcessPoolExecutor() as executor:
  my_results = executor.map(get_double_value, a)

print(time.time()-t)

我真的很想明白:

(我已经阅读了许多描述多处理和多线程的帖子，我得到的症结之一是多线程用于 I/O 进程和多处理用于 CPU 进程?)

(I have gone through many post that describe multiprocessing and multi-threading and one of the crux that I got is multi-threading is for I/O process and multiprocessing for CPU processes ? )

推荐答案

正如你所说:我已经阅读了许多描述多处理和多线程的帖子，我得到的症结之一是多线程是为了我/O 进程和 CPU 进程的多处理".

As you say: "I have gone through many post that describe multiprocessing and multi-threading and one of the crux that I got is multi-threading is for I/O process and multiprocessing for CPU processes".

你需要弄清楚，你的程序是IO-bound还是CPU-bound，然后应用正确的方法来解决你的问题.随机或同时使用各种方法通常只会让事情变得更糟.

You need to figure out, if your program is IO-bound or CPU-bound, then apply the correct method to solve your problem. Applying various methods at random or all together at the same time usually makes things only worse.

这篇关于Python 中的 Multiprocessing 中的多线程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！