为什么 pool.map 比普通地图慢? [英] Why is pool.map slower than normal map?

查看:110
本文介绍了为什么 pool.map 比普通地图慢?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我正在尝试以下代码:

import multiprocessing
import time
import random

def square(x):
    return x**2

pool = multiprocessing.Pool(4)

l = [random.random() for i in xrange(10**8)]

now = time.time()
pool.map(square, l)
print time.time() - now

now = time.time()
map(square, l)
print time.time() - now

并且 pool.map 版本的运行速度始终比正常的 map 版本慢几秒(19 秒对 14 秒).

and the pool.map version consistently runs several seconds more slowly than the normal map version (19 seconds vs 14 seconds).

我看过这些问题:为什么是多处理.Pool.map 比内置地图慢?多处理.Pool() 比使用普通函数慢他们似乎将其归结为 IPC 开销或磁盘饱和,但我觉得在我的示例中,这些显然不是问题;我没有向/从磁盘写入/读取任何内容,并且计算时间足够长,与多处理节省的总时间相比,IPC 开销似乎应该很小(我估计是这样,因为我正在做工作在 4 个内核而不是 1 个内核上,我应该将计算时间从 14 秒减少到大约 3.5 秒).我不认为我的 CPU 饱和了;检查 cat/proc/cpuinfo 显示我有 4 个内核,但即使我多处理到只有 2 个进程,它仍然比普通 map 函数慢(甚至比 4 个进程慢).还有什么可能会减慢多处理版本的速度?我是否误解了 IPC 开销的缩放方式?

I've looked at the questions: Why is multiprocessing.Pool.map slower than builtin map? and multiprocessing.Pool() slower than just using ordinary functions and they seem to chalk it up to to either IPC overhead or disk saturation, but I feel like in my example those aren't obviously the issue; I'm not writing/reading anything to/from disk, and the computation is long enough that it seems like IPC overhead should be small compared to the total time saved by the multiprocessing (I'm estimating that, since I'm doing work on 4 cores instead of 1, I should cut the computation time down from 14 seconds to about 3.5 seconds). I'm not saturating my cpu I don't think; checking cat /proc/cpuinfo shows that I have 4 cores, but even when I multiprocess to only 2 processes it's still slower than just the normal map function (and even slower than 4 processes). What else could be slowing down the multiprocessed version? Am I misunderstanding how IPC overhead scales?

如果相关,这段代码是用 Python 2.7 编写的,我的操作系统是 Linux Mint 17.2

If it's relevant, this code is written in Python 2.7, and my OS is Linux Mint 17.2

推荐答案

pool.map 将列表拆分为 N 个作业(其中 N 是列表的大小)并将这些作业分派给进程.

pool.map splits a list into N jobs (where N is the size of the list) and dispatches those to the processes.

单个进程所做的工作显示在您的代码中:

The work a single process is doing is shown in your code:

def square(x):
    return x**2

此操作在现代 CPU 上花费的时间很少,无论数量有多大.

This operation takes very little time on modern CPUs, no matter how big the number is.

在您的示例中,您正在创建一个巨大的列表并对每个元素执行不相关的操作.当然,与针对快速循环优化的常规 map 函数相比,IPC 开销会更大.

In your example you're creating a huge list and performing an irrelevant operation on every single element. Of course the IPC overhead will be greater compared to the regular map function which is optimized for fast looping.

为了让您的示例按预期工作,只需向 square 函数添加一个 time.sleep(0.1) 调用.这模拟了一个长时间运行的任务.当然,您可能希望减小列表的大小,否则需要很长时间才能完成.

In order to see your example working as you expect, just add a time.sleep(0.1) call to the square function. This simulates a long running task. Of course you might want to reduce the size of the list or it will take forever to complete.

这篇关于为什么 pool.map 比普通地图慢?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆