池映射未使用所有可用资源的可能原因 [英] Possible reasons why Pool map is not using all available resources

问题描述

我正在运行以下代码

from multiprocessing import Pool


def loop_f(x, num_loops):
    for i in range(num_loops):
        f(x)
    return 

def f(x):
    result = 0 
    for i in range(x):
        result = result*i
    return result

x = 200000
num_times=200
for i in range(8):
    p = Pool(i +1)
    print(i+1)
    %time res=p.map(f, [x]*num_times)

现在，当我运行此代码时，我看到性能改进在第 4 个进程后停止

Now when I run this code I see that the performance improvement stops after the 4th process

Timing when using  1  processes
CPU times: user 9.08 ms, sys: 13.4 ms, total: 22.5 ms
Wall time: 1.17 s
Timing when using  2  processes
CPU times: user 0 ns, sys: 12.1 ms, total: 12.1 ms
Wall time: 598 ms
Timing when using  3  processes
CPU times: user 5.51 ms, sys: 5.6 ms, total: 11.1 ms
Wall time: 467 ms
Timing when using  4  processes
CPU times: user 9.1 ms, sys: 479 µs, total: 9.58 ms
Wall time: 348 ms
Timing when using  5  processes
CPU times: user 4.15 ms, sys: 4.51 ms, total: 8.66 ms
Wall time: 352 ms
Timing when using  6  processes
CPU times: user 6.85 ms, sys: 2.74 ms, total: 9.59 ms
Wall time: 343 ms
Timing when using  7  processes
CPU times: user 2.79 ms, sys: 7.16 ms, total: 9.95 ms
Wall time: 349 ms
Timing when using  8  processes
CPU times: user 9.06 ms, sys: 427 µs, total: 9.49 ms
Wall time: 362 ms

但是当我检查我的系统时，我应该可以访问 8 个处理器内核.

But when I check my system, I should have access to at 8 processor cores.

import multiprocessing
import os

print(multiprocessing.cpu_count())
print(len(os.sched_getaffinity(0)))

8
8

那么发生了什么，或者可能发生了什么?如何最大限度地提高系统的性能?

So what's happening, or possibly happening? How can I maximize my system's performance?

推荐答案

我的机器实际上只有 4 个内核:https://ark.intel.com/content/www/us/en/ark/products/75056/intel-xeon-processor-e3-1270-v3-8m-cache-3-50-ghz.html

My machine actually only has 4 cores: https://ark.intel.com/content/www/us/en/ark/products/75056/intel-xeon-processor-e3-1270-v3-8m-cache-3-50-ghz.html

import multiprocessing
import os

print(multiprocessing.cpu_count())
print(len(os.sched_getaffinity(0)))

不报告核心数只报告线程数

Does not report the number of cores only the number of threads

这篇关于池映射未使用所有可用资源的可能原因的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！