python 3.6 multiprocessing.Pool() 开始使用 Windows 非常慢 [英] python 3.6 multiprocessing.Pool() very slow to start using Windows

问题描述

在 Windows 10 上运行 Python 3.6.4(64 位)，AMD Threadripper 16 核 CPU，64 GB RAM，快速 SSD.在我开始此测试之前，没有其他任何东西在运行(<2% CPU)或使用 RAM(55GB 空闲).除了调用 mp.Pool() 来设置工作进程之外，一切都运行得很快.设置后，运行 pool.map() 会按预期快速运行.注意:在本地运行并使用和不使用虚拟环境进行测试.

Running Python 3.6.4 (64 bit) on Windows 10, AMD Threadripper 16 core CPU, 64 GB RAM, fast SSD. Nothing else is running (<2% CPU) or using RAM (55GB Free) before I start this test. Everything runs fast except calling mp.Pool() to setup the worker processes. Once setup, running pool.map() is fast as expected. Notes: Running local and tested with and w/o virtual env.

任何修复、解决方法、想法或解释将不胜感激.谢谢.

Any fixes, workarounds, ideas or explanations would be appreciated. Thanks.

import multiprocessing as mp
import time

for x in range(2,15):
    t0 = time.perf_counter()
    with mp.Pool(processes=x) as pool:
        pass
    print('Done {0} processes in {1:.2f}s'.format(x, time.perf_counter() - t0))

<小时>

Done 2 processes in 0.79s
Done 3 processes in 1.34s
Done 4 processes in 2.18s
...
Done 12 processes in 6.44s
Done 13 processes in 5.45s
Done 14 processes in 5.73s

推荐答案

几个小时后，我确定是 Bitdefender 设置导致了问题.在防病毒软件下，有一个名为仅扫描新文件和已更改文件"的设置.如果设置为关闭，则会出现性能问题.打开它，时间下降到每 5 个进程 0.11 秒.出于文档目的，我正在运行 Bitdefender AV Plus 2018(截至今天).我向 Bitdefender 报告了这个问题，并将其升级到下一层支持.

After several hours, I determined it was a Bitdefender setting that was causing the problem. Under antivirus there is a setting called, "Scan only new and changed files." If set to Off, the performance problem happens. Turn it on and the timings go down to 0.11s per 5 processes. For documentation purposes, I am running Bitdefender AV Plus 2018 (up to date as of today). I reported the problem to Bitdefender and it was escalated to the next tier of support.

感谢大家的投入.

这篇关于python 3.6 multiprocessing.Pool() 开始使用 Windows 非常慢的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！