多处理.将池进程锁定到单个内核 [英] multiprocessing.Pool processes locked to a single core

问题描述

我正在Ubuntu 12.04上的Python中使用multiprocessing.Pool，但遇到了一个奇怪的问题；当我在Pool上调用map_async时，我生成了8个进程，但是它们都为在8核计算机的单个核上占据优势而苦苦挣扎.完全相同的代码用尽了Macbook Pro中的两个核心以及其他Ubuntu 12.04台式机的所有四个核心(在所有情况下，均使用htop进行测量).

I'm using multiprocessing.Pool in Python on Ubuntu 12.04, and I'm running into a curious problem; When I call map_async on my Pool, I spawn 8 processes, but they all struggle for dominance over a single core of my 8-core machine. The exact same code uses up both of my cores in my Macbook Pro, and all four cores of my other Ubuntu 12.04 desktop (as measured with htop, in all cases).

我的代码太长了，无法发布所有内容，但重要的部分是:

My code is too long to post all of, but the important part is:

P = multiprocessing.Pool()
results = P.map_async( unwrap_self_calc_timepoint, zip([self]*self.xLen,xrange(self.xLen)) ).get(99999999999)
P.close()
P.join()
ipdb.set_trace()

其中，unwrap_self_calc_timepoint是包装器函数，用于根据

where unwrap_self_calc_timepoint is a wrapper function to pass the necessary self argument to a class, based on the advice of this article.

所有三台计算机都使用Python 2.7.3，我真的不知道从哪里开始寻找为什么一台Ubuntu计算机运行正常的原因.关于如何开始缩小问题范围的任何帮助都将有所帮助.谢谢！

All three computers are using Python 2.7.3, and I don't really know where to start in hunting down why that one Ubuntu computer is acting up. Any help as to how to begin narrowing the problem down would be helpful. Thank you!

推荐答案

我遇到了同样的问题，在我的情况下，解决方案是告诉linux在整个处理器上运行，而不是仅在一个处理器上运行:尝试在以下位置添加以下两行:代码的开头:

I had the same problem, in my case the solution was to tell linux to work on the whole processors instead on only one : try adding the 2 following lines at the beginning of your code :

import os os.system("taskset -p 0xfffff %d" % os.getpid())

import os os.system("taskset -p 0xfffff %d" % os.getpid())

这篇关于多处理.将池进程锁定到单个内核的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！