IPython.parallel 不使用多核? [英] IPython.parallel not using multicore?

问题描述

我正在试验 IPython.parallel 并且只想在不同的引擎上启动几个 shell 命令.

I am experimenting with IPython.parallel and just want to launch several shell command on different engines.

我有以下笔记本:

单元格 0:

from IPython.parallel import Client
client = Client()
print len(client)
5

并启动命令:

单元格 1:

%%px --targets 0 --noblock
!python server.py

单元格 2:

%%px --targets 1 --noblock
!python mincemeat.py 127.0.0.1

单元格 3:

%%px --targets 2 --noblock
!python mincemeat.py 127.0.0.1

它的作用是使用 MapReduce 的 mincemeat 实现.当我启动第一个 !python mincemeat.py 127.0.0.1 时，它使用了大约 100% 的一个内核，然后当我启动第二个时，每个内核都下降到 50%.我在机器上有 4 个内核(+虚拟内核)，可以在直接从终端启动时使用它们，但不能在笔记本中使用.

What it does is it uses the mincemeat implementation of MapReduce. When I launch the first !python mincemeat.py 127.0.0.1 it uses roughly 100 % of one core, then when I launch the second it drops to 50 % each. I have 4 cores (+virtual cores) on the machine and can use them when launching directly from the terminal but not in the Notebook.

有什么我遗漏的吗?我想为每个 !python mincemeat.py 127.0.0.1 命令使用一个内核.

Is there something I am missing? I would like to use one core per !python mincemeat.py 127.0.0.1 command.

为清楚起见，这里还有另一件事不使用多核:

For clarity, here's another thing that's not using multiple cores:

单元格 1:

%%px --targets 0 --noblock

a = 0
for i in xrange(100000):
    for j in xrange(10000):
        a += 1

单元格 2:

%%px --targets 0 --noblock

a = 0
for i in xrange(100000):
    for j in xrange(10000):
        a += 1

我想我错过了什么.我相信如果可用，这两个单元应该运行一个不同的内核.然而，情况似乎并非如此.CPU 使用率再次表明它们共享相同的内核并使用了 50%.我做错了什么?

I suppose I am missing something. I believe those two cells should run one different cores if available. However, it does not seem to be the case. Again the CPU usage shows that they share the same core and use 50 % of it. What did I do wrong?

推荐答案

聊天讨论总结:

CPU 关联 是一种将进程固定到特定 CPU 内核的机制，这里的问题是有时导入 numpy 最终会将 Python 进程固定到 CPU 0，由于链接到特定的 BLAS 库.您可以通过运行此单元来取消锁定所有引擎:

CPU affinity is a mechanism for pinning a process to a particular CPU core, and the issue here is that sometimes importing numpy can end up pinning Python processes to CPU 0, as a result of linking against particular BLAS libraries. You can unpin all of your engines by running this cell:

%%px
import os
import psutil
from multiprocessing import cpu_count

p = psutil.Process(os.getpid())
p.set_cpu_affinity(range(cpu_count()))
print p.get_cpu_affinity()

它使用 multiprocessing.cpu_count 来获取 CPU 的数量，然后将每个引擎与所有 CPU 相关联.

Which uses multiprocessing.cpu_count to get the number of CPUs, and then associates every engine with all CPUs.

IPython 笔记本 探索问题.

An IPython notebook exploring the issue.

这篇关于IPython.parallel 不使用多核?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！