假定自动线程scipy和numpy函数没有利用多个内核 [英] Supposed automatically threaded scipy and numpy functions aren't making use of multiple cores
问题描述
我正在运行Mac OS X 10.6.8，并且正在使用Enthought Python发行版.我希望numpy函数能够同时利用我的两个核心.我遇到的问题与此帖子类似: python/numpy中的多线程blas 但是按照那个海报的步骤进行操作之后，我仍然遇到同样的问题.这是我的numpy.show_config():
I am running Mac OS X 10.6.8 and am using the Enthought Python Distribution. I want for numpy functions to take advantage of both my cores. I am having a problem similar to that of this post: multithreaded blas in python/numpy but after following through the steps of that poster, I still have the same problem. Here is my numpy.show_config():
lapack_opt_info:
libraries = ['mkl_lapack95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_mc', 'mkl_mc3', 'pthread']
library_dirs = ['/Library/Frameworks/EPD64.framework/Versions/1.4.2/lib']
define_macros = [('SCIPY_MKL_H', None)]
include_dirs = ['/Library/Frameworks/EPD64.framework/Versions/1.4.2/include']
blas_opt_info:
libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_mc', 'mkl_mc3', 'pthread']
library_dirs = ['/Library/Frameworks/EPD64.framework/Versions/1.4.2/lib']
define_macros = [('SCIPY_MKL_H', None)]
include_dirs = ['/Library/Frameworks/EPD64.framework/Versions/1.4.2/include']
lapack_mkl_info:
libraries = ['mkl_lapack95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_mc', 'mkl_mc3', 'pthread']
library_dirs = ['/Library/Frameworks/EPD64.framework/Versions/1.4.2/lib']
define_macros = [('SCIPY_MKL_H', None)]
include_dirs = ['/Library/Frameworks/EPD64.framework/Versions/1.4.2/include']
blas_mkl_info:
libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_mc', 'mkl_mc3', 'pthread']
library_dirs = ['/Library/Frameworks/EPD64.framework/Versions/1.4.2/lib']
define_macros = [('SCIPY_MKL_H', None)]
include_dirs = ['/Library/Frameworks/EPD64.framework/Versions/1.4.2/include']
mkl_info:
libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_mc', 'mkl_mc3', 'pthread']
library_dirs = ['/Library/Frameworks/EPD64.framework/Versions/1.4.2/lib']
define_macros = [('SCIPY_MKL_H', None)]
include_dirs = ['/Library/Frameworks/EPD64.framework/Versions/1.4.2/include']
与原始帖子的评论一样，我删除了设置变量MKL_NUM_THREADS=1的行.但是即使那样，应该利用多线程的numpy和scipy函数一次也只使用我的一个内核.还有什么我应该改变的吗?
As in the original post's comments, I deleted the line that set the variable MKL_NUM_THREADS=1. But even then the numpy and scipy functions that should take advantage of multi-threading are only using one of my cores at a time. Is there something else I should change?
为澄清起见，我正在尝试获取一个单独的计算(例如numpy.dot())以根据MKL实现自行使用多线程，我不是在尝试利用numpy计算这一事实释放对GIL的控制，从而使与其他功能的多线程更加容易.
To clarify, I am trying to get one single calculation such as numpy.dot() to use multi-threading on its own as per the MKL implementation, I am not trying to take advantage of the fact that numpy calculations release control of the GIL, hence making multi-threading with other functions easier.
这是一个应该使用多线程但不在我的机器上使用的小脚本:
Here is a small script that should make use of multi-threading but does not on my machine:
import numpy as np
a = np.random.randn(1000, 10000)
b = np.random.randn(10000, 1000)
np.dot(a, b) #this line should be multi-threaded
推荐答案
本文似乎暗示着numpy根据预计的操作速度智能地使某些操作并行进行:
This article seems to imply that numpy intelligently makes certain operations parallel, depending on predicted speedup of the operation:
- 如果您的numpy/scipy是使用其中一种进行编译的，则dot()将在不执行任何操作的情况下并行计算(如果更快)."
也许您的小型(-ish)测试用例不会根据numpy的启发式方法来确定何时并行化特定dot()调用而显示出显着的提速?也许尝试一个大得可笑的操作，看看是否同时使用了两个内核?
Perhaps your small(-ish) test case won't show significant speedup according to numpy's heuristic for determining when to parallelize a particular dot() call? Maybe try a ridiculously large operation and see if both cores are utilized?
请注意，您的处理器/机器配置是否实际上支持BLAS?
As a side note, does your processor/machine configuration actually support BLAS?
这篇关于假定自动线程scipy和numpy函数没有利用多个内核的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
