如何限制没有终端或多处理库的python脚本使用的CPU数量? [英] How to limit number of CPU's used by a python script w/o terminal or multiprocessing library?

问题描述

我的主要问题是这里.由于还没有人给出解决方案，我决定找到一种解决方法.我正在寻找一种方法来使用 python 代码限制 python 脚本 CPU 使用率(不是优先级，而是 CPU 内核的数量).我知道我可以使用多处理库(池等)来做到这一点，但我不是使用多处理运行它的人.所以，我不知道该怎么做.我也可以通过终端执行此操作，但此脚本正在由另一个脚本导入.不幸的是，我没有通过终端调用它的奢侈.

My main problem is issued here. Since no one has given a solution yet, I have decided to find a workaround. I am looking for a way to limit a python scripts CPU usage (not priority but the number of CPU cores) with python code. I know I can do that with multiprocessing library (pool, etc.) but I am not the one who is running it with multiprocessing. So, I don't know how to that. And also I could do that via terminal but this script is being imported by another script. Unfortunately, I don't have the luxury of calling it through terminal.

tl;dr:如何限制 python 脚本的 CPU 使用率(核心数)，该脚本正在由另一个脚本导入，我什至不知道它为什么并行运行，没有通过终端运行它.请检查下面的代码片段.

tl;dr: How to limit CPU usage (number of cores) of a python script, which is being imported by another script and I don't even know why it runs in parallel, without running it via terminal. Please check the code snippet below.

导致问题的代码片段:

from sklearn.datasets import load_digits
from sklearn.decomposition import IncrementalPCA
import numpy as np

X, _ = load_digits(return_X_y=True)

#Copy-paste and increase the size of the dataset to see the behavior at htop.
for _ in range(8):
    X = np.vstack((X, X))

print(X.shape)

transformer = IncrementalPCA(n_components=7, batch_size=200)

#PARTIAL FIT RUNS IN PARALLEL! GOD WHY?
---------------------------------------
transformer.partial_fit(X[:100, :])
---------------------------------------
X_transformed = transformer.fit_transform(X)

print(X_transformed.shape)

版本:

• Python 3.6
• joblib 0.13.2
• scikit-learn 0.20.2
• numpy 1.16.2

更新:不起作用.感谢您澄清@Darkonaut.可悲的是，我已经知道这行不通，并且我已经在问题标题上明确说明了，但我猜人们没有阅读.我想我做错了.我已根据@Ben Chaliah Ayoub 答案更新了代码片段.似乎什么都没有改变.而且我还想指出一点:我不想在多个内核上运行这段代码.这行 transformer.partial_fit(X[:100, :]) 在多个内核上运行(出于某种原因)，它没有 n_jobs 或任何东西.另请注意，我的第一个示例和我的原始代码未使用池或类似的东西进行初始化.我一开始就无法设置核心数(因为没有这样的地方).但是现在它有了一个位置，但它仍然在多个内核上运行.随意自己测试一下.(下面的代码)这就是我寻找解决方法的原因.

UPDATE: Doesn't work. Thank you for clarification @Darkonaut . The sad thing is, I already knew this wouldn't work and I already clearly stated on the question title but people don't read I guess. I guess I am doing it wrong. I've updated the code snippet based on the @Ben Chaliah Ayoub answer. Nothing seems to be changed. And also I want to point out to something: I am not trying to run this code on multiple cores. This line transformer.partial_fit(X[:100, :]) running on multiple cores (for some reason) and it doesn't have n_jobs or anything. Also please note that my first example and my original code is not initialized with a pool or something similar. I can't set the number of cores in the first place (Because there is no such place). But now there is a place for it but it is still running on multiple cores. Feel free to test it yourself. (Code below) That's why I am looking for a workaround.

from sklearn.datasets import load_digits
from sklearn.decomposition import IncrementalPCA
import numpy as np
from multiprocessing import Pool, cpu_count
def run_this():
    X, _ = load_digits(return_X_y=True)
    #Copy-paste and increase the size of the dataset to see the behavior at htop.
    for _ in range(8):
        X = np.vstack((X, X))
    print(X.shape)
    #This is the exact same example taken from sckitlearn's IncrementalPCA website.
    transformer = IncrementalPCA(n_components=7, batch_size=200)
    transformer.partial_fit(X[:100, :])
    X_transformed = transformer.fit_transform(X)
    print(X_transformed.shape)
pool= Pool(processes=1)
pool.apply(run_this)

更新:因此，我尝试在我的代码中使用 this 设置 blas 线程在导入 numpy 但它没有工作(再次)之前.还有其他建议吗?最新阶段的代码可以在下面找到.

UPDATE: So, I have tried to set blas threads using this in my code before importing numpy but it didn't work (again). Any other suggestions? The latest stage of code can be found below.

致谢: @Amir

from sklearn.datasets import load_digits
from sklearn.decomposition import IncrementalPCA
import os
os.environ["OMP_NUM_THREADS"] = "1" # export OMP_NUM_THREADS=1
os.environ["OPENBLAS_NUM_THREADS"] = "1" # export OPENBLAS_NUM_THREADS=1
os.environ["MKL_NUM_THREADS"] = "1" # export MKL_NUM_THREADS=1
os.environ["VECLIB_MAXIMUM_THREADS"] = "1" # export VECLIB_MAXIMUM_THREADS=1
os.environ["NUMEXPR_NUM_THREADS"] = "1" # export NUMEXPR_NUM_THREADS=1

import numpy as np

X, _ = load_digits(return_X_y=True)

#Copy-paste and increase the size of the dataset to see the behavior at htop.
for _ in range(8):
    X = np.vstack((X, X))

print(X.shape)
transformer = IncrementalPCA(n_components=7, batch_size=200)

transformer.partial_fit(X[:100, :])

X_transformed = transformer.fit_transform(X)

print(X_transformed.shape)

推荐答案

我通过设置 BLAS 环境变量(来自 这个链接).但这不是我实际问题的答案.我的第一次尝试(第二次更新)是错误的.我需要设置线程数不是在导入 numpy 库之前，而是在库(IncrementalPCA)导入 numpy 之前.
那么，示例代码中的问题是什么?这不是一个实际问题，而是 numpy 库使用的 BLAS 库的一个特性.尝试使用多处理库限制它不起作用，因为默认情况下 OpenBLAS 设置为使用所有可用线程.
致谢: @Amir 和 @Darkonaut来源:OpenBLAS 1，OpenBLAS 2, 解决方案

I solved the problem in the example code given in the original question by setting BLAS environmental variables (from this link). But this is not the answer to my actual question. My first try (second update) was wrong. I needed to set the number of threads not before importing the numpy library but before the library (IncrementalPCA) importing the numpy.
So, what was the problem in the example code? It wasn't an actual problem but a feature of BLAS library used by numpy library. Trying to limit it with multiprocessing library didn't work because by default OpenBLAS is set to use all available threads.
Credits: @Amir and @Darkonaut Sources: OpenBLAS 1, OpenBLAS 2, Solution

import os
os.environ["OMP_NUM_THREADS"] = "1" # export OMP_NUM_THREADS=1
os.environ["OPENBLAS_NUM_THREADS"] = "1" # export OPENBLAS_NUM_THREADS=1
os.environ["MKL_NUM_THREADS"] = "1" # export MKL_NUM_THREADS=1
os.environ["VECLIB_MAXIMUM_THREADS"] = "1" # export VECLIB_MAXIMUM_THREADS=1
os.environ["NUMEXPR_NUM_THREADS"] = "1" # export NUMEXPR_NUM_THREADS=1
from sklearn.datasets import load_digits
from sklearn.decomposition import IncrementalPCA


import numpy as np

X, _ = load_digits(return_X_y=True)

#Copy-paste and increase the size of the dataset to see the behavior at htop.
for _ in range(8):
    X = np.vstack((X, X))

print(X.shape)
transformer = IncrementalPCA(n_components=7, batch_size=200)

transformer.partial_fit(X[:100, :])

X_transformed = transformer.fit_transform(X)

print(X_transformed.shape)

但是您可以通过检查哪一个来明确设置正确的 BLAS 环境您的 numpy 构建使用如下:

But you can explicitly set the correct BLAS environment by checking which one is used by your numpy build like this:

>>>import numpy as np
>>>np.__config__.show()

给出了这些结果...

blas_mkl_info:
  NOT AVAILABLE
blis_info:
  NOT AVAILABLE
openblas_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
blas_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
lapack_mkl_info:
  NOT AVAILABLE
openblas_lapack_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
lapack_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]

...意思是我的 numpy 构建使用了 OpenBLAS.我只需要编写 os.environ["OPENBLAS_NUM_THREADS"] = "2" 以限制 numpy 库的线程使用.

...meaning OpenBLAS is used by my numpy build. And all I need to write is os.environ["OPENBLAS_NUM_THREADS"] = "2" in order to limit thread usage by the numpy library.

这篇关于如何限制没有终端或多处理库的python脚本使用的CPU数量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！