如何在没有终端或多处理库的情况下限制 python 脚本使用的 CPU 数量? [英] How to limit number of CPU's used by a python script w/o terminal or multiprocessing library?

问题描述

我的主要问题在这里.由于还没有人给出解决方案，我决定找到一种解决方法.我正在寻找一种使用 python 代码限制 python 脚本 CPU 使用率的方法(不是优先级，而是 CPU 内核数).我知道我可以使用多处理库(池等)来做到这一点，但我不是使用多处理运行它的人.所以，我不知道该怎么做.而且我也可以通过终端来做到这一点，但是这个脚本是由另一个脚本导入的.不幸的是，我无法通过终端调用它.

My main problem is issued here. Since no one has given a solution yet, I have decided to find a workaround. I am looking for a way to limit a python scripts CPU usage (not priority but the number of CPU cores) with python code. I know I can do that with multiprocessing library (pool, etc.) but I am not the one who is running it with multiprocessing. So, I don't know how to that. And also I could do that via terminal but this script is being imported by another script. Unfortunately, I don't have the luxury of calling it through terminal.

tl;dr:如何限制由另一个脚本导入的 Python 脚本的 CPU 使用率(核心数)，我什至不知道它为什么并行运行，没有通过终端运行它.请检查下面的代码片段.

tl;dr: How to limit CPU usage (number of cores) of a python script, which is being imported by another script and I don't even know why it runs in parallel, without running it via terminal. Please check the code snippet below.

导致问题的代码片段:

from sklearn.datasets import load_digits
from sklearn.decomposition import IncrementalPCA
import numpy as np

X, _ = load_digits(return_X_y=True)

#Copy-paste and increase the size of the dataset to see the behavior at htop.
for _ in range(8):
    X = np.vstack((X, X))

print(X.shape)

transformer = IncrementalPCA(n_components=7, batch_size=200)

#PARTIAL FIT RUNS IN PARALLEL! GOD WHY?
---------------------------------------
transformer.partial_fit(X[:100, :])
---------------------------------------
X_transformed = transformer.fit_transform(X)

print(X_transformed.shape)

版本:

• Python 3.6
• joblib 0.13.2
• scikit-learn 0.20.2
• numpy 1.16.2

更新:不起作用.感谢您澄清@Darkonaut.可悲的是，我已经知道这行不通，而且我已经在问题标题中明确说明了，但我猜人们不会阅读.我想我做错了.我已经根据 @Ben Chaliah Ayoub answer 更新了代码片段.似乎什么都没有改变.而且我还想指出一点:我不是要在多个内核上运行此代码.这行 transformer.partial_fit(X[:100, :]) 在多个内核上运行(出于某种原因)，它没有 n_jobs 或任何东西.另请注意，我的第一个示例和我的原始代码没有用池或类似的东西初始化.我不能首先设置核心数(因为没有这样的地方).但是现在它有一个位置，但它仍然在多个内核上运行.随意自己测试一下.(下面的代码)这就是我寻找解决方法的原因.

UPDATE: Doesn't work. Thank you for clarification @Darkonaut . The sad thing is, I already knew this wouldn't work and I already clearly stated on the question title but people don't read I guess. I guess I am doing it wrong. I've updated the code snippet based on the @Ben Chaliah Ayoub answer. Nothing seems to be changed. And also I want to point out to something: I am not trying to run this code on multiple cores. This line transformer.partial_fit(X[:100, :]) running on multiple cores (for some reason) and it doesn't have n_jobs or anything. Also please note that my first example and my original code is not initialized with a pool or something similar. I can't set the number of cores in the first place (Because there is no such place). But now there is a place for it but it is still running on multiple cores. Feel free to test it yourself. (Code below) That's why I am looking for a workaround.

from sklearn.datasets import load_digits
from sklearn.decomposition import IncrementalPCA
import numpy as np
from multiprocessing import Pool, cpu_count
def run_this():
    X, _ = load_digits(return_X_y=True)
    #Copy-paste and increase the size of the dataset to see the behavior at htop.
    for _ in range(8):
        X = np.vstack((X, X))
    print(X.shape)
    #This is the exact same example taken from sckitlearn's IncrementalPCA website.
    transformer = IncrementalPCA(n_components=7, batch_size=200)
    transformer.partial_fit(X[:100, :])
    X_transformed = transformer.fit_transform(X)
    print(X_transformed.shape)
pool= Pool(processes=1)
pool.apply(run_this)

更新:所以，我尝试在我的代码中使用 this 设置 blas 线程在导入 numpy 之前，但它不起作用(再次).还有其他建议吗?最新阶段的代码可以在下面找到.

UPDATE: So, I have tried to set blas threads using this in my code before importing numpy but it didn't work (again). Any other suggestions? The latest stage of code can be found below.

致谢:@Amir

from sklearn.datasets import load_digits
from sklearn.decomposition import IncrementalPCA
import os
os.environ["OMP_NUM_THREADS"] = "1" # export OMP_NUM_THREADS=1
os.environ["OPENBLAS_NUM_THREADS"] = "1" # export OPENBLAS_NUM_THREADS=1
os.environ["MKL_NUM_THREADS"] = "1" # export MKL_NUM_THREADS=1
os.environ["VECLIB_MAXIMUM_THREADS"] = "1" # export VECLIB_MAXIMUM_THREADS=1
os.environ["NUMEXPR_NUM_THREADS"] = "1" # export NUMEXPR_NUM_THREADS=1

import numpy as np

X, _ = load_digits(return_X_y=True)

#Copy-paste and increase the size of the dataset to see the behavior at htop.
for _ in range(8):
    X = np.vstack((X, X))

print(X.shape)
transformer = IncrementalPCA(n_components=7, batch_size=200)

transformer.partial_fit(X[:100, :])

X_transformed = transformer.fit_transform(X)

print(X_transformed.shape)

推荐答案

我通过设置 BLAS 环境变量(来自 此 链接).但这不是我实际问题的答案.我的第一次尝试(第二次更新)是错误的.我需要设置线程数不是在导入 numpy 库之前，而是在库 (IncrementalPCA) 导入 numpy 之前.
那么，示例代码中的问题是什么?这不是实际问题，而是 numpy 库使用的 BLAS 库的一个功能.尝试使用多处理库对其进行限制不起作用，因为默认情况下 OpenBLAS 设置为使用所有可用线程.
致谢: @Amir 和 @Darkonaut来源:OpenBLAS 1，OpenBLAS 2、解决方案

I solved the problem in the example code given in the original question by setting BLAS environmental variables (from this link). But this is not the answer to my actual question. My first try (second update) was wrong. I needed to set the number of threads not before importing the numpy library but before the library (IncrementalPCA) importing the numpy.
So, what was the problem in the example code? It wasn't an actual problem but a feature of BLAS library used by numpy library. Trying to limit it with multiprocessing library didn't work because by default OpenBLAS is set to use all available threads.
Credits: @Amir and @Darkonaut Sources: OpenBLAS 1, OpenBLAS 2, Solution

import os
os.environ["OMP_NUM_THREADS"] = "1" # export OMP_NUM_THREADS=1
os.environ["OPENBLAS_NUM_THREADS"] = "1" # export OPENBLAS_NUM_THREADS=1
os.environ["MKL_NUM_THREADS"] = "1" # export MKL_NUM_THREADS=1
os.environ["VECLIB_MAXIMUM_THREADS"] = "1" # export VECLIB_MAXIMUM_THREADS=1
os.environ["NUMEXPR_NUM_THREADS"] = "1" # export NUMEXPR_NUM_THREADS=1
from sklearn.datasets import load_digits
from sklearn.decomposition import IncrementalPCA


import numpy as np

X, _ = load_digits(return_X_y=True)

#Copy-paste and increase the size of the dataset to see the behavior at htop.
for _ in range(8):
    X = np.vstack((X, X))

print(X.shape)
transformer = IncrementalPCA(n_components=7, batch_size=200)

transformer.partial_fit(X[:100, :])

X_transformed = transformer.fit_transform(X)

print(X_transformed.shape)

但是您可以通过检查哪一个来明确设置正确的 BLAS 环境由您的 numpy 构建使用，如下所示:

But you can explicitly set the correct BLAS environment by checking which one is used by your numpy build like this:

>>>import numpy as np
>>>np.__config__.show()

给出这些结果...

blas_mkl_info:
  NOT AVAILABLE
blis_info:
  NOT AVAILABLE
openblas_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
blas_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
lapack_mkl_info:
  NOT AVAILABLE
openblas_lapack_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
lapack_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]

...意味着我的 numpy 构建使用了 OpenBLAS.我需要编写的只是 os.environ["OPENBLAS_NUM_THREADS"] = "2" 以限制 numpy 库的线程使用.

...meaning OpenBLAS is used by my numpy build. And all I need to write is os.environ["OPENBLAS_NUM_THREADS"] = "2" in order to limit thread usage by the numpy library.

这篇关于如何在没有终端或多处理库的情况下限制 python 脚本使用的 CPU 数量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！