从NumPy链接Cython包装的C函数与BLAS [英] Link Cython-wrapped C functions against BLAS from NumPy

问题描述

我想在Cython扩展中使用在使用BLAS子例程的.c文件中定义的某些C函数，例如

I want to use inside a Cython extension some C functions defined in .c files that uses BLAS subroutines, e.g.

cfile.c

double ddot(int *N, double *DX, int *INCX, double *DY, int *INCY);

double call_ddot(double* a, double* b, int n){
    int one = 1;
    return ddot(&n, a, &one, b, &one);
}

（比方说，这些函数的作用不只是调用一个BLAS子例程）

(Let’s say the functions do more than just call one BLAS subroutine)

pyfile.pyx

pyfile.pyx

cimport numpy as np
import numpy as np

cdef extern from "cfile.c":
    double call_ddot(double* a, double* b, int n)

def pyfun(np.ndarray[double, ndim=1] a):
    return call_ddot(&a[0], &a[0], <int> a.shape[0])

setup.py：

from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
from Cython.Distutils import build_ext
import numpy

setup(
    name  = "wrapped_cfun",
    packages = ["wrapped_cfun"],
    cmdclass = {'build_ext': build_ext},
    ext_modules = [Extension("wrapped_cfun.cython_part", sources=["pyfile.pyx"], include_dirs=[numpy.get_include()])]
)

我想要这个包裹可以链接到已安装的NumPy或SciPy使用的同一BLAS库，并且希望可以将其从numpy或scipy作为依赖项在不同操作系统下从PIP安装，而无需任何其他与BLAS相关的依赖项。

I want this package to link against the same BLAS library that the installed NumPy or SciPy are using, and would like it to be installable from PIP under different operating systems using numpy or scipy as dependencies, without any additional BLAS-related dependency.

setup.py 是否有任何可让我完成此任务的方式，使其可以与任何BLAS实现一起使用？

Is there any hack for setup.py that would allow me to accomplish this, in a way that it could work with any BLAS implementation?

更新：
使用MKL，我可以通过修改扩展名对象指向 libmkl_rt ，如果安装了MKL，则可以从numpy中提取它，例如：
Extension（ wrapped_cfun。 cython_part，sources = [ pyfile.pyx]，include_dirs = [numpy.get_include（）]，extra_link_args = [-L {python lib dir的路径}，-l：libmkl_rt。{so，dll，dylib }]）
但是，相同的技巧不适用于OpenBLAS（例如 -l：libopenblasp-r0.2.20.so ）。指向 libblas。{so，dll，dylib} 如果该文件是指向libopenblas的链接将不起作用，但可以正常工作，它是指向libmkl_rt的链接。

Update: With MKL, I can make it work by modifying the Extension object to point to libmkl_rt, which can be extracted from numpy if MKL is installed, e.g.: Extension("wrapped_cfun.cython_part", sources=["pyfile.pyx"], include_dirs=[numpy.get_include()], extra_link_args=["-L{path to python's lib dir}", "-l:libmkl_rt.{so, dll, dylib}"]) However, the same trick does not work for OpenBLAS (e.g. -l:libopenblasp-r0.2.20.so). Pointing to libblas.{so,dll,dylib} will not work if that file is a link to libopenblas, but works fine it it's a link to libmkl_rt.

更新2：
看来OpenBLAS的C函数在结尾处用下划线命名，例如不是 ddot ，而是 ddot _ 。如果将 ddot 更改为 ddot _ l：libopenblas 的代码将起作用.c文件中的c $ c>。我仍然想知道是否存在某种（理想的运行时）机制来检测c文件中应使用的名称。

Update 2: It seems OpenBLAS names their C functions with an underscore at the end, e.g. not ddot but ddot_. The code above with l:libopenblas will work if I change ddot to ddot_ in the .c file. I'm still wondering if there is some (ideally run-time) mechanism to detect which name should be used in the c file.

推荐答案

依靠链接器/加载器提供正确的blas功能的一种替代方法是模拟必要的blas符号的解析（例如 ddot ）并使用包装好的 blas- scipy 在运行时提供的功能。

An alternative to depending on linker/loader to provide the right blas-functionality, would be to emulate resolution of the necessary blas-symbols (e.g. ddot) and to use the wrapped blas-function provided by scipy during the runtime.

不确定，这种方法优于常规构建方式，但希望将其引入您的注意，即使只是因为我发现这种方法很有趣。

Not sure, this approach is superior to the "normal way" of building, but wanted to bring it to your attention, even if only because I find this approach interesting.

这个想法简而言之：

1. 将显式函数指针定义为 ddot -功能，在下面的代码段中称为 my_ddot 。 / li>
   
2. 在要使用<$的位置使用 my_ddot 指针c $ c> ddot -否则。


3. 在加载cython模块时初始化 my_ddot 指针

1. Define an explicit function-pointer to ddot-functionality, called my_ddot in the snippet below.
2. Use my_ddot-pointer where you would use ddot-otherwise.
3. Initialize my_ddot-pointer when the cython-module is loaded with the functionality provided by scipy.

这是一个可以正常工作的原型（我使用C-code-verbatim使代码段独立且轻松可在木星笔记本中测试，请相信您将其转换为所需的格式/类似）：

Here is a working prototype (I use C-code-verbatim to make the snippet standalone and easily testable in a jupiter-notebook, trust you to transform it to format you need/like):

%%cython
# h-file:
cdef extern from *:
    """
    // blas-functionality,
    // will be initialized by cython when module is loaded:
    typedef double (*ddot_t)(int *N, double *DX, int *INCX, double *DY, int *INCY);
    extern ddot_t my_ddot;

    double call_ddot(double* a, double* b, int n);
    """
    ctypedef double (*ddot_t)(int *N, double *DX, int *INCX, double *DY, int *INCY)
    ddot_t my_ddot
    double call_ddot(double* a, double* b, int n)    

# init the functions of the c-library
# with blas-function provided by scipy
from scipy.linalg.cython_blas cimport ddot
my_ddot=ddot

# a simple function to demonstrate, that it works
def ddot_mult(double[:]a, double[:]b):
    cdef int n=len(a)
    return call_ddot(&a[0], &b[0], n)

#-------------------------------------------------
# c-file, added so the example is complete    
cdef extern from *:
    """  
    ddot_t my_ddot;
    double call_ddot(double* a, double* b, int n){
        int one = 1;
        return my_ddot(&n, a, &one, b, &one);
    }
    """
    pass

现在<$可以使用c $ c> ddot_mult ：

import numpy as np
a=np.arange(4, dtype=float)

ddot_mult(a,a)  # 14.0 as expected!

此方法的优点是，distutils无需忙碌，您可以保证使用与scipy相同的blas功能。

An advantage of this approach is, that there is no hustle with distutils and you have a guarantee, to use the same blas-functionality as scipy.

另一种好处：无需重新编译/重新链接就可以在运行时切换使用的引擎（mkl，open_blas甚至是自己的实现）。

Another perk: One could switch the used engine (mkl, open_blas or even an own implementation) during the runtime without the need to recompile/relink.

一方面，还有一些额外的样板代码，还有这样的危险，即某些符号的初始化会被忘记。

On there other hand, there is some additional amount of boilerplate-code and also the danger, that initialization of some symbols will be forgotten.

这篇关于从NumPy链接Cython包装的C函数与BLAS的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！