从C ++返回Eigen :: Matrix数组到Python，无需复制 [英] Return Array of Eigen::Matrix from C++ to Python without copying

问题描述

我有一些C ++代码可以生成和处理 Eigen 矩阵的数组.最后，我想在python中使用这些矩阵，并认为这可能是 pybind11 的工作.

I have some C++ code that generates and manipulates arrays of Eigen matrices. In the end I want to use those matrices in python and thought this might be a job for pybind11.

基本上我想要在python中返回的是两个嵌套列表/numpy数组 mat_a(I，4，4)和 mat_b(J，K，4，4).因为我必须在C ++中做很多线性代数工作，所以我想使用Eigen，而我使用的数据结构是 std :: array< std :: array< Eigen :: Matrix4f，2> ;, 3>>mat_b//对于J = 3，K = 2 .现在的问题是如何有效地将其转换为python?

Basically what I want back in python are two nested lists / numpy arrays mat_a(I, 4, 4) and mat_b(J, K, 4, 4). Because I have to do a lot of linear algebra stuff in C++ I wanted to use Eigen and the data structure I used is std::array<std::array<Eigen::Matrix4f, 2>, 3>>> mat_b // for J=3, K=2. The problem now is how to get this to python efficiently?

另外，我想对多个输入 x = [x_0，x_1，...，x_N]进行这些计算，并期望 mat_a(N，I，4，4)和 mat_b(N，J，K，4，4)作为结果.每个 x_i 的计算都是独立的，但我认为也许也可以在C ++中通过 x_i 编写此循环更快.另一方面，如果在C ++中只有固定大小的数组，则任务变得更容易，则此循环也可以移至python.

Additionally I want to perform those calculations for multiple inputs x = [x_0, x_1, ..., x_N] and than expect mat_a(N, I, 4, 4) and mat_b(N, J, K, 4, 4) as result. The calculations for each x_i are independent but I thought maybe it is faster to write this loop over x_i in C++ as well. If on the other hand the task gets easier if we only have fixed sized arrays in C++ this loop can also move to python.

这是我的问题的一些伪代码(I = 5，J = 3，K = 2):

Here is some dummy code of my problem (I=5, J=3, K=2) :

// example.cpp
#include <pybind11/pybind11.h>
#include <pybind11/eigen.h>
#include <pybind11/stl.h>
#include <pybind11/functional.h>
#include <pybind11/stl_bind.h>

#include <array>
#include <vector>
#include <Eigen/Dense>


Eigen::Matrix4f get_dummy(){
    Eigen::Matrix4f mat_a;
    mat_a << 1, 2, 3, 4,
             5, 6, 7, 8,
             9, 8, 7, 6,
             5, 4, 3, 2;
    return mat_a;
}

std::pair< std::vector<std::array<Eigen::Matrix4f, 5> >,
           std::vector<std::array<std::array<Eigen::Matrix4f, 2>, 3> > >  get_matrices(std::vector<float> & x){

    std::vector<std::array<Eigen::Matrix4f, 5> > mat_a(x.size());
    std::vector< std::array< std::array< Eigen::Matrix4f, 2>, 3> > mat_b(x.size());

    //    for (u_int i=0; i< x.size(); i++)
    //        do_stuff(x[i], mat_a[i], mat_b[i]);
    mat_a[0][0] = get_dummy();

    return std::make_pair(mat_a, mat_b);
    }


PYBIND11_MODULE(example, m) {
    m.def("get_dummy", &get_dummy, pybind11::return_value_policy::reference_internal);
    m.def("get_matrices", &get_matrices, pybind11::return_value_policy::reference_internal);
}

我通过以下方式编译代码:

I compile the code via:

c++ -O3 -Wall -shared -std=c++14 -fPIC `python3 -m pybind11 --includes` example.cpp -o example`python3-config --extension-suffix`

并且比在python中使用它还要好:

And than use it in python:

import numpy as np
import example

x = np.zeros(1000)

mat_a, mat_b = get_matrices(x)

print(np.shape(mat_a))
print(np.shape(mat_b))
print(mat_a[0][0])

如果我只想返回一个 Eigen :: Matrix ，它可以快速且据我所知无需复制.但是当我尝试使用 std :: array/std :: vector 嵌套 Eigen:Matrices 时，pybind返回一个嵌套的numpy数组列表，而不是一个多维数组.这是预料之中的，实际上我对它的工作效果印象深刻，但是对我来说，这似乎很慢，尤其是随着数组尺寸的增加.

If I just want to return a single Eigen::Matrix it works fast and as far as I can tell without copying. But when I try to nest the Eigen:Matrices with std::array/std::vector pybind returns a nested list of numpy arrays instead of one multidimensional array. This is as expected and I am actually impressed how well this works but it seems rather slow to me especially as the dimensions of the arrays grow.

问题是我该如何改进它以获得多维的numpy数组而无需不必要的复制.

The question is how can I improve this to get multidimensional numpy arrays without unnecessary copying.

我尝试过但没有用的某些道路(对我而言，这并不意味着它们通常无法使用；我只是无法弄清楚)

Some roads I tried but did not work (for me, what doesn't mean that they do not work in general; I just could not figure it out):

• 使用 Eigen :: Tensor 代替 Eigen:Matrix
的数组
• 在python中创建矩阵，并通过引用将其传递给C ++
• 为array< array< array< Matrix4f，K> ;, J>构建自定义包装器

推荐答案

您最好的选择可能是在python端创建数据，以便对其进行引用和垃圾回收.

Your best option may be to create the data on python side so it gets refcounted and garbage collected.

test.py

import example
import numpy as np

array = np.zeros((3, 2, 4, 4), 'f4')

example.do_math(array, 3, 2)
print(array[0, 0])

example.cpp

#define PY_SSIZE_T_CLEAN
#include <Python.h>

#include <Eigen/Dense>

Eigen::Matrix4f get_dummy() {
    Eigen::Matrix4f mat_a;
    mat_a << 1, 2, 3, 4,
             5, 6, 7, 8,
             9, 8, 7, 6,
             5, 4, 3, 2;
    return mat_a;
}

PyObject * example_meth_do_math(PyObject * self, PyObject * args, PyObject * kwargs) {
    static char * keywords[] = {"array", "rows", "cols", NULL};

    PyObject * array;
    int rows, cols;

    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "Oii", keywords, &array, &rows, &cols)) {
        return NULL;
    }

    Py_buffer view = {};
    if (PyObject_GetBuffer(array, &view, PyBUF_SIMPLE)) {
        return NULL;
    }

    Eigen::Matrix4f * ptr = (Eigen::Matrix4f *)view.buf;

    for (int i = 0; i < rows; ++i) {
        for (int j = 0; j < cols; ++j) {
            ptr[i * cols + j] = get_dummy();
        }
    }

    PyBuffer_Release(&view);
    Py_RETURN_NONE;
}

PyMethodDef module_methods[] = {
    {"do_math", (PyCFunction)example_meth_do_math, METH_VARARGS | METH_KEYWORDS, NULL},
    {},
};

PyModuleDef module_def = {PyModuleDef_HEAD_INIT, "example", NULL, -1, module_methods};

extern "C" PyObject * PyInit_example() {
    PyObject * module = PyModule_Create(&module_def);
    return module;
}

setup.py

from setuptools import Extension, setup

ext = Extension(
    name='example',
    sources=['./example.cpp'],
    extra_compile_args=['-fpermissive'],
    include_dirs=['.'], # add the path of Eigen
    library_dirs=[],
    libraries=[],
)

setup(
    name='example',
    version='0.1.0',
    ext_modules=[ext],
)

从这里开始添加第二个参数并使用两个数组进行计算应该很简单.

It should be trivial from here to add a second parameter and use the two arrays for the calculation.

您可以使用 python setup.py development 构建它.

如果要分发它，则可以使用 python setup.py bdist_wheel 创建一个wheel文件.

if you want to distribute it you can create a wheel file with python setup.py bdist_wheel.

我使用 numpy 创建数据，这确保了数据的基础内存是C连续的.

I used numpy to create the data, this ensures the underlying memory of the data is C contiguous.

此示例保持简单，它使用Matrix4f指针迭代3x2矩阵数组.随意将 ptr 强制转换为 Eigen :: Array< Eigen :: Matrix4f>，3、2> .您无法将其强制转换为 std :: vector ，因为 std :: vector 的内部数据包含一个指针.

This example was kept simple and it uses a Matrix4f pointer to iterate a 3x2 array of matrices. Feel free to cast the ptr to an Eigen::Array<Eigen::Matrix4f>, 3, 2>. You cannot cast it to an std::vector since the internal data of an std::vector contains a pointer.

请注意， std :: vector< std :: array< ...>> 在内存中没有单个连续的数组.改用 Eigen :: Array .

Please note that std::vector<std::array<...>> does not have a single contiguous array in the memory. Use Eigen::Array instead.

这里是使用 Eigen Array Map :

PyObject * example_meth_do_math(PyObject * self, PyObject * args, PyObject * kwargs) {
    static char * keywords[] = {"array", NULL};

    PyObject * array;

    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O", keywords, &array)) {
        return NULL;
    }

    Py_buffer view = {};
    if (PyObject_GetBuffer(array, &view, PyBUF_SIMPLE)) {
        return NULL;
    }

    Eigen::Map<Eigen::Array<Eigen::Matrix4f, 2, 3>> array_map((Eigen::Matrix4f *)view.buf, 2, 3);

    for (int i = 0; i < 2; ++i) {
        for (int j = 0; j < 3; ++j) {
            array_map(i, j) = get_dummy();
        }
    }

    PyBuffer_Release(&view);
    Py_RETURN_NONE;
}

这篇关于从C ++返回Eigen :: Matrix数组到Python，无需复制的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！