C / C ++载体的快速转换为numpy的数组 [英] Fast conversion of C/C++ vector to Numpy array

问题描述

我用SWIG粘合起来，一些C ++ code到Python（2.6），以及胶部分包括一张code，其将从C数据（百万值）大领域++侧到numpy的阵列。我可以拿出实现对类的迭代器，然后最好的方法提供了一个Python的方法：

I'm using SWIG to glue together some C++ code to Python (2.6), and part of that glue includes a piece of code that converts large fields of data (millions of values) from the C++ side to a Numpy array. The best method I can come up with implements an iterator for the class and then provides a Python method:

def __array__(self, dtype=float):
    return np.fromiter(self, dtype, self.size())

问题是，每个迭代接下来调用是非常昂贵的，因为它要经过三四SWIG包装。它需要的时间太长了。我可以保证，在C ++数据连续存储（因为他们生活在一个std ::向量），它只是感觉像numpy的应该可以拿一个指向数据沿着它包含的值的号码的开头，并直接读取它。

The problem is that each iterator next call is very costly, since it has to go through about three or four SWIG wrappers. It takes far too long. I can guarantee that the C++ data are stored contiguously (since they live in a std::vector), and it just feels like Numpy should be able to take a pointer to the beginning of that data alongside the number of values it contains, and read it directly.

有没有办法通过一个指针 internal_data_ [0] 和值 internal_data_.size（）到numpy的，以便它可以直接没有所有的Python开销访问或复制的数据？

Is there a way to pass a pointer to internal_data_[0] and the value internal_data_.size() to numpy so that it can directly access or copy the data without all the Python overhead?

推荐答案

所以看起来唯一真正的解决办法是什么基地关闭 pybuffer.i ，可以从复制C ++到已有的缓冲区。如果你将它添加到痛饮包含文件：

So it looks like the only real solution is to base something off pybuffer.i that can copy from C++ into an existing buffer. If you add this to a SWIG include file:

%insert("python") %{
import numpy as np
%}

/*! Templated function to copy contents of a container to an allocated memory
 * buffer
 */
%inline %{
//==== ADDED BY numpy.i
#include <algorithm>

template < typename Container_T >
void copy_to_buffer(
        const Container_T& field,
        typename Container_T::value_type* buffer,
        typename Container_T::size_type length
        )
{
//    ValidateUserInput( length == field.size(),
//            "Destination buffer is the wrong size" );
    // put your own assertion here or BAD THINGS CAN HAPPEN

    if (length == field.size()) {
        std::copy( field.begin(), field.end(), buffer );
    }
}
//====

%}

%define TYPEMAP_COPY_TO_BUFFER(CLASS...)
%typemap(in) (CLASS::value_type* buffer, CLASS::size_type length)
(int res = 0, Py_ssize_t size_ = 0, void *buffer_ = 0) {

    res = PyObject_AsWriteBuffer($input, &buffer_, &size_);
    if ( res < 0 ) {
        PyErr_Clear();
        %argument_fail(res, "(CLASS::value_type*, CLASS::size_type length)",
                $symname, $argnum);
    }
    $1 = ($1_ltype) buffer_;
    $2 = ($2_ltype) (size_/sizeof($*1_type));
}
%enddef


%define ADD_NUMPY_ARRAY_INTERFACE(PYVALUE, PYCLASS, CLASS...)

TYPEMAP_COPY_TO_BUFFER(CLASS)

%template(_copy_to_buffer_ ## PYCLASS) copy_to_buffer< CLASS >;

%extend CLASS {
%insert("python") %{
def __array__(self):
    """Enable access to this data as a numpy array"""
    a = np.ndarray( shape=( len(self), ), dtype=PYVALUE )
    _copy_to_buffer_ ## PYCLASS(self, a)
    return a
%}
}

%enddef

那么你可以做一个集装箱numpy的-able以

then you can make a container "Numpy"-able with

%template(DumbVectorFloat) DumbVector<double>;
ADD_NUMPY_ARRAY_INTERFACE(float, DumbVectorFloat, DumbVector<double>);

然后在Python，只是做：

Then in Python, just do:

# dvf is an instance of DumbVectorFloat
import numpy as np
my_numpy_array = np.asarray( dvf )

这只有开销一个Python＆LT; - > C ++编译调用，而不是N个，将导致从一个典型的长度为N阵列

This has only the overhead of a single Python <--> C++ translation call, not the N that would result from a typical length-N array.

一个稍微完整的版本是我的在GitHub上 PyTRT项目。

A slightly more complete version of this code is part of my PyTRT project at github.

这篇关于C / C ++载体的快速转换为numpy的数组的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！