快速转换C / C ++向量到Numpy数组 [英] Fast conversion of C/C++ vector to Numpy array

问题描述

我使用SWIG将一些C ++代码粘合到Python（2.6），并且该粘合剂的一部分包括一段代码，用于将大数据字段（数百万个值）从C ++端转换为Numpy数组。最好的方法我可以实现一个迭代器的类，然后提供一个Python方法：

  def __array __（self， dtype = float）：
 return np.fromiter（self，dtype，self.size（））
  

问题是每个迭代器 next 调用是非常昂贵的，因为它必须经历大约三个或四个SWIG包装器。它需要太长时间。我可以保证C ++数据是连续存储的（因为他们住在std :: vector），它只是觉得像Numpy应该能够采取一个指针，该数据的开头，包含的值的数量旁边，和

有一种方法可以传递 internal_data_ [0] 的指针， c $ c> internal_data_.size（）到numpy，以便它可以直接访问或复制数据而不需要所有的Python开销？

解决方案看起来像唯一真正的解决方案是基于 pybuffer.i ，可以从C + +复制到现有的缓冲区。如果将它添加到SWIG包含文件中：

 ％insert（python）％{
 import numpy as np 
％} 
 
 / *！模板函数将容器的内容复制到已分配的内存
 * buffer 
 * / 
％inline％{
 // ==== ADDED BY numpy.i 
 #include< algorithm> 
 
 template<类型名Container_T> 
 void copy_to_buffer（
 const Container_T& field，
 typename Container_T :: value_type * buffer，
 typename Container_T :: size_type length 
）
 {
 // ValidateUserInput（length == field.size（），
 //目标缓冲区大小错误）; 
 //在这里放你自己的断言或者BAD THINGS CAN HAPPEN 
 
 if（length == field.size（））{
 std :: copy（field.begin ，field.end（），buffer）; 
} 
} 
 // ==== 
 
％} 
 
％define TYPEMAP_COPY_TO_BUFFER（CLASS ...）
％typemap（in）（CLASS :: value_type * buffer，CLASS :: size_type length）
（int res = 0，Py_ssize_t size_ = 0，void * buffer_ = 0）{
 
 res = PyObject_AsWriteBuffer（$ input，& buffer_，& size_）; 
 if（res< 0）{
 PyErr_Clear（）; 
％argument_fail（res，（CLASS :: value_type *，CLASS :: size_type length），
 $ symname，$ argnum）; 
} 
 $ 1 =（$ 1_ltype）buffer_; 
 $ 2 =（$ 2_ltype）（size_ / sizeof（$ * 1_type））; 
} 
％enddef 
 
 
％define ADD_NUMPY_ARRAY_INTERFACE（PYVALUE，PYCLASS，CLASS ...）
 
 TYPEMAP_COPY_TO_BUFFER（CLASS）
 
％template（_copy_to_buffer_ ## PYCLASS）copy_to_buffer< CLASS> ;; 
 
％extend CLASS {
％insert（python）％{
 def __array __（self）：
允许访问此数据为numpy array
a = np.ndarray（shape =（len（self），），dtype = PYVALUE）
 _copy_to_buffer_ ## PYCLASS（self，a）
 return a 
 ％} 
} 
 
％enddef 
  

可以用

 ％template（DumbVectorFloat）DumbVector< double> 
 ADD_NUMPY_ARRAY_INTERFACE（float，DumbVectorFloat，DumbVector< double>）; 
  

然后在Python中执行：

 ＃dvf是DumbVectorFloat的一个实例
 import numpy as np 
 my_numpy_array = np.asarray（dvf）
  pre> 
 
 
这只具有单个Python C ++转换调用的开销，而不是典型长度为N的数组产生的N.。 p> 
 
 
此代码的稍微完整的版本是我的在github上的PyTRT项目。
 
I'm using SWIG to glue together some C++ code to Python (2.6), and part of that glue includes a piece of code that converts large fields of data (millions of values) from the C++ side to a Numpy array. The best method I can come up with implements an iterator for the class and then provides a Python method:
def __array__(self, dtype=float):
    return np.fromiter(self, dtype, self.size())
The problem is that each iterator next call is very costly, since it has to go through about three or four SWIG wrappers. It takes far too long. I can guarantee that the C++ data are stored contiguously (since they live in a std::vector), and it just feels like Numpy should be able to take a pointer to the beginning of that data alongside the number of values it contains, and read it directly.

Is there a way to pass a pointer to internal_data_[0] and the value internal_data_.size() to numpy so that it can directly access or copy the data without all the Python overhead?
解决方案
So it looks like the only real solution is to base something off pybuffer.i that can copy from C++ into an existing buffer. If you add this to a SWIG include file:
%insert("python") %{
import numpy as np
%}

/*! Templated function to copy contents of a container to an allocated memory
 * buffer
 */
%inline %{
//==== ADDED BY numpy.i
#include <algorithm>

template < typename Container_T >
void copy_to_buffer(
        const Container_T& field,
        typename Container_T::value_type* buffer,
        typename Container_T::size_type length
        )
{
//    ValidateUserInput( length == field.size(),
//            "Destination buffer is the wrong size" );
    // put your own assertion here or BAD THINGS CAN HAPPEN

    if (length == field.size()) {
        std::copy( field.begin(), field.end(), buffer );
    }
}
//====

%}

%define TYPEMAP_COPY_TO_BUFFER(CLASS...)
%typemap(in) (CLASS::value_type* buffer, CLASS::size_type length)
(int res = 0, Py_ssize_t size_ = 0, void *buffer_ = 0) {

    res = PyObject_AsWriteBuffer($input, &buffer_, &size_);
    if ( res < 0 ) {
        PyErr_Clear();
        %argument_fail(res, "(CLASS::value_type*, CLASS::size_type length)",
                $symname, $argnum);
    }
    $1 = ($1_ltype) buffer_;
    $2 = ($2_ltype) (size_/sizeof($*1_type));
}
%enddef


%define ADD_NUMPY_ARRAY_INTERFACE(PYVALUE, PYCLASS, CLASS...)

TYPEMAP_COPY_TO_BUFFER(CLASS)

%template(_copy_to_buffer_ ## PYCLASS) copy_to_buffer< CLASS >;

%extend CLASS {
%insert("python") %{
def __array__(self):
    """Enable access to this data as a numpy array"""
    a = np.ndarray( shape=( len(self), ), dtype=PYVALUE )
    _copy_to_buffer_ ## PYCLASS(self, a)
    return a
%}
}

%enddef
then you can make a container "Numpy"-able with
%template(DumbVectorFloat) DumbVector<double>;
ADD_NUMPY_ARRAY_INTERFACE(float, DumbVectorFloat, DumbVector<double>);
Then in Python, just do:
# dvf is an instance of DumbVectorFloat
import numpy as np
my_numpy_array = np.asarray( dvf )
This has only the overhead of a single Python <--> C++ translation call, not the N that would result from a typical length-N array.

A slightly more complete version of this code is part of my PyTRT project at github.

                        
这篇关于快速转换C / C ++向量到Numpy数组的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！