将大型复杂数组从Python传递到C ++-我最好的选择是什么? [英] Passing big complex arrays from Python to C++ - what's my best option?
问题描述
2017/06/13 我尝试按照建议的方法使用boost，但是在花了3天以上的时间尝试使其进行编译和链接后，失败了，我认为这种愚蠢的痛苦方式可能是最快且痛苦程度降低的....所以现在我的代码只是保存一堆C ++然后读取的硕大的文本文件(将数组和数字的复数/虚数部分拆分开).优雅...不...有效...是.
我有一些科学的代码，当前是用Python编写的，但由于循环内的数字3d集成步骤而使其速度变慢.为了克服这个问题，我在C ++中重写了这一特定步骤. (Cython等不是选项).
长话短说:我想尽可能方便，轻松地将数个非常大的复数数组从python代码传输到C ++集成器.我可以手动使用文本或二进制文件来完成此任务，但痛苦的是，但是在我着手之前，我想知道我是否还有更好的选择?
我将Visual Studio用于C ++，将anaconda用于python(不是我的选择！)
是否有任何文件格式或方法可以快速方便地从python保存复数数组，然后在C ++中重新创建它?
非常感谢， 本
我多次使用的简单解决方案是将您的"C ++端"构建为dll(在Linux/OS X上为共享对象)，提供一个简单的方法. ，类似 C的入口点(直整数，指针& co.，没有STL内容)，并将数据通过ctypes传递.
这避免了boost/SIP/Swig/...构建的噩梦，可以保持零拷贝(使用ctypes可以将直指针指向您的numpy数据)，并允许您做任何您想做的事情(尤其是在构建过程中) -方面-无需花钱的distutils，无需提升，也无需任何操作-使用可以在C ++方面构建类似于C的dll的工具进行构建.让您的C ++算法可以从其他语言中调用也具有很好的副作用(实际上，任何一种语言都可以通过某种方式与C库进行接口).
此处是一个快速的人工示例. C ++方面只是:
extern "C" {
double sum_it(double *array, int size) {
double ret = 0.;
for(int i=0; i<size; ++i) {
ret += array[i];
}
return ret;
}
}
必须将其编译为dll(在Windows上)或.so(在Linux上)，并确保导出sum_it函数(对于gcc是自动的，需要使用VC ++的.def文件)./p>
在Python方面，我们可以有一个类似的包装器
import ctypes
import os
import sys
import numpy as np
path = os.path.dirname(__file__)
cdll = ctypes.CDLL(os.path.join(path, "summer.dll" if sys.platform.startswith("win") else "summer.so"))
_sum_it = cdll.sum_it
_sum_it.restype = ctypes.c_double
def sum_it(l):
if isinstance(l, np.ndarray) and l.dtype == np.float64 and len(l.shape)==1:
# it's already a numpy array with the right features - go zero-copy
a = l.ctypes.data
else:
# it's a list or something else - try to create a copy
arr_t = ctypes.c_double * len(l)
a = arr_t(*l)
return _sum_it(a, len(l))
可确保正确封送数据；然后调用该函数就像
import summer
import numpy as np
# from a list (with copy)
print summer.sum_it([1, 2, 3, 4.5])
# from a numpy array of the right type - zero-copy
print summer.sum_it(np.array([3., 4., 5.]))
有关如何使用它的更多信息，请参见 ctypes文档 .另请参见 numpy中的相关文档.
对于复数，情况稍微复杂一些，因为ctypes中没有内置函数；如果我们想在C ++端使用std::complex<double>(非常有保证的可以与numpy复杂布局配合使用，即两个双精度数的序列)，我们可以将C ++编写为:
extern "C" {
std::complex<double> sum_it_cplx(std::complex<double> *array, int size) {
std::complex<double> ret(0., 0.);
for(int i=0; i<size; ++i) {
ret += array[i];
}
return ret;
}
}
然后，在Python端，我们必须复制c_complex布局以获取返回值(或能够构建没有numpy的复杂数组):
class c_complex(ctypes.Structure):
# Complex number, compatible with std::complex layout
_fields_ = [("real", ctypes.c_double), ("imag", ctypes.c_double)]
def __init__(self, pycomplex):
# Init from Python complex
self.real = pycomplex.real
self.imag = pycomplex.imag
def to_complex(self):
# Convert to Python complex
return self.real + (1.j) * self.imag
从ctypes.Structure继承会启用ctypes编组魔术，这是根据_fields_成员执行的；构造函数和其他方法只是为了在Python方面易于使用.
然后，我们必须告诉ctypes返回类型
_sum_it_cplx = cdll.sum_it_cplx
_sum_it_cplx.restype = c_complex
最后以与上一个类似的方式编写我们的包装器:
def sum_it_cplx(l):
if isinstance(l, np.ndarray) and l.dtype == np.complex and len(l.shape)==1:
# the numpy array layout for complexes (sequence of two double) is already
# compatible with std::complex (see https://stackoverflow.com/a/5020268/214671)
a = l.ctypes.data
else:
# otherwise, try to build our c_complex
arr_t = c_complex * len(l)
a = arr_t(*(c_complex(r) for r in l))
ret = _sum_it_cplx(a, len(l))
return ret.to_complex()
如上所述进行测试
# from a complex list (with copy)
print summer.sum_it_cplx([1. + 0.j, 0 + 1.j, 2 + 2.j])
# from a numpy array of the right type - zero-copy
print summer.sum_it_cplx(np.array([1. + 0.j, 0 + 1.j, 2 + 2.j]))
产生预期的结果:
(3+3j)
(3+3j)
2017/06/13 EDIT: I tried using boost as was suggested, but after spending more than 3 days trying to get it to compile and link, and failing, I decided that the stupid painful way was probably the fastest and less painfull.... so now my code just saves a mess of gigantic text files (splitting arrays and the complex/ imaginary parts of the numbers across files) that C++ then reads. Elegant... no.... effective... yes.
I have some scientific code, currently written in Python, that is being slowed down by a numerical 3d integration step within a loop. To overcome this I am re-writing this particular step in C++. (Cython etc is not an option).
Long story short: I want to transfer several very large arrays of complex numbers from the python code to the C++ integrator as conveniently and painlessly as possible. I could do this manually and painfully using text or binary files - but before I embark on this, I was wondering if I have any better options?
I'm using visual studio for C++ and anaconda for python (not my choice!)
Is there any file format or method that would make it quick and convenient to save an array of complex numbers from python and then recreate it in C++?
Many thanks, Ben
An easy solution that I used many times is to build your "C++ side" as a dll (=shared object on Linux/OS X), provide a simple, C-like entrypoint (straight integers, pointers & co., no STL stuff) and pass the data through ctypes.
This avoids boost/SIP/Swig/... build nightmares, can be kept zero-copy (with ctypes you can pass a straight pointer to your numpy data) and allow you to do whatever you want (especially on the build-side - no friggin' distutils, no boost, no nothing - build it with whatever can build a C-like dll) on the C++ side. It has also the nice side-effect of having your C++ algorithm callable from other languages (virtually any language has some way to interface with C libraries).
Here's a quick artificial example. The C++ side is just:
extern "C" {
double sum_it(double *array, int size) {
double ret = 0.;
for(int i=0; i<size; ++i) {
ret += array[i];
}
return ret;
}
}
This has to be compiled to a dll (on Windows) or a .so (on Linux), making sure to export the sum_it function (automatic with gcc, requires a .def file with VC++).
On the Python side, we can have a wrapper like
import ctypes
import os
import sys
import numpy as np
path = os.path.dirname(__file__)
cdll = ctypes.CDLL(os.path.join(path, "summer.dll" if sys.platform.startswith("win") else "summer.so"))
_sum_it = cdll.sum_it
_sum_it.restype = ctypes.c_double
def sum_it(l):
if isinstance(l, np.ndarray) and l.dtype == np.float64 and len(l.shape)==1:
# it's already a numpy array with the right features - go zero-copy
a = l.ctypes.data
else:
# it's a list or something else - try to create a copy
arr_t = ctypes.c_double * len(l)
a = arr_t(*l)
return _sum_it(a, len(l))
which makes sure that the data is marshaled correctly; then invoking the function is as trivial as
import summer
import numpy as np
# from a list (with copy)
print summer.sum_it([1, 2, 3, 4.5])
# from a numpy array of the right type - zero-copy
print summer.sum_it(np.array([3., 4., 5.]))
See the ctypes documentation for more information on how to use it. See also the relevant documentation in numpy.
For complex numbers, the situation is slightly more complicated, as there's no builtin for it in ctypes; if we want to use std::complex<double> on the C++ side (which is pretty much guaranteed to work fine with the numpy complex layout, namely a sequence of two doubles), we can write the C++ side as:
extern "C" {
std::complex<double> sum_it_cplx(std::complex<double> *array, int size) {
std::complex<double> ret(0., 0.);
for(int i=0; i<size; ++i) {
ret += array[i];
}
return ret;
}
}
Then, on the Python side, we have to replicate the c_complex layout to retrieve the return value (or to be able to build complex arrays without numpy):
class c_complex(ctypes.Structure):
# Complex number, compatible with std::complex layout
_fields_ = [("real", ctypes.c_double), ("imag", ctypes.c_double)]
def __init__(self, pycomplex):
# Init from Python complex
self.real = pycomplex.real
self.imag = pycomplex.imag
def to_complex(self):
# Convert to Python complex
return self.real + (1.j) * self.imag
Inheriting from ctypes.Structure enables the ctypes marshalling magic, which is performed according to the _fields_ member; the constructor and extra methods are just for ease of use on the Python side.
Then, we have to tell ctypes the return type
_sum_it_cplx = cdll.sum_it_cplx
_sum_it_cplx.restype = c_complex
and finally write our wrapper, in a similar fashion to the previous one:
def sum_it_cplx(l):
if isinstance(l, np.ndarray) and l.dtype == np.complex and len(l.shape)==1:
# the numpy array layout for complexes (sequence of two double) is already
# compatible with std::complex (see https://stackoverflow.com/a/5020268/214671)
a = l.ctypes.data
else:
# otherwise, try to build our c_complex
arr_t = c_complex * len(l)
a = arr_t(*(c_complex(r) for r in l))
ret = _sum_it_cplx(a, len(l))
return ret.to_complex()
Testing it as above
# from a complex list (with copy)
print summer.sum_it_cplx([1. + 0.j, 0 + 1.j, 2 + 2.j])
# from a numpy array of the right type - zero-copy
print summer.sum_it_cplx(np.array([1. + 0.j, 0 + 1.j, 2 + 2.j]))
yields the expected results:
(3+3j)
(3+3j)
这篇关于将大型复杂数组从Python传递到C ++-我最好的选择是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
