从ctypes数组获取数据到numpy [英] Getting data from ctypes array into numpy

问题描述

我正在使用Python(通过ctypes)包装的C库来运行一系列计算.在运行的不同阶段，我想将数据放入Python，尤其是numpy数组.

I am using a Python (via ctypes) wrapped C library to run a series of computation. At different stages of the running, I want to get data into Python, and specifically numpy arrays.

我正在使用的包装对数组数据进行两种不同类型的返回(这对我特别感兴趣):

The wrapping I am using does two different types of return for array data (which is of particular interest to me):

• ctypes数组:当我执行type(x)时(其中x是ctypes数组，我得到一个<class 'module_name.wrapper_class_name.c_double_Array_12000'>作为回报.我知道此数据是一个从文档复制内部数据，我可以轻松地将其放入numpy数组:

• ctypes Array: When I do type(x) (where x is the ctypes array, I get a <class 'module_name.wrapper_class_name.c_double_Array_12000'> in return. I know that this data is a copy of the internal data from the documentation and I am able to get it into a numpy array easily:

>>> np.ctypeslib.as_array(x)

这将返回数据的一维numpy数组.

This returns a 1D numpy array of the data.

• ctype指向数据的指针:在这种情况下，从库的文档中可以理解，我正在获取指向已存储并直接用于库的数据的指针.乳清我做type(y)(其中y是指针)我得到<class 'module_name.wrapper_class_name.LP_c_double'>.在这种情况下，我仍然能够像y[0][2]这样的数据建立索引，但是我只能通过一个超级笨拙的方法将其放入numpy中:

• ctype pointer to data: In this case from the library's documentation, I understand that I am getting a pointer to the data stored and used directly to the library. Whey I do type(y) (where y is the pointer) I get <class 'module_name.wrapper_class_name.LP_c_double'>. With this case I am still able to index through the data like y[0][2], but I was only able to get it into numpy via a super awkward:

>>> np.frombuffer(np.core.multiarray.int_asbuffer(
    ctypes.addressof(y.contents), array_length*np.dtype(float).itemsize))

我在旧的numpy邮件列表线程中找到了来自Travis Oliphant ，但不是numpy文档中的内容.如果不尝试这种方法，请尝试以上操作:

I found this in an old numpy mailing list thread from Travis Oliphant, but not in the numpy documentation. If instead of this approach I try as above I get the following:

>>> np.ctypeslib.as_array(y)
...
...  BUNCH OF STACK INFORMATION
...
AttributeError: 'LP_c_double' object has no attribute '__array_interface__'

此np.frombuffer方法是执行此操作的最佳方法还是唯一方法?我愿意接受其他建议，但仍必须使用numpy，因为我还有很多其他后处理代码，这些代码依赖于我要用于此数据的numpy功能.

Is this np.frombuffer approach the best or only way to do this? I am open to other suggestions but must would still like to use numpy as I have a lot of other post-processing code that relies on numpy functionality that I want to use with this data.

推荐答案

从ctypes指针对象创建NumPy数组是有问题的操作.尚不清楚谁真正拥有指针指向的内存.什么时候可以再次释放它?有效期有多长?只要有可能，我都会尽量避免这种构造.在Python中创建数组并将其传递给C函数比使用不知道Python的C函数分配的内存要容易得多，也更安全.通过执行后者，您在某种程度上会否定使用高级语言来处理内存管理的优势.

Creating NumPy arrays from a ctypes pointer object is a problematic operation. It is unclear who actually owns the memory the pointer is pointing to. When will it be freed again? How long is it valid? Whenever possible I would try to avoid this kind of construct. It is so much easier and safer to create arrays in the Python code and pass them to the C function than to use memory allocated by a Python-unaware C function. By doing the latter, you negate to some extent the advantages of having a high-level language taking care of the memory management.

如果您确实确定有人照顾了内存，则可以创建一个暴露Python缓冲区协议"的对象，然后使用此缓冲区对象创建一个NumPy数组.您通过未公开的int_asbuffer()函数提供了一种在帖子中创建缓冲区对象的方法:

If you are really sure that someone takes care of the memory, you can create an object exposing the Python "buffer protocol" and then create a NumPy array using this buffer object. You gave one way of creating the buffer object in your post, via the undocumented int_asbuffer() function:

buffer = numpy.core.multiarray.int_asbuffer(
    ctypes.addressof(y.contents), 8*array_length)

(请注意，我用8代替了np.dtype(float).itemsize.在任何平台上，它总是8.)创建缓冲区对象的另一种方法是通过ctypes从Python C API调用PyBuffer_FromMemory()函数:

(Note that I substituted 8 for np.dtype(float).itemsize. It's always 8, on any platform.) A different way to create the buffer object would be to call the PyBuffer_FromMemory() function from the Python C API via ctypes:

buffer_from_memory = ctypes.pythonapi.PyBuffer_FromMemory
buffer_from_memory.restype = ctypes.py_object
buffer = buffer_from_memory(y, 8*array_length)

对于这两种方式，您都可以通过通过

创建一个NumPy数组.

For both these ways, you can create a NumPy array from buffer by

a = numpy.frombuffer(buffer, float)

(我实际上不明白为什么您使用.astype()而不是frombuffer的第二个参数；此外，我想知道为什么您使用np.int，而您之前说过数组包含double.)

(I actually do not understand why you use .astype() instead of a second parameter to frombuffer; furthermore, I wonder why you use np.int, while you said earlier that the array contains doubles.)

恐怕它不会比这容易得多，但是还不错，你不觉得吗?您可以将所有丑陋的细节隐藏在包装函数中，而不必担心.

I'm afraid it won't get much easier than this, but it isn't that bad, don't you think? You could bury all the ugly details in a wrapper function and don't worry about it any more.

这篇关于从ctypes数组获取数据到numpy的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！