在Cython中返回结构数组 [英] Returning an array of structs in Cython

问题描述

我正在尝试在Cython中返回一个结构数组.

I'm trying to return an array of structs in Cython.

// .pyx

from libc.stdint cimport uint8_t

cdef extern from "<apriltag.h>":
    cdef struct apriltag_detection:
        int id
        double c[2]
        double p[4][2]

    ctypedef apriltag_detection apriltag_detection_t

cdef extern from "tag36h11_detector/tag36h11_detector.h":
    apriltag_detection_t* scan_frame(int width, int height, uint8_t* data);

cdef class Detection:
    # how do I "link" this to the struct defined above?
    def __cinit__(self):
        pass
    def __dealloc__(self):
        pass

def detect(width, height, frame):
    return scan_frame(width, height, frame)

理想情况下，我想在Python代码中调用detect函数，并获取Detection对象的列表，其中Detection是C结构apriltag_detection的包装类，其类型定义为apriltag_detection_t.

Ideally, I would like to call the detect function in Python code, and get a list of Detection objects where Detection is a wrapper class for the C struct apriltag_detection which is typedef'd to apriltag_detection_t.

我收到以下编译错误:

tag36h11_detector.pyx:22:21:无法将'apriltag_detection_t *'转换为Python对象

tag36h11_detector.pyx:22:21: Cannot convert 'apriltag_detection_t *' to Python object

在文档的任何地方，找不到指向结构或结构数组的指针的引用.

I can't find a reference to returning a pointer to a struct, or an array of structs, anywhere in the documentation.

更新3

// .h
typedef struct detection_payload {
    int size;
    apriltag_detection_t** detections;
} detection_payload_t;

我正在尝试将上述结构转换为包含size的Python对象和包含apriltag_detection_t对象的Python列表.

I'm trying to convert the above struct into a Python object that holds size and a Python list that holds apriltag_detection_t objects.

// .pyx

cdef extern from "<apriltag.h>":
    cdef struct apriltag_detection:
        int id
        double c[2]
        double p[4][2]

    ctypedef apriltag_detection apriltag_detection_t

cdef extern from "tag36h11_detector/tag36h11_detector.h":
    cdef struct detection_payload:
        int size
        apriltag_detection_t** detections
    ctypedef detection_payload detection_payload_t
    detection_payload* scan_frame(int width, int height, uint8_t* data)

...


cdef class Detection:
    cdef apriltag_detection* _d

    def __cinit__(self):
        self._d = NULL
    cdef _setup(self, apriltag_detection* d):
        self._d = d
    def __dealloc__(self):
        self._d = NULL
    property id:
        def __get__(self):
            return self._d.id
    property c:
        def __get__(self):
            return self._d.c
    property p:
        def __get__(self):
            return self._d.p

cdef Detection_create(apriltag_detection_t* d):
    return Detection()._setup(d)

cdef class DetectionPayload:
    cdef detection_payload* _p

    def __cinit__(self):
        self._p = NULL
    cdef _setup(self, detection_payload* p):
        self._p = p
        self.size = p.size
        self.detections = []
        for i in range(0, self.size):
            apriltag_detection_t* detection = self._p.detections[i]
            d = Detection_create(detection)
            self.detections+=[d]
    def __dealloc__(self):
        _p = NULL
    property size:
        def __get__(self):
            return self.size
    property detections:
        def __get__(self):
            return self.detections

我在网上遇到了一些语法错误:

I'm getting several syntax errors on the line:

apriltag_detection_t* detection = self._p.detections[I]

具体来说，在指针apriltag_detection_t*

更新2

现在可以编译和导入了.在数组上仍然没有进展

This compiles and imports fine now. Still no progress on arrays

from libc.stdint cimport uint8_t

cdef extern from "<apriltag.h>":
    cdef struct apriltag_detection:
        int id
        double c[2]
        double p[4][2]

    ctypedef apriltag_detection apriltag_detection_t

cdef extern from "tag36h11_detector/tag36h11_detector.h":
    apriltag_detection_t* scan_frame(int width, int height, uint8_t* data);

cdef class Detection:
    cdef apriltag_detection* _d

    def __cinit__(self):
        self._d = NULL
    cdef _setup(self, apriltag_detection* d):
        self._d = d
    def __dealloc__(self):
        self._d = NULL
    property id:
        def __get__(self):
            return self._d.id
    property c:
        def __get__(self):
            return self._d.c
    property p:
        def __get__(self):
            return self._d.p

cdef Detection_create(apriltag_detection_t* d):
    return Detection()._setup(d)

def detect(width, height, frame):
    cdef apriltag_detection_t* detection = scan_frame(width, height, frame)
    return Detection_create(detection)

更新1

我尝试了下面的链接，这是我到目前为止的内容.

I tried following the post linked below, and this is what I have so far.

from libc.stdint cimport uint8_t

cdef extern from "<apriltag.h>":
    cdef struct apriltag_detection:
        int id
        double c[2]
        double p[4][2]

    ctypedef apriltag_detection apriltag_detection_t

cdef extern from "tag36h11_detector/tag36h11_detector.h":
    apriltag_detection_t* scan_frame(int width, int height, uint8_t* data);

cdef class Detection:
    cdef apriltag_detection* _d;
    def __cinit__(self):
        self._d = NULL
    def _setup(self, apriltag_detection* d):
        self._d = d
    def __dealloc__(self):
        self._d = NULL
    property id:
        def __get__(self):
            return self._t.id
    property c:
        def __get__(self):
            return self._t.c
    property p:
        def __get__(self):
            return self._t.p

cdef Detection_create(apriltag_detection_t* d):
    return Detection()._setup(d)

def detect(width, height, frame):
    return <Detection>scan_frame(width, height, frame)

尽管这比以前更近了，但我仍然收到错误消息:

Although this is closer than I was before, I'm still getting the error:

tag36h11_detector.pyx:33:30:无法将'apriltag_detection_t *'转换为Python对象

tag36h11_detector.pyx:33:30: Cannot convert 'apriltag_detection_t *' to Python object

在线

cdef Detection_create(apriltag_detection_t* d):
    return Detection()._setup(d)

此外，我不知道如何返回Python列表...

Moreover, I have no idea how to return a Python list...

推荐答案

您似乎已经解决了您的问题，但是我仍然想回答这个问题.首先是因为我想尝试一下，其次是因为我认为您在内存管理方面遇到了一些问题，我想指出一点.

You seem to have solved your problems, but I would like to answer the question nevertheless. First because I would like to try it out and second because I think you have some problems with memory management and I would like to point it out.

我们将包装以下简单的C接口:

We will wrap the following simple C-interface:

//creator.h
typedef struct {
   int mult;
   int add;
} Result;

typedef struct {
   int size;
   Result *arr;
} ResultArray;

ResultArray create(int size, int *input){
   //whole file at the end of the answer
}

处理输入数组并返回结构的C数组以及该数组中元素的数量.

which processes the input-array and return a C-array of structs along with the number of elements in this array.

我们的包装pyx文件如下所示:

Our wrapping pyx-file looks then like following:

#result_import.pyx (verion 0)
cdef extern from "creator.h":
     ctypedef struct Result:
          int mult
          int add
     ctypedef struct ResultArray:
          int size
          Result *arr
     ResultArray create(int size, int *input)

def create_structs(int[::1] input_vals):
    pass

最值得注意的部分:我使用memoryview(int[::1])传递我的输入数组，这有两个优点:

The most notable part: I use a memoryview (int[::1]) to pass my input array and this has two advantages:

1. 在python端，可以使用任何支持内存视图的东西(从Python3开始为numpy，array)，这比使用numpy数组更灵活
2. 确保(通过[::1])输入是连续的

1. On the python-side it is possible to use anything that supports memory views (numpy, array since Python3) which is more flexible as to use a numpy array
2. It is ensured (through [::1]) that the input is continuous

在测试脚本中，我使用numpy，但内置数组也可以使用:

In the test-script I use numpy, but one could also go with build-in array:

#test.py
import result_import
import numpy as np

a=np.array([1,2,3],dtype='int32')
result=result_import.create_structs(a)
for i,el in enumerate(result):
    print  i, ": mult:", el.mult, " add:", el.add

现在什么都没有了，但是一切都设置好了.

Now nothing works yet, but everything is set up.

第一种情况:我们只想拥有普通的python对象，别无所求！一种可能性是:

First scenario: we just wanna have normal python-objects nothing fancy! One possibility would be:

#result_import.pyx (verion 1)
#from cpython cimport array needed for array.array in Python2
from libc.stdlib cimport free
....
class PyResult:
    def __init__(self, mult, add):
       self.mult=mult
       self.add=add


def create_structs(int[::1] input_vals):
    cdef ResultArray res=create(len(input_vals), &input_vals[0])
    try:
        lst=[]
        for i in range(res.size):
            lst.append(PyResult(res.arr[i].mult, res.arr[i].add))
    finally:
        free(res.arr)
    return lst  

我将整个数据转换为python-objects并使用一个简单的列表.很简单，但是有两点值得注意:

I convert the whole data to python-objects and use a simple list. Very simple, but there are two things worth noting:

1. 内存管理:我负责释放res.arr中分配的内存.因此，我使用try...finally来确保即使抛出异常也会发生这种情况.
2. 仅将此指针设置为NULL是不够的，我必须调用free -function.

1. Memory management: I'm responsible for freeing the memory allocated in res.arr. So I use try...finally to make sure it happens even if an exception is thrown.
2. It is not enough to set this pointer to NULL, I have to call free-function.

现在我们的test.py可以工作了-很好！

Now our test.py works - nice!

第二种情况:但是，如果我只需要一些元素并将其全部转换-则效率很低.此外，我将所有元素两次保留在内存中(至少一段时间)-这是幼稚方法的缺点.因此，我想稍后在程序中的某个位置创建PyResult -objects.

Second scenario: however if I need only some of the elements and convert them all - it is just inefficient. Furthermore, I keep all elements twice in the memory (at least for some time) - this is a downside of the naive approach. So I would like to create PyResult-objects on demand somewhere later in program.

让我们写一个包装列表:

Let's write a wrapper list:

#result_import.pyx (verion 2)
...
cdef class WrappingList:
    cdef int size
    cdef Result *arr

    def __cinit__(self):
        self.size=0
        self.arr=NULL

    def __dealloc__(self):
        free(self.arr)
        print "deallocated"#just a check

    def __getitem__(self, index):
        if index<0 or index>=self.size:
            raise IndexError("list index out of range")
        return PyResult(self.arr[index].mult, self.arr[index].add)


def create_structs(int[::1] input_vals):
    cdef ResultArray res=create(len(input_vals), &input_vals[0])
    lst=WrappingList()
    lst.size, lst.arr=res.size, res.arr
    return lst 

因此，类WrappingList的行为非常类似于列表，保留整个C数组而不进行复制，并且仅在需要时创建PyResult对象.值得一提的事情:

So the class WrappingList behaves a lot like a list, keeps the whole C-array without copying and creates PyResult-objects only when needed. Things worth mentioning:

1. __dealloc__在销毁WrapperingList对象时被调用-这是我们释放C代码给我们的内存的地方.在test.py的结尾处，我们应该看到已取消分配"，否则出现了问题...
2. __getitem__用于迭代.

1. __dealloc__ is called when the WrapperingList-object is destroyed - this is the place where we release the memory which was given to us by the C-code. At the end of test.py we should see "deallocated", otherwise something gone wrong...
2. __getitem__ is used for iteration.

第三种情况::Python代码不仅应读取结果，还应更改结果，以便将更改后的数据传递回C代码.为此，让我们将PyResult用作代理:

Third scenario: Python code should not only read the results but also change them, so the changed data can be passed back to the C-code. For that let's make PyResult a proxy:

#result_import.pyx (verion 3, last)
...
cdef class PyResult:
    cdef Result *ptr #ptr to my element
    def __init__(self):
       self.ptr=NULL

    @property
    def mult(self):
        return self.ptr.mult

    @mult.setter
    def mult(self, value):
        self.ptr.mult = value

    @property
    def add(self):
        return self.ptr.add

    @add.setter
    def add(self, value):
        self.ptr.add = value


cdef class WrappingList:  
    ...
    def __getitem__(self, index):
        if index>=self.size:
            raise IndexError("list index out of range")
        res=PyResult()
        res.ptr=&self.arr[index]
        return res

现在，PyResult对象拥有指向相应元素的指针，并可以直接在C数组中对其进行更改.但是，我要提一些陷阱:

Now, the PyResult-object holds a pointer to the corresponding element and can change it directly in the C-array. However, there are some pitfalls, I should mention:

1. 这有点不安全:PyResult的生存期不应超过父对象WrappingList-对象的生存时间.您可以通过在PyResult -class中添加对parent的引用来解决此问题.
2. 访问元素(add或mult)的成本非常高，因为每次必须创建，注册并删除新的Python对象时，它的成本都很高.

1. It's a little bit unsafe: PyResult should not live longer than the parent-WrappingList-object. You can fix it by adding a reference to parent in the PyResult-class.
2. Accessing the elements (add or mult) is quite costly, because every time a new Python-object must be created, registered and then deleted.

让我们更改测试脚本，以查看正在使用的代理:

Let's change the test-script, to see the proxies in action:

#test.py(second version)
import result_import
import numpy as np

a=np.array([1,2,3],dtype='int32')
result=result_import.create_structs(a)
for i,el in enumerate(result):
    print  i, ": mult:", el.mult, " add:", el.add
    el.mult, el.add=42*i,21*i

# now print changed values:
for i,el in enumerate(result):
    print  i, ": mult:", el.mult, " add:", el.add

还有很多需要改进的地方，但我想一个答案就足够了:)

There is still a lot to improve, but I guess enough for one answer:)

附件:

草率的creator.h-需要检查malloc的结果:

Sloppy creator.h - one needs to check the result of malloc:

//creator.h
typedef struct {
   int mult;
   int add;
} Result;

typedef struct {
   int size;
   Result *arr;
} ResultArray;

ResultArray create(int size, int *input){
   ResultArray res;
   res.size=size;
   res.arr=(Result *)malloc(size*sizeof(Result));//todo: check !=0
   for(int i=0;i<size;i++){
       res.arr[i].mult=2*input[i];
       res.arr[i].add=2+input[i]; 
   }
   return res;
}

setup.py:

from distutils.core import setup, Extension
from Cython.Build import cythonize

setup(ext_modules=cythonize(Extension(
            name='result_import',
            sources = ["result_import.pyx"]
    )))

这篇关于在Cython中返回结构数组的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！