使用python回调的SWIG引用计数错误 [英] SWIG reference count error with python callback

问题描述

我在Python中包装一个C函数，其中的函数允许回调到python。我有包装工作，但有一个引用计数问题，我想帮助理解和修复。 [

I'm wrapping a C function in python where the function allows a callback into python. I have the wrapper working, but with a reference counting problem that I'd like help to understand and fix. [There are few working examples of language callbacks using SWIG that I could find so I could easily have got something quite wrong.]

一个简单的C函数func（）是一个简单的函数，需要回调的是在cb_test.i中定义的。 func（）创建一个C数组，并将该数组作为参数传递给回调。 func（）被包装以允许函数从python调用，并且回调在python中被定义。一个简单的客户端在test.py中执行。该程序的工作原理是这样的[我已经把示例降低到最小，但它仍然不'短']：

A simple C function, func(), that requires a callback is defined in cb_test.i. The func() creates a C array and passes that array as a param to the callback. The func() is wrapped to allow both the function to be called from python, and also for the callback to be defined in python. A simple client executing this is in test.py. The program works like this [I've cut the example down to a minimum but it's still not 'short']:

• func从python调用，使用python回调函数my_cb（）。


• func（）的一个％pythoncode包装器创建一个％array_class数组py_vals []，稍后用于优化数据


• SWIG包装器具有callback_func typedef的typemap，将其映射到cb_func（），它在cb_test.i中定义。 python回调存储在py_cb_func全局中，并由cb_func（）调用。


• C func（）是一个简单的函数，修改它们并打印新值。


• cb_func（）主要是在python中执行my_cb（）的trampoline。数组参数必须从C复制到python数据，并且在回调执行后再次反向。我们使用py_vals []为此目的，因为它只分配一次。


• 在python中执行my_cb（）。

• func() is called from python, with a python callback, my_cb().
• a %pythoncode wrapper for func() creates an %array_class array, py_vals[], which is used later to optimise data marshalling between the C and python callbacks.
• the SWIG wrapper has a typemap for the callback_func typedef which maps it onto cb_func(), which is defined in cb_test.i. The python callback is stored in the py_cb_func global, and invoked by cb_func().
• the C func() is a simple function which seeds a local array with square values, allows the callback to modify them, and prints the new values.
• cb_func() is mainly a trampoline which executes my_cb() in python. The array param has to be copied from C to python data, and again in reverse after the callback has executed. We use py_vals[] for this purpose since it is only allocated once.
• my_cb() is executed in python.

我的主要问题是％pythoncode包装器中明显的引用计数问题。我发现有必要指出一个临时变量在py_vals []，以避免python崩溃在_wrap_func（）中的某处。如果cb_test.use_tmp_hack变量设置为False，python将崩溃。任何人都可以指出为什么python崩溃，以及如何可以正确地改变代码。因为问题是一个python变量，看起来不超出范围，我有点困惑。

My main problem is an apparent reference count problem in the %pythoncode wrapper. I have found it necessary to point a temp variable at py_vals[] to avoid python crashing somewhere in _wrap_func(). If the cb_test.use_tmp_hack variable is set to False, python will crash. Can anyone point out why python crashes, and how the code can be changed 'properly'. Since the problem is with a python variable that doesn't appear to go out of scope, I am a bit confused.

我的第二个问题是，有更好的方法的编组数组之间的C和python回调？我宁愿只是复制指针之间的数据，而不是复制数组。但是，C func（）创建并拥有C数组，不能更改。

My second question is, is there a better way of marshalling the array between the C and python callbacks? I would prefer to just copy pointers to data between them rather than copying the array. But, the C func() creates and owns the C array and that cannot be changed.

非常感谢

test.py

"""Test for SWIG callback reference count problem
"""

import cb_test

# If False, python crashes from a reference count problem
cb_test.use_tmp_hack = True

""" Callback for func.
    Multiplies a small array (seeded to square values) by a constant factor.
    The C func() prints the modified array.
"""
def my_cb(vals, py_cb_data):
    factor = py_cb_data
    for i in range(cb_test.dims):
        vals[i] = vals[i] * factor
    return True;


# func uses my_cb as the callback function, with 4 as the callback data
cb_test.func(my_cb, 4)

cb_test.i

cb_test.i

%module cb_test

/* Wrap callback_func param with cb_func, from where the python callback will be executed */
%typemap(in) callback_func {
  py_cb_func = $input;
  $1 = cb_func;
}

#ifdef SOLUTION
/* This block solves the reference counting problem */
/* The PyObject * fields in cb_data_struct require their python refcnts managed. */
%typemap(memberin) PyObject * {
  Py_DecRef($1);
  $1 = $input;
  Py_IncRef($1);
}
#endif

%inline %{
  #include "stdio.h"

  #define dims 6

  /* Define a C function and callback. We're wrapping both in Python. */
  typedef void (*callback_func) (short Vals[], void *cb_args);

  /* Func seeds a small array with square values, which the callback modifies.
     The modified array is printed to stdout */
  void func(callback_func cb, void *cb_args)
  {
    int i;
    short vals[dims];

    for (i = 0; i < dims; i++)
      vals[i] = i * i;
    cb(vals, cb_args);
    for (i = 0; i < dims; i++)  printf("%d ", vals[i]);  printf("\n");
  }

  /* The cb_data struct for callback_func */
  typedef struct cb_data_struct {
    PyObject *py_vals;
    PyObject *py_cb_data;
  } cb_data_struct;
%}

%{
  static PyObject *py_cb_func;
  static void cb_func(short Vals[], cb_data_struct *cb_data)
  {
    PyObject *py_cb_data;
    short *py_vals;
    unsigned int i;

    /* py_vals must be a ushortArray object */
    SWIG_ConvertPtr(cb_data->py_vals, &py_vals, SWIGTYPE_p_ushortArray, 0);

    /* Copy the C input array to the py_vals Python wrapping object */
    for (i = 0; i < dims; i++)
      py_vals[i] = Vals[i];

    /* Pass python callback data back to python unmodified */
    py_cb_data = cb_data->py_cb_data;

    /* The Python callback will modify the array */
    PyObject_CallFunctionObjArgs(py_cb_func, cb_data->py_vals, py_cb_data, NULL);

    /* Copy the modified py_vals back to the C array */
    for (i = 0; i < dims; i++)
      Vals[i] = py_vals[i];
  }
%}


/* Define a C array type for use in python code */
%include carrays.i
%array_class(unsigned short, ushortArray);

/* Marshal the python objects enabling the C callback_func to execute the Python callback function */
%pythoncode %{
use_tmp_hack = True

def func(cb_func, py_cb_data):
  cb_data = cb_data_struct()

  # If use_tmp_hack is set to False by the client, python crashes; the py_vals
  # array disappears in the setters for py_vals.
  # Using tmp_vals to bump the reference count seems to work around that.
  if use_tmp_hack:
    tmp_vals = ushortArray(dims)
    cb_data.py_vals = tmp_vals
  else:
    cb_data.py_vals = ushortArray(dims)

  cb_data.py_cb_data = py_cb_data
  _cb_test.func(cb_func, cb_data)
%}

推荐答案

崩溃的问题在于cb_data_struct的PyObject *成员的setters。我假设Python解释器会以某种方式管理引用计数，但似乎有必要在SWIG中这样做。我相信我已经解决了使用这个在设置器中使用的成员的typemap。它会递减先前值的计数，并增加新值。

The crashing problem lies in the setters for the PyObject * members of cb_data_struct. I assumed that the Python interpreter would somehow manage the reference counting, but it appears necessary to do that in SWIG. I believe I've solved that using this typemap which is used in the setters for the members. It decrements the count of prior values and increments the new value.

%typemap(memberin) PyObject * {
  Py_DecRef($1);
  $1 = $input;
  Py_IncRef($1);
}

我已经将这个块添加到问题中的代码，很明显。

I've added this block to the code in the question, hopefully obviously.

看起来很辛苦，我不认为有更有效的方法来编译数组在C和Python回调之间。

After looking quite hard, I don't think there is a more efficient way to marshal arrays between the C and Python callbacks.

[这个解决方案带有健康警告，因为我不是SWIG的经验。我有，虽然，验证，当计数减少为零，对象被删除，如预期。]

[This solution does come with a health warning because I'm not that experienced in SWIG. I have though, verified that objects get deleted when the count reduces to zero, as expected.]

这篇关于使用python回调的SWIG引用计数错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！