将3维numpy数组传递给C [英] Passing 3-dimensional numpy array to C

问题描述

出于速度目的，我正在为Python程序编写C扩展，并尝试传递3维numpy数组时遇到一些非常奇怪的行为.它适用于二维数组，但是我敢肯定我正在尝试用指针弄乱某些东西，以使其能够与三维数组一起使用.但这是奇怪的部分.如果我只是传递3-D数组，它会因 Bus Error (总线错误)而崩溃.如果(在Python中)我先将变量创建为2D数组，然后用3D数组覆盖，则它可以完美运行.如果变量首先是一个空数组，然后是3D数组，则它会崩溃，并显示 Seg Fault .那怎么可能发生呢?

I'm writing a C extension to my Python program for speed purposes, and running into some very strange behaviour trying to pass in a 3-dimensional numpy array. It works with a 2-dimensional array, but I'm sure I'm screwing something up with the pointers trying to get it to work with the 3rd dimension. But here's the weird part. If I just pass in a 3-D array, it crashes with a Bus Error. If (in Python) I create my variable as a 2D array first, and then overwrite it with a 3D array, it works perfectly. If the variable is an empty array first and then a 3D array, it crashes with a Seg Fault. How can that possibly happen?

还有，有人可以帮助我使3D阵列正常工作吗?还是我应该放弃并传递2D数组并自己重塑形状?

Also, can anyone help me get a 3D array working? Or should I just give up and pass in a 2D array and reshape it myself?

这是我的C代码:

static PyObject* func(PyObject* self, PyObject* args) {
  PyObject *list2_obj;
  PyObject *list3_obj;
  if (!PyArg_ParseTuple(args, "OO", &list2_obj, &list3_obj))
    return NULL;

  double **list2;
  double ***list3;

  //Create C arrays from numpy objects:
  int typenum = NPY_DOUBLE;
  PyArray_Descr *descr;
  descr = PyArray_DescrFromType(typenum);
  npy_intp dims[3];
  if (PyArray_AsCArray(&list2_obj, (void **)&list2, dims, 2, descr) < 0 || PyArray_AsCArray(&list3_obj, (void ***)&list3, dims, 3, descr) < 0) {
    PyErr_SetString(PyExc_TypeError, "error converting to c array");
    return NULL;
  }
  printf("2D: %f, 3D: %f.\n", list2[3][1], list3[1][0][2]);
}

这是我的Python代码，调用上述函数:

And here's my Python code that calls the above function:

import cmod, numpy
l2 = numpy.array([[1.0,2.0,3.0], [4.0,5.0,6.0], [7.0,8.0,9.0], [3.0, 5.0, 0.0]])

l3 = numpy.array([[2,7, 1], [6, 3, 9], [1, 10, 13], [4, 2, 6]])  # Line A
l3 = numpy.array([])                                             # Line B

l3 = numpy.array([[[2,7, 1, 11], [6, 3, 9, 12]],
                 [[1, 10, 13, 15], [4, 2, 6, 2]]])

cmod.func(l2, l3)

因此，如果我同时注释掉A行和B行，它会因总线错误而崩溃.如果有行A，但注释了行B，则它可以正确运行且没有错误.如果有行B，但行A被注释掉，它会打印正确的数字，但会出现段故障.最后，如果两行同时出现，它还会打印正确的数字，然后显示Seg错误.这到底是怎么回事?

So, if I comment out both Line A and B, it crashes with a Bus error. If Line A is there, but Line B is commented out, it runs correctly with no errors. If Line B is there but Line A is commented out, it prints the correct numbers but then Seg faults. Finally if both lines are present it also prints the correct numbers and then Seg faults. What in the hell is going on here?

编辑:好的.哇.因此，我在Python中使用了int，但在C中将其称为double.这在1D和2D数组上工作得很好.但不是3D.因此，我将l3的Python定义更改为浮点数，现在它都可以正常运行(非常感谢Bi Rico ).

Ok. Wow. So I was using int in Python but calling them double in C. And that was working fine with 1D and 2D arrays. But not 3D. So I changed the Python definition of l3 to be floats, and now it all works fantastically (Thank you very much Bi Rico).

但是，现在，A行和& B！现在，如果两行都被注释掉，该程序将运行.如果存在B行，但注释掉了A行，则它起作用，如果两者都未注释，则同上.但是，如果存在A行而B被注释掉了，那么我又会遇到那美妙的Bus错误.我真的很想在将来避免这些情况，所以有人知道为什么声明Python变量会产生这种影响吗?

But now, more strange behaviour with Lines A & B! Now if both Lines are commented out, the program works. If Line B is present but A is commented out, it works, and ditto if both are uncommented. But if Line A is present and B is commented out, I get that fantastic Bus error again. I'd really like to avoid these in the future, so does anyone have any clue why the declaration of a Python variable can have this kind of impact?

嗯，尽管这些错误很疯狂，但它们都是由于我传入的3维numpy数组而引起的.如果我仅传入一维或二维数组. ，它的行为符合预期，而其他Python变量的操作则无济于事.这使我相信问题出在Python的引用计数中.在C代码中，引用计数的减少要比3-D数组减少的更多，并且当该函数返回时，Python会尝试清理对象，并尝试删除NULL指针.这只是我的猜测，而我试图Py_INCREF();我能想到的一切都无济于事.我想我只会使用2D数组并在C中重塑它.

EDIT 2: Well, as insane as these errors are, they're all due to the 3-dimensional numpy array I pass in. If I only pass in 1- or 2-D arrays, it behaves as expected, and manipulation of the other Python variables does nothing. This leads me to believe that the problem lies somewhere in Python's reference counting. In the C-code the reference count is decreased more than it should for the 3-D arrays, and when that function returns Python tries to clean up objects, and attempts to delete a NULL pointer. This is just my guess, and I've tried to Py_INCREF(); everything I could think of to no avail. I guess I'll just be using a 2D array and reshaping it in C.

推荐答案

我已经在评论中提到了这一点，但是我希望将其冲洗掉一点有助于使其更加清晰.

I already mentioned this in a comment, but I hope flushing it out a little helps make it more clear.

在C语言中使用numpy数组时，最好明确说明数组的类型.具体来说，您似乎将指针声明为double ***list3，但是它们以您在python代码中创建l3的方式，您将获得一个dtype npy_intp的数组(我认为).您可以在创建数组时通过显式使用dtype来解决此问题.

When you're working with numpy arrays in C it's good to be explicit about the typing of your arrays. Specifically it looks like you're declaring your pointers as double ***list3, but they way you're creating l3 in your python code you'll get an array with dtype npy_intp (I think). You can fix this by explicitly using the dtype when creating your arrays.

import cmod, numpy
l2 = numpy.array([[1.0,2.0,3.0],
                  [4.0,5.0,6.0],
                  [7.0,8.0,9.0],
                  [3.0, 5.0, 0.0]], dtype="double")

l3 = numpy.array([[[2,7, 1, 11], [6, 3, 9, 12]],
                  [[1, 10, 13, 15], [4, 2, 6, 2]]], dtype="double")

cmod.func(l2, l3)

另一个说明，由于python的工作方式，"A行"和"B行"几乎不可能对C代码产生任何影响.我知道这似乎与您的经验相抵触，但是我很确定这一点.

Another note, because of the way python works it's nearly impossible for "line A" and "line B" to have any effect on the C code what so ever. I know that this seems to conflict with your empirical experience, but I'm pretty sure on this point.

我对此不太确定，但是根据我在C方面的经验，总线错误和段错误不是确定性的.它们取决于内存分配，对齐方式和地址.在某些情况下，代码似乎可以正常运行10次，即使没有任何变化，第11次运行也会失败.

I'm a little less sure about this, but based on my experience with C, bus-errors and segfaults are not deterministic. They depend on memory allocation, alignment, and addresses. In some situation code seems to run fine 10 times, and fails on the 11th run even though nothing has changed.

您是否考虑过使用 cython ?我知道这不是每个人的选择，但是如果选择的话，您可以使用

Have you considered using cython? I know it's not an option for everyone, but if it is an option you could get nearly C level speedups using typed memoryviews.

这篇关于将3维numpy数组传递给C的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！