在没有gil的情况下使用指向类方法的函数指针 [英] Using function pointers to methods of classes without the gil

问题描述

我的一部分作品需要进行大量计算，但是它们通常非常简单明了，并且原则上可以很容易地与Cython的prange并行完成，需要nogil。但是，考虑到我试图编写自己的Cython代码，重点是将cdef类作为构建基块，因此遇到了一个问题。

Part of my works requires a lot of calculations, but they are often fairly straight-forward and can in principle quite easily be done in parallel with Cython's prange, requiring nogil. However, given that I tried to write my Cython code with a focus on having cdef classes as building blocks I encountered a problem.

假设我有一个数值积分例程或类似，它以函数指针作为输入

Let's say I got an numerical integration routine or similar which takes a function pointer as an input

ctypedef double (* someFunctionPointer) (double tt) nogil

cdef double integrationRoutine(someFunctionfointer ff) nogil:
    # Doing something
    # ...
    return result

现在，我的每个集成点实际上都是一个更大的仿真（很多参数，例如我加载/设置/输入等等），该仿真在一个类中实现。所以我最初的方法是做类似的事情

Now each of my integration points is actually a "larger" simulation (lots of parameters and so on I load/set/input or whatever), which is implemented in a class. So my initial approach was doing something like

cdef class SimulationClass:

    cdef double simulationFunctionPointer(SimulationClass self, double tt) nogil:
        # ...

将 simulationFunctionPointer改为 integrationRoutine，就可以了。

where I though I could just hand "simulationFunctionPointer" to "integrationRoutine" and would be fine. This does of course not work because of the self argument.

我所有的解决方法都需要

All my work-arounds either require to

• 不使用cdef类，而是使用类似C的结构（如果SimulationClass引用了许多其他类，参数等，则有点奇怪）


• 使用gil（因为我想使用SimulationClass；我编写了一些独立的函数，将SimulationClass当作void *，但是随后我需要再次将其强制转换为SimulationClass，这需要gil）

任何建议或想法如何解决此问题？我的第一种方法是否可以在其他语言（如C ++）中使用？

Any advice or ideas how to approach this problem? Is my first approach possible in other languages like C++?

干杯

推荐答案

您可以在需要GIL的块周围使用和gil：，然后在重要的块附近使用和nogil：内部块将占用您的大部分运行时间。举个简单的例子

You can use with gil: around the blocks that need the GIL, and then with nogil: around the important inner blocks that will take most of your run time. To give a trivial example

from cython.parallel import prange

cdef class Simulation:
    cdef double some_detail

    def __cinit__(self,double some_detail):
        self.some_detail = some_detail

    def do_long_calculation(self, double v):
        with nogil:
            pass # replace pass with something long and time-consuming
        return v*self.some_detail


def run_simulations(int number_of_simulations):
    cdef int n
    for n in prange(number_of_simulations,nogil=True):
        with gil: # immediately get the gil back to do the "pythony bits"
            sim = Simulation(5.3*n)
            sim.do_long_calculation(1.2) # and release again in here"

do_long_calculation 中的 nogil 部分的运行时间长于您设置并通过模拟的部分（可以在平行带有 do_long_calculation 的el，但本身没有），这是相当有效的。

Provided the nogil section in do_long_calculation runs from longer than the section where you set up and pass the simulations (which can run in parallel with do_long_calculation, but not with itself) this is reasonably efficient.

关于将绑定方法转换为函数指针的其他小注释：在Cython中，您确实很难做到这一点。我最好的解决方法是使用 ctypes （或者也可以使用cffi），它可以将所有可调用的Python转换为函数指针。他们执行此操作的方式似乎涉及一些运行时代码生成，而您可能不想复制。您可以将此方法与Cython结合使用，但可能会增加函数调用的开销（因此请确保 do_long_calculation 实际上很长！）

An additional small comment about turning a bound method into a function pointer: you really struggle to do this in Cython. The best workround I have is to use ctypes (or possibly also cffi) which can turn any Python callable into a function pointer. The way they do this appears to involve some runtime code generation which you probably don't want to replicate. You can combine this method with Cython, but it probably adds a bit of overhead to the function call (so make sure do_long_calculation is actually long!)

以下工作原理（记入 http ：//osdir.com/ml/python-cython-devel/2009-10/msg00202.html ）

import ctypes
# define the function type for ctypes
ftype = ctypes.CFUNCTYPE(ctypes.c_double,ctypes.c_double)

S = Simulation(3.0)
f = ftype(S.do_long_calculation) # create the ctypes function pointer

cdef someFunctionPointer cy_f_ptr = (<someFunctionPointer*><size_t>ctypes.addressof(f))[0] # this is pretty awful!

这篇关于在没有gil的情况下使用指向类方法的函数指针的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！