我可以在Cython中使用此并行迭代器模式吗? [英] Can I use this parallel iterator pattern with Cython?

问题描述

在C ++ 11中，我一直在使用以下模式来实现带有并行迭代器的图形数据结构.节点只是索引，边缘是邻接数据结构中的条目.为了遍历所有节点，将一个函数(lambda，closure ...)传递给parallelForNodes方法，并以每个节点为参数进行调用.迭代细节很好地封装在该方法中.

With C++11 I have been using the following pattern for implementing a graph data structure with parallel iterators. Nodes are just indices, edges are entries in an adjacency data structure. For iterating over all nodes, a function (lambda, closure...) is passed to a parallelForNodes method and called with each node as an argument. Iteration details are nicely encapsulated in the method.

现在，我想在Cython上尝试相同的概念. Cython提供了cython.parallel.prange函数，该函数使用OpenMP在一定范围内并行化循环.为了使并行性起作用，需要使用nogil=True参数停用Python的Global Interpreter Lock.如果没有GIL，则不允许使用Python对象，这很棘手.

Now I would like to try the same concept with Cython. Cython provides the cython.parallel.prange function which uses OpenMP for parallelizing a loop over a range. For parallelism to work, Python's Global Interpreter Lock needs to be deactivated with the nogil=True parameter. Without the GIL, using Python objects is not allowed, which makes this tricky.

是否可以在Cython中使用这种方法?

Is it possible to use this approach with Cython?

class Graph:

    def __init__(self, n=0):
        self.n = n
        self.m = 0  
        self.z = n  # max node id
        self.adja = [[] for i in range(self.z)]
        self.deg = [0 for i in range(self.z)]

    def forNodes(self, handle):
        for u in range(self.z):
            handle(u)

    def parallelForNodes(self, handle):
        # first attempt which will fail...
        for u in prange(self.z, nogil=True):
            handle(u)


# usage 

def initialize(u):
    nonlocal ls
    ls[u] = 1

G.parallelForNodes(initialize)

推荐答案

首先，没有GIL的东西就不能是Python对象.

Firstly, things cannot be Python objects without the GIL.

from cython.parallel import prange

cdef class Graph:
    cdef int n, m, z

    def __cinit__(self, int n=0):
        self.z = n  # max node id

    cdef void parallelForNodes(self, void (*handle)(int) nogil) nogil:
        cdef int u
        for u in prange(self.z, nogil=True):
            handle(u)

最大的收获是我们的函数指针也是 nogil.

The biggest catch there is that our function pointer was also nogil.

parallelForNodes本身不一定必须是nogil，但是没有理由不这样做.

parallelForNodes does not have to be nogil itself, but there's no reason for it not to be.

然后我们需要一个 C 函数来调用:

Then we need a C function to call:

cdef int[100] ls
cdef void initialize(int u) nogil:
    global ls
    ls[u] = 1

就可以了！

Graph(100).parallelForNodes(initialize)

# Print it!
cdef int[:] ls_ = ls
print(list(ls_))

这篇关于我可以在Cython中使用此并行迭代器模式吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！