如何在cdef中等待？ [英] How to await in cdef?

问题描述

我有这个Cython代码（简体）：

I have this Cython code (simplified):

class Callback:
    async def foo(self):
        print('called')

cdef void call_foo(void* callback):
    print('call_foo')
    asyncio.wait_for(<object>callback.foo())

async def py_call_foo():
    call_foo(Callback())

async def example():
    loop.run_until_complete(py_call_foo())

发生了什么：我得到了 RuntimeWarning：协程回调从未等待过.foo 。而且，实际上，它从未被调用过。但是，会调用 call_foo 。

What happens though: I get RuntimeWarning: coroutine Callback.foo was never awaited. And, in fact, it is never called. However, call_foo is called.

任何想法，正在发生什么/如何使其真正等待 Callback.foo 完成？

Any idea what's going on / how to get it to actually wait for Callback.foo to complete?

在上面的示例中，缺少了一些重要的细节：特别是，实际上很难从 call_foo 中获取返回值。实际的项目设置是这样的：

In the example above some important details are missing: In particular, it is really difficult to get hold of return value from call_foo. The real project setup has this:

1. 具有规则的Bison解析器。规则引用了特制结构，我们称之为 ParserState 。此结构包含对回调的引用，当规则匹配时由解析器调用。

1. Bison parser that has rules. Rules are given a reference to specially crafted struct, let's call it ParserState. This struct contains references to callbacks, which are called by parser when rules match.

在Cython代码中，有一个类，我们称其为 Parser ，该软件包的用户应该扩展以创建其自定义解析器。此类具有一些方法，然后需要从 ParserState 的回调中调用。

In Cython code, there's a class, let's call it Parser, that users of the package are supposed to extend to make their custom parsers. This class has methods which then need to be called from callbacks of ParserState.

解析应该被认为是发生这样的情况：

Parsing is supposed to happen like this:

async def parse_file(file, parser):
    cdef ParserState state = allocate_parser_state(
        rule_callbacks,
        parser,
        file,
    )
    parse_with_bison(state)

回调具有大致的形状：

ctypedef void(callback*)(char* text, void* parser)

我必须承认不知道 asyncio 是如何实现 await 的，所以我不知道通常是否可以用我已有的设置来做到这一点。不过，我的最终目标是，多个Python函数能够迭代地或多或少地同时解析不同的文件。

I have to admit I don't know how exactly asyncio implements await, and so I don't know if it is in general possible to do this with the setup that I have. My ultimate goal though is that multiple Python functions be able to iteratively parse different files, all at the same time more or less.

推荐答案

TLDR：

协程必须经过 await 或通过事件循环运行。 cdef 函数无法，但可以构造并返回协程。

Coroutines must be await'ed or run by an event loop. A cdef function cannot await, but it can construct and return a coroutine.

您的实际问题是将同步代码与异步代码混合在一起。例子：

Your actual problem is mixing synchronous with asynchronous code. Case in point:

async def example():
    loop.run_until_complete(py_call_foo())

这类似于将子例程放入线程中，但是从不启动它。
即使启动时也是如此，这是一个僵局：同步部分将阻止异步部分运行。

This is similar to putting a subroutine in a Thread, but never starting it. Even when started, this is a deadlock: the synchronous part would prevent the asynchronous part from running.

异步定义协程类似于 def ...：yield 生成器：仅对其进行实例化。您必须与之交互才能真正运行它：

An async def coroutine is similar to a def ...: yield generator: calling it only instantiates it. You must interact with it to actually run it:

def foo():
     print('running!')
     yield 1

bar = foo()  # no output!
print(next(bar))  # prints `running!` followed by `1`

同样，当您拥有 async def 协程时，您必须 await 或在事件循环中安排它。由于 asyncio.wait_for 会生成一个协程，而您永远不会等待或安排它，它不会运行。这是 RuntimeWarning 的原因。

Similarly, when you have an async def coroutine, you must either await it or schedule it in an event loop. Since asyncio.wait_for produces a coroutine, and you never await or schedule it, it is not run. This is the cause of the RuntimeWarning.

请注意将协程放入 asyncio.wait_for 纯粹是添加超时。它会产生一个异步包装器，必须 $ 。

Note that the purpose of putting a coroutine into asyncio.wait_for is purely to add a timeout. It produces an asynchronous wrapper which must be await'ed.

async def call_foo(callback):
    print('call_foo')
    await asyncio.wait_for(callback.foo(), timeout=2)

asyncio.get_event_loop().run_until_complete(call_foo(Callback()))

异步函数需要异步指令

异步编程的关键是合作：只有一个协程执行直到产生控制权。然后，另一个协程执行，直到产生控制权。这意味着任何协程阻塞而没有产生控制权也会阻塞所有其他协程。

Asynchronous functions need asynchronous instructions

The key for asynchronous programming is that it is cooperative: Only one coroutine executes until it yields control. Afterwards, another coroutine executes until it yields control. This means that any coroutine blocking without yielding control blocks all other coroutines as well.

通常，如果某项执行的工作没有 await 上下文，它正在阻止。值得注意的是， loop.run_until_complete 被阻止。您必须从同步函数中调用它：

In general, if something performs work without an await context, it is blocking. Notably, loop.run_until_complete is blocking. You have to call it from a synchronous function:

loop = asyncio.get_event_loop()

# async def function uses await
async def py_call_foo():
    await call_foo(Callback())

# non-await function is not async
def example():
    loop.run_until_complete(py_call_foo())

example()

协程的返回值

协程可以像常规函数一样返回结果。

async def make_result():
    await asyncio.sleep(0)
    return 1

如果您从另一个协程等待，您将直接获得返回值： / p>

If you await it from another coroutine, you directly get the return value:

async def print_result():
    result = await make_result()
    print(result)  # prints 1

asyncio.get_event_loop().run_until_complete(print_result())

到从内部的协程获得价值常规子例程，使用 run_until_complete 运行协程：

To get the value from a coroutine inside a regular subroutine, use run_until_complete to run the coroutine:

def print_result():
    result = asyncio.get_event_loop().run_until_complete(make_result())
    print(result)

print_result()

A cdef / cpdef 函数不能是协程

Cython仅通过Python函数通过 和 await 的收益支持协程。即使是经典的协程，也无法使用 cdef ：

A cdef/cpdef function cannot be a coroutine

Cython supports coroutines via yield from and await only for Python functions. Even for a classical coroutine, a cdef is not possible:

Error compiling Cython file:
------------------------------------------------------------
cdef call_foo(callback):
    print('call_foo')
    yield from asyncio.wait_for(callback.foo(), timeout=2)
   ^
------------------------------------------------------------

testbed.pyx:10:4: 'yield from' not supported here

您很好打电话同步 cdef 来自协程的函数。通过 cdef 函数计划一个协程，您非常好。
，但是您不能从 cdef 函数内部等待，也不能等待一个 cdef 函数。如果您需要这样做，例如使用示例 def 函数。

You are perfectly fine calling a synchronous cdef function from a coroutine. You are perfectly fine scheduling a coroutine from a cdef function. But you cannot await from inside a cdef function, nor await a cdef function. If you need to do that, as in your example, use a regular def function.

但是您可以构造并在 cdef 函数中返回协程。这使您可以在外部协程中等待：

You can however construct and return a coroutine in a cdef function. This allows you to await the result in an outer coroutine:

# inner coroutine
async def pingpong(what):
    print('pingpong', what)
    await asyncio.sleep(0)
    return what

# cdef layer to instantiate and return coroutine
cdef make_pingpong():
    print('make_pingpong')
    return pingpong('nananana')

# outer coroutine
async def play():
    for i in range(3):
        result = await make_pingpong()
        print(i, '=>', result)

asyncio.get_event_loop().run_until_complete(play())

等待， make_pingpong 不是协程。它只是协程的工厂。

Note that despite the await, make_pingpong is not a coroutine. It is merely a factory for coroutines.

这篇关于如何在cdef中等待？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！