将带有回调的函数转换为Python生成器？ [英] Turn functions with a callback into Python generators?

问题描述

Scipy最小化函数（仅用作示例），可以选择在每个步骤中添加一个回调函数。所以我可以做类似的事情，

The Scipy minimization function (just to use as an example), has the option of adding a callback function at each step. So I can do something like,

def my_callback(x):
    print x
scipy.optimize.fmin(func, x0, callback=my_callback)

是否可以使用回调函数创建fmin的生成器版本，以便我可以做，

Is there a way to use the callback function to create a generator version of fmin, so that I could do,

for x in my_fmin(func,x0):
    print x

似乎可以结合使用收益和发送，但我可以很想任何事情。

It seems like it might be possible with some combination of yields and sends, but I can quite think of anything.

推荐答案

如注释中所指出的，您可以使用 队列 。缺点是您仍然需要某种方法来访问最终结果（最后的 fmin 返回值）。下面的示例使用一个可选的回调函数来做一些事情（另一个选择就是也产生它，尽管您的调用代码必须区分迭代结果和最终结果）：

As pointed in the comments, you could do it in a new thread, using Queue. The drawback is that you'd still need some way to access the final result (what fmin returns at the end). My example below uses an optional callback to do something with it (another option would be to just yield it also, though your calling code would have to differentiate between iteration results and final results):

from thread import start_new_thread
from Queue import Queue

def my_fmin(func, x0, end_callback=(lambda x:x), timeout=None):

    q = Queue() # fmin produces, the generator consumes
    job_done = object() # signals the processing is done

    # Producer
    def my_callback(x):
        q.put(x)
    def task():
        ret = scipy.optimize.fmin(func,x0,callback=my_callback)
        q.put(job_done)
        end_callback(ret) # "Returns" the result of the main call

    # Starts fmin in a new thread
    start_new_thread(task,())

    # Consumer
    while True:
        next_item = q.get(True,timeout) # Blocks until an input is available
        if next_item is job_done:
            break
        yield next_item

更新： 阻止下一次迭代的执行，直到使用者完成对最后一个迭代的处理为止，还必须使用 task_done 和 join 。

Update: to block the execution of the next iteration until the consumer has finished processing the last one, it's also necessary to use task_done and join.

    # Producer
    def my_callback(x):
        q.put(x)
        q.join() # Blocks until task_done is called

    # Consumer
    while True:
        next_item = q.get(True,timeout) # Blocks until an input is available
        if next_item is job_done:
            break
        yield next_item
        q.task_done() # Unblocks the producer, so a new iteration can start

请注意，没有必要 maxsize = 1

更新2：还要注意，除非所有项目都已添加到队列中。最终由该生成器检索，创建的线程将死锁（它将永远阻塞并且其资源将永远不会释放）。生产者正在等待队列，并且由于它存储了对该队列的引用，因此即使消费者使用了它，它也不会被gc回收。然后，队列将变得无法访问，因此没有人能够释放锁。

Update 2: Also note that, unless all items are eventually retrieved by this generator, the created thread will deadlock (it will block forever and its resources will never be released). The producer is waiting on the queue, and since it stores a reference to that queue, it will never be reclaimed by the gc even if the consumer is. The queue will then become unreachable, so nobody will be able to release the lock.

一个干净的解决方案是未知的，如果有的话（因为这取决于代替 fmin 使用的特定函数）。可以使用超时进行变通，如果 put 阻塞太长时间，则使生产者引发异常：

A clean solution for that is unknown, if possible at all (since it would depend on the particular function used in the place of fmin). A workaround could be made using timeout, having the producer raises an exception if put blocks for too long:

    q = Queue(maxsize=1)

    # Producer
    def my_callback(x):
        q.put(x)
        q.put("dummy",True,timeout) # Blocks until the first result is retrieved
        q.join() # Blocks again until task_done is called

    # Consumer
    while True:
        next_item = q.get(True,timeout) # Blocks until an input is available
        q.task_done()                   # (one "task_done" per "get")
        if next_item is job_done:
            break
        yield next_item
        q.get() # Retrieves the "dummy" object (must be after yield)
        q.task_done() # Unblocks the producer, so a new iteration can start

这篇关于将带有回调的函数转换为Python生成器？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！