使用龙卷风未来获取网址，两种不同的方式可获得不同的结果 [英] Use tornado future to fetch url, two different ways get different results

问题描述

我想使用龙卷风获取批处理网址.所以我的代码如下所示:

I want to use tornado to fetch batch urls. So my code shows below:

from tornado.concurrent import Future
from tornado.httpclient import AsyncHTTPClient    
from tornado.ioloop import IOLoop    


class BatchHttpClient(object):    
    def __init__(self, urls, timeout=20):    
        self.async_http_client = AsyncHTTPClient()    
        self.urls = urls    
        self.timeout = 20    

    def __mid(self):    
        results = []    
        for url in self.urls:    
            future = Future()    

            def f_callback(f1):    
                future.set_result(f1.result())    

            f = self.async_http_client.fetch(url)    
            f.add_done_callback(f_callback)    
            results.append(future)    
        return results    

    def get_batch(self):    
        results = IOLoop.current().run_sync(self.__mid)    
        return results    


urls = ["http://www.baidu.com?v={}".format(i) for i in range(10)]    
batch_http_client = BatchHttpClient(urls)    
print batch_http_client.get_batch()    

运行代码时，发生错误:

When I run the code, an error occurs:

ERROR:tornado.application:Exception in callback <function f_callback at 0x7f35458cae60> for <tornado.concurrent.Future object at 0x7f35458c9650>
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 317, in _set_done
    cb(self)
  File "/home/q/www/base_data_manager/utils/async_util.py", line 21, in f_callback
    future.set_result(f1.result())
  File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 271, in set_result
    self._set_done()
  File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 315, in _set_done
    for cb in self._callbacks:
TypeError: 'NoneType' object is not iterable

但是，如果我将代码更改为:

But if I change the code like:

class BatchHttpClient(object):
    def __init__(self, urls, timeout=20):
        self.async_http_client = AsyncHTTPClient()
        self.urls = urls
        self.timeout = 20

    def _get_batch(self, url):
        future = Future()
        f = self.async_http_client.fetch(url)
        def callback(f1):
            print future
            print f1.result()

            future.set_result(f1.result())
            print '---------'
        f.add_done_callback(callback)
        return future

    def __mid(self):
        results = []
        for url in self.urls:
            results.append(self._get_batch(url))
        return results

    def get_batch(self):
        results = IOLoop.current().run_sync(self.__mid)
        return results


urls = ["http://www.baidu.com?v={}".format(i) for i in range(10)]
batch_http_client = BatchHttpClient(urls)
for result in batch_http_client.get_batch():
    print result.body

然后它起作用. 我要做的只是添加一个中间函数，为什么结果不同.

Then it works. What I do is just add a mid-function,why the results are different.

推荐答案

在您的第一个代码段中，问题在于，在执行回调时，future的值是循环设置的最后一个值.换句话说，当它执行时:

In your first code snippet, the problem is that by the time your callbacks execute, the value of future is the last value set by the loop. In other words, when this executes:

def f_callback(f1):    
    future.set_result(f1.result())    

future的值始终相同.如果添加print future，您会看到此信息:对象的地址将始终相同.

the value of future is always the same. You can see this if you add a print future: the object's address will always be the same.

在第二个片段中，每个future和每个回调都在循环调用的函数中创建.因此，每个回调都从一个新作用域获取一个future 的值，从而解决了该问题.

In your second snippet, each future and each callback are created in a function called by the loop. So each callback gets its value for future from a new scope, which fixes the problem.

解决此问题的另一种方法是像这样修改__mid:

Another way to fix the issue would be to modify __mid like this:

def __mid(self):
    results = []
    for url in self.urls:
        future = Future()

        def make_callback(future):
            def f_callback(f1):
                future.set_result(f1.result())
            return f_callback

        f = self.async_http_client.fetch(url)
        f.add_done_callback(make_callback(future))
        results.append(future)
    return results

通过在make_callback(future)中创建回调，回调中future的值来自每个回调的不同作用域.

By creating the callback in make_callback(future), the value of future in the callbacks comes from a different scope for each callback.

这篇关于使用龙卷风未来获取网址，两种不同的方式可获得不同的结果的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！