一次击中多个API，龙卷风和python [英] Hitting multiple APIs at once, tornado and python

问题描述

我正在尝试创建一个API，该API将从其他多个API收集响应并将结果合并为一个响应。我想异步发送get请求，以便它更快地运行，但是即使我使用协程和yield，我的代码似乎仍然一次发出每个请求。想知道这也许是因为我使用请求库而不是龙卷风的AsyncHTTPClient，还是因为我在循环内调用self.path_get，还是因为将结果存储在实例变量中？

I'm trying to make an API that will collect responses from several other API's and combine the results into one response. I want to sent the get requests asynchronously so that it runs faster, but even though I'm using coroutines and yielding, my code still seems to be making each request one at a time. Wondering if maybe it's because I'm using the requests library instead of tornado's AsyncHTTPClient, or because I'm calling self.path_get inside of a loop, or because I'm storing results in an instance variable?

API正在击中JSON对象的返回数组，我想将它们全部组合成一个数组并将其写入响应。

The API's I'm hitting return arrays of JSON objects, and I want to combine them all into one array and write that to the response.

from tornado import gen, ioloop, web
from tornado.gen import Return
import requests


PATHS = [
    "http://firsturl",
    "http://secondurl",
    "http://thirdurl"
]


class MyApi(web.RequestHandler):

    @gen.coroutine
    def get(self):
        self.results = []
        for path in PATHS:
            x = yield self.path_get(path)

        self.write({
            "results": self.results,
        })

    @gen.coroutine
    def path_get(self, path):
        resp = yield requests.get(path)
        self.results += resp.json()["results"]
        raise Return(resp)


ROUTES = [
    (r"/search", MyApi),
]


def run():
    app = web.Application(
        ROUTES,
        debug=True,
    )

    app.listen(8000)

    ioloop.IOLoop.current().start()


if __name__ == "__main__":
    run()

推荐答案

代码无法正常工作的原因很多。首先，请求通常会阻止事件循环，并且不执行其他任何操作。将请求替换为 AsyncHTTPClient.fetch 。同样，您产生每个请求的方式也会使请求按顺序而不是像您想的那样同时进行。下面是一个示例代码重组示例：

There's many reasons why your code doesn't work. To begin, requests generally blocks the event loop and doesn't let anything else execute. Replace requests with AsyncHTTPClient.fetch. Also, the way you were yielding each request would also make the requests sequential and not concurrently like you thought. Here's an example of how your code could be restructured:

import json
from tornado import gen, httpclient, ioloop, web

# ...

class MyApi(web.RequestHandler):

    @gen.coroutine
    def get(self):
        futures_list = []
        for path in PATHS:
            futures_list.append(self.path_get(path))

        yield futures_list
        result = json.dumps({'results': [x.result() for x in futures_list]})
        self.write(result)

    @gen.coroutine
    def path_get(self, path):
        request = httpclient.AsyncHTTPClient()
        resp = yield request.fetch(path)
        result = json.loads(resp.body.decode('utf-8'))
        raise gen.Return(result)

正在发生的事情是我们正在创建一个 期货 从 gen.coroutine 函数返回，并产生整个列表，直到请求的结果可用为止。然后，一旦所有请求完成，就会迭代 futures_list 并将结果用于创建一个新列表，该列表将附加到JSON对象上。

What's happening is we're creating a list of Futures that get returned from gen.coroutine functions and yielding the entire list until the results from the request are available. Then once all the requests are complete, futures_list is iterated and the results are used to create a new list which is appended to a JSON object.

这篇关于一次击中多个API，龙卷风和python的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！