如何使用asyncio和aiohttp异步通过API响应进行分页 [英] How paginate through api response asynchronously with asyncio and aiohttp

问题描述

我正在尝试使用python异步进行api调用。我在列表中有多个端点，每个端点将返回分页结果。我可以通过多个端点进行异步设置，但是无法返回每个端点的分页结果。

I'm trying to make api calls with python asynchronously. I have multiple endpoints in a list and each endpoint will return paginated results. I'm able to set up going though the multiple endpoints asynchronously, however am not able to return the paginated results of each endpoint.

从调试中，我发现 fetch_more（）函数运行while循环，但实际上并没有得到通过session.get（）异步。所以基本上。函数 fetch_more（）旨在从api调用中获取每个端点的其余结果，但是我发现无论是否包含<$ c，返回的结果数都是相同的$ c> fetch_more（）函数。我曾尝试寻找与asyncio分页的示例，但运气不太好。

From debugging, I found that the fetch_more() function runs the while loop, but doesn't actually get past the async with session.get(). So basically. The function fetch_more() is intended to get remaining results from the api call for each endpoint, however I find that the same number of results are returned with or without the fetch_more() function. I've tried looking for examples of pagination with asyncio but have not have much luck.

据我了解，我不应该在while循环内进行请求，但是，我不确定为获得分页结果而应该采取的方法。

From my understanding, I should not be doing a request inside a while loop, however, I'm not sure a way around that in order to get paginated results.

if __name__ == 'main':

    starter_fun(url, header, endpoints):

starter_func(url, header, endpoints):

    loop = asyncio.get_event_loop() #event loop
    future = asyncio.ensure_future(fetch_all(url, header, endpoints))
    loop.run_until_complete(future) #loop until done

async def fetch_all(url, header, endpoints):

    async with ClientSession() as session:
        for endpoint in endpoints:
           task = asyncio.ensure_future(fetch(url, header, endpoint))
           tasks.append(task)
        res = await asyncio.gather(*tasks) # gather task responses
        return res

async def fetch(url, header, endpoint): 

    total_tasks = []
    async with session.get(url, headers=header, params=params, ssl=False) as response:
        response_json = await response.json()
        data = response_json['key']
       tasks = asyncio.ensure_future(fetch_more(response_json, data, params, header, url, endpoint, session)) //this is where I am getting stuck
        total_tasks.append(tasks)
    return data


//function to get paginated results of api endpoint
async def fetch_more(response_json, data, params, header, url, endpoint, session): //this is where I am getting stuck

    while len(response_json['key']) >= params['limit']:
        params['offset'] = response_json['offset'] + len(response_json['key'])
        async with session.get(url, headers=header, params=params, ssl=False) as response_continued:
            resp_continued_json = await response_continued.json()
            data.extend(resp_continued_json[kebab_to_camel(endpoint)])
   return data

当前我是使用或不使用 fetch_more 函数可以得到1000个结果，但是使用 fetch_more 可以得到更多的结果。有关如何异步分页的任何想法？

Currently I am getting 1000 results with or without the fetch_more function, however it should be a lot more with the fetch_more. Any idea as to how to approach asynchronously paginating?

推荐答案

from aiohttp import web

 async def fetch(self size: int = 10):

    data = "some code to fetch data here"

    def paginate(_data, _size):
        import itertools

        while True:
            i1, i2 = itertools.tee(_data)
            _data, page = (itertools.islice(i1, _size, None),
                              list(itertools.islice(i2, _size)))
            if len(page) == 0:
                break
            yield page

    return web.json_response(list(paginate(_data=data, _size=size)))

这篇关于如何使用asyncio和aiohttp异步通过API响应进行分页的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！