如何使用 Boto3 分页 [英] How to use Boto3 pagination

问题描述

背景:

列出 IAM 用户的 AWS 操作默认返回最多 50 个.

The AWS operation to list IAM users returns a max of 50 by default.

阅读下面的文档(链接)我运行以下代码并通过设置MaxItems"返​​回完整的数据集到 1000.

Reading the docs (links) below I ran following code and returned a complete set data by setting the "MaxItems" to 1000.

paginator = client.get_paginator('list_users')
response_iterator = paginator.paginate(
 PaginationConfig={
     'MaxItems': 1000,
     'PageSize': 123})
for page in response_iterator:
    u = page['Users']
    for user in u:
        print(user['UserName'])

http://boto3.readthedocs.io/en/latest/guide/paginators.htmlhttps://boto3.readthedocs.io/en/latest/reference/services/iam.html#IAM.Paginator.ListUsers

问题:

如果MaxItems"例如，设置为 10，循环遍历结果的最佳方法是什么?

If the "MaxItems" was set to 10, for example, what would be the best method to loop through the results? the

我使用以下内容进行了测试，但它仅在 'IsTruncated' == False 之前循环了 2 次迭代并导致KeyError: 'Marker'".不知道为什么会这样，因为我知道有 200 多个结果.

I tested with the following but it only loops 2 iterations before 'IsTruncated' == False and results in "KeyError: 'Marker'". Not sure why this is happening because I know there are over 200 results.

marker = None

while True:
    paginator = client.get_paginator('list_users')
    response_iterator = paginator.paginate( 
        PaginationConfig={
            'MaxItems': 10,
            'StartingToken': marker})
    #print(response_iterator)
    for page in response_iterator:
        u = page['Users']
        for user in u:
            print(user['UserName'])
            print(page['IsTruncated'])
            marker = page['Marker']
            print(marker)
        else:
            break

推荐答案

(答案重写)**注意**，分页器包含一个与文档不符的错误(反之亦然).MaxItems 当总项目数超过 MaxItems 数量时不会返回 Marker 或 NextToken.确实PageSize 是控制Marker/NextToken 指示符返回的那个.

(Answer rewrite) **NOTE **, the paginator contains a bug that doesn't tally with the documentation (or vice versa). MaxItems doesn't return the Marker or NextToken when total items exceed MaxItems number. Indeed PageSize is the one that controlling return of Marker/NextToken indictator.

import sys
import boto3
iam = boto3.client("iam")
marker = None
while True:
    paginator = iam.get_paginator('list_users')
    response_iterator = paginator.paginate( 
        PaginationConfig={
            'PageSize': 10,
            'StartingToken': marker})
    for page in response_iterator:
        print("Next Page : {} ".format(page['IsTruncated']))
        u = page['Users']
        for user in u:
            print(user['UserName'])
    try:
        marker = response_iterator['Marker']
        print(marker)
    except KeyError:
        sys.exit()

您的代码不起作用并不是您的错误.分页器中的 MaxItems 似乎变成了阈值"；指标.具有讽刺意味的是，原始 boto3.iam.list_users 中的 MaxItems 仍然像前面提到的那样工作.

It is not your mistake that your code doesn't works. MaxItems in the paginator seems become a "threshold" indicator. Ironically, the MaxItems inside original boto3.iam.list_users still works as mentioned.

如果你检查 boto3.iam.list_users，你会发现要么省略了 Marker，否则你必须输入一个值.显然，分页器不是所有 boto3 类 list_* 方法的包装器.

If you check boto3.iam.list_users, you will notice either you omit Marker, otherwise you must put a value. Apparently, paginator is NOT a wrapper for all boto3 class list_* method.

import sys
import boto3
iam = boto3.client("iam")
marker = None
while True:
    if marker:
        response_iterator = iam.list_users(
            MaxItems=10,
            Marker=marker
        )
    else:
        response_iterator = iam.list_users(
            MaxItems=10
        )
    print("Next Page : {} ".format(response_iterator['IsTruncated']))
    for user in response_iterator['Users']:
        print(user['UserName'])

    try:
        marker = response_iterator['Marker']
        print(marker)
    except KeyError:
        sys.exit()

您可以跟进我在 boto3 github 中提交的问题.根据成员的说法，你可以在 paginate() 之后调用 build_full_result ，这将显示出想要的行为.

You can follow up the issue I filed in boto3 github. According to the member, you can call build_full_result after paginate(), that will show the desire behavior.

这篇关于如何使用 Boto3 分页的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！