如何使用Boto3分页 [英] How to use Boto3 pagination

问题描述

背景:

列出IAM用户的AWS操作默认最多返回50.

The AWS operation to list IAM users returns a max of 50 by default.

阅读下面的文档(链接)，我运行了以下代码，并通过将"MaxItems"设置为1000返回了完整的设置数据.

Reading the docs (links) below I ran following code and returned a complete set data by setting the "MaxItems" to 1000.

paginator = client.get_paginator('list_users')
response_iterator = paginator.paginate(
 PaginationConfig={
     'MaxItems': 1000,
     'PageSize': 123})
for page in response_iterator:
    u = page['Users']
    for user in u:
        print(user['UserName'])

http://boto3.readthedocs.io/en/latest/guide/paginators.html https://boto3.readthedocs.io/en/latest/reference/services/iam.html#IAM.Paginator.ListUsers

问题:

例如，如果将"MaxItems"设置为10，那么遍历结果的最佳方法是什么?

If the "MaxItems" was set to 10, for example, what would be the best method to loop through the results? the

我使用以下代码进行了测试，但它仅在"IsTruncated" == False之前循环执行2次迭代，并导致"KeyError:'Marker'".不知道为什么会这样，因为我知道结果超过200.

I tested with the following but it only loops 2 iterations before 'IsTruncated' == False and results in "KeyError: 'Marker'". Not sure why this is happening because I know there are over 200 results.

marker = None

while True:
    paginator = client.get_paginator('list_users')
    response_iterator = paginator.paginate( 
        PaginationConfig={
            'MaxItems': 10,
            'StartingToken': marker})
    #print(response_iterator)
    for page in response_iterator:
        u = page['Users']
        for user in u:
            print(user['UserName'])
            print(page['IsTruncated'])
            marker = page['Marker']
            print(marker)
        else:
            break

谢谢.

推荐答案

(答案重写) **注意**，分页器包含一个与文档不符的错误(反之亦然).当项目总数超过MaxItems数量时，MaxItems不返回Marker或NextToken.的确，PageSize是控制Marker/NextToken指示符返回的控制符.

(Answer rewrite) **NOTE **, the paginator contains a bug that doesn't tally with the documentation (or vice versa). MaxItems doesn't return the Marker or NextToken when total items exceed MaxItems number. Indeed PageSize is the one that controlling return of Marker/NextToken indictator.

import sys
import boto3
iam = boto3.client("iam")
marker = None
while True:
    paginator = iam.get_paginator('list_users')
    response_iterator = paginator.paginate( 
        PaginationConfig={
            'PageSize': 10,
            'StartingToken': marker})
    for page in response_iterator:
        print("Next Page : {} ".format(page['IsTruncated']))
        u = page['Users']
        for user in u:
            print(user['UserName'])
    try:
        marker = page['Marker']
        print(marker)
    except KeyError:
        sys.exit()

代码不起作用不是您的错误.分页器中的MaxItems似乎成为阈值"指示器.具有讽刺意味的是，原始boto3.iam.list_users内部的MaxItems仍然可以按照所述方式工作.

It is not your mistake that your code doesn't works. MaxItems in the paginator seems become a "threshold" indicator. Ironically, the MaxItems inside original boto3.iam.list_users still works as mentioned.

如果选中boto3.iam.list_users，则会发现您省略了Marker，否则必须输入一个值.显然，所有boto3类list_ *方法的 paginator都不是包装器.

If you check boto3.iam.list_users, you will notice either you omit Marker, otherwise you must put a value. Apparently, paginator is NOT a wrapper for all boto3 class list_* method.

import sys
import boto3
iam = boto3.client("iam")
marker = None
while True:
    if marker:
        response_iterator = iam.list_users(
            MaxItems=10,
            Marker=marker
        )
    else:
        response_iterator = iam.list_users(
            MaxItems=10
        )
    print("Next Page : {} ".format(response_iterator['IsTruncated']))
    for user in response_iterator['Users']:
        print(user['UserName'])

    try:
        marker = response_iterator['Marker']
        print(marker)
    except KeyError:
        sys.exit()

您可以继续我在boto3 github中提出的问题.根据该成员的说法，您可以在paginate()之后调用build_full_result，这将显示期望的行为.

You can follow up the issue I filed in boto3 github. According to the member, you can call build_full_result after paginate(), that will show the desire behavior.

这篇关于如何使用Boto3分页的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！