如何使用 list_objects_v2 从 S3 获取 1000 多个对象? [英] How to get more than 1000 objects from S3 by using list_objects_v2?

问题描述

我在 s3 上有超过 500,000 个对象.我正在尝试获取每个对象的大小.我为此使用了以下 python 代码

I have more than 500,000 objects on s3. I am trying to get the size of each object. I am using the following python code for that

import boto3

bucket = 'bucket'
prefix = 'prefix'

contents = boto3.client('s3').list_objects_v2(Bucket=bucket,  MaxKeys=1000, Prefix=prefix)["Contents"]

for c in contents:
    print(c["Size"])

但它只是给了我前 1000 个对象的大小.根据文档，我们不能得到超过 1000.有什么办法可以得到更多吗?

But it just gave me the size of the top 1000 objects. Based on the documentation we can't get more than 1000. Is there any way I can get more than that?

推荐答案

内置的 boto3 Paginator 类是克服 list-objects-v2 1000 条记录限制的最简单方法代码>.这可以实现如下

The inbuilt boto3 Paginator class is the easiest way to overcome the 1000 record limitation of list-objects-v2. This can be implemented as follows

s3 = boto3.client('s3')

paginator = s3.get_paginator('list_objects_v2')
pages = paginator.paginate(Bucket='bucket', Prefix='prefix')

for page in pages:
    for obj in page['Contents']:
        print(obj['Size'])

欲知更多详情:https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Paginator.ListObjectsV2

这篇关于如何使用 list_objects_v2 从 S3 获取 1000 多个对象?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！