如何通过使用list_objects_v2从S3获取1000个以上的对象? [英] How to get more than 1000 objects from S3 by using list_objects_v2?
本文介绍了如何通过使用list_objects_v2从S3获取1000个以上的对象?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我在s3上有超过500,000个对象.我正在尝试获取每个对象的大小.我为此使用了以下python代码
I have more than 500,000 objects on s3. I am trying get the size of each object. I am using the following python code for that
import boto3
bucket = 'bucket'
prefix = 'prefix'
contents = boto3.client('s3').list_objects_v2(Bucket=bucket, MaxKeys=1000, Prefix=prefix)["Contents"]
for c in contents:
print(c["Size"])
但这只是给了我前1000个对象的大小.根据文档,我们不能获得更多1000.有什么办法可以使我获得更多?
But it just gave me the size of top 1000 objects. Based on the documentation we can't get more 1000. Is there any way I can get more than that?
推荐答案
使用响应中返回的ContinuationToken作为后续调用的参数,直到响应中返回的IsTruncated值为false.
Use the ContinuationToken returned in the response as a parameter for subsequent calls, until the IsTruncated value returned in the response is false.
这可以分解为一个整洁的生成器函数:
This can be factored into a neat generator function:
def get_all_s3_objects(s3, **base_kwargs):
continuation_token = None
while True:
list_kwargs = dict(MaxKeys=1000, **base_kwargs)
if continuation_token:
list_kwargs['ContinuationToken'] = continuation_token
response = s3.list_objects_v2(**list_kwargs)
yield from response.get('Contents', [])
if not response.get('IsTruncated'): # At the end of the list?
break
continuation_token = response.get('NextContinuationToken')
for file in get_all_s3_objects(boto3.client('s3'), Bucket=bucket, Prefix=prefix):
print(file['size'])
这篇关于如何通过使用list_objects_v2从S3获取1000个以上的对象?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文