AWS S3列表键以字符串开头 [英] AWS S3 list keys begins with a string

问题描述

我正在AWS Lambda函数中使用python列出以特定ID开头的s3存储桶中的键

I am using python in AWS Lambda function to list keys in a s3 bucket that begins with a specific id

for object in mybucket.objects.all():
            file_name = os.path.basename(object.key)
            match_id = file_name.split('_', 1)[0]

问题是，如果s3存储桶中有数千个文件，则迭代效率很低，有时lambda函数会超时

The problem is if a s3 bucket has several thousand files the iteration is very inefficient and sometimes lambda function times out

这是示例文件名

https://s3.console.aws.amazon.com/s3/object/bucket-name/012345_abc_happy.jpg

我只想迭代包含"012345"的对象.在键名中关于如何实现这一目标的任何好的建议

i want to only iterate objects that contains "012345" in the key name Any good suggestion on how i can accomplish that

推荐答案

这是解决问题的方法.

S3将所有内容存储为对象，并且没有文件夹或文件名.一切都是为了方便用户.

S3 stores everything as objects and there is no folder or filename. It is all for user convenience.

aws s3 ls s3://bucket/folder1/folder2/filenamepart --recursive

aws s3 ls s3://bucket/folder1/folder2/filenamepart --recursive

将获得与该名称匹配的所有s3对象名称.

will get all s3 objects name that matches to that name.

import boto3
s3 = boto3.resource('s3')
my_bucket = s3.Bucket('bucketname')
for obj in my_bucket.objects.filter(Prefix='012345'):
    print(obj)

要加快列表的速度，您可以并行运行多个脚本.

To speed up the list you can run multiple scripts parallelly.

希望有帮助.

这篇关于AWS S3列表键以字符串开头的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！