如何获取 S3 存储桶中所有不同前缀的列表? [英] How to get a list of all distinct prefixes in S3 bucket?

问题描述

如果我有一个如下的目录结构并且前缀是/folder1，

/folder1/folder11/folder12/folder13/*.files/folder21/folder22/folder23/*.files/folder31/folder32/*.files

我想动态地遍历这些目录，以便分别读取每个叶文件夹中的文件，即我需要一个列表

<预><代码>[/folder1/folder11/folder12/folder13/,/folder1/folder21/folder22/folder23/,/folder1/folder31/folder32/]

除了递归遍历每个前缀、获取下一级前缀、连接、获取下一级等，直到到达最后一个(叶)文件夹，还有没有更好的方法来获取它?

解决方案

你可以在循环中使用它来获取前缀

def get_list_of_prefixes_from_prefix(bucket, prefix):""获取给定桶的前缀列表和前缀""list_of_prefixes = []paginator = boto3.resource('s3').meta.client.get_paginator('list_objects')对于 paginator.paginate(Bucket=bucket, Prefix=prefix, Delimiter='/') 中的结果:# 打印(结果)如果结果为CommonPrefixes":prefixes = [f['Prefix'] for f in result['CommonPrefixes']]list_of_prefixes.extend(前缀)返回 list_of_prefixeslist_of_prefixes = get_list_of_prefixes_from_prefix('my-bucket', 'my-prefix/')

If I have a directory structure as below and the prefix is /folder1,

/folder1/folder11/folder12/folder13/*.files
               /folder21/folder22/folder23/*.files
               /folder31/folder32/*.files

I want to loop through these directories dynamically in order to read files in each of the leaf folder separately, i.e. I'd need a list

[
 /folder1/folder11/folder12/folder13/, 
 /folder1/folder21/folder22/folder23/,
 /folder1/folder31/folder32/
]

Is there a better way to get it other than loop through each prefix recursively, get next level prefix, concatenate, get next level, etc., until you get to the last (leaf) folder?

解决方案

you can use this in a loop to get just the prefixes

def get_list_of_prefixes_from_prefix(bucket, prefix):
    """gets list of prefixes for given bucket and prefix"""
    list_of_prefixes = []
    paginator = boto3.resource('s3').meta.client.get_paginator('list_objects')
    for result in paginator.paginate(Bucket=bucket, Prefix=prefix, Delimiter='/'):
        # print(result)
        if 'CommonPrefixes' in result:
            prefixes = [f['Prefix'] for f in result['CommonPrefixes']]
            list_of_prefixes.extend(prefixes)
    return list_of_prefixes

list_of_prefixes = get_list_of_prefixes_from_prefix('my-bucket', 'my-prefix/')

这篇关于如何获取 S3 存储桶中所有不同前缀的列表?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！