如何在短时间内列出Google存储桶中的所有文件? [英] How to list all files in Google Storage bucket in a short time?

问题描述

我有一个Google存储桶，其中包含超过20k个文件名.有什么办法可以在短时间内列出存储桶中的所有文件名?

I have a Google Storage bucket that contains more than 20k+ filenames. Is there any way to list all the filenames in the bucket in a short time?

推荐答案

这取决于您所指的空头"，但是:

It depends on what you mean by "short", but:

您可以加快清单桶的速度的一件事就是分片清单操作.例如，如果存储桶中的对象以英语字母字符开头，则可以并行列出每个字母并合并结果.您可以在bash中使用gsutil这样操作:

One thing you can do to speed up listing a bucket is to shard the listing operation. For example, if your bucket has objects that begin with English alphabetic characters you could list each letter in parallel and combine the results. You could do this with gsutil in bash like this:

gsutil ls gs://your-bucket/a* > a.out &
gsutil ls gs://your-bucket/b* > b.out &
...
gsutil ls gs://your-bucket/b* > z.out &
wait
cat ?.out > listing.out

如果存储桶中的对象具有不同的命名，则必须调整分片的方式.

If your bucket has objects with different naming you'd have to adjust how you do the sharding.

这篇关于如何在短时间内列出Google存储桶中的所有文件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！