如何通过CLI对超过1000个对象使用AWS S3同步 [英] How to use aws s3 sync from the CLI for more than 1000 objects

问题描述

我正在尝试使用aws-cli s3 sync或aws-cli s3 cp --recursive将S3存储桶(实际上是数字海洋空间)的内容同步到本地硬盘上

I'm attempting syncing the contents of an S3 bucket (actually digital ocean space) to my local hard drive with aws-cli s3 sync or aws-cli s3 cp --recursive

我尝试同时使用aws-cli sync 和 cp 命令，但都在1000个对象之后停止.我知道sync会使用--page-size标志(

I've tried using both the aws-cli sync and cp commands, but both stop after 1000 objects. I know sync mentions it's limited to 1000 objects with the --page-size flag (https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html), but it seems like everything I've seen talks about syncing entire buckets, but my sync's stop at 1000 objects and I have 7 million to sync.

aws s3 cp s3://MYBUCKET ~/Documents/temp_space --source-region https://sfo2.digitaloceanspaces.com --profile MYPROFILE --endpoint=https://sfo2.digitaloceanspaces.com --recursive

aws sync cp s3://MYBUCKET ~/Documents/temp_space --source-region https://sfo2.digitaloceanspaces.com --profile MYPROFILE --endpoint=https://sfo2.digitaloceanspaces.com

我希望能够同步整个存储桶，而不仅仅是同步1000个对象.

I expect to be able to sync the entire bucket, not just sync 1000 objects.

推荐答案

‘page_size’’参数限制了请求的数量或结果，而不是总数.

The ‘’’—page-size’’’ parameter limits the number or results in a request, not the total number.

例如，在您的目录中包含要复制到s3存储桶中的5,000个对象的情况下.您的命令看起来像 aws s3 cp.s3://您的存储桶.这会将当前目录的所有内容(5,000个对象)复制到s3存储桶中.

By way of example, in a scenario where you have a directory with 5,000 objects that you wish to copy to a s3 bucket. Your command would look something like aws s3 cp . s3://your-bucket. This will copy all contents of our current directory, 5,000 objects, to our s3 bucket.

默认(且最大) -page-size 是1,000个对象，因此，因为我们没有指定 -page-size ，以便完成所有复制将5,000个对象存储到您的s3存储桶中，在后台，AWSCLI将处理5个请求(5 x 1,000个对象)以复制所有5,000个对象.

The default (and maximum) —page-size is 1,000 objects so, because we haven’t specified a —page-size, in order to accomplish copying all 5,000 objects to your s3 bucket, under the hood, the AWSCLI is going to handle making 5 requests (5 x 1,000 objects) to copy all 5,000 objects.

通常，您应该能够简单地忽略此可选参数，并运行 aws s3 cp 或 aws s3 sync 而不会出现问题.如果遇到请求超时的问题，则可以添加 -page-size 参数，该参数的值小于1,000，以解决超时问题.

Generally, you should be able to simply ignore this optional parameter and run your aws s3 cp or aws s3 sync without issue. If you face issues with the request timing out, then you can add the —page-size parameter with a value less than 1,000 to address the time-out issue.

这篇关于如何通过CLI对超过1000个对象使用AWS S3同步的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！