AWS s3api列表对象返回多少个对象? [英] how many objects are returned by aws s3api list-objects?

问题描述

我正在使用:

aws s3api list-objects --endpoint-url https://my.end.point/ --bucket my.bucket.name --query 'Contents[].Key' --output text

以获取存储桶中的文件列表.

to get the list of files in a bucket.

aws s3api list-object 文档页面表示此命令最多只能返回1000个对象，但是我注意到在我的情况下，它返回了存储桶中所有文件的名称.例如，当我运行以下命令时:

The aws s3api list-object documentation page says that this command returns only up to a 1000 objects, however I noticed that in my case it returns the names of all files in my bucket. For example when I run the following command:

aws s3api list-objects --endpoint-url https://my.end.point/ --bucket my.bucket.name --query 'Contents[].Key' --output text | tr "\t" "\n" | wc -l

我显示了13512，这意味着返回了超过一万三千个文件名.

I get 13512 displayed, meaning that more than 13 thousand file names were returned.

我错过了什么吗?

我使用以下aws cli版本:

I use the following aws cli version:

aws-cli/1.10.57 Python/2.7.3 Linux/3.2.0-4-amd64 botocore/1.4.47

推荐答案

返回存储桶中的部分或全部(最多1000个)对象.您可以使用请求参数作为选择条件，以返回存储桶中对象的子集. [1]

Returns some or all (up to 1000) of the objects in a bucket. You can use the request parameters as selection criteria to return a subset of the objects in a bucket. [1]

我认为(最多1000个)"部分文档说明中的高度误导.它指的是cli发送的每个基础HTTP请求的最大页面大小. --page-size选项的文档清楚说明了这一点:

I think that the part "(up to 1000)" in the documentation's description is highly misleading. It refers to the maximal page size per underlying HTTP request which is sent by the cli. The documentation of the --page-size option makes this clear:

要进入AWS服务调用的每个页面的大小.这不会影响命令输出中返回的项目数.设置较小的页面大小会导致对AWS服务的更多调用，每个调用中检索的项目更少.这可以帮助防止AWS服务调用超时.

The size of each page to get in the AWS service call. This does not affect the number of items returned in the command's output. Setting a smaller page size results in more calls to the AWS service, retrieving fewer items in each call. This can help prevent the AWS service calls from timing out.

在阅读有关分页的AWS文档[2]时，它变得更加清晰:

It gets even clearer when reading the AWS documentation about pagination [2] which describes:

对于可以返回大量项目的命令，AWS命令行界面(AWS CLI)添加了三个选项，当AWS CLI调用服务的API进行填充时，可使用这些选项来控制输出中包含的项目数列表.

For commands that can return a large list of items, the AWS Command Line Interface (AWS CLI) adds three options that you can use to control the number of items included in the output when the AWS CLI calls a service's API to populate the list.

默认情况下，AWS CLI使用的页面大小为1000 ，并检索所有可用项目.例如，如果您在包含3,500个对象的Amazon S3存储桶上运行aws s3api列表对象，则CLI会对Amazon S3进行四次调用，在后台为您处理特定于服务的分页逻辑，并在最后返回所有3,500个对象输出.

By default, the AWS CLI uses a page size of 1000 and retrieves all available items. For example, if you run aws s3api list-objects on an Amazon S3 bucket that contains 3,500 objects, the CLI makes four calls to Amazon S3, handling the service-specific pagination logic for you in the background and returning all 3,500 objects in the final output.

正如Ankit正确指出的那样，使用--max-items选项是限制结果并停止自动分页的正确解决方案:

As Ankit already stated correctly, using the --max-items option is the correct solution to limit the result and stop the automatic pagination:

要一次在AWS CLI输出中包含较少的项目，请使用--max-items选项. AWS CLI仍如上所述处理服务的分页，但一次仅输出指定数量的项目. [2]

To include fewer items at a time in the AWS CLI output, use the --max-items option. The AWS CLI still handles pagination with the service as described above, but prints out only the number of items at a time that you specify. [2]

参考文献

[1] https://docs.aws .amazon.com/cli/latest/reference/s3api/list-objects.html
[2] https://docs.aws.amazon. com/cli/latest/userguide/cli-usage-pagination.html

References

[1] https://docs.aws.amazon.com/cli/latest/reference/s3api/list-objects.html
[2] https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-pagination.html

这篇关于AWS s3api列表对象返回多少个对象?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！