并非使用 Amazon SDK 从 S3 检索的所有数据(仅 1000 个元素) [英] Not all data (only 1000 elements) retrieved from S3 using Amazon SDK

问题描述

我在使用 Amazon SDK 从 S3 检索数据时遇到问题.问题是它只检索了 1000 个元素，而实际上我在 aws_bucket_data -> currentDataDirectory 中有 10,000 个元素.我不使用setMaxKeys(...)，所以结果似乎很奇怪.

I have a problem with retrieving the data from S3 using Amazon SDK. The problem is that it retrieves only 1000 elements, while indeed I have 10,000 elements in the aws_bucket_data -> currentDataDirectory. I do not use setMaxKeys(...), so the result seems to be weird.

BasicAWSCredentials credentials = new BasicAWSCredentials("...", "...");
        client = new AmazonS3Client(credentials);

ListObjectsRequest listObjectsRequest = new ListObjectsRequest()
                    .withBucketName(aws_bucket_data)
                    .withPrefix(currentDataDirectory);

ObjectListing objectListing = client.listObjects(listObjectsRequest);

System.out.println(objectListing.getObjectSummaries().size());

我该如何解决这个问题?

How can I solve this problem?

推荐答案

AWS S3 API 限制为每个响应最多 1000 个键.

AWS S3 API has a limit of maximum 1000 keys per response.

您必须执行多个请求才能检索所有对象.

You will have to do multiple requests to retrieve all of your objects.

您可以在此处查看 API:

You can take a look at the API here:

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html

我找到了一个可以检索所有对象的示例:

I have found a example to retrieve all your objects:

如何列出所有 AWS使用 Java 的存储桶中的 S3 对象

这篇关于并非使用 Amazon SDK 从 S3 检索的所有数据(仅 1000 个元素)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！