我们能否迭代Amazon S3中的完整对象集 [英] Could we iterate over the complete set of objects in Amazon S3

问题描述

我尝试打印S3存储桶中所有对象的元数据.但是，它不会返回超过1000个对象的结果.我曾尝试实现 objectListing.isTruncated()，但它没有帮助.这是我列出超过1000个对象的示例代码.

I have tried to print the metadata of all the objects in S3 bucket. However, it does not return the results of more than 1000 objects. I have tried implementing the objectListing.isTruncated() and it did not help. Here is a sample code of what I did to list more than 1000 objects.

 ListObjectsRequest listObjectsRequest = new ListObjectsRequest()
            .withBucketName(bucketName);
    ObjectListing objectListing;
    do {
        objectListing = s3client.listObjects(listObjectsRequest);
        for (S3ObjectSummary objectSummary :
                objectListing.getObjectSummaries()) {
            System.out.println( " - " + objectSummary.getKey() + "  " +
                    "(size = " + objectSummary.getSize() +
                    ")");

            listObjectsRequest.setMarker(objectListing.getNextMarker());
        }
        listObjectsRequest.setMarker(objectListing.getNextMarker());
    } while (objectListing.isTruncated());

推荐答案

这解决了我的问题.我已经设置了一个标记并截断了我的列表，并且能够打印所有对象(超过1000个).

This solved my problem. I had setup a marker and truncated my list and was able to print all the objects (more than 1000).

 ListObjectsRequest listObjectsRequest = new ListObjectsRequest()
     .withBucketName(bucketName);
 ObjectListing objectListing;
 do {
     objectListing = s3.listObjects(listObjectsRequest);
     System.out.println("Enter the path where to save yout file");
     Scanner scan = new Scanner(System.in);
     String path = scan.nextLine();
     fileOne = new File(path);
     fw = new FileWriter(fileOne.getAbsoluteFile(), true);
     bw = new BufferedWriter(fw);
     bw.write("Writing data to file");
     bw.write("\n");
     for (S3ObjectSummary objectSummary: objectListing.getObjectSummaries()) {
         String key = objectSummary.getKey();
         String dummyKey = key.substring(2);
         if (dummyKey.equalsIgnoreCase("somestring")) {
             S3Object s3object = s3.getObject(new GetObjectRequest(bucketName, key));
             BufferedReader reader = new BufferedReader(new InputStreamReader(s3object.getObjectContent()));
             String line;
             int i = 0;
             while ((line = reader.readLine()) != null) {
                 if (i > 0) {
                     bw.append(line + "," + s3object.getKey().substring(0, 2));
                     bw.append(objectSummary.getLastModified().toString());
                     bw.newLine();
                 }
                 i++;
                 System.out.println(line);
             }
         }
         //                    bw.close();
     }
     listObjectsRequest.setMarker(objectListing.getNextMarker());
 } while (objectListing.isTruncated());

这篇关于我们能否迭代Amazon S3中的完整对象集的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！