如何从gcs中获取文件列表？ [英] How to fetch the file list from gcs?

问题描述

遵循google的入门，我使用以下代码获取列表在远程目录中的所有文件

  class GCSFileStorage {
 String bucket =bucket_name; 
 String remoteDirectoryPath =remote / path; 
 int fetchBlockSize = 1024 * 1024; 
 GcsService gcsService = 
 GcsServiceFactory.createGcsService（RetryParams.getDefaultInstance（））; 
 
列表< String> list（）{
 List< String> filenames = new List（）; 
 ListResult listResult = gcsService.list（bucket，ListOptions.DEFAULT）; 
 while（listResult.hasNext（））{
 ListItem listItem = listResult.next（）; 
文件名+ = listItem.getName（）; 
} 
返回文件名; 
} 
} 
 
 GCSFileStorage gcs = new GCSFileStorage（）; 
 gcs.list（）; 
  

但是这段代码会失败并产生一个异常：

  java.io.IOException：com.google.appengine.tools.cloudstorage.RetriesExhaustedException：
 ... 
导致：java.io.IOException：java .lang.NullPointerException 
 ... 
导致：java.lang.NullPointerException 
在com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService $ BlobStorageAdapter。< init>（LocalRawGcsService .java：123）
 at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService $ BlobStorageAdapter.getInstance（LocalRawGcsService.java:184）
  

我怀疑我应该以gcs授权，这可能是失败的原因。然而，我还没有找到正确的方法来启动gcs工作所需的一切。

我使用的https://cloud.google.com/appengine/docs/java/googlecloudstorageclient/ =nofollow>客户端是特定于App Engine的。它是通过依赖关系添加的。 $ b

  com.google.appengine.tools:appengine-gcs-client:0.5 
 
 code> 

改为 REST API客户端应该被使用。它的依赖关系是


  com.google.apis：google-api-services-storage：v1-rev44-1.20.0 
  

然后，获取文件列表的代码可能如下所示：



  import com.google.api.client.googleapis.auth.oauth2.GoogleCredential; 
 import com.google.api.client.http.javanet.NetHttpTransport; 
 import com.google.api.client.json.jackson2.JacksonFactory; 
导入com.google.api.services.storage.Storage; 
导入com.google.api.services.storage.StorageScopes; 
导入com.google.api.services.storage.model.Objects; 
 import com.google.api.services.storage.model.StorageObject; 
导入com.google.common.collect.Lists; 
 
 import java.io.File; 
 import java.io.IOException; 
 import java.security.GeneralSecurityException; 
 import java.util.LinkedList; 
 import java.util.List; 
 
 
 class GCSFileStorage {
 String bucket =bucket_name; 
 String remoteDirectoryPath =remote / path; 
存储空间
 
 public GCSFileStorage（）抛出GeneralSecurityException，IOException {
 storage = setupStorage（）; 
} 
 
列表< String> list（）抛出IOException {
 List< String> allItems = new LinkedList< String>（）; 
对象响应= storage.objects（）.list（bucket）。 
 setPrefix（remoteDirectoryPath）.execute（）; 
 for（StorageObject obj：response.getItems（））{
 allItems.add（obj.getName（））; 
 
 while（response.getNextPageToken（）！= null）{
 String pageToken = response.getNextPageToken（）; 
 response = storage.objects（）。list（bucket）。 
 setPrefix（remoteDirectoryPath）.setPageToken（pageToken）.execute（）; 
 for（StorageObject obj：response.getItems（））{
 allItems.add（obj.getName（））; 
} 
} 
返回allItems; 
} 
 
 
存储setupStorage（）抛出GeneralSecurityException，IOException {
 GoogleCredential凭证=新的GoogleCredential.Builder（）。 
 setTransport（新的NetHttpTransport（））。 
 setJsonFactory（new JacksonFactory（））。 
 setServiceAccountId（your_account_id）。 
 setServiceAccountScopes（
 Lists.newArrayList（StorageScopes.DEVSTORAGE_FULL_CONTROL））。 
 setServiceAccountPrivateKeyFromP12File（
 new File（/ local / path / to / private / key.p12））。 
 build（）; 
 
返回新的存储。 
 Builder（新的NetHttpTransport（），
新的JacksonFactory（），凭据）。 
 setApplicationName（foo）。build（）; 
} 
} 
  




Following google's Getting Started I use following code to get the list of all files in a remote directory

class GCSFileStorage {
    String bucket = "bucket_name";
    String remoteDirectoryPath = "remote/path";
    int fetchBlockSize = 1024 * 1024;
    GcsService gcsService =
      GcsServiceFactory.createGcsService(RetryParams.getDefaultInstance());

    List<String> list() {
        List<String> filenames = new List();
        ListResult listResult = gcsService.list(bucket, ListOptions.DEFAULT);
        while (listResult.hasNext()) {
            ListItem listItem = listResult.next();
            filenames += listItem.getName();
        }
        return filenames;
    }
}

GCSFileStorage gcs = new GCSFileStorage();
gcs.list();

But this code fails with an exception:

java.io.IOException: com.google.appengine.tools.cloudstorage.RetriesExhaustedException:
...
Caused by: java.io.IOException: java.lang.NullPointerException
...
Caused by: java.lang.NullPointerException
    at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService$BlobStorageAdapter.<init>(LocalRawGcsService.java:123)
    at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService$BlobStorageAdapter.getInstance(LocalRawGcsService.java:184)

I suspect that I somehow should authorize in gcs and this may be the reason of failure. However I haven't found proper way to init everything that gcs needs for work.

解决方案

As @ozarov mentioned the client I was using is specific for App Engine. It was added through dependency

com.google.appengine.tools:appengine-gcs-client:0.5

Instead REST API client should be used. Its dependency is

com.google.apis:google-api-services-storage:v1-rev44-1.20.0

Then the code to fetch files list may look as follows

import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.services.storage.Storage;
import com.google.api.services.storage.StorageScopes;
import com.google.api.services.storage.model.Objects;
import com.google.api.services.storage.model.StorageObject;
import com.google.common.collect.Lists;

import java.io.File;
import java.io.IOException;
import java.security.GeneralSecurityException;
import java.util.LinkedList;
import java.util.List;


class GCSFileStorage {
    String bucket = "bucket_name";
    String remoteDirectoryPath = "remote/path";
    Storage storage

    public GCSFileStorage() throws GeneralSecurityException, IOException {
        storage = setupStorage();
    }

    List<String> list() throws IOException {
        List<String> allItems = new LinkedList<String>();
        Objects response = storage.objects().list(bucket).
            setPrefix(remoteDirectoryPath).execute();
        for (StorageObject obj: response.getItems()) {
            allItems.add(obj.getName());
        }
        while (response.getNextPageToken() != null) {
            String pageToken = response.getNextPageToken();
            response = storage.objects().list(bucket).
                setPrefix(remoteDirectoryPath).setPageToken(pageToken).execute();
            for (StorageObject obj: response.getItems()) {
                allItems.add(obj.getName());
            }
        }
        return allItems;
    }


    Storage setupStorage() throws GeneralSecurityException, IOException {
        GoogleCredential credential = new GoogleCredential.Builder().
            setTransport(new NetHttpTransport()).
            setJsonFactory(new JacksonFactory()).
            setServiceAccountId("your_account_id").
            setServiceAccountScopes(
                Lists.newArrayList(StorageScopes.DEVSTORAGE_FULL_CONTROL)).
            setServiceAccountPrivateKeyFromP12File(
                new File("/local/path/to/private/key.p12")).
            build();

        return new Storage.
            Builder(new NetHttpTransport(),
                new JacksonFactory(), credential).
            setApplicationName("foo").build();
    }
}

这篇关于如何从gcs中获取文件列表？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！