从Google Cloud Storage流式传输文件 [英] Stream file from Google Cloud Storage

问题描述

以下是从Google Cloud Storage下载文件的代码:

Here is a code to download File from Google Cloud Storage:

@Override
public void write(OutputStream outputStream) throws IOException {
    try {
        LOG.info(path);
        InputStream stream = new ByteArrayInputStream(GoogleJsonKey.JSON_KEY.getBytes(StandardCharsets.UTF_8));
        StorageOptions options = StorageOptions.newBuilder()
                .setProjectId(PROJECT_ID)
                .setCredentials(GoogleCredentials.fromStream(stream)).build();
        Storage storage = options.getService();
        final CountingOutputStream countingOutputStream = new CountingOutputStream(outputStream);
        byte[] read = storage.readAllBytes(BlobId.of(BUCKET, path));
        countingOutputStream.write(read);
    } catch (Exception e) {
        e.printStackTrace();
    } finally {
        outputStream.close();
    }
}

这有效，但是这里的问题是，它必须先缓冲所有字节，然后再流回此方法的客户端.这会导致很多延迟，尤其是当存储在GCS中的文件很大时.

This works but the problem here is that it has to buffer all the bytes first before it streams back to the client of this method. This is causing a lot of delays especially when the file stored in the GCS is big.

是否可以从GCS获取文件并将其直接流传输到OutputStream ，这里的OutputStream是用于Servlet的.

Is there a way to get the File from GCS and stream it directly to the OutputStream, this OutputStream here btw is for a Servlet.

推荐答案

请澄清一下，您是否需要OutputStream或InputStream?一种查看方式是将存储在Google Cloud Storage对象中的数据作为文件存储，并且您有一个InputStream可以读取该文件.如果可行，请继续阅读.

Just to clarify, do you need an OutputStream or an InputStream ? One way to look at this is that the data stored in Google Cloud Storage object as a file and you having an InputStream to read that file. If that works, read on.

Storage API中没有提供InputStream或OutputStream的现有方法.但是这里有 Cloud Storage客户端库中的2个API ，它们公开了一个ReadChannel对象，该对象是从ReadableByteChannel扩展(来自Java NIO API)的.

There is no existing method in Storage API which provides an InputStream or an OutputStream. But the there are 2 APIs in the Cloud Storage client library which expose a ReadChannel object which is extended from ReadableByteChannel (from java NIO API).

ReadChannel reader(String bucket, String blob, BlobSourceOption... options);
ReadChannel reader(BlobId blob, BlobSourceOption... options);

使用此示例的简单示例(摘自

A simple example using this (taken from StorageSnippets.java):

/**
   * Example of reading a blob's content through a reader.
   */
  // [TARGET reader(String, String, BlobSourceOption...)]
  // [VARIABLE "my_unique_bucket"]
  // [VARIABLE "my_blob_name"]
  public void readerFromStrings(String bucketName, String blobName) throws IOException {
    // [START readerFromStrings]
    try (ReadChannel reader = storage.reader(bucketName, blobName)) {
      ByteBuffer bytes = ByteBuffer.allocate(64 * 1024);
      while (reader.read(bytes) > 0) {
        bytes.flip();
        // do something with bytes
        bytes.clear();
      }
    }
    // [END readerFromStrings]
  }

您还可以使用

You can also use the newInputStream() method to wrap an InputStream over the ReadableByteChannel.

即使您需要OutputStream，您也应该能够将数据从InputStream或更好地从ReadChannel对象复制到OutputStream.

Even if you need an OutputStream, you should be able to copy data from the InputStream or better from the ReadChannel object into the OutputStream.

以以下示例运行此示例:PROGRAM_NAME <BUCKET_NAME> <BLOB_PATH>

Run this example as: PROGRAM_NAME <BUCKET_NAME> <BLOB_PATH>

import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.channels.Channels;
import java.nio.channels.WritableByteChannel;

import com.google.cloud.ReadChannel;
import com.google.cloud.storage.Bucket;
import com.google.cloud.storage.BucketInfo;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;

/**
 * An example which reads the contents of the specified object/blob from GCS
 * and prints the contents to STDOUT.
 *
 * Run it as PROGRAM_NAME <BUCKET_NAME> <BLOB_PATH>
 */
public class ReadObjectSample {
  private static final int BUFFER_SIZE = 64 * 1024;

  public static void main(String[] args) throws IOException {
    // Instantiates a Storage client
    Storage storage = StorageOptions.getDefaultInstance().getService();

    // The name for the GCS bucket
    String bucketName = args[0];
    // The path of the blob (i.e. GCS object) within the GCS bucket.
    String blobPath = args[1];

    printBlob(storage, bucketName, blobPath);
  }

  // Reads from the specified blob present in the GCS bucket and prints the contents to STDOUT.
  private static void printBlob(Storage storage, String bucketName, String blobPath) throws IOException {
    try (ReadChannel reader = storage.reader(bucketName, blobPath)) {
      WritableByteChannel outChannel = Channels.newChannel(System.out);
      ByteBuffer bytes = ByteBuffer.allocate(BUFFER_SIZE);
      while (reader.read(bytes) > 0) {
        bytes.flip();
        outChannel.write(bytes);
        bytes.clear();
      }
    }
  }
}

这篇关于从Google Cloud Storage流式传输文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！