在 Apache Beam 中从 GCS 读取文件 [英] Read a file from GCS in Apache Beam

问题描述

我需要从 GCS 存储桶中读取文件.我知道我必须使用 GCS API/客户端库，但我找不到任何与之相关的示例.

I need to read a file from a GCS bucket. I know I'll have to use GCS API/Client Libraries but I cannot find any example related to it.

我一直在参考 GCS 文档中的这个链接:GCS 客户端库.却真的打不开.如果有人可以提供一个真正有帮助的例子.谢谢.

I have been referring to this link in the GCS documentation: GCS Client Libraries. But couldn't really make a dent. If anybody can provide an example that would really help. Thanks.

推荐答案

好的.如果您想简单地从 GCS 读取文件，而不是作为 PCollection 而是作为常规文件，并且如果您在使用 GCS Java 客户端库时遇到问题，您还可以使用 Apache Beam 文件系统 API:

OK. If you want to simply read files from GCS, not as a PCollection but as regular files, and if you are having trouble with the GCS Java client libraries, you can also use the Apache Beam FileSystems API:

首先，您需要确保在 beam-sdks-java-extensions-google-cloud-platform-core 上的 pom.xml 中有一个 Maven 依赖项> 其中包含 gs:// 文件系统的实现:

First, you need to make sure that you have a Maven dependency in your pom.xml on beam-sdks-java-extensions-google-cloud-platform-core which contains implementation of the gs:// filesystem:

<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-extensions-google-cloud-platform-core</artifactId>
</dependency>

然后设置 FileSystems API(默认设置在所有管道中，但如果您在管道外使用它，则需要手动完成).

Then set up the FileSystems API (it is set up by default in all pipelines, but if you're using it outside a pipeline, you need to do it manually).

PipelineOptions options = PipelineOptionsFactory.create();
// ...Optionally fill in options such as GCP credentials...
// (see GcpOptions class)
FileSystems.setDefaultPipelineOptions(options);

然后就可以使用了:

ReadableByteChannel chan = FileSystems.open(FileSystems.matchNewResource(
  "gs://path/to/your/file", false /* is_directory */));
try (InputStream stream = Channels.newInputStream(chan)) {
  // Use regular Java utilities to work with the input stream.
}

这篇关于在 Apache Beam 中从 GCS 读取文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！