在Apache Beam中寻找与文件模式匹配的新文件 [英] Watching for new files matching a filepattern in Apache Beam

问题描述

我在GCS或其他受支持的文件系统上有一个目录，外部进程正在向该目录中写入新文件.

I have a directory on GCS or another supported filesystem to which new files are being written by an external process.

我想编写一个Apache Beam流传输管道，该管道持续监视此目录中的新文件，并在每个新文件到达时对其进行读取和处理.这可能吗?

I would like to write an Apache Beam streaming pipeline that continuously watches this directory for new files and reads and processes each new file as it arrives. Is this possible?

推荐答案

从Apache Beam 2.2.0开始，这是可能的.几个API支持此用例:

This is possible starting with Apache Beam 2.2.0. Several APIs support this use case:

如果您使用的是TextIO或AvroIO，他们会通过TextIO.read().watchForNewFiles()显式支持此功能，而在readAll()上也是如此，例如:

If you're using TextIO or AvroIO, they support this explicitly via TextIO.read().watchForNewFiles() and the same on readAll(), for example:

PCollection<String> lines = p.apply(TextIO.read()
    .from("gs://path/to/files/*")
    .watchForNewFiles(
        // Check for new files every 30 seconds
        Duration.standardSeconds(30),
        // Never stop checking for new files
        Watch.Growth.<String>never()));

如果您使用其他文件格式，则可以将FileIO.match().continuously()和FileIO.matchAll().continuously()与FileIO.readMatches()结合使用，它们支持相同的API.

If you're using a different file format, you may use FileIO.match().continuously() and FileIO.matchAll().continuously() which support the same API, in combination with FileIO.readMatches().

API支持指定检查新文件的频率以及何时停止检查(支持的条件例如:如果在给定时间内没有新输出出现"，在观察到N个输出之后"，开始检查"及其组合.)

The APIs support specifying how often to check for new files, and when to stop checking (supported conditions are e.g. "if no new output appears within a given time", "after observing N outputs", "after a given time since starting to check" and their combinations).

请注意，此功能目前仅在Direct运行器和Dataflow运行器中有效，并且仅在Java SDK中可用.通常，它可以在支持 Splittable DoFn 的任何运行程序中运行(请参阅

Note that right now this feature currently works only in the Direct runner and the Dataflow runner, and only in the Java SDK. In general, it will work in any runner that supports Splittable DoFn (see capability matrix).

这篇关于在Apache Beam中寻找与文件模式匹配的新文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！