Google Cloud Dataprep可以监控GCS路径中是否有新文件吗? [英] Can Google Cloud Dataprep monitor a GCS path for new files?

问题描述

Google Cloud Dataprep看起来很棒，我们已经使用它手动导入了静态数据集，但是我想多次执行它，以便它可以使用上载到GCS路径的新文件.我可以看到您可以为Dataprep设置时间表，但是在导入设置中的任何地方都看不到它将如何处理新文件.

Google Cloud Dataprep seems great and we've used it to manually import static datasets, however I would like to execute it more than once so that it can consume new files uploaded to a GCS path. I can see that you can setup a schedule for Dataprep, but I cannot see anywhere in the import setup how it would process new files.

这可能吗?似乎很明显的需求-希望我错过了明显的东西.

Is this possible? Seems like an obvious need - hopefully I've missed something obvious.

推荐答案

您可以通过在数据集期间单击文件夹左侧的+图标来将GCS路径添加为数据集.当为使用此数据集的流设置计划作业时，该目录中的所有文件(包括新文件)将在每次计划作业运行时获取.

You can add a GCS path as a dataset by clicking on the + icon left of the folder during the dataset (see screenshot). When you set up a scheduled job for a flow that uses this dataset, all files in that directory (including new files) will be picked up on each scheduled job run.

这篇关于Google Cloud Dataprep可以监控GCS路径中是否有新文件吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！