通过数据流将pubsub数据写入gcs [英] Write pubsub data to gcs through dataflow

问题描述

我想通过数据流流作业使用pubsub中的数据，并将其存储到每小时目录中的GCS中.

I would like to consume data from pubsub through dataflow streaming job and store it into GCS in hourly directories.

什么是最好的方法?

我尝试使用WindowedFilenamePolicy，但是它添加了一个附加的group by并减慢了写入时的写入操作.数据流可以正确缓冲数据，但是花费太长时间才能将数据写入临时存储区.

I tried using WindowedFilenamePolicy but it adds an additional group by and slows down the write operation at the time of writes. Dataflow buffers the data correctly but takes too long to write data in temp bucket.

对于这种相当常见的情况，有什么最佳实践吗?

Any best practice for such fairly common case?

关于， 帕里

推荐答案

将Google提供的数据流模板用于流传输管道

Using the Google-Provided Dataflow Template for the streaming pipeline from Cloud Pub/Sub to Google Cloud Storage files, you can easily do it by setting the outputDirectory to gs://<BUCKET>/YYYY/MM/DD/HH/ and it will automatically replace YYYY, MM, DD and HH for the values of the interval window.

这篇关于通过数据流将pubsub数据写入gcs的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！