TextIO可以写入从窗口maxTimestamp派生的前缀吗? [英] Can TextIO write to prefixes derived from the window maxTimestamp?

问题描述

我正在处理PubSub消息的窗口化流，我想将其归档到GCS.我希望存档文件具有从窗口时间戳派生的前缀(例如gs://bucket/messages/2015/01/messages-2015-01-01.json). TextIO.Write是否可以实现此功能，还是我需要实现自己的FileBasedSink?

I am processing a windowed stream of PubSub messages and I would like to archive them to GCS. I'd like the archived files to have a prefix that's derived from the window timestamp (something like gs://bucket/messages/2015/01/messages-2015-01-01.json). Is this possible with TextIO.Write, or do I need to implement my own FileBasedSink?

推荐答案

这可以通过最近添加的用于TextIO中的窗口写入的功能来完成.请参阅文档有关TextIO 的信息，请参见withWindowedWrites和to(FilenamePolicy). AvroIO中也存在此功能.

This can be done with the recently added feature for windowed writes in TextIO. Please see the documentation for TextIO, in particular see withWindowedWrites and to(FilenamePolicy). This feature is also present in AvroIO.

这篇关于TextIO可以写入从窗口maxTimestamp派生的前缀吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！