如何在Google Dataflow中创建个性化WindowFn [英] How to create a personalised WindowFn in google dataflow
问题描述
我想以不同的方式创建一个不同的WindowFn
,以便根据另一个字段而不是根据输入条目的时间戳将Windows分配给我的任何输入元素.我知道Google DataFlow SDK中的预定义WindowFn
使用时间戳作为分配窗口的条件.
I'd like to create a different WindowFn
in a such way to assign Windows to any of my input elements based on another field instead of based on my input entry's timestamp. I know the pre-defined WindowFn
's from Google DataFlow SDK use the timestamp as a criteria to assign window.
更具体地说,我想创建一种SlidingWindows
,但与其考虑将时间戳作为窗口分配标准,不如将其他字段视为该条件.
More specifically I'd like to create a kind of SlidingWindows
but instead of considering timestamp as the Window assignment criteria I'd like to consider another field as that criteria.
如何创建自定义的WindowFn
?创建自己的WindowFn
时应考虑哪些要点?
How could I create my customised WindowFn
? What are the points that I should consider when creating my own WindowFn
?
谢谢.
推荐答案
To create a new WindowFn, you just need to inherit from WindowFn or a subclass and override the various abstract methods.
在您的情况下,您不需要窗口合并,因此您可以从NonMergingWindowFn继承,并且您的代码可能类似于
In your case, you don't need window merging, so you can inherit from NonMergingWindowFn, and your code could look something like
public class MyWindowFn extends NonMergingWindowFn<ElementT, IntervalWindow> {
public Collection<W> assignWindows(AssignContext c) {
return setOfWindowsElementShouldBeIn(c.element());
}
public boolean isCompatible(WindowFn other) {
return other instanceof MyWindowFn;
}
public Coder<IntervalWindow> windowCoder() {
return IntervalWindow.getCoder();
}
public W getSideInputWindow(final BoundedWindow window) {
// You may not need this if you won't ever be using PCollections windowed
// with this as side inputs. If that's the case, just throw.
// Otherwise you'll need to figure out how to map the main input windows
// into the windows generated by this WindowFn.
}
}
这篇关于如何在Google Dataflow中创建个性化WindowFn的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!