水印卡住 [英] Watermark getting stuck
问题描述
我正在通过pub/sub将数据摄取到以无界模式运行的数据流管道.数据基本上是从跟踪设备捕获的时间戳的坐标.这些消息分批到达，其中每批可能是1..n条消息.在一段时间内，可能没有消息到达，稍后可能会(或不会)重新发送.我们使用每个坐标的时间戳(以UTC表示)作为pub-sub消息的属性.并通过时间戳标签读取管道:
I am ingesting data via pub/sub to a dataflow pipeline which is running in unbounded mode. The data are basically coordinates with timestamps captured from tracking devices. Those messages arrive in batches, where each batch might be 1..n messages. For a certain period there might be no messages arriving, which might be resent later on (or not). We use the time-stamp (in UTC) of each coordinate as an attribute for the pub-sub message. And read the pipeline via a Timestamp label:
pipeline.apply(PubsubIO.Read.topic("new").timestampLabel("timestamp")
坐标和延迟的示例如下:
An example of coordinates and delay looks like:
36 points wait 0:02:24
36 points wait 0:02:55
18 points wait 0:00:45
05 points wait 0:00:01
36 points wait 0:00:33
36 points wait 0:00:43
36 points wait 0:00:34
消息可能看起来像:
2013-07-07 09:34:11;47.798766;13.050133
在第一批处理后，水印为空，在第二批处理后，我可以在管道诊断中看到一个水印，只是没有更新，尽管有新消息到达.另外，根据堆栈驱动程序日志记录，PubSub没有未送达或未确认的消息.
After the first batch the Watermark is empty, after the second batch I can see a Watermark in the Pipeline diagnostics, just it doesn't get updated, although new messages arrive. Also according to stackdriver logging PubSub has no undelivered or unacknowledged messages.
随着带有新事件时间的消息到达，水印是否应该向前移动?
Shouldn't the watermark move forward as messages with new event time arrive?
根据> PubsubIO的水印启发式是什么在GCD上运行? WaterMark还应该每2分钟向前移动一次吗?
According to What is the watermark heuristic for PubsubIO running on GCD? the WaterMark should also move forward every 2minutes which it doesn't?
[..]如果我们没有在更多信息中看到订阅数据 不到两分钟(并且没有积压)，我们将水印提前到 接近实时. [..]
[..] In the case that we have not seen data on the subscription in more than two minutes (and there's no backlog), we advance the watermark to near real time. [..]
更新以解决Bens问题:
是否有我们可以查看的工作ID?
Is there a job ID that we could look at?
是的，我刚刚在EET时间09:52 CET(世界标准时间07:52)重新启动了整个设置，作业ID为2017-05-05_00_49_11-11176509843641901704.
Yes I just restarted the whole setup at 09:52 CET which is 07:52 UTC, with job ID 2017-05-05_00_49_11-11176509843641901704.
您使用的是哪个版本的SDK?
What version of the SDK are you using?
1.9.0
您如何发布带有时间戳标签的消息?
How are you publishing the messages with the timestamp labels?
我们使用python脚本发布使用pub sub sdk的数据. 来自那里的消息可能看起来像:
We use a python script to publish the data which is using the pub sub sdk. A message from there might look like:
{'数据':{timestamp; lat; long; ele}，'timestamp':'2017-05-05T07:45:51Z'}
{'data': {timestamp;lat;long;ele}, 'timestamp': '2017-05-05T07:45:51Z'}
我们将timestamp属性用于数据流中的timestamplabel.
We use the timestamp attribute for the timestamplabel in dataflow.
水印卡在哪里?
What is the watermark stuck at?
对于这项工作，水印现在停留在09:57:35(我将在10:10左右发布)，尽管会发送新数据，例如在
For this job the watermark is now stuck at 09:57:35 (I am posting this around 10:10), although new data is sent e.g. at
10:05:14
10:05:43
10:06:30
我还可以看到，将数据发布到pub sub的延迟可能会超过10秒，例如在10:07:47，我们发布的时间戳最高为10:07:26.
I can also see that it may happen that we publish data to pub sub with delay of more than 10 seconds e.g. at 10:07:47 we publish data with a highest timestamp of 10:07:26.
几个小时后，水印出现了，但是我看不出为什么它被延迟了/一开始没有移动.
After a few hours the watermark catches up but I cannot see why it is delayed /not moving in the beginning.
推荐答案
这是PubSub水印跟踪逻辑中的一个极端情况，它有两个解决方法(请参见下文).本质上，如果2分钟没有输入 ，则水印将前进到当前时间.但是，如果数据到达的速度快于每2分钟一次，但QPS仍然很低，那么就没有足够的数据来使估计的水印保持最新.
This is an edge-case in the PubSub watermark tracking logic that has two work arounds (see below). Essentially, if there is no input for 2 minutes, then the watermark will advance to the current time. But, if data is arriving faster than every 2 minutes but still at a very low QPS, then there isn't enough data to have a keep the estimated watermark up to date.
正如我提到的，有几种解决方法:
As I mentioned, there are several work arounds:
- 如果您处理更多数据，则该问题自然会得到解决.
- 或者，如果您注入额外的消息(例如每秒2条)，则它将为水印提供足够的数据以更快地前进.这些只需要有时间戳记，并且可以立即从管道中过滤掉.
这篇关于水印卡住的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
