Bigquery流式传输数据延迟 [英] Bigquery streaming data delays

问题描述

我正在将数据流式传输到bigquery表中，由于某些原因，我最近几次检查数据不能立即得到。流数据最终可用于查询，有时它看起来像超过一个小时不同步。但数据最终会在一些延迟后出现。

这是否预期？如果是的话，它有什么方法？该表按日期进行分区，我们不会将数据加载到任何特定的分区，我们仅在当天流式传输数据。

更新： p>

当我说数据不可用时，我的意思是运行以下查询似乎不匹配数据被推送到Bigquery的日期/时间。

 从[nproj.reporting]中选择max（dateTime）; 
  

解决方案

我感到你的痛苦。我真的找到了这家伙的问题（和答案） 照亮。

I'm streaming data into a bigquery table, for some reason last few times I checked the data isn't available instantly. The streamed data eventually is available to query, sometimes it looks like it's more than an hour out of sync. But data does eventually appear after some delays.

Is this expected? If it is, what are the ways around it? The table is partitioned by date and we are not "load"-ing data into any particular partition, we are streaming data only for the current day.

Update:

When I say data is not available, I mean running following query doesn't seem to match date/time the data was pushed to Bigquery.

select   max(dateTime) from [nproj.reporting] ;

解决方案

I feel your pain. I really found this guy's question (and answer) to be illuminating.

这篇关于Bigquery流式传输数据延迟的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！