“数据流似乎卡住了". [英] "The Dataflow appears to be stuck"
问题描述
我正在运行上个月运行的数据流.来自同一客户端，具有相同的数据流版本(0.7.0dev0).不幸的是，它以前所未有的神秘方式失败了.
I am running a dataflow that I last ran a few months ago. From the same client, with same dataflow version (0.7.0dev0). Unfortunately it fails in mysterious ways that it did not do before.
我正在开始工作，第一步是:
I am starting the job, and the first stage is:
(8733429d016bc2fb):执行从数据存储区/拆分查询中读取的操作+从数据存储区/GroupByKey/Reify中读取的操作+从数据存储区/GroupByKey/Write中读取的操作
(8733429d016bc2fb): Executing operation read from datastore/Split Query+read from datastore/GroupByKey/Reify+read from datastore/GroupByKey/Write
但是1小时后会出现以下错误:
But it gives the following error after 1 hour:
(e88cb3c076926976):工作流失败.原因:(e88cb3c07692626f): 数据流似乎卡住了.请通过以下方式与数据流小组联系 http://stackoverflow.com/questions/tagged/google-cloud-dataflow .
(e88cb3c076926976): Workflow failed. Causes: (e88cb3c07692626f): The Dataflow appears to be stuck. Please reach out to the Dataflow team at http://stackoverflow.com/questions/tagged/google-cloud-dataflow.
如果有帮助，则JobID为 2017-08-21_00_30_03-3588685705436948852 .我将升级到该库的较新版本，但这需要对API进行更多更改，并弄清楚如何使所有部分重新工作.所以我现在正在努力.我希望一个以前可用且当前失败的简单用例"可能比更改更多内容更容易调试.
if would help, JobID is 2017-08-21_00_30_03-3588685705436948852. I would upgrade to a newer version of the library, but that involves a bunch more API changes and figuring out how to get all the pieces working again. So I'm working at it now. I was hoping that "a simple use case that previously worked and currently fails" might be easier to debug than changing even-more-things.
我不确定如何调试或进一步调查.几个月前，它使用相同的代码即可工作，但现在不起作用(数据集扩大了4-5倍，记录200-300K，没有什么疯狂的...)
I'm not sure how to debug or investigate further. It worked a few months ago with the same code, but doesn't work now (with a 4-5x larger dataset, of 200-300K records, nothing crazy...)
推荐答案
此问题已通过升级到2.0.0来解决(感谢Ben Chambers！)似乎0.7.0不再适用于云数据流.
This was fixed by upgrading to 2.0.0 (thanks Ben Chambers!) It seems that 0.7.0 no longer worked well with cloud dataflow.
这篇关于“数据流似乎卡住了".的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
