谷歌的数据流和QUOT;工作流程失败"没有理由 [英] Google Dataflow "Workflow failed" with no reason

问题描述

我正在Google Cloud Platform上运行Dataflow-Jobs，我遇到的一个新错误是工作流失败"，没有任何说明.我得到的日志如下:

I am running Dataflow-Jobs on Google Cloud Platform and one new Error I get is "Workflow failed" without any explanations. The logs I get are the following:

 2017-08-25 (00:06:01) Executing operation ReadNewXXXFromStorage/Read+JsonStringsToXXX+RemoveLanguagesFromXXX...
 2017-08-25 (00:06:01) Executing operation ReadOldXYZ_ABC_1234_123_ns_123123123123123/GroupByKey/Create
 2017-08-25 (00:06:01) Starting 1 workers in europe-west1-b...
 2017-08-25 (00:06:01) Executing operation ReadOldXYZ_ABC_1234_123_ns_123123123123123/ParDo(SplitQuery)+ReadOldXYZ...
 2017-08-25 (00:06:48) Workflow failed.
 2017-08-25 (00:06:48) Stopping worker pool...
 2017-08-25 (00:06:58) Worker pool stopped.

我应该如何找出问题所在?对象的权限应该不成问题，因为类似的作业可以成功运行.当我尝试从Google Cloud Console重新运行模板时，收到消息:

How am I supposed to find out whats going wrong? It should not be a problem with rights on the object, as similar jobs run successfully. When I try to rerun the template from Google Cloud Console, I get the message:

找不到此模板的元数据文件

No metadata file found for this template

但是我能够启动模板，现在它可以成功运行了.这可能与超出配额有关吗?我们刚刚增加了Dataflow的CPU和IP配额，并且我将并行运行的作业从5增加到15，以便能够使用配额.当我在不运行任何其他Job的情况下重新运行模板时，一切似乎都正常运行.

But I am able to start the template and now it runs successfully. May this have to do with exceeded quotas? We just increased our CPU and IP-Quota for Dataflow and I increased our parallel running jobs from 5 to 15 to be able to use the quota. When I rerun the template without any other Jobs running, everything seems to work fine.

高度赞赏任何输入.谢谢

Any Input is highly appreciated. Thanks

似乎作业由于超出CPU配额而失败，但是通常我们会得到一个错误说明，上面写着无法产生足够的工人".不过，在我减少了每个工作的最大工人人数之后，一切工作正常，这样就不能超过我们的配额.

Seems like the Jobs failed because of exceeded CPU-Quota, but usually we would get an error-description where it says "could not spawn enough workers". Nevertheless, Everything works fine after I reduced the maximum number of workers per job, so that our quota cannot be exceeded.

推荐答案

我认为没有为该模板找到元数据文件"应被视为警告，而不是错误.模板可以具有与之关联的元数据"文件，该文件允许参数验证.如果不存在此类文件，则不会验证参数，但其他所有内容都将正常运行-消息仅是这种情况的指示.

I believe the "No metadata file found for this template" should be considered a warning, not an error. A template is able to have a "metadata" file associated with it which allows validation of parameters. If no such file is present, the parameters aren't validated, but everything else works as normal -- the message is just the indicator of this situation.

这听起来像是问题是由于其他原因导致作业无法完成.根据您的描述和修改，听起来这是因为没有足够的配额来运行作业.

It sounds like the problem was the job being unable for other reasons. Based on your description and the edit, it sounds like this was because of lack of quota to run the job.

这篇关于谷歌的数据流和QUOT;工作流程失败"没有理由的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！