为什么数据流步骤无法启动? [英] Why do Dataflow steps not start?

问题描述

我有一个线性的三步数据流管道-由于某种原因，最后一步开始了，但是前两步在Not started中挂了很长时间，之后我放弃并取消了工作.我不确定是什么原因造成的，因为该管道在过去已成功运行，并且令我惊讶的是，在日志中未显示任何错误，从而阻止了前两个步骤的启动.会导致这种情况的原因是什么?如何防止这种情况发生?

I have a linear three step Dataflow pipeline - for some reason the last step started, but the preceding two steps hung in Not started for a long time before I gave up and killed the job. I'm not sure what caused this, as this same pipeline had successfully run in the past, and I'm surprised it didn't show any errors in the logs as to what was preventing the first two steps from starting. What can cause such a situation and how can I prevent it from happening?

推荐答案

这是由于工作程序启动错误而发生的.某些Dataflow步骤似乎不需要工作人员(例如，写入GCS)，这就是为什么该步骤能够启动的原因-即该步骤开始并不意味着正确地创建了工作人员.默认情况下，作业启动"未显示在作业日志中-您需要在作业日志中单击指向Stackdriver的链接，然后在日志下拉列表中添加worker-startup才能查看其中的任何错误.

This was happening because of an error in the worker start up. Certain Dataflow steps do not seem to require workers (e.g. writing to GCS), which is why that step was able to start - i.e. that step starting does not imply that workers are being created correctly. Worker start up is not displayed in the job logs by default - you need to click the link to Stackdriver in the job logs and then add worker-startup in the logs drop down in order to see any of those errors.

这篇关于为什么数据流步骤无法启动?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！