气流工作器 - 连接中断:IncompleteRead(读取 0 个字节) [英] Airflow worker - Connection broken: IncompleteRead(0 bytes read)

问题描述

使用 Airflow worker 和 webserver/scheduler 作为在 EC2 上的 Kubernetes Engine 上运行的 Docker 镜像

Using Airflow worker and webserver/scheduler as a Docker images running on Kubernetes Engine on EC2

我们有一个具有 KubernetesPodOperator 的任务，它是资源密集型的，每 15 分钟运行一次.

We have a task which has KubernetesPodOperator which is resource intensive and runs every 15min.

在airflow-worker 中作为电子邮件收到这些错误

Try 2 out of 3
Exception:
('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
Log: Link
Host: airflow-worker-deployment-123456789
Log file: /usr/local/airflow/logs/DAG_NAME/TASK_NAME/2019-03-14T10:50:00+00:00.log
Mark success: Link

知道它可以是什么吗?

推荐答案

所以，迟到总比不到好

这是因为 KubernetesPodOperator 中的已知错误.为避免这种行为，您必须将操作符 get_logs 参数设置为 False.默认值为 True.

it is because of known bug in KubernetesPodOperator. to avoid this behavior you have to set operators get_logs parameter to False. default value is True.

详情在这里https://issues.apache.org/jira/browse/AIRFLOW-3534https://issues.apache.org/jira/browse/AIRFLOW-5571

这篇关于气流工作器 - 连接中断:IncompleteRead(读取 0 个字节)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！