在 Docker 上运行时，气流给出日志文件不存在错误 [英] Airflow giving log file does not exist error while running on Docker

问题描述

调度程序和网络服务器正在不同的容器上运行，当我运行 DAG 并检查网络服务器上的日志时，它向我显示了这个特定错误.

The scheduler and the webserver are being run on different containers and when I run a DAG and check the logs on the webserver, it shows me this particular error.

*** Log file does not exist: /usr/local/airflow/logs/indexing/index_articles/2019-12-31T00:00:00+00:00/1.log
*** Fetching from: http://465e0f4a4332:8793/log/indexing/index_articles/2019-12-31T00:00:00+00:00/1.log
*** Failed to fetch log file from worker. HTTPConnectionPool(host='465e0f4a4332', port=8793): Max retries exceeded with url: /log/indexing/index_articles/2019-12-31T00:00:00+00:00/1.log (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0a143700d0>: Failed to establish a new connection: [Errno 111] Connection refused'))

我按照其他类似问题中的说明设置了气流变量，我在 cfg 文件中更改的唯一变量是这些.

I set the airflow variables as mentioned in this other similar question and the only variables that I'm changing on the cfg files are these.

AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres:5432/airflow
AIRFLOW__CORE__LOAD_EXAMPLES=False
AIRFLOW__CORE__BASE_URL = http://{hostname}:8080

我手动检查并正确生成了日志文件，我假设唯一的问题是无法通过网络服务器容器公开访问 url.我不确定我在哪里搞砸了，我正在本地运行和测试它.

I manually checked and log files are being generated properly, I'm assuming the only problem is the url not being publically accessible through the webserver container. I'm not sure where I'm messing it up and I'm running and testing this in the local.

推荐答案

问题是因为 docker 容器不共享文件系统.这由响应的第一行指示.

The problem is because the docker containers do not share a filesystem. This is indicated by the first line of the response.

Airflow 然后回退到尝试通过 HTTP 获取日志文件，如响应的第二行所示.其他答案尝试通过覆盖 HOSTNAME_CALLABLE 函数来解决此问题，但是除非主机通过 HTTP 公开日志文件，否则这将不起作用.

Airflow then falls back to attempting to fetch the log-file over HTTP, as indicted by the second line of the response. Other answers try to fix this by overriding the HOSTNAME_CALLABLE function, however this will not work unless the host is exposing the logfiles over HTTP.

解决方案是通过挂载共享卷来解决第一个问题.

The solution is to fix the first problem by mounting a shared volume.

在您的 docker-compose.yml 文件中，添加一个名为 logs-volume 的新卷.

In your docker-compose.yml file, add a new volume called logs-volume.

volumes:
  logs-volume:

然后，同样在 docker-compose.yml 文件中，将此卷添加到所需的日志目录中，在您的情况下为 /usr/local/airflow/logs/，对于每个服务:

Then, also in the docker-compose.yml file, add mount this volume to the required log directory, in your case /usr/local/airflow/logs/, for each service:

services: 
  worker:
    volumes:
      - logs-volume:/usr/local/airflow/logs
  webserver:
    volumes:
      - logs-volume:/usr/local/airflow/logs

这篇关于在 Docker 上运行时，气流给出日志文件不存在错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！