气流需要mysql吗? [英] Does airflow require mysql?

问题描述

我正在尝试将我们的气流版本升级到 1.10.0.当我这样做时，我收到一个错误，抱怨它无法连接到 mysql:

I am trying to upgrade our version of airflow to 1.10.0. When I do, I get an error that complains it cannot connect to mysql:

worker_1     | sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (2002, 'Can\'t connect to local MySQL server through socket \'/var/run/mysqld/mysqld.sock\' (2 "No such file or directory")') (Background on this error at: http://sqlalche.me/e/e3q8)

当我尝试从我们的系统中完全删除 mysql 时，我得到以下信息:

When I try to remove mysql from our systems altogether, I get the following instead:

scheduler_1  | [2018-10-25 17:22:19,399] {{celery_executor.py:113}} ERROR - No module named 'MySQLdb'

Mysql 没有出现在我们设置的环境变量中，也没有出现在airflow.cfg 中.似乎这个版本的气流由于其他原因需要 mysql.这是真的?

Mysql appears in no environment variable we have set, nor does it appear in airflow.cfg. It appears as if this version of airflow requires mysql for some other reason. Is this true?

更新这类似于在此处提出的问题，但是我更感兴趣的是气流为什么要调用 mysql.

Update This is similar to the issue raised here, but I'm more interested in why airflow is calling mysql at all.

我还应该指出，我们确实将 sqlalchemy 连接显式设置为 postgres 数据库.

I should point out also that we do explicitly set the sqlalchemy connection to a postgres database.

AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgres://airflow:airflow@postgres/airflow

当气流尝试写入任务运行的结果时发生错误(将某些内容标记为失败).

The error is happening when airflow is trying to write the result of a task run (marking something as failure).

更新

这是我使用的定义气流图像的 dockerfile.注意没有提到mysql:

This is the dockerfile I use which defines the airflow image. Note no mention of mysql:

# SOURCE: https://github.com/puckel/docker-airflow

FROM python:3.6-jessie

# Never prompts the user for choices on installation/configuration of packages
ENV DEBIAN_FRONTEND noninteractive
ENV TERM linux

# Airflow
ARG AIRFLOW_VERSION=1.10.0
ARG AIRFLOW_HOME=/usr/local/airflow

# Define en_US.
ENV LANGUAGE en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LC_ALL en_US.UTF-8
ENV LC_CTYPE en_US.UTF-8
ENV LC_MESSAGES en_US.UTF-8
ENV PYTHONPATH ${AIRFLOW_HOME}
ENV AIRFLOW_GPL_UNIDECODE yes

COPY ./requirements.txt .

RUN set -ex \
    && buildDeps=' \
        python3-dev \
        libkrb5-dev \
        libsasl2-dev \
        libssl-dev \
        libffi-dev \
        build-essential \
        libblas-dev \
        liblapack-dev \
        libpq-dev \
        git \
    ' \
    && apt-get update -yqq \
    && apt-get upgrade -yqq \
    && apt-get install -yqq --no-install-recommends \
        $buildDeps \
        python3-pip \
        python3-requests \
        apt-utils \
        curl \
        rsync \
        netcat \
        locales \
        vim \
    && sed -i 's/^# en_US.UTF-8 UTF-8$/en_US.UTF-8 UTF-8/g' /etc/locale.gen \
    && locale-gen \
    && update-locale LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 \
    && useradd -ms /bin/bash -d ${AIRFLOW_HOME} airflow \
    && pip install -U pip setuptools wheel \
    && pip install Cython \
    && pip install pytz \
    && pip install pyOpenSSL \
    && pip install ndg-httpsclient \
    && pip install pyasn1 \
    && pip install apache-airflow[crypto,celery,postgres,hive,jdbc]==$AIRFLOW_VERSION \
    && pip install 'celery[redis]>=4.1.1,<4.2.0' \
    && pip install -r requirements.txt \
    && apt-get purge --auto-remove -yqq $buildDeps \
    && apt-get autoremove -yqq --purge \
    && apt-get clean \
    && rm -rf \
        /var/lib/apt/lists/* \
        /tmp/* \
        /var/tmp/* \
        /usr/share/man \
        /usr/share/doc \
        /usr/share/doc-base

COPY script/entrypoint.sh /entrypoint.sh
COPY celery_healthcheck.sh ${AIRFLOW_HOME}
COPY config/airflow.cfg ${AIRFLOW_HOME}/airflow.cfg
COPY dags ${AIRFLOW_HOME}/dags
COPY operators ${AIRFLOW_HOME}/operators
COPY models ${AIRFLOW_HOME}/models
COPY constants.py ${AIRFLOW_HOME}/constants.py
COPY envconsul ${AIRFLOW_HOME}/envconsul
COPY *.hcl ${AIRFLOW_HOME}/

RUN chown -R airflow: ${AIRFLOW_HOME}

EXPOSE 8080 5555 8793

USER airflow
WORKDIR ${AIRFLOW_HOME}

推荐答案

想通了.原来这个其他 env var (AIRFLOW__CELERY__RESULT_BACKEND) 设置了一个错字.我将它设置为 AIRFLOW__CELERY__CELERY_RESULT_BACKEND.我不清楚为什么这在 1.9 中有效，并且在更新时突然开始抛出此错误，但是当我修复 var 时，它现在可以工作了.

Figured it out. Turns out this other env var (AIRFLOW__CELERY__RESULT_BACKEND) was set with a typo. I had it set to AIRFLOW__CELERY__CELERY_RESULT_BACKEND. I'm not clear why that worked in 1.9 and suddenly started throwing this error when updating, but when I fixed the var it now works.

这篇关于气流需要mysql吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！