配置气流以与CeleryExecutor配合使用 [英] configuring Airflow to work with CeleryExecutor

问题描述

我尝试将Airbnb AirFlow配置为使用CeleryExecutor，如下所示：

I try to configure Airbnb AirFlow to use the CeleryExecutor like this:

我在以下位置更改了 executer airflow.cfg从 SequentialExecutor 到 CeleryExecutor ：

I changed the executer in the airflow.cfg from SequentialExecutor to CeleryExecutor:

# The executor class that airflow should use. Choices include
# SequentialExecutor, LocalExecutor, CeleryExecutor
executor = CeleryExecutor

但是我出现以下错误：

airflow.configuration.AirflowConfigException: error: cannot use sqlite with the CeleryExecutor

请注意， sql_alchemy_conn 的配置如下：

sql_alchemy_conn = sqlite:////root/airflow/airflow.db

我查看了Airflow的GIT（ https://github.com/airbnb/airflow/blob/master/airflow/configuration.py ）

I looked at Airflow's GIT (https://github.com/airbnb/airflow/blob/master/airflow/configuration.py)

，发现下面的代码抛出了这个例外：

and found that the following code throws this exception:

def _validate(self):
        if (
                self.get("core", "executor") != 'SequentialExecutor' and
                "sqlite" in self.get('core', 'sql_alchemy_conn')):
            raise AirflowConfigException("error: cannot use sqlite with the {}".
                format(self.get('core', 'executor')))

它从此 validate 方法看来， sql_alchemy_conn 不能包含 sqlite

It seems from this validate method that the sql_alchemy_conn cannot contain sqlite.

您是否知道如何在不使用sqllite的情况下配置 CeleryExecutor ？请注意，我根据需要下载了rabitMQ以便与CeleryExecuter一起使用。

Do you have any idea how to configure the CeleryExecutor without sqllite? please note that I downloaded rabitMQ for working with the CeleryExecuter as required.

推荐答案

AirFlow表示 CeleryExecutor 需要默认数据库SQLite以外的其他后端。例如，您必须使用 MySQL 或 PostgreSQL 。

It is said by AirFlow that the CeleryExecutor requires other backend than default database SQLite. You have to use MySQL or PostgreSQL, for example.

必须更改 airflow.cfg 中的 sql_alchemy_conn 遵循SqlAlchemy连接字符串结构（请参见 SqlAlchemy文档）

The sql_alchemy_conn in airflow.cfg must be changed to follow the SqlAlchemy connection string structure (see SqlAlchemy document)

例如，

sql_alchemy_conn = postgresql+psycopg2://airflow:airflow@127.0.0.1:5432/airflow

这篇关于配置气流以与CeleryExecutor配合使用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！