Docker上的气流-路径问题 [英] Airflow on Docker - Path issue

问题描述

使用气流，我尝试简单的DAG工作.

Working with airflow I try simple DAG work.

我编写了自定义运算符和其他要导入到DAG逻辑所在的主文件中的文件.

I wrote custom operators and other files that I want to import into the main file where the DAG logic is.

这里是文件夹的结构:

├── airflow.cfg
├── dags
│   ├── __init__.py
│   ├── dag.py
│   └── sql_statements.sql
├── docker-compose.yaml
├── environment.yml
└── plugins
    ├── __init__.py
    └── operators
        ├── __init__.py
        ├── facts_calculator.py
        ├── has_rows.py
        └── s3_to_redshift.py

我直接在 compose文件中设置了卷，因为当我登录到容器的终端时可以看到它们.

I setup the volume right in the compose file since I can see them when I log into the container's terminal.

我在线上关注了一些教程，在其中添加了一些 __ init __.py .

I followed some tutorials online from where I have added some __init__.py.

两个空的 __ init __ 都放入

• /plugins/operators :

from operators.facts_calculator import FactsCalculatorOperator
from operators.has_rows import HasRowsOperator
from operators.s3_to_redshift import S3ToRedshiftOperator

__all__ = [
    'FactsCalculatorOperator',
    'HasRowsOperator',
    'S3ToRedshiftOperator'
]

• /插件:

from airflow.plugins_manager import AirflowPlugin

import operators

# Defining the plugin class
class CustomPlugin(AirflowPlugin):
    name = "custom_plugin"
    # A list of class(es) derived from BaseOperator
    operators = [
        operators.FactsCalculatorOperator,
        operators.HasRowsOperator,
        operators.S3ToRedshiftOperator
    ]
    # A list of class(es) derived from BaseHook
    hooks = []
    # A list of class(es) derived from BaseExecutor
    executors = []
    # A list of references to inject into the macros namespace
    macros = []
    # A list of objects created from a class derived
    # from flask_admin.BaseView
    admin_views = []
    # A list of Blueprint object created from flask.Blueprint
    flask_blueprints = []
    # A list of menu links (flask_admin.base.MenuLink)
    menu_links = []

但是我不断从我的IDE中收到错误消息(说在操作符的 __ init __ 中没有名为操作符的模块或未解析的引用操作符).

But I keep getting errors from my IDE (saying No module named operators or Unresolved reference operators inside the operator's __init__).

由于一切都无法在网络服务器上启动.

Since everything fails to launch on the webserver.

任何想法如何进行设置?我哪里错了?

Any idea how to set this up ? Where I'm wrong ?

推荐答案

您是否正在使用 puckel的图片?

如果是，则需要取消注释docker-compose文件中的#-./plugins:/usr/local/airflow/plugins 行(可能有多个)本地或芹菜).其余的设置对我来说都很好.

If you are, you need to uncomment the # - ./plugins:/usr/local/airflow/plugins lines (may there are more than one) in the docker-compose files (either Local or Celery). The rest of your setup looks fine to me.

这篇关于Docker上的气流-路径问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！