气流：任务排队但未运行 [英] Airflow: Tasks queued but not running
问题描述
我是气流的新手，正在尝试设置气流以运行ETL管道。我能够安装
I am new to airflow and trying to setup airflow to run ETL pipelines. I was able to install
- 气流
- postgres
- celery
- rabbitmq
- airflow
- postgres
- celery
- rabbitmq
我能够测试运行turag dag。当我尝试调度作业时，调度程序可以将其拾取并排队，这些作业可以在UI上看到，但任务未运行。有人可以帮我解决这个问题吗？我相信我在这里缺少最基本的气流概念。以下是airflow.cfg
I am able to test run the turtorial dag. When i try to schedule the jobs, scheduler is able to pick it up and queue the jobs which i could see on the UI but tasks are not running. Could somebody help me fix ths issue? I believe i am missing most basic airflow concept here. below is the airflow.cfg
这是我的配置文件：
[core]
airflow_home = /root/airflow
dags_folder = /root/airflow/dags
base_log_folder = /root/airflow/logs
executor = CeleryExecutor
sql_alchemy_conn = postgresql+psycopg2://xxxx.amazonaws.com:5432/airflow
api_client = airflow.api.client.local_client
[webserver]
web_server_host = 0.0.0.0
web_server_port = 8080
web_server_worker_timeout = 120
worker_refresh_batch_size = 1
worker_refresh_interval = 30
[celery]
celery_app_name = airflow.executors.celery_executor
celeryd_concurrency = 16
worker_log_server_port = 8793
broker_url = amqp://rabbit:rabbit@x.x.x.x/rabbitmq_vhost
celery_result_backend = db+postgresql+psycopg2://postgres:airflow@xxx.amazonaws.com:5432/airflow
flower_host = 0.0.0.0
flower_port = 5555
default_queue = default
DAG：这是我使用的教程dag
，我的dag的开始日期是-'start_date'：datetime（2017，4，11），
and the start date for my dag is -- 'start_date': datetime(2017, 4, 11),
推荐答案
运行气流的所有三个组件，即：
have your run all the three components of airflow, namely:
airflow webserver
airflow scheduler
airflow worker
如果仅运行前两个，则任务将是已排队，但未执行。
If you only run the previous two, the tasks will be queued, but not executed. airflow worker will provide the workers that actually execute the dags.
此外，芹菜4.0.2目前与气流1.7或1.8不兼容。改用芹菜3。
Also btw, celery 4.0.2 is not compatible with airflow 1.7 or 1.8 currently. Use celery 3 instead.
这篇关于气流：任务排队但未运行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
