损坏的DAG：（...）没有名为docker的模块 [英] Broken DAG: (...) No module named docker

问题描述

我所有的BigQuery连接器都在运行，但是我希望在Cloud Composer而不是App Engine Flexible上安排一些在Docker容器中的现有脚本。

下面的脚本似乎遵循我可以找到的示例：

  import datetime 
从airflow导入DAG 
从airflow导入模型
从airflow.operators.docker_operator导入DockerOperator 
 
昨天= datetime.datetime.combine（
 datetime.datetime.today（）-datetime.timedelta（1） ，
 datetime.datetime.min.time（））
 
 default_args = {
＃将开始日期设置为昨天将立即启动DAG 
'start_date'：昨天， 
＃如果任务失败，请等待至少5分钟后重试一次。
'retries'：1，1，
'retry_delay'：datetime.timedelta（minutes = 5），
 } 
 
 schedule_interval = '45 09 * * *'
 
 dag = DAG（'xxx-merge'，default_args = default_args，schedule_interval = schedule_in时间段）
 
 hfan = DockerOperator（
 task_id ='hfan'，
 image ='gcr.io/yyyyy/xxxx'
）
  

...但是尝试运行时，它会在Web UI中告诉我：

 破损的DAG：[/home/airflow/gcs/dags/xxxx.py]没有名为docker 
   
 
 
也许Docker没有配置为在Cloud Composer运行的Kubernetes集群内工作吗？还是我只是缺少语法上的内容？
解决方案
我通过在以下位置安装docker-py == 1.10.6来解决它
 
 
 
但是，要使DockerOperator正常工作需要付出更多的努力，因为作曲者的工作人员无法访问Docker守护进程。转到GCP控制台并执行以下步骤；获得群集凭据）之后。
 
 
 
将当前的部署配置导出到文件中
 
 
 
  kubectl获取部署流程- o yaml --export> airflow-worker-config.yaml  
 
 
 
 
编辑airflow-worker-config.yaml（示例链接）将docker.socker和docker安装到运行中的docker-socker和docker权限
 
 
 
 
应用部署设置
 
 
 
  kubectl apply -f airflow-worker- config.yaml  
 
 
 
 
I have BigQuery connectors all running, but I have some existing scripts in Docker containers I wish to schedule on Cloud Composer instead of App Engine Flexible.

I have the below script that seems to follow the examples I can find:
import datetime
from airflow import DAG
from airflow import models
from airflow.operators.docker_operator import DockerOperator

yesterday = datetime.datetime.combine(
    datetime.datetime.today() - datetime.timedelta(1),
    datetime.datetime.min.time())

default_args = {
    # Setting start date as yesterday starts the DAG immediately
    'start_date': yesterday,
    # If a task fails, retry it once after waiting at least 5 minutes
    'retries': 1,
    'retry_delay': datetime.timedelta(minutes=5),
}

schedule_interval = '45 09 * * *'

dag = DAG('xxx-merge', default_args=default_args, schedule_interval=schedule_interval)

hfan = DockerOperator(
   task_id = 'hfan',
   image   = 'gcr.io/yyyyy/xxxx'
 )
...but when trying to run it tells me in the web UI:
Broken DAG: [/home/airflow/gcs/dags/xxxx.py] No module named docker
Is it perhaps that the Docker is not configured to work inside the Kubernetes cluster that Cloud Composer runs?  Or am I just missing something in the syntax?
解决方案
I got it resolved by installing docker-py==1.10.6 in the PyPI section of composer.

However, to get DockerOperator to work properly requires a bit more effort as the composer workers do not have access to the Docker daemon. Head to the GCP console and perform the following steps; after getting cluster credentials).

Export current deployment config to file

kubectl get deployment airflow-worker -o yaml --export > airflow-worker-config.yaml
Edit airflow-worker-config.yaml (example link) to mount docker.sock and docker, grant privileged access to airflow-worker to run docker commands
Apply deployment settings

kubectl apply -f airflow-worker-config.yaml


                        
这篇关于损坏的DAG：（...）没有名为docker的模块的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！