Helm稳定/气流-使用Helm图表的共享持久卷的气流部署的自定义值失败 [英] Helm stable/airflow - Custom values for Airflow deployment with Shared Persistent Volume using Helm chart failing

问题描述

我想在共享持久卷中的Kubernetes上部署Airflow，在其中Pod可以访问相同的DAG. 根据文档(

I want to deploy Airflow on Kubernetes where pods have access to the same DAGs, in a Shared Persistent Volume. According to the documentation (https://github.com/helm/charts/tree/master/stable/airflow#using-one-volume-for-both-logs-and-dags), it seems I have to set and pass these values to Helm: extraVolume, extraVolumeMount, persistence.enabled, logsPersistence.enabled, dags.path, logs.path.

我在安装官方Helm图表时传递的任何自定义值都会导致类似以下错误:

Any custom values I pass when installing the official Helm chart results in errors similar to:

Error: YAML parse error on airflow/templates/deployments-web.yaml: error converting YAML to JSON: yaml: line 69: could not find expected ':'

• 工作正常:microk8s.helm install --namespace "airflow" --name "airflow" stable/airflow
• 不起作用:
• Works fine: microk8s.helm install --namespace "airflow" --name "airflow" stable/airflow
• Not working:

microk8s.helm install --namespace "airflow" --name "airflow" stable/airflow \
--set airflow.extraVolumes=/home/*user*/github/airflowDAGs \
--set airflow.extraVolumeMounts=/home/*user*/github/airflowDAGs \
--set dags.path=/home/*user*/github/airflowDAGs/dags \
--set logs.path=/home/*user*/github/airflowDAGs/logs \
--set persistence.enabled=false \
--set logsPersistence.enabled=false

• 也无法正常工作:microk8s.helm install --namespace "airflow" --name "airflow" stable/airflow --values=values_pv.yaml，带有values_pv.yaml: https://pastebin .com/PryCgKnC
  • 请将/home/*user*/github/airflowDAGs更改为计算机上的路径以复制错误.
  • Also not working: microk8s.helm install --namespace "airflow" --name "airflow" stable/airflow --values=values_pv.yaml, with values_pv.yaml: https://pastebin.com/PryCgKnC
    • Please change /home/*user*/github/airflowDAGs to a path on your machine to replicate the error.
    1. 由于默认values.yaml中的以下几行，可能会出错:
    1. Maybe it is going wrong because of these lines in the default values.yaml:

    ## Configure DAGs deployment and update
    dags:
      ##
      ## mount path for persistent volume.
      ## Note that this location is referred to in airflow.cfg, so if you change it, you must update airflow.cfg accordingly.
      path: /home/*user*/github/airflowDAGs/dags
    

    如何在Kubernetes部署中配置airflow.cfg?在非容器化的Airflow部署中，可以在~/airflow/airflow.cfg中找到此文件.

    How do I configure airflow.cfg in a Kubernetes deployement? In a non-containerized deployment of Airflow, this file can be found in ~/airflow/airflow.cfg.

    1. airflow.cfg中的第69行是指:
    1. Line 69 in airflow.cfg refers to: https://github.com/helm/charts/blob/master/stable/airflow/templates/deployments-web.yaml#L69

    其中包含git. .yaml是否配置错误，并且错误地尝试使用git pull，但是由于未指定git路径，因此失败了?

    Which contains git. Are the .yaml wrongly configured, and it falsely is trying to use git pull, but since no git path is specified, this fails?

    • 操作系统:Ubuntu 18.04(单机)
    • MicroK8s:v1.15.4修订版:876
    • microk8s.kubectl version:v1.15.4
    • microk8s.helm version:v2.14.3
    • OS: Ubuntu 18.04 (single machine)
    • MicroK8s: v1.15.4 Rev:876
    • microk8s.kubectl version: v1.15.4
    • microk8s.helm version: v2.14.3

    如何正确地将正确的值传递给Airflow Helm图表，以便能够在Kubernetes上部署Airflow，并且Pod可以访问相同的DAG并登录到共享持久卷上?

    How do I correctly pass the right values to the Airflow Helm chart to be able to deploy Airflow on Kubernetes with Pods having access to the same DAGs and logs on a Shared Persistent Volume?

    推荐答案

    不确定是否已解决此问题，但是如果您还没有解决，那么我认为有一种非常简单的方法可以解决您的工作.

    Not sure if you have this solved yet, but if you haven't I think there is a pretty simple way close to what you are doing.

    所有部署，服务和Pod都需要持久的卷信息-本地驻留在哪里以及每种kube类型应放在何处.看起来图表的values.yaml提供了一种方法.我只会在下面用dags显示它，但是我认为它也应该与日志大致相同.

    All of the Deployments, Services, Pods need the persistent volume information - where it lives locally and where it should go within each kube kind. It looks like the values.yaml for the chart provides a way to do this. I'll only show this with dags below, but I think it should be roughly the same process for logs as well.

    因此，基本步骤是:1)告诉kube"volume"(目录)在您的计算机上的位置，2)告诉kube将其放在容器中的位置，以及3)告诉airflow在哪里寻找dag.因此，您可以从头盔存储库中复制values.yaml文件，并使用以下内容进行更改.

    So the basic steps are, 1) tell kube where the 'volume' (directory) lives on your computer, 2) tell kube where to put that in your containers, and 3) tell airflow where to look for the dags. So, you can copy the values.yaml file from the helm repo and alter it with the following.

    1. airflow部分

    首先，您需要创建一个包含本地目录中项目的卷(这是下面的extraVolumes).然后，需要将其挂载-幸运的是将其放在此处会将其模板化到所有kube文件中.一旦创建了该卷，就应该告诉它挂载dags.因此，基本上，extraVolumes创建该卷，然后extraVolumeMounts装入该卷.

    First, you need to create a volume containing the items in your local directory (this is the extraVolumes below). Then, that needs to be mounted - luckily putting it here will template it into all kube files. Once that volume is created, then you should tell it to mount dags. So basically, extraVolumes creates the volume, and extraVolumeMounts mounts the volume.

    airflow:
      extraVolumeMounts: # this will get the volume and mount it to that path in the container                                                                                                                                                               
      - name: dags
        mountPath: /usr/local/airflow/dags  # location in the container it will put the directory mentioned below.
    
      extraVolumes: # this will create the volume from the directory
      - name: dags
        hostPath:
          path: "path/to/local/directory"  # For you this is something like /home/*user*/github/airflowDAGs/dags
    
    

    1. 告诉dag居住在容器中的气流配置(与上面的yaml部分相同).

    airflow:
      config:
        AIRFLOW__CORE__DAGS_FOLDER: "/usr/local/airflow/dags"  # this needs to match the mountPath in the extraVolumeMounts section
    

    1. 安装helm和新的values.yaml文件.

    helm install --namespace "airflow" --name "airflow" -f local/path/to/values.yaml stable/airflow
    

    最后，这应该允许气流在dags文件夹中查看您的本地目录.如果添加一个新文件，它应该显示在容器中-尽管可能需要一分钟才能显示在UI中-我不认为dagbag进程一直在运行吗?无论如何，希望这会有所帮助！

    In the end, this should allow airflow to see your local directory in the dags folder. If you add a new file, it should show up in the container - though it may take a minute to show up in the UI - I don't think the dagbag process is constantly running? Anyway, hope this helps!

    这篇关于Helm稳定/气流-使用Helm图表的共享持久卷的气流部署的自定义值失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！