气流：PythonOperator：为什么要包含'ds'arg？ [英] Airflow: PythonOperator: why to include 'ds' arg?

问题描述

在定义一个函数以供以后用作python_callable时，为什么将'ds'作为函数的第一个参数？

While defining a function to be later used as a python_callable, why is 'ds' included as the first arg of the function?

例如：

def python_func(ds, **kwargs):
    pass

我查看了Airflow文档，但找不到任何解释。

I looked into the Airflow documentation, but could not find any explanation.

推荐答案

这与 provide_context = True 参数有关。根据Airflow文档，

This is related to the provide_context=True parameter. As per Airflow documentation,

如果设置为true，Airflow将传递一组可以在函数中使用的关键字参数。这组kwarg与您在jinja模板中可以使用的完全对应。为此，您需要在函数头中定义** kwargs。

if set to true, Airflow will pass a set of keyword arguments that can be used in your function. This set of kwargs correspond exactly to what you can use in your jinja templates. For this to work, you need to define **kwargs in your function header.

ds 是这些关键字参数之一，并以 YYYY-MM-DD。对于在文档中标记为（模板）的参数，可以使用’{{ds}}’默认变量来传递执行日期。您可以在此处阅读有关默认变量的更多信息：

ds is one of these keyword arguments and represents execution date in format "YYYY-MM-DD". For parameters that are marked as (templated) in the documentation, you can use '{{ ds }}' default variable to pass the execution date. You can read more about default variables here:

https://pythonhosted.org/airflow/code.html?highlight=pythonoperator#default-variables （已淘汰）

https://airflow.incubator.apache.org/concepts.html?highlight=python_callable

PythonOperator没有模板化参数，所以要做类似

PythonOperator doesn't have templated parameters, so doing something like

python_callable=print_execution_date('{{ ds }}')

不起作用。要在PythonOperator的可调用函数中打印执行日期，您将必须执行以下操作：

won't work. To print execution date inside the callable function of your PythonOperator, you will have to do it as

def print_execution_date(ds, **kwargs):
    print(ds)

或

def print_execution_date(**kwargs):
    print(kwargs.get('ds'))

希望这会有所帮助。

这篇关于气流：PythonOperator：为什么要包含'ds'arg？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！