如何在作曲家/气流中清楚列出所有连接? [英] How to list all connections clearly within composer/airflow?

问题描述

我正在尝试使用基础架构作为代码的方法创建一个作曲者环境.为此，我需要以编程方式存储和检索气流变量，并将它们的版本保持在某个地方.

I am trying to create a composer environment with the approach of infrastructure as code. For this, I need to store and retrieve airflow variables programmatically and keep them versioned somewhere.

在上一篇文章， Ed Morton 编写了一个脚本来将表转换为JSON，但是存在与方法有关的问题使用以下命令时，作曲家/气流会输出数据:

In a previous post, Ed Morton wrote a script to convert the table to a JSON, but there is a problem related to the way composer/airflow outputs the data when using the following command :

gcloud composer environments run `$COMPOSER_ENV` --location <location> connections -- --list

输出示例为:

╒════════════════════════════════╤═════════════════════════════╤════════════════════════════════╤════════╤════════════════╤══════════════════════╤════════════════════════════════╕
│ Conn Id                        │ Conn Type                   │ Host                           │ Port   │ Is Encrypted   │ Is Extra Encrypted   │ Extra                          │
╞════════════════════════════════╪═════════════════════════════╪════════════════════════════════╪════════╪════════════════╪══════════════════════╪════════════════════════════════╡
│ 'airflow_db'                   │ 'mysql'                     │ 'airflow-sqlp...rvice.default' │ None   │ True           │ False                │ None                           │
├────────────────────────────────┼─────────────────────────────┼────────────────────────────────┼────────┼────────────────┼──────────────────────┼────────────────────────────────┤

如您所见，问题出在Host，而Extra列包含省略号...，该省略号删去了诸如'airflow-sqlp...rvice.default'这样的长文本.

As you can see, the problem is the Host, and Extra columns contain an ellipsis ... that abridges long text such as here 'airflow-sqlp...rvice.default'.

如何获取上述(composer)实用工具输出的信息的完整版本?

How do I get a full version of the information output by the above-mentioned (composer) utility?

我正在使用composer-1.12.1-airflow-1.10.9.不幸的是，使用CLI将连接导出到JSON的很好的功能只是在最新版本的气流中可用..

I am using composer-1.12.1-airflow-1.10.9. Unfortunately the nice feature of exporting the connections to a JSON using the CLI is only available in the latest version of airflow.

推荐答案

在@savsr的出色解决方案中，以下是他解决方案之前的步骤.您可以在Cloud Shell中运行以下命令.

Adding to the excellent solution of @savsr, here are the steps before his solution. You can run the following commands in Cloud Shell.

根据Google Cloud Platform 文档:

According to the Google Cloud Platform docs:

您可以通过执行以下命令来获取所有吊舱:

You can get all the pods by performing the following command:

kubectl get pods --all-namespaces

下一步，寻找一个名称类似于airflow-worker-1a2b3c-x0yz的豆荚.

Next, look for a pod with a name like airflow-worker-1a2b3c-x0yz.

让我们假设此Pod具有命名空间composer-1-6-0-airflow-example-namespace.然后，您将连接到它，并通过执行以下命令进入bash shell:

Let's assume that this pod has namespace composer-1-6-0-airflow-example-namespace. Then you'll connect to it and get into a bash shell by performing the following command:

kubectl -n composer-1-6-0-airflow-example-namespace \ 
  exec -it airflow-worker-1a2b3c-x0yz -c airflow-worker -- /bin/bash

现在您位于特定的气流工作人员内部，在bash shell中，您可以执行@savsr提到的命令.

Now that you are inside this specific airflow worker, in a bash shell, you can perform the commands that @savsr has mentioned.

我尝试过，对我有用:

echo "select * from connection limit 3;" | airflow shell

此外，您可以执行一个更严格的查询，该查询将为您提供所需的主机名:

Also, you may perform a more limiting query that will give you exactly the host names that you want:

echo "select host from connection limit 3;" | airflow shell

在我的示例中产生了:

host
airflow-sqlproxy-service.default
localhost
NULL

这篇关于如何在作曲家/气流中清楚列出所有连接?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！