Google Cloud Composer 和 Google Cloud SQL [英] Google Cloud Composer and Google Cloud SQL

问题描述

我们有哪些方法可以从新推出的 Google Cloud Composer 连接到 Google Cloud SQL (MySQL) 实例?目的是将数据从 Cloud SQL 实例获取到 BigQuery(可能通过 Cloud Storage 中间步骤).

What ways do we have available to connect to a Google Cloud SQL (MySQL) instance from the newly introduced Google Cloud Composer? The intention is to get data from a Cloud SQL instance into BigQuery (perhaps with an intermediary step through Cloud Storage).

1. Cloud SQL 代理能否以某种方式在托管 Composer 的 Kubernetes 集群的 Pod 上公开?

1. Can the Cloud SQL proxy be exposed in some way on pods part the Kubernetes cluster hosting Composer?

如果不能，是否可以使用 Kubernetes Service Broker 引入 Cloud SQL Proxy?-> https://cloud.google.com/kubernetes-engine/docs/concepts/add-on/service-broker

If not can the Cloud SQL Proxy be brought in by using the Kubernetes Service Broker? -> https://cloud.google.com/kubernetes-engine/docs/concepts/add-on/service-broker

是否应该使用 Airflow 来调度和调用 GCP API 命令，例如 1) 将 mysql 表导出到云存储 2) 将 mysql 导出读入 bigquery?

Should Airflow be used to schedule and call GCP API commands like 1) export mysql table to cloud storage 2) read mysql export into bigquery?

也许我还缺少其他方法来完成这项工作

Perhaps there are other methods that I am missing to get this done

推荐答案

Cloud SQL 代理提供对 Cloud SQL 第二代实例的安全访问，而无需将 IP 地址列入白名单或配置 SSL."-Google CloudSQL-Proxy 文档

"The Cloud SQL Proxy provides secure access to your Cloud SQL Second Generation instances without having to whitelist IP addresses or configure SSL." -Google CloudSQL-Proxy Docs

CloudSQL 代理似乎是最推荐的连接 CloudSQL 的方式.所以在 Composer 中，从 1.6.1 版本开始，我们可以创建一个新的 Kubernetes Pod 来运行 gcr.io/cloudsql-docker/gce-proxy:latest 镜像，通过一个服务公开它，然后在 Composer 中创建一个 Connection 来使用在运算符中.

CloudSQL Proxy seems to be the recommended way to connect to CloudSQL above all others. So in Composer, as of release 1.6.1, we can create a new Kubernetes Pod to run the gcr.io/cloudsql-docker/gce-proxy:latest image, expose it through a service, then create a Connection in Composer to use in the operator.

开始设置:

• 遵循 Google 的文档

使用来自 Arik 的中篇帖子

• 检查 pod 是否已创建 kubectl get pods --all-namespaces

检查服务是否已创建kubectl get services --all-namespaces

跳转到一个工作节点 kubectl --namespace=composer-1-6-1-airflow-1-10-1-exec -itairflow-worker-<some-uid>bash

• 测试mysql连接mysql -u composer -p --host .default.svc.cluster.local

注意事项:

• Composer 现在使用命名空间来组织豆荚

不同命名空间中的 Pod 不相互交谈 除非你给他们完整路径 ..svc.cluster.local

Pods in different namespaces don't talk to each other unless you give them the full path <k8-service-name>.<k8-namespace-name>.svc.cluster.local

使用完整路径创建新的 Composer Connection 将启用成功连接

Creating a new Composer Connection with the full path will enable successful connection

这篇关于Google Cloud Composer 和 Google Cloud SQL的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！