Google Cloud Composer和Google Cloud SQL [英] Google Cloud Composer and Google Cloud SQL

问题描述

我们可以通过哪些方式从新推出的Google Cloud Composer连接到Google Cloud SQL（MySQL）实例？目的是将数据从Cloud SQL实例获取到BigQuery（可能通过Cloud Storage进行中间步骤）。

1. Can




2. 如果不能，则可以通过Kubernetes引入Cloud SQL代理吗？服务经纪人？ -> https://cloud.google.com/kubernetes -engine / docs / concepts / add-on / service-broker




3. 应该使用Airflow计划和调用GCP API命令，例如1）将mysql表导出到云存储2）将mysql导出到bigquery吗？




4. 也许我缺少其他方法来完成此操作

解决方案

Cloud SQL代理提供了对Cloud SQL Second Generation实例的安全访问，而无需将IP地址列入白名单或配置SSL。 - Google CloudSQL代理文档

CloudSQL代理似乎是连接到CloudSQL的首选方法。因此，在Composer中，从1.6.1版开始，我们可以创建一个新的Kubernetes Pod来运行gcr.io/cloudsql-docker/gce-proxy:latest映像，通过服务公开它，然后在Composer中创建一个Connection以使用

要进行设置：

• 关注 Google的文档




• 使用来自阿里克的中号帖子

  
  
  
  
  

  • 检查是否已创建广告连播 kubectl get pods --all-namespaces

  
  

  • 检查是否已创建服务 kubectl get services --all-namespaces

  
  

  • 跳转到工作节点 kubectl --namespace = composer-1-6-1-airflow -1-10-1-< some-uid> exec -it airflow-worker-< some-uid> bash

    
    
    
    
    
    • 测试mysql连接 mysql -u composer -p --host< service-name> .default.svc.cluster.local
    
    
  
  

注意：

• Composer现在使用命名空间来组织广告连播




• 位于不同名称空间的豆荚不要互相交谈除非您为其提供完整路径< k8-service-name>。< k8-namespace-name> .svc.cluster.local




• 使用以下命令创建新的 Composer连接完整路径将启用成功连接

What ways do we have available to connect to a Google Cloud SQL (MySQL) instance from the newly introduced Google Cloud Composer? The intention is to get data from a Cloud SQL instance into BigQuery (perhaps with an intermediary step through Cloud Storage).

1. Can the Cloud SQL proxy be exposed in some way on pods part the Kubernetes cluster hosting Composer?

2. If not can the Cloud SQL Proxy be brought in by using the Kubernetes Service Broker? -> https://cloud.google.com/kubernetes-engine/docs/concepts/add-on/service-broker

3. Should Airflow be used to schedule and call GCP API commands like 1) export mysql table to cloud storage 2) read mysql export into bigquery?

4. Perhaps there are other methods that I am missing to get this done

解决方案

"The Cloud SQL Proxy provides secure access to your Cloud SQL Second Generation instances without having to whitelist IP addresses or configure SSL." -Google CloudSQL-Proxy Docs

CloudSQL Proxy seems to be the recommended way to connect to CloudSQL above all others. So in Composer, as of release 1.6.1, we can create a new Kubernetes Pod to run the gcr.io/cloudsql-docker/gce-proxy:latest image, expose it through a service, then create a Connection in Composer to use in the operator.

To get set up:

• Follow Google's documentation

• Test the connection using info from Arik's Medium Post

  • Check that the pod was created kubectl get pods --all-namespaces

  • Check that the service was created kubectl get services --all-namespaces

  • Jump into a worker node kubectl --namespace=composer-1-6-1-airflow-1-10-1-<some-uid> exec -it airflow-worker-<some-uid> bash

    • Test mysql connection mysql -u composer -p --host <service-name>.default.svc.cluster.local

Notes:

• Composer now uses namespaces to organize pods

• Pods in different namespaces don't talk to each other unless you give them the full path <k8-service-name>.<k8-namespace-name>.svc.cluster.local

• Creating a new Composer Connection with the full path will enable successful connection

这篇关于Google Cloud Composer和Google Cloud SQL的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！