如何实时将Mysql同步到BigQuery中？ [英] How to Sync Mysql into Bigquery in realtime?

问题描述

目前我有一些脚本首先删除表并将表从MySQL上传到Bigquery。很多时候它失败了。另外它每天只运行一次。我正在寻找一些可扩展的实时解决方案。我们将非常感谢您的帮助：）

阅读这些系列文章，从Wepay，他们详细了解他们如何同步他们的MySQL数据库到BigQuery，使用Airflow：

• https://wecode.wepay.com/posts/wepays-data-warehouse-bigquery-airflow


• https://wecode.wepay.com/posts/airflow-wepay


• （第三个是关于BigQuery的）
  
  $ b

  总结（引用）：
  
  

  
  
  • 设置认证，连接，DAG。
  
  
  • 定义要从MySQL获取哪些列并加载到BigQuery中。 b $ b
  • 选择如何加载数据：增量或完全加载数据。
  
  
  • 重复删除。
    
  
  

  Currently I have some script which first deletes the table and upload the table from MySQL to Bigquery. And many time it had failed. Plus it run only once a day. I am looking for some scalable and realtime solution. Your Help will be much appreciated :)

  解决方案

  Read these series of posts from Wepay, where they detail how they sync their MySQL databases to BigQuery, using Airflow:

  • https://wecode.wepay.com/posts/wepays-data-warehouse-bigquery-airflow
  • https://wecode.wepay.com/posts/airflow-wepay
  • (3rd one is about BigQuery)

  As a summary (quoting):

  • Setup authentication, connections, DAG.
  • Define which columns to pull from MySQL and load into BigQuery.
  • Choose how to load the data: incrementally, or fully.
  • De-duplicating.

  这篇关于如何实时将Mysql同步到BigQuery中？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！