如何将 Mysql 实时同步到 Bigquery? [英] How to Sync Mysql into Bigquery in realtime?

问题描述

目前我有一些脚本，它首先删除表并将表从 MySQL 上传到 Bigquery.而且很多时候都失败了.此外，它每天只运行一次.我正在寻找一些可扩展的实时解决方案.您的帮助将不胜感激:)

Currently I have some script which first deletes the table and upload the table from MySQL to Bigquery. And many time it had failed. Plus it run only once a day. I am looking for some scalable and realtime solution. Your Help will be much appreciated :)

推荐答案

阅读来自 Wepay 的这些系列文章，其中详细介绍了如何使用 Airflow 将 MySQL 数据库同步到 BigQuery:

Read these series of posts from Wepay, where they detail how they sync their MySQL databases to BigQuery, using Airflow:

• https://wecode.wepay.com/posts/wepays-数据仓库-bigquery-airflow
• https://wecode.wepay.com/posts/airflow-wepay
• (第三个是关于 BigQuery)

作为总结(引用):

• 设置身份验证、连接、DAG.
• 定义要从 MySQL 中提取哪些列并加载到 BigQuery 中.
• 选择加载数据的方式:增量加载或完全加载.
• 去重.

这篇关于如何将 Mysql 实时同步到 Bigquery?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！