如何以流模式将数据从AWS RDS加载到Google BigQuery? [英] how to load data from AWS RDS to Google BigQuery in streaming mode?

问题描述

如何以流模式将数据从AWS RDS加载到Google BigQuery?说明:我在RDS(SQL Server)中有数据，并希望将这些数据实时加载到Google BigQuery中.

how to load data from AWS RDS to Google BigQuery in streaming mode? Description: I have data in RDS (SQL Server), and wanted to load this data into Google BigQuery in real-time.

推荐答案

没有直接方法可以将更改从Amazon RDS插入到Google Cloud BigQuery.可以使用这样的管道完成

There is no direct way to insert changes from Amazon RDS to Google Cloud BigQuery. It could be done with a pipeline like this

Amazon RDS ---- Lambda/DMS ----> Kinesis数据流----- Lambda ----> BigQuery

Amazon RDS ----Lambda/DMS----> Kinesis Data Streams -----Lambda----> BigQuery

1. 使用此.

1. Read changes from Amazon RDS to Kinesis Data Streams using Lambda or use Cloud DMS. You can also push it to Kinesis Firehose for aggregating/batching records.
2. Use Lambda to read from Kinesis streams/Firehose to insert into BigQuery using tabledata.insertAll (BQ streaming API). Code will be something similar to this.

这篇关于如何以流模式将数据从AWS RDS加载到Google BigQuery?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！