从DynamoDB迁移到Spanner / BigTable [英] Migration from DynamoDB to Spanner/BigTable

问题描述

我有一个用例，需要将70 TB的数据从DynamoDB迁移到BigTable和Spanner。具有单个索引的表将进入BigTable，否则将进入Spanner。

I have a use case where I need to migrate 70 TB of data from DynamoDB to BigTable and Spanner. Tables with a single index will go to BigTable else they will go to Spanner.

我可以通过将数据导出到S3-> GCS轻松处理历史负载。 >扳手/大表。但是具有挑战性的部分是要处理DynamoDB上同时发生的增量流负载。 DynamoDB中有300个表。

I can easily handle the historical loads by exporting the data to S3 --> GCS --> Spanner/BigTable. But the challenging part is to handle the incremental streaming loads simultaneously happening on DynamoDB. There are 300 tables in DynamoDB.

如何以最佳方式处理此问题？

How to handle this thing in the best possible manner? Has anyone done this before?

推荐答案

可以使用lambda完成一种方法来捕获dynamodb更改，并将更改发布到GCP pub。 / sub，然后让Dataflow流管道处理传入的pub / sub消息，并根据表将其写入Spanner或BigTable

One approach could be done using lambdas to capture the dynamodb changes, posting the changes to GCP pub/sub, and then having a Dataflow streaming pipeline processing the incoming pub/sub messages, writing it to Spanner or BigTable depending on the table

DynamoDB-> Spanner基本解决方案记录在这里： https://cloud.google.com/solutions/migrating -dynamodb-to-cloud-spanner

The basic DynamoDB->Spanner solution is documented here: https://cloud.google.com/solutions/migrating-dynamodb-to-cloud-spanner

这可以适应不同表的不同目的地。

This could be adapted to handle the different destinations for different tables.

这篇关于从DynamoDB迁移到Spanner / BigTable的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！