仅将新记录从 AWS DynamoDB 复制到 AWS Redshift [英] Copying only new records from AWS DynamoDB to AWS Redshift

问题描述

我看到有大量示例和文档可将数据从 DynamoDB 复制到 Redshift，但我们正在研究增量复制过程，其中仅将新行从 DynamoDB 复制到 Redshift.我们每天都会运行这个复制过程，所以不需要每天杀死整个 redshift 表.有人对此主题有任何经验或想法吗?

I see there is tons of examples and documentation to copy data from DynamoDB to Redshift, but we are looking at an incremental copy process where only the new rows are copied from DynamoDB to Redshift. We will run this copy process everyday, so there is no need to kill the entire redshift table each day. Does anybody have any experience or thoughts on this topic?

推荐答案

Dynamo DB 有一个名为 流:

Dynamo DB has a feature (currently in preview) called Streams:

Amazon DynamoDB Streams 维护按时间排序的项目序列日志中任何 DynamoDB 表中的级别更改持续时间为 24小时.使用 Streams API，开发人员可以查询更新，接收变更前后的项目级数据，并使用为他们构建的应用程序构建创造性的扩展DynamoDB.

Amazon DynamoDB Streams maintains a time ordered sequence of item level changes in any DynamoDB table in a log for a duration of 24 hours. Using the Streams APIs, developers can query the updates, receive the item level data before and after the changes, and use it to build creative extensions to their applications built on top of DynamoDB.

此功能将允许您在新更新进入时对其进行处理并按照您的意愿使用它们，而不是在 DynamoDB 之上设计一个导出系统.

This feature will allow you to process new updates as they come in and do what you want with them, rather than design an exporting system on top of DynamoDB.

您可以在 阅读和处理 DynamoDB 流 文档.

这篇关于仅将新记录从 AWS DynamoDB 复制到 AWS Redshift的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！