将数据从 Amazon dynamoDB 加载到 redshift [英] Loading data from Amazon dynamoDB to redshift

问题描述

我们有一个生产中的 DynamoDB 表正在不断更新，我们希望将所有记录从 dynamoDB 加载到 redshift.

We have a DynamoDB table in production that is being continuously updated,we want to load all the records from dynamoDB to redshift.

我们尝试使用复制命令，但由于新记录不断插入到表中，复制命令将永远运行.

we tried using the copy command,but since new records are continuously being inserted in table the copy command runs forever.

我们想知道将数据从 live dynamodb 加载到 redshift 的最佳方式是什么.

We want to know what is the best way to load the data from live dynamodb to redshift.

推荐答案

考虑查看 DynamoDB Streams 基于解决方案.Streams 提供了在每个 DynamoDB 分区上发生的数据平面事件的有序日志(因此每个主键的事件都是绝对有序的).您可以使用 Kinesis 客户端库 和 DynamoDB Streams Kinesis Adapter 处理流到 Redshift.

Consider looking into a DynamoDB Streams based solution. Streams provides an ordered log of the data plane events transpiring on each DynamoDB partition (so events for each primary key are absolutely ordered). You can use Kinesis Client Library and DynamoDB Streams Kinesis Adapter to process the Stream to Redshift.

DynamoDB Streams 目前处于预览阶段，但应该很快就会普遍可用.

DynamoDB Streams is currently in preview, but should be generally available soon.

这篇关于将数据从 Amazon dynamoDB 加载到 redshift的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！