AWS DynamoDB流进入Redshift [英] AWS DynamoDB Stream into Redshift

问题描述

我们希望将数据作为流从DynamoDB NoSQL连续移入Redshift数据库。
我很难理解AWS中的所有新术语/技术。

1）

解决方案

Amazon Kinesis可以实时收集，处理和分析视频和数据流。

• 使用 Kinesis视频流 >捕获，处理和存储视频流以进行分析和机器学习。


• 使用 Kinesis Data Streams 来构建自定义应用程序，以使用流行的流处理来分析数据流框架。


• 使用 Kinesis Data Firehose 将数据流加载到AWS数据存储中。


• 使用 Kinesis Data Analytics（分析），以使用SQL分析数据流。

DynamoDB流与Kinesis Data Stream相同，但是由DynamoDB中的新数据/更改后的数据自动生成。这样，可以在将新数据添加到DynamoDB表或更改数据时通知应用程序。

A Kinesis Data Firehose 可以自动将流输出到Redshift（以及其他目的地）中。

AWS Lambda 可以运行代码而无需置备或管理服务器。您只需为自己消耗的计算时间付费-代码未运行时无需付费。您几乎可以为任何类型的应用程序或后端服务运行代码-只需执行零管理即可。

Lambda对于检查流中的数据很有用。例如，它可以用于处理不需要的数据格式或跳过数据。

将它们放在一起，可以在其中添加/修改数据。 DynamoDB 。这将导致发送包含有关更改信息的 DynamoDB流。 AWS Lambda函数可以检查数据并处理/删除消息。然后，如果可以将数据转发到 Kinesis Data Firehose ，以将数据自动插入到 Amazon Redshift 。

示例：

• 银行交易记录存储在DynamoDB中


• DynamoDB流将其发送到Lambda函数


• Lambda函数查看交易，还检索有关银行帐户的信息。如果帐户中有足够余额，则函数将退出并且不执行任何操作。


• 如果帐户中有 insufficient 余额，它可以通过Amazon SES发送电子邮件告知帐户持有人。然后，它可以将数据发送到Firehose，并将其存储在Redshift中，以报告过期帐户。

一起使用这些系统的好处是他们可以以最少的编码提供丰富的应用程序功能。在此示例中，仅Lambda函数需要编码-其余功能通过将各种组件链接在一起而起作用。另外，它完全没有服务器-也就是说，无需在Amazon EC2实例上运行应用程序。

We would like to move data from DynamoDB NoSQL into Redshift Database continously as a stream. I am having hard time understand all the new terms/technologies in AWS. There is

1) DynamoDB Streams

2) AWS Lambda

3) AWS Kinesis Firehose

Can someone provide a brief summary of each. What are DynamoDB streams? How does this differ from AmazonKinesis? After reading all the resources, this is my hypothesis understanding, please verify below.

(a) I assume DynamoDB Streams, create the streaming data of NoSQL, and start sending it out. It is the Sender.

(b) Lambda allows people for only time consumed, it is the time for rent Server which handles the DynamoDB Stream.

(c) Kinesis FireHose Converts the DynamoDB Stream, and places into Redshift.

(d) AmazonQuickSight is their business intelligence tool,

Is that the correct understanding of the glossary terms? Reviewing Stack link, wanted more thorough information.

解决方案

Amazon Kinesis can collect, process, and analyze video and data streams in real time.

• Use Kinesis Video Streams to capture, process, and store video streams for analytics and machine learning.
• Use Kinesis Data Streams to build custom applications that analyze data streams using popular stream processing frameworks.
• Use Kinesis Data Firehose to load data streams into AWS data stores.
• Use Kinesis Data Analytics to analyze data streams with SQL.

DynamoDB streams are effective the same as a Kinesis Data Stream, but it is automatically generated by new/changed data in DynamoDB. This allows applications to be notified when new data is added to a DynamoDB table, or when data is changed.

A Kinesis Data Firehose can automatically output a stream into Redshift (amongst other destinations).

AWS Lambda can run code without provisioning or managing servers. You pay only for the compute time you consume — there's no charge when your code isn't running. You can run code for virtually any type of application or backend service — all with zero administration.

Lambda is useful for inspecting data coming through a stream. For example, it could be used to manipulate the data format or skip-over data that is not required.

Putting it all together, you could have data added/modified in DynamoDB. This would cause a DynamoDB Stream to be sent that contains information about the change. An AWS Lambda function could inspect the data and manipulate/drop the message. If could then forward the data to Kinesis Data Firehose to automatically insert the data into Amazon Redshift.

Here's an example:

• A bank transaction is stored in DynamoDB
• DynamoDB Streams sends it to a Lambda function
• The Lambda function looks at the transaction and also retrieves information about the bank account. If there is sufficient balance in the account, the function exits and does nothing.
• If there is insufficient balance in the account, it could send an email via Amazon SES telling the account holder. It could then send the data to Firehose that stores it in Redshift for reporting of overdue accounts.

The benefit of using these systems together is that they can provide rich application functionality with minimal coding. In this example, only the Lambda function needed coding -- the rest worked via linking together various components. Also, it was totally serverless — that is, there was no need to run an application on an Amazon EC2 instance.

这篇关于AWS DynamoDB流进入Redshift的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！